Problem 01 is live
Write a better harness by practicing.
AI engineering moved to the systems around the model. Same model for everyone. A real environment. A hidden eval. Improve the loop until the number goes up.
How it works
Read the trace. Change the loop.
- 01
Start from a weak loop
The starter is already in the editor, with traces that fail. You do not burn a run just to watch it lose. Read those, then change the loop.
- 02
Read a failing trace
System prompt, every tool call, every result, the grader verdict. This is how you debug a harness.
- 03
Edit and submit
Thirty hidden tasks. Same model. Pass is 85 percent. A careful loop lands higher, with tokens still in budget.
Open the first problem.
The starter is already in the editor. Read a failing trace, then change the loop.
Start problem 01FAQ
Questions
What is a harness?
The software around the model: the loop, the tools, the context, retries, when to stop. Not the model. That file is what you write here.
Why pin the model?
If everyone picks a different model, you cannot tell who built the better loop. Same model, same environment, same hidden eval. The only variable is the harness.
What is a trace?
The full transcript of one task: system prompt, every message, every tool call and result, the final answer, the grader verdict. You debug from this, not from the score.
What is Run vs Submit?
Run is three visible tasks so you can see the failures. Submit is thirty hidden tasks. The number that counts is Submit.
What counts as a pass?
85% on the hidden set, inside the step, token, and time budgets.
What do I submit?
One Python file. from harness import llm, tools, Task. The starter is a naive ReAct loop.
Do I need an account?
Run is free without an account. Submit (the hidden set) needs a login — 5 per day.
Can I pick a stronger model?
No. Switching models is not the skill we measure.
Won't this burn money?
A cheap pinned model. Run is three tasks. Submit is five a day.
Won't the score bounce around?
Temperature 0. Deterministic fixtures. We always show the number.
Won't I sit and wait?
Tasks fan out. Results stream into the trace as they finish.