Method

Setup

The agent works alone in a sandbox for up to 6 hours, offline except for its model API. Harbor orchestrates the runs, and the sandboxes run on Modal.

Grading

1. The agent submits a prediction script, pipeline/predict.sh, with the files it needs to run.
2. A fresh sandbox with no network connection runs predict.sh twice on sealed held-out data.
3. The outcome is graded against a baseline model. A submission that passes the validity checks and beats the baseline can score up to 100 points; one that fails any check scores 0. The checks:
   • predict.sh finishes offline within the time and memory limits;
   • both runs give the same output;
   • every output file has the required format, with one prediction per input row and values in range;
   • panels meet the task's size and diversity rules;
   • on tasks that test it, a row's prediction does not depend on which other rows are in the batch.

The hidden data differ from the training data in a way that matters for the task, such as new antibody families, antigens, strains or lineages.

Aggregation

We run every agent three times on each task. We average those three runs to get the agent's score on the task, then average those 25 task scores. Runs that fail a check and runs refused by a provider's safety filter count as 0.