Antler Gate 1: a review model that ranks on par with Opus 5.5 at a sixtieth of the cost

Antler Gate 1 predicts the outcome of a repository check before the check runs. It takes the check’s text and the pull request diff and returns P(fail) from a single output token. We built it by post-training GLM 5.3 Flash, an open-weight mixture-of-experts model with 320B parameters (18B active per token), with supervised fine-tuning on check verdicts from our internal repositories. On pull requests reviewed after the training cutoff, for checks with review history:

Claude Opus 5.5 was the strongest general-purpose model we tested as a gate. Before post-training, GLM 5.3 Flash was well behind it (AUC 0.909). After post-training, the 95% confidence interval of the AUC difference between Antler Gate and Opus is −0.015 to +0.013, so on this test set any gap is at most about 0.015 either way. Read from the test-set curves, Antler Gate skips more passing runs at each recall level in the table, and a call costs about a sixtieth as much at list prices. It is also cheaper than the base model, mostly because it reads a shorter diff excerpt (about 5k input tokens instead of 11k), and faster, because it no longer generates reasoning tokens.

Background

HeyDeer lets every repository define its own review rules. A rule lives in .agents/checks/*.md: a requirement, the files it applies to, an incorrect and a correct example. When a pull request arrives, each matching check runs as its own agent, which reads the diff, explores the repository and decides whether the change violates the rule. A verifier then confirms or rejects that verdict.

Most check runs pass. Many of them are runs where the rule plainly does not apply, such as a copy-review check on a database migration or a styling check on a backend refactor, and the agent still spends a full run to reach that conclusion.

We wanted to know whether a small model could predict a check’s verdict from the same input the agent starts with. If it can, runs that are very likely to pass could be skipped.

Problem setup

The gate gets the input a check agent starts with:

• the check’s own text (its requirement, scope and examples),
• the list of changed files the check matches,
• the pull request title,
• the diff of those files.

It returns the probability that the check will fail, and a threshold turns that into run or skip. A skipped run saves its cost; a skipped run that would have failed is a missed violation. We report three numbers per threshold:

• Fail recall: of the checks that did fail, how many the gate would still have run.
• Passes skipped: of the checks that passed, how many the gate would have skipped.
• Check cost skipped: the same, weighted by what each run actually cost.

We weight by cost because check runs differ in cost by more than an order of magnitude, mostly with diff size. Two gates that skip the same number of runs can save very different amounts.

Data

The data comes from our internal repositories, where HeyDeer reviews every pull request with checks our teams wrote. For each check in a review we have:

• the final status (passed or failed) after the verifier,
• the exact head and merge-base commits,
• the check text the agent was given, with a content digest,
• the token usage of the agent run, which we price at API list rates.

An example is one pull request head and one check, labelled with the check’s final status after verification and weighted by the run’s cost. There is no manual labelling. Four processing steps mattered:

1. Rebuilding inputs from git. We regenerated each diff with git diff merge-base..head, limited to the files the check matches, so training inputs are identical to what the gate would receive.
2. Handling label drift. Verdicts depend on the check configuration and the verifier prompt, and the fail rate shifted whenever those changed. We kept only reviews from the current configuration.
3. Time split by pull request. A pull request is reviewed at several heads, and consecutive heads are near-duplicates, so a random split leaks. Each pull request goes, with all its heads, to train, validation or test by the date of its last review. Validation and test come after training.
4. Deduplication. Reviews of the same head with the same check digest are kept once.

Training

Antler Gate is a rank-16 LoRA adapter on GLM 5.3 Flash, trained with supervised fine-tuning through Fireworks’ serverless training API and served on their multi-LoRA serverless deployment, which bills per token and has no idle cost.

One-token output

The target is fail or pass. GLM 5.3 Flash normally reasons before answering, so each assistant turn is rendered with an empty thinking block, <think></think>fail, and the loss covers only the verdict and the end-of-turn token. The trained model answers with no reasoning tokens and one output token, which keeps calls short and makes the first token’s probability usable as a score.

Prompt structure

The pull request is written by the author under review, so the system prompt treats its title and diff as untrusted data and the check text as the only requirement. Each part sits in its own tagged block (<check>, <matched_files>, <pr_title>, <diff>), which keeps instructions written into a diff from being read as instructions.

Diff truncation

Diff length has a long tail. We cap the diff at 24,000 characters, cutting only between files, and mark the block truncated="true" when something was left out. Cutting inside a file leaves partial hunks that are easy to misread; the flag tells the model that a large change has more it can’t see. The base model and Opus were given up to 200,000 characters in the comparison below.

Class balance

Failing checks are the minority, and they are the class we can least afford to get wrong. We kept every failing example and subsampled passing ones. Rebalancing moves the scores, but the threshold is chosen afterwards on validation data, so only the ranking matters.

Scoring with log-probabilities

At inference we read the log-probabilities of the verdict token at temperature 0 and normalize over the two answers:

This gives a continuous score from one call. The baselines report a 0–100 likelihood that tends to land on a few values, which limits how finely a threshold can be set; the log-probability score has no such steps.

One bug we hit: several tokens decode to the same word (fail, " fail", "\tfail"), and the top log-probabilities often include more than one of them. We first stripped whitespace and stored them in a dictionary, so a low-probability variant could overwrite the high-probability one, and the scores came out inverted even though the generated answers were correct. The probabilities of all tokens that decode to the same answer have to be summed.

Threshold selection

For Antler Gate, the threshold for a target fail recall is chosen on the validation split and then applied to the test split; the results are in the evaluation below. The baselines were run on the test split only.

Evaluation

All models were scored on the same test set, made of pull requests reviewed after the training cutoff. The baselines were used zero-shot:

• GLM 5.3 Flash (base) and Opus 5.5 received the same check text, matched files and title, and a longer diff (up to 200,000 characters), and returned a violation likelihood from 0 to 100 through a JSON schema. The base model ran with low reasoning effort.
• Jev, TypeSafe AI’s decision model on Vercel AI Gateway, got the same input through the decisions API as a predicate (the probability that a reviewer finds a violation). Its 32k-token context meant capping the diff at 70,000 characters.

Ranking (AUC)

AUC measures how well a score separates failing from passing checks across all thresholds.

The second row is the relevant one for deployment, since a gate like this would only be enabled for checks that have review history.

We bootstrapped the test set by pull request, resampling all checks of a pull request together because they share a diff. On checks with history, the 95% confidence interval of Antler Gate’s AUC gain over the base model is +0.021 to +0.068, and of its difference from Opus is −0.015 to +0.013. An interval that includes zero does not by itself make the two models equal; it bounds the gap this test set could hide at about 0.015 AUC in either direction.

Savings at a fixed recall

To compare all four models the same way, this table reads each model’s curve on the test set at the given fail recall, so the threshold is picked on the test set itself. It describes how well each score separates failures from passes, not what a deployed threshold would achieve. Values are passes skipped / check cost skipped on checks with history:

The base model’s identical rows at 95–99% come from its score steps: the next threshold up would drop below the target recall. The same effect explains most of Antler Gate’s lead over Opus at equal AUC, since Opus’s scores also cluster on a few low values. The 99% row depends on the few lowest-scored failures in the test set, so we would not read its margin precisely.

With thresholds chosen on validation

For Antler Gate we also have validation scores, so we chose each threshold on the validation split and applied it unchanged to the test split, as a deployment would:

Recall on the test split met or exceeded every target. Savings are a little lower than the test-curve figures above, as expected when the threshold is fixed in advance: at a 99% target, Antler Gate skipped 58% of passing runs and 16.2% of check cost.

Cost per call

The gate runs on every check, so its own cost is subtracted from what it saves. Jev is the cheapest per call, but it ranks lowest and skips well below Antler Gate and Opus at every recall level. Antler Gate costs about a sixtieth of an Opus call. Nearly all of a gate call’s cost is input: Antler Gate’s advantage over Opus comes from the lower per-token price of a hosted open model and from its shorter diff excerpt, and its advantage over the base model almost entirely from the shorter excerpt. The one-token output matters for latency and for scoring rather than for price. Calls take about a second, and a gate could run while the review environment starts, so it would not have to add latency.

Where the gain comes from

We split AUC into two parts. Within a single check, post-training improved how Antler Gate ranks pull requests against the rule, closing most of the gap between the base model and Opus; Opus is still slightly ahead on this measure. Across checks, Antler Gate learned how often each check fails in practice and what kinds of changes it reacts to. That information is in the review history rather than in the rule text, and it is what brings Antler Gate level with Opus overall.

Deployment requirements

Antler Gate 1 is a research result, and HeyDeer runs every matching check today. A production gate would need to:

• Fail open: run the check if the gate errors, times out or sees a truncated diff.
• Show skips: list each skipped check in the review with its reason, with a way to run it anyway.
• Gate per check: apply only to checks with enough review history, and run new checks in full.
• Set the threshold by recall: start from the fail recall a team accepts, then read off the savings.

Conclusion

Supervised post-training with a one-token target was enough to bring an 18B-active open model level with Opus 5.5 on predicting check verdicts for checks with history, at about a sixtieth of the cost per call. Most of the work was in the data: rebuilding inputs from git, controlling label drift and splitting by time. On the modelling side, the one-token target and log-probability scoring mattered most, because they give a fine-grained threshold at the cost of a single output token.