---
title: "Higher F1, Worse Product"
slug: "compile-before-fine-tune"
language: "en"
date: "2026-10-28T09:14:46.634Z"
updated: "2026-10-28T09:14:46.000Z"
description: "A compiled payment-risk program beat frontier models on F1 and cost, then failed the precision target. The failure is the useful part."
canonical: "https://fix.slopit.io/compile-before-fine-tune/"
tags: ["machine-learning", "evaluation", "cost", "engineering"]
---

Seldon published an experiment on payment-risk classification, and the headline is the kind of number a vendor would frame as a win: higher F1 and recall than both GPT-5.6 Sol and Luna, at roughly 1/124th the inference cost of Sol and 1/27th of Luna.

The catch is in the same table. Precision was 21.64%. Sol's was 65.17%.

If the decision is "block this payment automatically," precision is not a secondary metric — it's the thing that determines how many legitimate customers you annoy. A model that catches more fraud while flagging three times as many good payments is not obviously better; it's a different operating point, and the business has to choose it explicitly.

To Seldon's credit, the post says this plainly. It also reports that when they tried to constrain the search to match Sol's precision, the search failed.

## The setup

The task: take a card payment and its history, return low/medium/high risk, treat only "high" as a fraud-positive. Data was IBM's TabFormer synthetic transactions, with fraud labels excluded from the inputs. Four approaches: direct Sol, direct Luna, a Seldon-compiled program without a precision floor, and a Seldon-compiled program constrained to maximize F1 while hitting at least Sol's 65.17% precision.

The unconstrained compile found a specialist decision model (Jev) configured in a particular way, and it beat both frontier baselines on F1 and recall at a fraction of the cost. The constrained compile did not find anything that met the precision floor. The best candidate reached 32.14% precision and 19.50% F1 — better than the unconstrained program's precision, nowhere near Sol's.

The post keeps that row in the table labeled as a failed selection, with a note that the cohort was different and it shouldn't be read as a controlled comparison. That's the most credible thing in it.

## Why the failure matters more than the win

The successful case is the one the vendor wants you to remember: a compiled program can be much cheaper and catch more fraud. The failed case is the one a practitioner should remember: an optimization target specified in the language of the business ("match Sol's precision") was not reachable by the search, and the system stopped rather than shipping something that looked good on F1.

That's the correct behavior, and it's also the answer to the obvious objection to "compilation." An ordinary compiler preserves program semantics — if the output behaves differently, the compiler is broken. Replacing a learned judgment with a different model does not preserve semantics; it changes behavior in ways only evaluation can reveal. The compiler analogy is useful for the *interface* (keep the function, swap the implementation) and misleading for the *guarantee*.

The post says this too: "matching the JSON schema does not prove matching behavior."

## The sample-size anecdote

The detail I'd pin to the wall: some saved internal training evaluations contained 32 cases and two frauds. One correct fraud prediction with no false positives produced an apparent 100% precision. On the subsequent 40-case development evaluation, the three published alternatives measured 19.72%, 0%, and 0%.

This is the whole problem with model evaluation in miniature. A benchmark that small can't distinguish a good classifier from a lucky one, and the number it produces is precise enough to look authoritative. The post uses it to explain why they added a 500-case selection stage — which is the right fix — but the anecdote is more persuasive than the methodology section.

## What generalizes

Three things, independent of the vendor.

**Define the operating target before you optimize.** "Make the model better" is underspecified; "maximize F1 subject to precision ≥ 65%" is testable and can fail, which is what you want. The failure is information.

**Compare against the cheapest credible baseline.** Here that was Luna, and it was much cheaper than Sol with similar F1. If you haven't priced the cheap model, you don't know what the expensive one is buying.

**Account for the whole pipeline.** The cost table excludes compilation, candidate selection, engineering, and review. Inference at $0.055 per 1,000 cases is striking; the total cost of the experiment that found that program is not in the table. That's not dishonesty — it's the normal boundary of a benchmark — but it's where the real decision lives.

## The honest summary

The experiment supports a narrow claim: on a synthetic payment-risk benchmark, a compiled program achieved higher F1 and recall at much lower inference cost, while sacrificing precision, and a constrained search did not recover the precision. It does not support "compilation beats fine-tuning," and the authors say so.

That's a good paper. The most useful sentence in it isn't the cost reduction; it's the admission that the constraint failed and the run stopped.

*Source: [Before you fine-tune, compile: lessons from a payment-risk experiment — Seldon](https://seldon-ai.com/blog/before-you-fine-tune-compile).*