There's a genre of blog post where someone builds a model, reports the holdout numbers, and quietly omits that the live numbers were worse. This is the opposite of that post. A builder of stock-pattern scoring models published the pipeline, the traps, and then a section headed, in effect: the live numbers are weaker than the holdout numbers, and I want that stated plainly.
The system scans about 6,000 US stocks nightly for sixteen bullish chart patterns and scores each detection from 0 to 40. The training pipeline has run nearly 800 times since May. One pass over all detectors takes about 45 minutes on a Mac. The fitted models are small: the median out-of-sample rank correlation with the two-week outcome is about 0.03, and a good one reaches 0.08 or 0.09.
At those effect sizes, most of the engineering isn't modelling. The training tree is about 88,000 lines of Python, and the part that fits a model is a small slice. The rest is the ruler — the splits, gates, ledgers, and audits that decide whether a model is real.
Six ways a model fools itself
The post lists six overfitting traps, and the useful thing is that they're different problems with different fixes.
Time leakage. Two detections ten days apart on the same stock share four days of price path; a shuffled split puts them on opposite sides of a fold. Purged, embargoed folds fix it. A subtler version survived until July: the early-stopping set was the same fold used for scoring.
Selection leakage. Fit fifty models and the best looks good whether or not signal exists. Deflating for the number of attempts is the fix — with the admission that across the first four October pipeline versions, the deflated verdict was false on 150 of 164 reads.
Calendar leakage. Too many models leaned on market-wide features and were learning the day, not the stock. When measured: median validation win-rate spread between top and bottom bands was 13.5 points; the within-date spread was 4.4; on the holdout both collapsed to about 2. Half the validation spread was market timing, and it didn't transfer. The fix was a date-share gate and testing correlation within trading days, treating the number of trading days (~400) rather than rows (~10,000) as the sample size.
Data-draw variance. Each night draws a different 90,000-row sample from a pool two or three times larger. Same screen, different seed: 52% of admitted features turned over. Two models of the same detector from different draws shared 29% of their top-5% holdout picks.
Holdout integrity. A frozen holdout protects nothing if it stops being frozen. Ours broke five ways, including an integrity check that compared nothing with nothing and passed, and a freeze that matched 0 of 33,013 rows without a word.
Tooling leaks. A research tool had a column called close_at_rec that was the close of the entry day, not the detection day. One day of the future, worth a 20.8-point win-rate split. The heuristic: a signal with the same sign on every detector in every period is more likely a leak than a law.
The result that matters
After all of that, the live read is this: on this year's production rows, at a one-week horizon within day and market, the score's top quintile carries a beta of about 1.0 against 0.4–0.7 in the bottom quintile, while the beat rate across quintiles is flat or falling. The pattern quality score ordered market exposure rather than outcome. Only one detector in sixteen ordered excess return the right way with an interval excluding zero.
That's a model that works in the way a leveraged index fund works: it wins when the market rises, and the holdout windows were mostly rising.
The author is explicit that both readings can be true — the holdout is a two-week horizon over a mostly-up period; the live read is one week including a bad stretch — and then says which one he weights. The live one.
Why this is worth reading
Three things generalize beyond trading.
Most of the code should be measurement. The ratio of ruler to model is the interesting number. If your evaluation code is smaller than your fitting code, you're probably measuring your optimism.
A holdout is a resource, not a test. Reading it fifty times and picking the best read is selection leakage at the pipeline level. This one spent reads cautiously, logged every one, and still says a Deploy rating is "the best of many reads of one window."
The honest disclosure is the product. The write-up ends with a note that the deploy rule doesn't yet measure the exposure the top band carries, and that adding a beta budget is the next change. That's a roadmap derived from a failure, published before the failure could be discovered by a customer.
The scores may or may not end up working. But the pipeline that produced them is now the most valuable part of the system, because it's the thing that can tell the difference between a model that predicts and a model that rides the market — and it's the only reason anyone would know.
Source: How We Train Stock Pattern Scores: 600 Runs, 6 Overfitting Traps — Stock Data Analytics.