Profitable in Simulation, Losing Live: An Honest Dry Run and Testing an AI Trading Model's Hit Rate

Same AI trading bot: live, it lost about $11 an hour, while the simulator stayed in the green.

I changed the simulator so it pays gas and gets orders rejected, then ran a 24-hour sample to test the Jev model's hit rate. It came out at 50.1%, lower than the 50.8% you'd get by always guessing the same side, and its confidence had nothing to do with outcomes.

The 53.4% I saw on a small sample the day before was luck, and I shut the project down under the exit rule I had written in advance.

This is the second jev-trader post. The engineering problems of putting the bot on Monad mainnet are covered in gas costs and margin accounting for an AI trading bot on Monad.

In dry run, gasMon was hardcoded to 0 and orders were never rejected. A ruler that always says you're making money can't help you judge anything before real money goes in.

The other problem is more fundamental. Which side to quote is entirely Jev's call, and if it has no predictive power, no optimization downstream has anything to stand on.

An honest simulator

The acceptance criterion was that the simulator's gas cost and rejection rate match live.

Gas limit uses the 438000 estimated at live startup, multiplied by the real-time gas price. The rejection rate defaults to 0.33, calibrated from live data.

Which blocks get rejected is decided by a deterministic hash of the block number, so replaying the same stretch of market gives identical results. A rejected order doesn't go on the book and the previous one stays live, which is what a live revert does.

The bug in this part was caught by a test.

JavaScript's xor leaves a signed integer, so whenever the remainder went negative, < rate was always true and the rejection rate became 67%. Adding >>> 0 to convert back to unsigned fixed it.

I originally thought the fill model was too optimistic. Checking the data, the live fill rate was 18% and the simulator's 9% to 11%. There was no evidence of optimism, and without evidence I didn't change it.

After the fixes I ran 245 blocks to verify: 31.8% rejected, gas working out to $13.4 an hour, P&L negative.

The gap from live's $11 an hour comes from the 14% of live blocks that were late, sent no order, and so paid no gas.

Record state, score offline

The most direct test is to record each Jev call and, 100 blocks later (the model's prediction horizon, about 30 seconds), check whether the mid price moved that way.

But asking Jev on every block costs $18 a day. And at this horizon, adjacent blocks are nearly the same sample.

So instead I save the model's input every 20 blocks and later replay them to the model offline for scoring. That brings the cost down to about $1 a day.

Two things in the scoring have to be right.

First, the comparison is against an "always guess the same side" baseline, not 50%. In a market trending one way, always guessing up beats 50%.

Second, the samples aren't independent. Sampling every 20 blocks with a 100-block horizon, adjacent samples share 80% of their future, so the error has to be computed from the effective sample size:

// Calls closer together than the horizon share most of their outcome, so they are not independent samples.
const gap = n > 1 ? (finalBlock - firstBlock) / (n - 1) : config.horizonBlocks;
const effective = Math.max(1, n * Math.min(1, gap / config.horizonBlocks));
const se = Math.sqrt(0.25 / effective) * 100;

Without this adjustment the error band is 2.2 times too narrow, and noise easily passes for signal.

53.4% on a small sample

After a few hours, a first rough look covered 2,420 calls, about 489 independent samples: hit rate 53.4%, baseline 51.4%, error ±4.5. Indistinguishable.

When the model said 80% to 100% chance of up, price actually went up 55.2% of the time; when it said 0% to 20%, 48.2%. A 7-point gap, about 1.4 standard deviations.

Even if 53.4% were real, it wouldn't be enough to profit.

As a directional edge that's about 0.49 bps per call, against a median spread of 3.62 bps.

I replayed 4.1 hours of real prices. Even a Jev that was right 100% of the time would cap directional profit at $3.15 an hour, not enough to cover $11 to $13 an hour in gas.

The intuitive fix is to send fewer orders. The model's horizon is 30 seconds, so its view doesn't change every 0.3 seconds. Cancelling and requoting only when its call flips could cut transaction count by one to two orders of magnitude.

But lowering frequency only saves gas. Without a signal there's nothing to earn.

So I didn't start building right away. I wrote the rule first: the 24-hour hit rate has to beat always guessing the same side by more than the error band before I build anything. If it sits around 50%, I don't.

The 24-hour test: 50.1%

The next day the test finished: 14,214 calls, about 2,871 independent samples, $1.00 in Jev fees.

hit rate        50.1%  (coin: 50% ± 1.9 at 95%)
always one side 50.8%  <- the number to beat
calibration: when it said buy with probability p, how often did price go up
  p 0-20%    n= 6567  went up 48.8%
  p 20-40%   n=  569  went up 51.3%
  p 40-60%   n=  545  went up 50.6%
  p 60-80%   n= 1754  went up 49.7%
  p 80-100%  n= 4779  went up 49.0%

The hit rate was below the baseline. Calibration, how well the stated probability matches how often price actually went up, was completely flat: confidence had no relationship to outcomes.

The previous day's 53.4% and the 7-point high-confidence gap both vanished once the sample grew sixfold.

Over the same period the simulator also ran a full 24 hours (with a mock model): 287,337 blocks, 33.1% rejected, $319.59 in gas, $13.3 an hour.

Whatever spread the quotes captured was eaten by price continuing to move against the position after fills, which has nothing to do with the model.

Following the rule I'd set the day before, I dropped the flip-only mode, stopped the simulator, and deleted everything on the server. Live funds had already been fully withdrawn before this.

The honest simulator itself stays, so the next time I test a different strategy I'll have a ruler I can trust.

Keep good-looking small-sample numbers out of your conclusions. The 53.4% was gone at six times the sample.

Write the exit rule before you see results, and let the rule decide when they come in.