The Reconciliation Alert Fired Twice: Missed and Double-Counted Interest When Bitfinex Positions Switch Sides

My reconciliation alert has fired twice: once at 0.970, with accrual missing from the denominator, and once at 0.488, with the same stretch of interest counted twice.

The ratio behind it is a 7-day "interest actually paid ÷ booked accrual." My Bitfinex auto-lending platform uses it to check whether the earnings users see are computed correctly.

The errors went in opposite directions, but the root cause was the same. Bitfinex splits a single lending position across credits and loans, and when a position switches sides it gets a new id but keeps its original opening time.

Both times the automated triage got the reasoning wrong. What actually found the cause was reconciling the numbers until they converged.

How the ratio works

The source of truth for "currently lent" is the position snapshot Bitfinex returns, not the bot's own order table.

The order table is a record of what the bot did. Cancellations, partial fills, and manual operations can all leave stale rows behind, and patching every kind of leftover just means maintaining a secondary projection.

Bitfinex pays interest once a day, already net of its 15% fee. On my side, from the position snapshots, I spread each position's amount × rate × seconds across the days it covers, second by second. That gives booked accrual, the interest that should have been earned, and I divide the two:

funding_recon_ratio = total interest actually paid over 7 days / total booked accrual over 7 days

Since payouts are net of the fee, a clean account should land around 0.85.

If the ratio stays outside its band for 3 days in a row, the alert fires. Too low means accrual is overcounted, and too high means a position is missing.

The ratio doesn't depend on the size of the amounts, and there's one per account per currency, so when an account starts drifting I know where to look.

One position, two homes

Bitfinex exposes lending positions through two endpoints, /funding/credits and /funding/loans. Money the borrower hasn't used yet sits on the loans side, and once they start using it, it moves to the credits side.

The two sides have separate id namespaces, so a position that switches sides gets a new id. It keeps its opening time MTS_OPENING, and its amount doesn't change.

Closed positions go into each side's hist, the history of finished positions. The credits hist is written to my credit history table and the loans hist isn't, because the id namespaces differ and mixing them would pollute the ledger.

After the 120-day bid-take lane for Bitfinex lending launched, most of its fills landed on the loans side.

Both of the bugs below have to do with that side.

First time, 0.970: closed positions vanish

About ten days after bid-take launched, the alert fired for three days on one production account, with a ratio of 0.970.

The read-only Claude Code triage pointed at "the daily materialization job computing pro-rata wrong for the new lane." It was half right about the direction and wrong about the mechanism.

The position snapshot mirrors live positions, and when a position leaves the exchange, its row is deleted. On the credits side the position later reaches the history table through hist. On the loans side it doesn't.

So the instant a bid-take position closed, the accrual for its entire lifetime disappeared from the books. That undercounted the denominator and pushed the ratio up.

There happened to be five such positions in the window. I added them back second by second, and the ratio went from 0.970 to 0.851.

The difference was exactly those five, so there was no need to look for a second cause. For reconciliation bugs, get the numbers to converge before touching anything.

The fix was a closed-position snapshot table. Before deleting a departing position, the job records its amount, rate, opening time, and closing time, and the daily materialization job reads that table as a third source.

The one thing to get right was that a side switch must not count as a close.

When a position moves from loans to credits, the old id disappears and a new one appears. That looks exactly like a close plus a new open, and writing it to the closed table would overlap with the new position. So if one id leaves and another arrives in the same cycle with the same opening time and amount, it's treated as a side switch and no close record is written.

I also considered having the materialization job remember the accrual it had already computed and only ever increase it. That would also block genuine corrections later, trading a visible error for an invisible one.

Second time, 0.488: a stretch counted twice

Two weeks later the alert fired again, on a different account, with a ratio of 0.488. This time the denominator was overcounted.

The triage's theory was "the user paused lending and the snapshot froze," but both configs were active, so that didn't hold up.

I laid out that account's daily accrual day by day. From a certain day on, the number of open positions went from 1 to 2, and accrual exactly doubled.

Then I matched up rows in the history table and the closed-position snapshots that shared an opening time and amount. A 120-day position had switched from credits to loans. The old credit's hist covered the stretch from opening to the switch, and the new loan started accruing from the opening time all over again, so every day in the overlap was counted twice.

Switches aren't rare, either: of the 6,528 hist credits from the last 60 days, 1,452 came from switches.

The other direction had a problem too. The switch detection I'd added the first time let one new id swallow every old id with the same key. Equal-sized positions from the same batch of fills often switch together, so only the first was recognized as a switch and the rest were recorded as closes.

Bitfinex already tells us which endpoint a position came from. I just hadn't stored it.

The fix was to add a from_loans column to both snapshot tables. For rows from the loans side, the materialization job looks for a hist credit with the same opening time and amount, and only starts accruing after that credit ends. If one of them is still open, the whole stretch isn't counted.

That single rule is correct in both directions. Switch detection was also changed so that one new id matches only one old id.

After recomputing, the 7-day ratios for the three affected accounts:

None of the loans close rows backfilled on production collided on key with a hist credit, so production wasn't hit by this bug.

Triage is only a lead

The first time, triage guessed the right lane but the wrong place where the math broke. The second time it was wrong about what was broken at all. Following its advice would have meant changing code that wasn't broken, both times.

MTS_OPENING is the economic opening time, not the birth time of that id.

Anywhere you use the opening time as the start of accrual and also sum across multiple ids, ask whether the same position could show up twice under different ids.

Store whatever the exchange already tells you the first time around. Reconstructing it later means guessing from times and amounts.