Skip to content
University of Chicago logo

University of Chicago

Research Collaborator·Apr 2026 - Jun 20263m

Worked with Prof. Haifeng Xu on AI Prophet, a benchmark for evaluating LLM forecasting capabilities and model performance.

Key contributions
  • Worked on AI Prophet, a benchmark tool for evaluating LLM forecasting capabilities and model performance.
  • Hardened the live trading path: fixed a ledger-divergence bug in the net-flip order flow and a silent double-fill risk in order persistence.
  • Corrected the betting strategy's edge maths for prediction-market books whose YES and NO asks do not sum to 1, and covered it with regression tests.

What I built

2 featured
  1. 01

    Keeping the internal ledger honest about the exchange

    2026

    Two independent paths let the bot's own record of its positions drift permanently away from what the exchange actually held. Both are silent at the moment they happen and compound on every later tick.

    Context

    AI Prophet turns LLM forecasts into live prediction-market orders. Ledger state is not held in memory between ticks: _live_ledger_state rebuilds it from the database each cycle, so anything the database fails to record, or records optimistically, becomes the bot's reality on the very next tick.

    Contributions
    • Traced a state-divergence bug in the NET-flip path, which sells the opposite side before buying the new one: the BUY leg executed even when the SELL leg raised or returned an ERROR status. The old position was still open on the exchange and its cash had never come back, so the BUY over-committed available funds — and the next tick read a database that had logged the SELL as attempted, diverging from the exchange for good.
    • Made the flip transactional from the caller's point of view: after catching the SELL exception, the routine now checks sell_status == ERROR and returns immediately with order_placed=False and an explanatory error string. The BUY is only ever attempted after a confirmed SELL.
    • Found a second, quieter failure in _save_order(), which short-circuited whenever signal_id was None — always the case for make_trade() calls. Those orders never reached the database at all, so _live_ledger_state had no record of them and would cheerfully place the same position again on the next tick.
    • Made BettingOrder.signal_id nullable so make_trade() rows persist without a parent signal, while the foreign key still constrains the normal process_forecasts path, then dropped the signal_id guard from _save_order() and left only the database-engine check.
    Impact

    Closed both paths by which the ledger could permanently disagree with real exchange state — the failure mode that matters most here, because a rebuilt-from-database ledger never self-corrects.

    PP
  2. 02

    Edge maths that survives a book that doesn't sum to 1

    2026

    The default strategy priced the NO side as 1 - yes_ask. That identity only holds on a perfectly normalised book, which is not what real prediction markets quote.

    Context

    DefaultBettingStrategy.evaluate decides which side of a market to take and how many shares to buy, by comparing the model's probability against the market's ask. Both halves of that decision depended on the assumption that the YES and NO asks sum to exactly 1.

    Contributions
    • Showed that using (1 - yes_ask) as the NO price is only correct when yes_ask + no_ask == 1. On an overround book (ask_sum > 1) or a dutch-book (ask_sum < 1) both the NO edge and the NO share count came out wrong, which corrupted side selection and sizing together.
    • Rewrote the decision to compute yes_edge = p_yes - yes_ask and no_edge = (1 - p_yes) - no_ask independently, pick the larger edge, and size the position by that edge at the actual ask price of the side chosen.
    • Renamed the local spread in both DefaultBettingStrategy and RebalancingStrategy to ask_sum — it had always held yes_ask + no_ask, not a bid-ask spread. The public max_spread keyword stays for backwards compatibility, with the misnomer now documented.
    • Added regression tests for the three cases the old code got wrong: the NO side on an overround book, the NO side on an underround book, and picking the larger-edge side.
    Impact

    Side selection and position sizing are now correct on books that do not sum to 1 — the ordinary case on live prediction markets — rather than systematically mispricing every NO-side bet.

    P