Learning Jev with Personal Data

I’ve been wanting to learn Jev, but there’s not much in my day-to-day work that I can use it for yet. So I thought I’d take some bet data I’ve been collecting and learn it while making a post. Every section has a point where you make a quick decision based on limited information, and your progress saves in the browser.

The subject is Jev, the first “System One” model from TypeSafe, and decision models in general. I didn’t want to use made-up examples, so everything here runs on my own data.

Since April, I’ve logged Rithmm’s sports bet picks so far, with 252 at by the time I was writing this. My original experiment with this was to see if I could use the Martingale betting theory in limited sets and profit more than I would have with money in a high-yield savings account. The answer is no. However, after reviewing the data, I found that Rithmm hit at a profitable rate on NBA picks with 60% or higher confidence. WNBA was in range as it went on, but MLB was not as successful. Jev leads me to some new experiments.

I sent the ones that settled as wins or losses through Jev and saved every answer. The question running through the whole course is pretty simple: can a decision model help me choose which picks to bet?

Start with a guess

Before any model gets involved, you try it. Here’s what I see before a game, minus the result.

What you just did is what Daniel Kahneman calls System 1 thinking in Thinking, Fast and Slow: a fast gut call without much deliberating. Rithmm’s confidence number is supposed to make that call easier. Across the 248 picks where it gave one, Rithmm averaged 63.8%, and the picks hit 58.1%. It also never went below 56% on anything, so as a filter it says yes to every pick it hands me.

What a decision model is

System 2 is the slow, effortful reasoning, and it’s what most of the LLM conversation for the last few years has been about: models that write, plan, and reason out loud. Jev goes the other way. You hand it some text or JSON, ask it questions, and get back typed answers with probabilities your code can branch on. It can’t write a sentence, summarize, or chat.

Every question is one of three types, and each maps to something you already write in code:

• A Noul (as in Bernoulli) is a yes/no question. It returns one number between 0 and 1, the probability the answer is yes. It’s an if.
• A Choice picks one option from a set you define. It returns the winner, a probability for every option, and a confidence. It’s a switch.
• A Score places something on ordered levels you describe. It returns a position, a probability per level, and a confidence. It’s what you sort by.

It’s also cheap enough that cost barely comes up. My whole run was 504 requests (every pick twice, more on that in a minute) and 330,106 input tokens. At $0.042 per million input tokens, with output tokens free, that’s about a penny and a half, and it finished in 8 seconds.

A pick becomes state

A request has two parts. The state is the thing being judged: a string, a JSON object, or an array of strings (text only for now). The questions are what you want to know about it. Every question sees the same state, each one is answered on its own, and they all come back in one response, so asking four questions doesn’t cost four round trips.

The state docs suggest keeping the state to facts and putting your framing in the questions. For a pick, the facts are easy: the date, sport, matchup, the pick, the odds, and recent form. Nothing about the result or what I bet. Then I asked every pick the same four questions: will it win, does recent form back it, what kind of bet is it, and how strong is the case at these odds.

Here’s the twist. I sent every pick twice, once with Rithmm’s projection in the state (“Tigers 5.9, Angels 3.2”) and once without it.

{
  "will_hit": {
    "type": "noul",
    "instructions": "Will this bet win?",
    "criteria": {
      "true": "The pick as written wins when the game is settled",
      "false": "The pick as written loses"
    }
  },
  "form_backs_pick": {
    "type": "noul",
    "instructions": "Does the player's or team's recent form support this pick?",
    "criteria": {
      "true": "The recent form record points the same direction as the pick",
      "false": "Recent form is mixed, thin, or points the other way"
    }
  },
  "bet_type": {
    "type": "choice",
    "instructions": "What kind of bet is this?",
    "criteria": {
      "moneyline": "A team to win the game outright",
      "spread": "A team to cover a point spread or run line",
      "game_total": "The combined score of both teams over or under a number",
      "player_over": "A player stat over a number (hits, points, rebounds, receptions, and so on)",
      "player_under": "A player stat under a number"
    }
  },
  "lean": {
    "type": "score",
    "instructions": "How strong is the case for placing this bet at these odds?",
    "criteria": [
      "Pass; the case doesn't justify the price",
      "Lean; some support, but a small stake at most",
      "Play; the case clearly beats the price"
    ]
  }
}

Rithmm’s projection is an opinion about the game, but sitting in the state it reads like a fact. With it, Jev’s pass/lean/play Score landed on “play” for 69 picks. Without it, Jev never said “play” once. Look at the Tigers moneyline: recent form 2/10, a lean of 0.07 on the facts alone, and 1.21 once Rithmm’s projected runs show up. It lost. Adding the projection also made Jev worse at telling winners from losers overall, which we’ll get to in a couple of sections.

If you want to compare Jev against another model, the lesson is to keep the other model out of the request and compare the two numbers in your own code.

Reading the three answers

The Noul is the simplest. “Does recent form support this pick?” came back 0.84 for a receptions over with 9/10 recent form, and 0.10 for a Sky moneyline with 1/10. That one number turns out to be one of the most useful things in the whole run, so keep it in mind.

The Choice is where Jev caught some data issues. I asked what kind of bet each pick was and compared the answer to the labels in my own data. The first time I checked, Jev “missed” 40 of them. It hadn’t. My bots had filed 39 run lines, game totals, and a moneyline as “combo” player props. Jev was right, my labels were wrong, and I fixed them. A few of them:

It also caught one more: I typed a total bases pick as an Under when I actually bet the Over. After fixing both, Jev matches my labels on all 252.

The Score is the probability-weighted average of the level positions, so with pass at 0, lean at 1, and play at 2, a 0.9 means “mostly lean.” The Score docs say to describe situations instead of degrees, which is why my levels read like “Lean; some support, but a small stake at most” instead of just “1.” The catch is that the average hides how the probability is spread.

That’s what confidence is for. The confidence docs put it at 1 when all the probability sits on one level and 0 when it’s spread about evenly. Check it before your code acts on the score.

Thresholds are break-even

A Noul gives you a probability and leaves the decision to you. The Noul docs suggest starting with 0.7 for yes and 0.3 for no, sending the middle to a person, and raising the bar when a false yes is expensive.

Betting hands you the line for free, because the odds already tell you what being wrong costs. At -150 you risk 150 to win 100, so you need to win 150 out of 250, or 60%, just to break even. At +120 it’s 100 out of 220, about 45%. Every pick has its own threshold, and your code can work it out:

const breakEven = odds < 0 ? -odds / (-odds + 100) : 100 / (odds + 100);

if (answers.will_hit.noul > breakEven + margin) placeBet(pick);

At zero margin, Jev on the facts alone picks 86 of the 248 picks and goes 57–29 for +19.42 units, compared with +6.43 for betting all of them. With Rithmm’s projection in the state, it bets 135 and finishes +16.57. Rithmm’s own number cleared break-even on every single pick, so its version of the rule is “bet everything.”

One thing I was careful about: zero margin is just break-even, so I didn’t tune it. Dragging the slider until the units peak is exactly the kind of fitting the docs warn about (“avoid fitting thresholds closely to a handful of test cases”). It’s fun to look at, but the best margin on these 248 picks won’t translate to the next 248.

Did the numbers mean anything?

There are two different questions you can ask about any set of probabilities, and a model can pass one and fail the other:

• Calibration: when it says 60%, do those picks hit about 60% of the time?
• Ranking: do the winners get higher numbers than the losers?

An independent evaluation of Jev found that its yes/no split probabilities ranked well but were poorly placed against a fixed 0.5 cutoff. Here’s how it looks on my picks.

Rithmm’s points sit on a flat line around 60% no matter what it said. Its most confident group (67% to 74%) hit 62.5%, about the same as its least confident group. That’s a ranking skill of 0.49, a coin flip, and its probabilities were slightly less accurate than using the break-even rates from the odds.

Jev on the facts alone climbs. Its least confident group hit 56%, and its most confident group hit 69%. It was too cautious overall (averaging 54% on picks that hit 58%), but it put them in roughly the right order. If you’re wondering what “ranking skill” means in practice, this is the easiest way to see it.

Where it breaks

I like what I saw. I also think the quickest way to get burned by a new model is to believe the first good chart, so here’s what I don’t trust yet.

Jev might have seen the results

Every one of these games happened between April and October, and Jev 1.14 became available on October 14. If any of these results ended up in its training data, the “will it win” numbers would look smart for the wrong reason. The gap is small, though. Jev averaged 54.5% on picks that hit and 52.8% on picks that missed, which doesn’t look like a model peeking. And a lot of the signal is sitting right there in the input: the raw recent form number (7/10 as 0.7) ranks winners at 0.56 all by itself, and Jev’s “does form back this pick?” Noul ranks at 0.59. I still can’t rule it out.

The sample is small

57–29 sounds great. Fewer than 1 in 200 random 86-bet slices of these picks do that well, so it probably isn’t pure luck. But Jev’s ranking skill ranges from 0.51 to 0.65, and the bottom of that range is barely better than a coin flip.

Typed doesn’t mean right

The launch post says Jev “can’t hallucinate” and “never makes type errors,” and both held up here: every answer came back well-formed. But “0.62 that this bet wins” is a valid answer whether or not Jev knows anything about baseball.

The benchmarks are the vendor’s

TypeSafe’s own launch post notes its workflow evals were made by its own team, “so some bias could exist.” Independent work is mixed so far. One evaluation ran 346,009 requests across 37 datasets for under $10 and found Choice probabilities well calibrated, while another paper found neither model it tested beat chance at zero-shot model routing.

Pin the version

I used jev-1.14.0, not jev-latest. The alias will move, and every number in this post could move with it.

So, can it help me choose?

On these 248 picks, a plain break-even rule on Jev’s facts-only read beat betting everything, and Rithmm’s own number couldn’t. I’m not betting on that until it holds up on games Jev couldn’t have seen.

What’s next

The only clean test is going forward: score each pick before the game, log the answer with the model version, and check back after a few hundred more. That’s what I’m building next.

Cheat sheet

• A Noul is an if: one condition, one probability, and the threshold lives in your code.
• A Choice is a switch. Describe every option and check confidence before you act.
• A Score is a sort. Describe each level as a situation and read the probabilities along with the number.
• The state holds facts. Another model’s projection is an opinion, so compare it in your code.
• In betting, the threshold is break-even, and the odds hand it to you.
• Check calibration and ranking separately, on your own data, with enough of it to mean something.
• Pin the model version and log it with every answer.

Baseline by sport

Before Jev becomes part of my daily tracking, here’s where things stand without it. This covers all 253 settled Rithmm picks from April 18 through October 18. “All picks” counts everything Rithmm sent me, including picks I only watched or skipped. “Bet” counts only the picks I put money on.

Football and basketball carried it. MLB gave nearly all of it back: Rithmm was barely better than a coin flip on baseball overall, and the baseball picks I actually bet went 5–14. WNBA and college football are too small to say anything about yet. This is the number Jev has to beat going forward, sport by sport.

Sources

• Introducing System One models and Jev, TypeSafe’s launch post
• TypeSafe docs: System One, State, Noul, Choice, Score, Confidence, Models
• Evaluating and Benchmarking the System One Model Jev by Deußer, Sparrenberg, and Sifa
• Fast Models, Slow Evidence by Jiawei Li