JevForge — decision tooling for web interaction

01 · Demo

Watch it choose, element by element

A live-capable autopilot on a mock travel site. Every control on the page is a real candidate from a recorded Mind2Web annotation. Two cursors work the same page — JevForge in orange, Jev-1.13 in gray — each glides to its own pick and clicks it; the amber ring marks the gold answer, and each verdict lands at its click point. Pages turn like real browsing; it runs all 48 held-out samples on a loop.

connecting to /v1/systemone · sample — / 48 · —

…

—

✈ FlightCenter — Fall fares ✕ ◈ JevForge demo ✕ ＋

‹ › ⟳ 🔒 https://www.flightcenter.com/deals ⟳ ⚡JevForge— ms ⚡Jev-1.13— ms autopilot

Task—

✈FlightCenterFARES·DEALS

Fall fares: fly warm for less

Hand-picked round trips under $298 — book by Oct 31 and lock in summer's lowest fares to 40+ sunny cities.

From

Depart

Today's lowest fares

Round trip, incl. taxes · updated 4 min ago

Destination	Dates	Fare

✦ Member Week

Extra 15% off select Caribbean routes — Nov sailings only.

Explore

Weekend escapes Business class upgrades Gift cards

The complete replay is scaled to the current viewport.

02 · Speed

System-One speed, measured

Same Jev contract, same request — three questions (choice + score + noul) answered per call. JevForge serves them from a local GPU in a single forward pass; Jev-1.13 round-trips a generation API. Lower is better.

JevForge 0.8B local · RTX 40900 ms

Jev-1.13 hosted API0 ms

same contract · 3 questions per request — choice, score and noul answered together in the recorded setup. It does not use autoregressive text decoding.

0 msmeasured request latency distribution1,200 ms

p50 50observed max 53

⚡

1 forward pass · 0 output tokens

The whole answer — a full distribution over candidates, a difficulty score and per-statement truths — is read straight off the logits. No autoregressive decoding, no prompt-injected reasoning or generated answer text.

21.6× fasterwarm p50 50 ms12 measured calls

03 · Results

Held-out, website-disjoint, honest

Fixed splits — a website never appears in both training and evaluation. Results below are reported as a bounded candidate-scoring comparison; one lower OOD score-MAE row is not a general superiority claim.

metric	split	Raw Qwen3.5-0.8B¹	JevForge 0.8B	Jev-1.13
choice top-1 ↑	test	0.235	0.579	0.543
	ood	0.340	0.637	0.610
noul accuracy ↑	test	—	0.826	0.910
	ood	—	0.860	0.825
noul Brier ↓	test	—	0.128	0.092
	ood	—	0.117	0.131
score MAE ↓	test	—	0.367	0.348
	ood	—	0.390	0.463

¹ Raw backbone = zero-shot JSON scoring on the first 200 records per split. noul = per-statement truth probability; Brier ↓ better; score MAE = mean absolute error of the 0–1 difficulty score.

04 · Architecture

One toolkit, the whole decision lifecycle

JevForge covers the path from raw task records to a calibrated, Jev-compatible service. Its first focus is the repeated decision loop inside web interaction: observe the page state, score the available actions, act, and repeat. The model is one output of that reproducible pipeline, not the whole project.

01 · DATABuild decision recordsnormalize raw states, questions, candidates and website-disjoint splits

02 · SYNTHESIZEComplete supervisioncreate auditable decision-label distributions with resumable caches

03 · TRAINFit and calibratetrain candidate scorers, select checkpoints and calibrate held-out probabilities

04 · SERVEEvaluate and inferrun frozen test/OOD reports and expose the Jev-compatible endpoint

JevForge candidate scoring: shared context, candidate rows, final-token pooling, shared scorer, question-level probabilities

Candidate framingshared state + question, paired with one candidate path per row

Position poolingthe last valid token becomes that candidate's representation

Shared scorerone two-layer GELU head assigns every candidate a comparable scalar

Question softmaxonly candidates from the same question compete; no text decoding

JevForge is a personal, independent research project exploring open structured-decision models and Jev-style interfaces.

choice

Which element should I act on?

A complete probability distribution over every candidate element — click targets, nav items and form fields — with ties and uncertainty preserved.

score

How hard is this page?

A 0–1 difficulty score per task, learned from recorded label distributions so downstream agents can decide when to slow down or escalate.

noul

Is this statement true here?

Per-statement truth probabilities over the page state — graded belief instead of a brittle boolean, so "maybe" survives the contract.

05 · Tutorial

Install and serve in three steps

Everything runs locally: weights from Hugging Face, an OpenAI-style endpoint on your own GPU, and the exact Jev contract your agents already speak.

Install

Clone the repo and install dependencies (Python 3.11+).

git clone https://github.com/zwliJay/jev-forge.git \
  && cd jev-forge \
  && pip install -r requirements.txt

Download weights

Pull the trained checkpoint from Hugging Face.

huggingface-cli download \
  AndeyTait/JevForge-0.8B \
  --local-dir ckpt

Serve & call

Start the Jev-compatible server, then ask it anything.

python -m jevforge.serve \
  --checkpoint-dir ckpt \
  --port 8123

# Jev contract in, probabilities out
curl localhost:8123/v1/systemone \
  -H 'content-type: application/json' \
  -d '{"state":"Task: open the privacy
    settings page. Elements: [e1] link
    Settings, [e2] button Submit, [e3]
    link Privacy","questions":{
      "next":{"type":"choice",
        "instructions":"Which element
        should be interacted with next?",
        "criteria":{"e1":"link Settings",
          "e2":"button Submit",
          "e3":"link Privacy"}},
      "is_link":{"type":"noul",
        "instructions":"e1 opens the
        privacy settings."}}}'

🛠 Train your own — the whole assembly line ▾

Three entry points take you from raw pages to a served model. Or run scripts/run_web_pipeline.sh for all of it in one shot.

# 1 · Mind2Web → Jev-contract decision records, website-disjoint splits
python -m jevforge.build_web

# 2 · complete missing decision labels — streaming, resumable cache
python -m jevforge.synthesize

# 3 · CE + Brier on complete candidate sets, cosine LR, temperature calibration
python -m jevforge.train

Preliminary RLCD path Starting from the supervised 0.8B checkpoint, JevForge updates complete candidate distributions with task utility + multiclass Brier reward + reference-policy KL. Grouped sampling uses a within-group reward baseline; browser hard-negative views keep every gold action and add confusable candidates only to training. The plain run moved choice top-1 from 57.87% to 58.75% on test and 63.73% to 64.25% OOD; one hard-negative view reached 64.77% OOD. The general supervised checkpoint remains the release default because ordered-score quality trades off in this first sweep. Results and method ↗

⚡ Recorded v1 recipe: 1,200 steps on one RTX 4080 SUPER. Reproduction time and cost depend on hardware and label-provider availability.