Open Source Lev: Running a Few More Rounds Increases Accuracy from 63.2% to 76.2%

Open Source Lev: Running a Few More Rounds Increases Accuracy from 63.2% to 76.2%

2026-11-17 10:40

Introduction: Same-weight iterative deepening by difficulty increases accuracy to 76.2% across four rounds. What is Lev? Lev is an open-source decision model developed by Lehigh University and others, based on the native loop foundation (Ouro/Nanbeige), where a single set of weights undergoes repeated computation across multiple rounds, each producing option probabilities. A lightweight stopper determines how many rounds to run per question—single round for simple queries, multiple rounds for complex ones. On JevBench, Lev improves accuracy from 63.2% at one round to 76.2% at four rounds, using only about 80% of the computation with less than 1 point of accuracy loss.

What is Lev?

Lev is an open-source decision model introduced by Lehigh University and others, built upon the native loop foundation (Ouro/Nanbeige). It reuses the same set of weights across multiple iterative rounds, reading out option probabilities after each round. A lightweight stopper dynamically decides how many rounds to execute per question—simple questions answered in one round, complex ones processed through additional iterations. On JevBench, Lev achieves a jump from 63.2% accuracy at one round to 76.2% at four rounds, using only approximately 80% of the computational budget with less than a 1-point accuracy drop.

Lev

Key Features of Lev

  • Single Forward Decision Output: Takes in state, question, and candidate options, directly outputs probability scores for each option without generating reasoning text.
  • Iterative Deepening via Looping: Repeats execution of the same decoding layers across multiple rounds, reading out option probabilities at each stage; the same weight set supports arbitrary depth of inference.
  • Per-Question Adaptive Termination: Uses an embedded lightweight stopper to predict whether the current answer matches the deepest round’s output, automatically adjusting the number of rounds based on question difficulty.
  • Latency-Accuracy Trade-off Switchable: A single weight checkpoint enables switching between millisecond-level fast answers and deep thinking modes—no need to deploy separate models for different compute budgets.
  • Label-Free Self-Calibration: The stopper can be calibrated using internal business traffic without human-labeled data—approximately 500 questions per fold suffice.

Technical Principles of Lev

  • Decision Model Interface: Shared encoding across state and multiple questions. Lev adopts Kev-style input structure: a single request includes shared state and several questions, each with its own candidate options. All content is packed into one sequence and processed via block causal masking, ensuring each question sees only the shared state and its own query. This allows state to be encoded once while enabling each question to be answered as if independently posed.
  • Native Loop Foundation: “Iterative thinking” occurs within hidden states natively, unlike post-hoc loop insertion in dense models. Lev directly leverages pre-trained, loop-aware language models Ouro and Nanbeige4.2 as its base, where the same decoding layer runs R times iteratively, with each subsequent round building upon the previous hidden state.
  • Per-Round Readout + Joint Training: A single weight set covers all depths universally. After each round, the shared pointer head reads end tokens and <decide> signals from each option.
  • Stopper Mechanism: Predicts “consistency with the deepest round’s answer,” not correctness. Traditional confidence-based early stopping fails when transferred to decision models. Thus, Lev’s stopper uses a per-depth logistic regression that predicts whether the current round’s output has stabilized compared to the deepest round. Since the “deepest answer” is the model’s own final output, calibrating the stopper and selecting thresholds require no human labels—direct calibration via unlabeled production traffic is possible. Thresholds are set to the minimal value guaranteeing at least 98% consistency with the deepest answer.

digging GitHub

Follow WeChat and reply "open source" to join the AI Open Source Project Community

How to Use Lev

  • Acquire Code and Models: Clone the repository from GitHub and download the corresponding base model, Lev checkpoint, and LoRA weights from Hugging Face.
  • Prepare Input: Pack the state, questions, and respective candidates into a single sequence; use block causal masking to ensure each question only accesses its own context and the shared state.
  • Select Iteration Depth: Define the maximum number of rounds; load the shared pointer head and LoRA adapter.
  • Iterative Inference and Readout: Run the same decoding layers iteratively; after each round, read out option probabilities. Computation continues from the prior hidden state—no need to re-read input.
  • Apply Stopper: Use the trained logistic regression stopper to determine whether the current answer matches the deepest round’s output; halt early if consistent—each question exits independently.
  • Calibrate Before Deployment: Fit temperature scaling and stop thresholds using your own unlabeled production requests across each depth level.

Core Advantages of Lev

  • One Weight Set, Multiple Depths: A single checkpoint switches between fast response and deep thinking via iteration count—no need to deploy separate models for different compute budgets.
  • Significant Gains on Hard Problems: On JevBench, accuracy rises from 63.2% (one round) to 76.2% (four rounds), with improvements primarily driven by standard and hard questions.
  • Continuation Without Re-reading: Computation resumes from the prior hidden state, eliminating latency overhead from re-reading inputs as seen in cascaded architectures.
  • Computation Optimized for Value: The stopper automatically allocates more rounds to harder problems—using only ~76%–85% of total compute with less than 1-point accuracy loss.
  • Label-Free Calibration: The stopper targets consistency with the deepest answer, enabling direct calibration using internal business traffic—~500 questions per fold suffice.
  • Superior Low-Latency Performance: Outperforms comparable cascade approaches by 7.0–8.8 points and best dense models by 5.4–7.5 points in the 39–46ms latency window.

Project Links for Lev

  • Official Website: https://openlair.github.io/lev/
  • GitHub Repository: https://github.com/OpenLAIR/lev
  • Hugging Face Model Hub: https://huggingface.co/collections/songdj/lev
  • arXiv Technical Paper: https://arxiv.org/pdf/2610.20951

Competitive Comparison with Similar Projects

Comparison Dimension Lev (illustrated with Lev-2.6B) Jev
Provider Type Academic team (Lehigh University et al.), fully open-source (code/weights/data) Commercial product by TypeSafe AI, closed-source, API-only deployment
Architecture Native loop foundation (Ouro), reusable layers across multiple rounds Dense, single forward pass with one-shot output, no looping
Parameter Scale 2.6B (also available in 1.4B, 3B) Not disclosed
JevBench Public Benchmark Accuracy 76.2% (four rounds); 63.2% (one round) Approx. 86.6% (200/231, third-party measurement)
Latency Local: 39–150 ms (depth-dependent) API p50: ~616 ms (including network transmission)
Compute Depth Adjustment Supported: One weight set enables multiple workpoints; hard questions auto-increase rounds Not supported: Fixed single forward pass
Calibration Requires per-depth temperature scaling + stopper calibration (label-free) Out-of-the-box ready; high-quality calibration (ECE 0.053)
Cost Weights free; self-hosting cost depends on in-house GPU resources ~$32 per million decisions (with examples), billed per input token
Data Compliance / On-Premise Deployment Supported: Fully offline self-deployment possible Not supported: Data must leave premises to reach hosted API
Maturity Level Academic prototype; base models limited to 1.4B–3B Production-grade commercial service, continuously evolving

Application Scenarios for Lev

  • Intelligent Customer Service Intent Routing: Simple queries resolved in one round with sub-second response; complex intents automatically increase iterations—same weight set balances speed and routing accuracy.
  • Content Moderation Tiered Processing: Clearly violating content rapidly approved/rejected; ambiguous cases undergo deeper analysis—overall saves 15%–24% compute and reduces false negatives.
  • Agent Tool Selection Optimization: Mechanical steps answered in one round to maintain flow; critical decisions undergo extra rounds to avoid tool misselection and wasted downstream effort.
  • E-commerce Recommendation Real-time Re-ranking: Regular traffic uses shallow ranking for speed; high-value positions (e.g., homepage first view) automatically deepen computation to improve ranking precision.
  • Financial Compliance Document Classification: Fully offline deployment satisfies data compliance requirements; stoppers calibrated via internal unlabeled traffic; hard samples automatically receive deeper processing to reduce misclassification.

Source: AI Tools Collection

#AI Tools

Disclaimer: Contains third-party opinions, does not constitute financial advice

Share To
X
Telegram
WeChat
QQ
Link
Recommended Reading

Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

09-24
Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

09-02
Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

09-02
Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

Open-source Retool is here, drag-and-drop backend with 60+ components

08-19
Open-source Retool is here, drag-and-drop backend with 60+ components

Claude Opus 5 is Here: Half the Price Approaching Fable 5

07-25
Claude Opus 5 is Here: Half the Price Approaching Fable 5

Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

07-25
Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks

07-25
Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks