While researching LLMs trained with “open architecture + open data,” one name kept coming up: BigCode. It built some of the first fully open-source code models, and has since become the referee that scores everyone else’s code models. This post covers the whole picture — from model builder to benchmark author.
What It Is
BigCode is an open scientific collaboration with a pure goal: responsibly train large language models for code.
- Launched: September 2022
- Stewards: Hugging Face + ServiceNow Research (NVIDIA joined for StarCoder2)
- Governance: Co-led by Leandro von Werra (HF) and Harm de Vries (ServiceNow), with Loubna Ben Allal as a core researcher
- Scale: 1,200+ community members
- Distinctive: It doesn’t just ship models — it publishes a Governance Card documenting data governance, environmental impact, and legal compliance. That’s rare among LLM projects.
A Two-Phase History
The key to understanding BigCode is to read its history in two phases.
Phase 1: Building Open Code Models (2022–2024)
| Time | Output | Notes |
|---|---|---|
| 2022 | The Stack v1 | 6.4 TB, 358 languages, open code dataset |
| 2023.05 | StarCoder | 15.5B, 80+ languages, first large-scale open code LLM |
| 2024.02 | The Stack v2 | 67.5 TB, 600+ languages, 7× v1 |
| 2024.02 | StarCoder2 | 3B / 7B / 15B, trained on Stack v2 |
| 2024 H2 | StarCoder2-15B-Instruct | Self-aligned via SelfCodeAlign — the first code instruct model with zero GPT-4 distillation |
The core philosophy of this phase was full transparency: traceable data (via Software Heritage persistent SWHIDs), open-source code, OpenRAIL-licensed weights, and even recorded training carbon footprint. This is what sets BigCode apart from CodeLlama / DeepSeek-Coder — those only open their weights while keeping training data closed.
Phase 2: Pivoting to Evaluation (Late 2024–Present)
The model line stalled, but the project didn’t — it pivoted to building evaluations for code models:
| Time | Output | What it is |
|---|---|---|
| 2024.06 paper / ICLR 2025 | BigCodeBench | Static benchmark of 1,140 real programming tasks |
| 2025.02 | BigCodeArena | Chatbot Arena with real code execution |
| 2025.10 | BigCodeReward + AutoCodeArena | Evaluating reward models on code judgment |
Current Pillars: Three Evaluation Efforts
1. BigCodeBench — Static Benchmark
Designed to go beyond HumanEval. HumanEval / MBPP are saturated (strong models hit 90%+), so they no longer differentiate models. BigCodeBench’s design:
- 1,140 function-level tasks requiring calls to 723 functions across 139 Python libraries
- Spanning 7 domains
- Two splits: Complete (code completion) and Instruct (plain natural language)
- Execution-based verification — it actually runs tests rather than asking humans to read code
How hard is it? Even top models score barely above 30% Pass@1:
| Model | BigCodeBench Pass@1 |
|---|---|
| DeepSeek-V3 | ~33–34 |
| GPT-4.1 | 32.8 |
| Qwen2.5-Max | 32.5 |
| Qwen2.5-Coder-32B | 30.8 |
| Claude-3.5-Sonnet | 30.4 |
On the Instruct split, 278 tasks are unsolved by any model and only 14 are solved by all — high ceiling, good discriminative power.
2. BigCodeArena — Human Combat Arena
Similar to LMSYS Chatbot Arena, but with a key difference: it actually executes the code before asking humans to vote. It supports multiple languages, frameworks, multi-turn dialogue, and interactive testing. This solves the old problem that “HumanEval is too easy, but humans reading code is unreliable.”
3. BigCodeReward / AutoCodeArena — Automated Evaluation
Evaluates how well reward models judge code (analogous to RewardBench for general RMs). The core finding: adding execution results dramatically improves a reward model’s judgment — reading code alone is unreliable. AutoCodeArena builds an automated benchmark on this insight, replacing costly human arenas.
A Notable Phenomenon
The BigCodeArena leaderboard features GPT, Claude, Qwen, DeepSeek, GLM, and Kimi — closed / open-weights models — while StarCoder2 itself is essentially absent from the top rankings. BigCode completed the transition from “the player being evaluated” to “the referee scoring everyone.”
Why the Pivot to Evaluation
Based on public information, my read:
The open-source code-model space got crushed. Qwen-Coder and DeepSeek-Coder, despite closed data, far outperform what an academic collaboration can sustain in both performance and iteration speed. Chasing StarCoder3 SOTA had poor ROI.
Code evaluation was a genuine gap. HumanEval is saturated; SWE-bench is repo-level and heavy. The middle ground — function-level + multi-tool calls + execution verification — was unoccupied, and BigCode seized it.
It fits the team’s strengths. BigCode’s evaluation-harness was already the de facto community standard for code evaluation. Building evaluations suited the team better than competing on training compute.
Historical Contribution and Current Position
| Dimension | Assessment |
|---|---|
| Open code model pioneer | StarCoder / StarCoder2 were the first large-scale fully open code LLMs — foundational work |
| Data transparency benchmark | The Stack v1/v2 + SWHID tracing — still unmatched |
| Current model line | Stalled; StarCoder2 has fallen behind open-weights rivals |
| Current evaluation line | Active and leading; BigCodeBench is now a de facto code-eval standard |