Let's Talk About BigCode

Fri, 10 Jul 2026 11:30:00 +0800

While researching LLMs trained with “open architecture + open data,” one name kept coming up: BigCode. It built some of the first fully open-source code models, and has since become the referee that scores everyone else’s code models. This post covers the whole picture — from model builder to benchmark author.

What It Is

BigCode is an open scientific collaboration with a pure goal: responsibly train large language models for code.

Launched: September 2022
Stewards: Hugging Face + ServiceNow Research (NVIDIA joined for StarCoder2)
Governance: Co-led by Leandro von Werra (HF) and Harm de Vries (ServiceNow), with Loubna Ben Allal as a core researcher
Scale: 1,200+ community members
Distinctive: It doesn’t just ship models — it publishes a Governance Card documenting data governance, environmental impact, and legal compliance. That’s rare among LLM projects.

A Two-Phase History

The key to understanding BigCode is to read its history in two phases.

Phase 1: Building Open Code Models (2022–2024)

Time	Output	Notes
2022	The Stack v1	6.4 TB, 358 languages, open code dataset
2023.05	StarCoder	15.5B, 80+ languages, first large-scale open code LLM
2024.02	The Stack v2	67.5 TB, 600+ languages, 7× v1
2024.02	StarCoder2	3B / 7B / 15B, trained on Stack v2
2024 H2	StarCoder2-15B-Instruct	Self-aligned via SelfCodeAlign — the first code instruct model with zero GPT-4 distillation

The core philosophy of this phase was full transparency: traceable data (via Software Heritage persistent SWHIDs), open-source code, OpenRAIL-licensed weights, and even recorded training carbon footprint. This is what sets BigCode apart from CodeLlama / DeepSeek-Coder — those only open their weights while keeping training data closed.

Phase 2: Pivoting to Evaluation (Late 2024–Present)

The model line stalled, but the project didn’t — it pivoted to building evaluations for code models:

Time	Output	What it is
2024.06 paper / ICLR 2025	BigCodeBench	Static benchmark of 1,140 real programming tasks
2025.02	BigCodeArena	Chatbot Arena with real code execution
2025.10	BigCodeReward + AutoCodeArena	Evaluating reward models on code judgment

Current Pillars: Three Evaluation Efforts

1. BigCodeBench — Static Benchmark

Designed to go beyond HumanEval. HumanEval / MBPP are saturated (strong models hit 90%+), so they no longer differentiate models. BigCodeBench’s design:

1,140 function-level tasks requiring calls to 723 functions across 139 Python libraries
Spanning 7 domains
Two splits: Complete (code completion) and Instruct (plain natural language)
Execution-based verification — it actually runs tests rather than asking humans to read code

How hard is it? Even top models score barely above 30% Pass@1:

Model	BigCodeBench Pass@1
DeepSeek-V3	~33–34
GPT-4.1	32.8
Qwen2.5-Max	32.5
Qwen2.5-Coder-32B	30.8
Claude-3.5-Sonnet	30.4

On the Instruct split, 278 tasks are unsolved by any model and only 14 are solved by all — high ceiling, good discriminative power.

2. BigCodeArena — Human Combat Arena

Similar to LMSYS Chatbot Arena, but with a key difference: it actually executes the code before asking humans to vote. It supports multiple languages, frameworks, multi-turn dialogue, and interactive testing. This solves the old problem that “HumanEval is too easy, but humans reading code is unreliable.”

3. BigCodeReward / AutoCodeArena — Automated Evaluation

Evaluates how well reward models judge code (analogous to RewardBench for general RMs). The core finding: adding execution results dramatically improves a reward model’s judgment — reading code alone is unreliable. AutoCodeArena builds an automated benchmark on this insight, replacing costly human arenas.

A Notable Phenomenon

The BigCodeArena leaderboard features GPT, Claude, Qwen, DeepSeek, GLM, and Kimi — closed / open-weights models — while StarCoder2 itself is essentially absent from the top rankings. BigCode completed the transition from “the player being evaluated” to “the referee scoring everyone.”

Why the Pivot to Evaluation

Based on public information, my read:

The open-source code-model space got crushed. Qwen-Coder and DeepSeek-Coder, despite closed data, far outperform what an academic collaboration can sustain in both performance and iteration speed. Chasing StarCoder3 SOTA had poor ROI.
Code evaluation was a genuine gap. HumanEval is saturated; SWE-bench is repo-level and heavy. The middle ground — function-level + multi-tool calls + execution verification — was unoccupied, and BigCode seized it.
It fits the team’s strengths. BigCode’s evaluation-harness was already the de facto community standard for code evaluation. Building evaluations suited the team better than competing on training compute.

Historical Contribution and Current Position

Dimension	Assessment
Open code model pioneer	StarCoder / StarCoder2 were the first large-scale fully open code LLMs — foundational work
Data transparency benchmark	The Stack v1/v2 + SWHID tracing — still unmatched
Current model line	Stalled; StarCoder2 has fallen behind open-weights rivals
Current evaluation line	Active and leading; BigCodeBench is now a de facto code-eval standard

BigCode on Svtter's Blog