OLMo vs LLM360: Two Routes to Open-Source LLMs

Fri, 10 Jul 2026 16:00:00 +0800

In the “fully open-source” LLM lane, two projects are almost always mentioned together: OLMo (Allen Institute for AI, USA) and LLM360 (MBZUAI, UAE). Both advocate releasing training data, code, weights, and even intermediate checkpoints — yet they differ in generation, scale, performance, and iteration cadence. This post lays them side by side so newcomers can build a full picture quickly.

TL;DR

Dimension	OLMo family	LLM360
Lead org	Allen Institute for AI (AI2), US nonprofit	MBZUAI (+ Petuum + Cerebras), UAE academic
Key figure	Dirk Groeneveld, Matt Gardner, et al.	Eric Xing — MBZUAI President / Petuum founder
Latest version	OLMo 3 / 3.1 (2025-11/12)	K2-V2 (early 2026, 70B dense)
Scale range	7B / 13B / 32B (+ multimodal Molmo)	7B (Amber/CrystalCoder) → 70B (K2-V2)
Performance	Strongest in the fully-open camp, approaching Qwen 3 at same size	Gen-1 was weak; K2-V2 catches up at 70B
Transparency	Fully open (data + code + checkpoints)	Fully open — the “360°” slogan goes further

One-liner: for benchmark performance pick OLMo; for studying the training process both work, and LLM360’s K2-V2 is a single-point flagship at 70B.

The OLMo Family (AI2)

One-liner positioning

The best-known fully-open LLM family in academia. Every generation releases pretraining data, code, and intermediate checkpoints. It has iterated from OLMo 1 (2024-02) to OLMo 3 (2025-11), and expanded into vision/multimodal (Molmo / OLMoVision).

OLMo 2 (7B / 13B, 2024-11)

Officially positioned as “the best fully-open model to date,” benchmarked against Llama 3.1, Gemma 2, and Qwen 2.5 at the same size. Core evaluations span 11 academic benchmarks: Average / ARC-C / HellaSwag / WinoGrande / MMLU / DROP / NQ / AGIEval / GSM8k / MMLUPro / TriviaQA.

Model	Avg	ARC-C	HellaSwag	MMLU	DROP	GSM8k
OLMo-2-1124-7B	62.9	79.8	83.8	63.7	60.8	67.5
OLMo-2-1124-13B	68.3	83.5	86.4	67.5	70.7	75.1
Llama-3.1-8B	61.8	79.5	81.6	66.9	56.4	56.5
Qwen-2.5-7B	67.4	89.5	89.7	74.4	55.8	81.5
Gemma-2-9B	67.8	89.5	87.3	70.6	63.0	70.1

A few key observations:

vs Llama-3.1-8B: OLMo-2-7B averages 62.9 vs 61.8 — roughly tied or slightly ahead, but at only ~1/4 the training FLOPs. Better compute efficiency.
vs Qwen-2.5-7B: trails by ~4-5 points (Qwen is clearly stronger on MMLU, ARC-C, GSM8k), but Qwen also uses ~4.5× the FLOPs.
Generational leap: nearly every benchmark doubled vs the original OLMo-7B (GSM8k 9.2 → 67.5, MMLU 28.3 → 63.7).
Weak spots: MMLUPro (31-35) and NQ (factual QA) are on the weaker side.

OLMo 3 / 3.1 (2025-11 / 2025-12)

This is OLMo’s biggest generational jump yet. It’s no longer just “a set of weights” — it open-sources the entire model flow: data, code, and checkpoints at every stage, all forkable.

Four product lines:

Line	Size	Purpose
OLMo 3-Base	7B / 32B	Base model — strongest fully-open base
OLMo 3-Think	7B / 32B	Flagship reasoning model with inspectable reasoning traces
OLMo 3-Instruct	7B	Chat / tool use
OLMo 3-RL Zero	7B	Starting point for RL research

OLMo 3-Think 32B core benchmarks:

Benchmark	OLMo 3-Think 32B	Qwen 3 32B	DeepSeek R1 Distill 32B
MATH	96.1	95.4	92.6
AIME 2024	76.8	80.8	70.3
AIME 2025	72.5	70.9	56.3
MMLU-Pro	75.9	—	—
GPQA (Diamond)	61.0	67.3	61.8
IFEval	89.0	86.5	78.7
HumanEvalPlus	91.4	91.2	92.3

OLMo 3-Think 7B: MATH 94.4 ties Qwen 3 8B; AIME 2024/2025 within a few points. Instruct 7B even beats Qwen 3 8B on Safety (87.3 vs 78.0).

The 2025-12-12 (3.1) update: extended the 32B Think’s RL training by another 21 days, yielding AIME +5, IFEval +4, IFBench +20, plus the first large-size Instruct (32B).

OLMo Hybrid (2026-03, an architecture-research branch)

Note this is not OLMo 4 (OLMo 4 is unannounced). It’s a controlled experiment: take the OLMo 3 7B recipe as the control, change only the architecture (transformer layers → hybrid SSM/attention, Mamba-style), keeping all data/code/hyperparams identical.

Metric	Result
Data efficiency	~2× — reaches the same accuracy as OLMo 3 with 49% fewer tokens
Compute equivalence	1.25×
Positioning	Architecture research, not a product line

It’s the first time hybrid SSM architecture entered the mainstream open-source LLM roadmap, providing controlled evidence that hybrid models scale more efficiently than pure transformers — alongside NVIDIA Nemotron-3 and Jamba, marking “hybrid architectures moving from paper to production.”

LLM360 (MBZUAI + Petuum + Cerebras)

Lead org & positioning

LLM360 is an open research initiative co-founded by three institutions:

Org	Role
MBZUAI (Mohammed bin Zayed University of AI)	Lead — Abu Dhabi, UAE, the world’s first graduate-level AI university. The K2 series is built by its Institute of Foundation Models (IFM)
Petuum, Inc.	Co-founder — an AI startup founded by Eric Xing
Cerebras Systems	Compute / hardware support (wafer-scale chips)

The key figure is Eric Xing: MBZUAI President, Petuum founder, former CMU professor. LLM360 and OLMo form an East/West contrast of the two academic open-source camps.

⚠️ Important clarification: the “K2 trillion-parameter, SWE-Bench 65.8%” you often see in search results refers to Moonshot AI’s Kimi K2 (a commercial open-weight MoE, 1T total params) — not an LLM360 product. They just share a name.

Generation 1: Amber & CrystalCoder (both 7B, 2023-12)

Model	Strength	Performance
Amber-7B (general base)	Relatively competitive on MMLU	ARC clearly behind peers; MT-Bench only 2.49 (weak)
AmberChat / AmberSafe	Chat / safety finetunes	MT-Bench 4.95 / 4.97
CrystalCoder-7B	Code	Clearly beats Amber on MMLU / HumanEval / MBPP; coding is the highlight

Honestly, Gen-1 LLM360 was on the weak side at 7B — its value was “transparency,” not “performance.”

New generation: K2-V2 (early 2026, 70B dense)

LLM360’s most important update since 2024:

Item	Detail
Release	Early 2026 (marking LLM360’s 2nd anniversary)
Params	~65–70B (dense)
License	Apache 2.0 (free commercial use worldwide)
Positioning	A “360-open” fully-transparent LLM, focused as a superior base for reasoning adaptation

Key benchmarks (mid-4 checkpoint):

Benchmark	K2-V2 (mid-4)
GPQA-Diamond	55.1
GSM8K	93.6
MATH	94.7

There’s also K2 Think V2 (an end-to-end reasoning system built on K2-V2, positioned as “the UAE’s first fully sovereign reasoning system,” with partners including G42 and Cerebras).

⚠️ Important: K2-V2’s GPQA-Diamond 55.1 / MATH 94.7 (mid-4 checkpoint) can’t be directly compared with OLMo 3-Think 32B — eval setups, checkpoint stages, and whether reasoning mode is on all differ. Always cite each project’s own technical report.

Side-by-side: how to choose

You want…	Recommendation
Strongest base + reasoning model in the fully-open camp	OLMo 3-Think 32B
A high-efficiency 7B at the same compute budget	OLMo 3-Think 7B
A single-point 70B fully-transparent research flagship	LLM360 K2-V2
Pure coding tasks with a small model	OLMo 3 family overall (HumanEvalPlus 91.4 leads; Gen-1 CrystalCoder is outdated)
To study the training process itself / do RL experiments	Both release checkpoints; OLMo 3-Think 32B is officially “the workhorse for RL research”
Just the absolute highest score	Look elsewhere — Qwen 3 / DeepSeek open-weights are stronger

My take

Both projects are benchmarks of open-science spirit, but differ clearly in engineering investment and iteration cadence:

OLMo’s generational progress is visible to the naked eye. From 1 → 2 → 3, each generation raises the ceiling of the fully-open camp by a big margin, and Gen-3 has begun catching Qwen 3. AI2’s pace feels more like a product team.
LLM360’s value is “the whole process is researchable.” Gen-1 Amber/CrystalCoder was genuinely weak, but K2-V2 jumping to 70B gives it real competitive teeth. The “360°” slogan isn’t marketing — even training logs are open, making it excellent reproduction/teaching material.
Both share a weak spot: general knowledge. Both lose to the strongest open-weights on MMLU / GPQA. That’s the price of full-open (including data): you can’t use high-quality proprietary data.

If you’re doing research reproduction or teaching, OLMo 2/3 + LLM360 K2-V2 are currently the two most complete textbook-grade open LLM stacks.

OLMo on Svtter's Blog