Baselines in progress

How well does AI know Canada?

CanBench is an open, bilingual benchmark for AI models. It tests the things Canadians actually ask about: our laws, our public services, Québec French and both official languages. The leaderboard below compares the world's leading models on them.

What it measures

Law

The Charter, federal statutes, federal vs provincial jurisdiction, Québec civil law.

Government services

CRA, EI, CPP, provincial health coverage, immigration steps.

Québec French

Usage, register and terminology as written in Québec, not translated from France or from English.

Official translation

English↔French translation of official text, following federal conventions.

Knowing Canada

Geography, history, institutions and the differences between provinces.

Safety

Refusals and harms judged in a Canadian legal context.

Leaderboard

Scores run from 0 (random guessing) to 100 (perfect). They are shown overall and separately for English and French.

Model	Maker	Overall	English	French
Claude	Anthropic	pending	pending	pending
GPT	OpenAI	pending	pending	pending
Gemini	Google	pending	pending	pending
Command A+	Cohere	pending	pending	pending
Mistral Small 4	Mistral AI	pending	pending	pending
Qwen3.8-27B	Alibaba	pending	pending	pending
Olmo 3	Ai2	pending	pending	pending
Apertus 1.5	ETH Zurich / EPFL	pending	pending	pending

How it works

Open. The test harness and part of the questions will be published, so anyone can see how scores are made and rerun them on the public questions.
Native, not translated. French questions are written and reviewed by Québec speakers. None is machine-translated from English.
Hard to game. The ranking uses only questions that are never published, and they are renewed every quarter. A model that does much better on the public questions than on the private ones has likely been trained on them, and the board will show it.
Built on existing work. Where Canadian benchmarks already exist, such as GRACE for federal law and QFrBLiMP for Québec French grammar, CanBench runs them instead of rewriting them.
Every source recorded. Each question cites the open Canadian source it comes from, such as the Justice Laws website or canada.ca.

Submit a model

Model makers will be able to submit models once the first results are out. To be told when that happens, write to support@suprun.ca.