The Falcon Series of Open Language Models

Why it matters

Falcon put a UAE government-backed research institute's name directly on one of the largest open pretraining runs completed anywhere at the time - a concrete, independently reproducible claim rather than a promotional one, published with a full technical account of the training infrastructure.

Key takeaways

• Falcon-180B was trained on over 3.5 trillion tokens - described in the paper as the largest openly documented pretraining run at time of publication.
• The paper reports Falcon-180B outperforming PaLM and Chinchilla and improving on contemporaries including LLaMA 2, while approaching PaLM-2-Large's performance at lower training and inference cost.
• Details a custom distributed-training codebase built to pretrain across up to 4,096 A100 GPUs on AWS cloud infrastructure with limited interconnect bandwidth.
• Released the models openly, and TII has since extended the family (including Falcon Mamba, an attention-free architecture) under continued open release.

UAE / MENA relevance

TII is a UAE government-backed research institute (part of Abu Dhabi's Advanced Technology Research Council). Falcon is the clearest example of the UAE funding, building and openly releasing frontier-scale AI infrastructure rather than only importing it.

Part of these reading paths

• Starting AI in the UAE