The genesis of this idea of an international collaboration—the Trillion Parameter Consortium (TPC)—goes back a few years when it became clear that 1) the emerging exascale platforms being deployed in the U.S. DOE laboratories (Frontier, Aurora, El Capitan, etc.) and similar systems around the world would be excellent platforms for training and evaluating large-scale O(1012 parameter) language models for science and engineering, and 2) building state-of-the-art LLMs will require large allocations of machine time (e.g., O(30-100) exaflop-days) for training and downstream tuning, alignment, and evaluation. In addition, given the scale of the effort to prepare datasets for training and the scale of cycles that need to be allocated to build and train a model, it became clear that while the community could develop a number of smaller models independently, and compete for cycles, a broader “AI for Science” community must work together if we are to create models that are at the scale of the largest private models.
The notion of bringing together a consortium of multiple groups interested in these goals was first articulated at the DOE/MEXT/ADAC meeting at R-CCS in Kobe Japan in February 2023 and then further explored at the Lusk Symposium at Argonne in April of 2023. Concurrently, Argonne, RIKEN, Oak Ridge, BSC, CSC, Together, AI2, and others have been planning independent projects to train LLMs on existing and newly deployed machines sited at various HPC Centers. Most of these efforts involve some set of academic and research institute partners, industry, and vendors, in addition to the labs and centers that host the machines. In these conversations it further became clear that there was much we could learn from each other that could enhance our individual efforts, and that new collaborations could result that would benefit many groups.
TPC organization and governance are described in TPC Introduction and Structure, published in February 2025.
For more information about TPC, see also the inaugural post about the consortium and join the TPC Slack workspace.