热门问题
时间线
聊天
视角
大型语言模型列表
维基媒体列表条目 来自维基百科,自由的百科全书
Remove ads
Remove ads
大型语言模型 (LLM) 是一种机器学习模型,专为语言生成等自然语言处理任务而设计。LLM 是具有许多参数的语言模型,并通过对大量文本进行自监督学习进行训练。
本页列出了值得注意的大型语言模型。
对于训练成本一列,1 petaFLOP-day = 1 petaFLOP/sec × 1 天 = 8.64×1019 FLOP。此外,仅列出最大模型的成本。
更多信息 名称, 发布日期[a] ...
名称 | 发布日期[a] | 开发者 | 参数量 (十亿) [b] | 语料库大小 | 训练成本 (petaFLOP-day) | 许可证[c] | 注解 |
---|---|---|---|---|---|---|---|
GPT-1 | 000000002018-06-01-00002018年6月 | OpenAI | 0.117 !0.117 | 1[1] | MIT[2] | 首个GPT模型,为仅解码器transformer。 在8个P600GPU上训练了30天。 | |
BERT | 000000002018-10-01-00002018年10月 | 0.340 !0.340[3] | 3300000000 !33亿单词[3] | 9 !9[4] | Apache 2.0[5] | 这是一个早期且有影响力的语言模型。[6] 仅用于编码器,因此并非为提示或生成而构建。[7] 在 64 个 TPUv2 芯片上训练耗时 4 天。[8] | |
T5 | 000000002019-10-01-00002019年10月 | 11 !11[9] | 340亿 tokens[9] | Apache 2.0[10] | 许多Google项目的基础模型,例如Imagen。[11] | ||
XLNet | 000000002019-06-01-00002019年6月 | 0.340 !0.340[12] | 3300000000 !330亿单词 | 330 | Apache 2.0[13] | 作为BERT的替代,设计为仅编码器 。在512个TPU v3芯片上训练了5.5天。[14] | |
GPT-2 | 000000002019-02-01-00002019年2月 | OpenAI | 1.5 !1.5[15] | 40 GB[16] (~10000000000 !100亿 tokens)[17] | 28[18] | MIT[19] | 在32个TPU v3芯片上训练了一周。[18] |
GPT-3 | 000000002020-05-01-00002020年5月 | OpenAI | 175 !175[20] | 300000000000 !3000亿 tokens[17] | 3640[21] | Proprietary | A fine-tuned variant of GPT-3, termed GPT-3.5, was made available to the public through a web interface called ChatGPT in 2022.[22] |
GPT-Neo | 000000002021-03-01-00002021年3月 | EleutherAI | 2.7 !2.7[23] | 825 GiB[24] | MIT[25] | The first of a series of free GPT-3 alternatives released by EleutherAI. GPT-Neo outperformed an equivalent-size GPT-3 model on some benchmarks, but was significantly worse than the largest GPT-3.[25] | |
GPT-J | 000000002021-06-01-00002021年6月 | EleutherAI | 6 !6[26] | 825 GiB[24] | 200[27] | Apache 2.0 | GPT-3-style language model |
Megatron-Turing NLG | 000000002021-10-01-00002021年10月 [28] | Microsoft and Nvidia | 530 !530[29] | 338600000000 !338.6 billion tokens[29] | 38000[30] | Restricted web access | Trained for 3 months on over 2000 A100 GPUs on the NVIDIA Selene Supercomputer, for over 3 million GPU-hours.[30] |
Ernie 3.0 Titan | 000000002021-12-01-00002021年12月 | Baidu | 260 !260[31] | 4 Tb | Proprietary | Chinese-language LLM. Ernie Bot is based on this model. | |
Claude[32] | 000000002021-12-01-00002021年12月 | Anthropic | 52 !52[33] | 400000000000 !400 billion tokens[33] | beta | Fine-tuned for desirable behavior in conversations.[34] | |
GLaM (Generalist Language Model) | 000000002021-12-01-00002021年12月 | 1200 !1200[35] | 1600000000000 !1.6 trillion tokens[35] | 5600[35] | Proprietary | Sparse mixture of experts model, making it more expensive to train but cheaper to run inference compared to GPT-3. | |
Gopher | 000000002021-12-01-00002021年12月 | DeepMind | 280 !280[36] | 300000000000 !300 billion tokens[37] | 5833[38] | Proprietary | Later developed into the Chinchilla model. |
LaMDA (Language Models for Dialog Applications) | 000000002022-01-01-00002022年1月 | 137 !137[39] | 1.56T words,[39] 168000000000 !168 billion tokens[37] | 4110[40] | Proprietary | Specialized for response generation in conversations. | |
GPT-NeoX | 000000002022-02-01-00002022年2月 | EleutherAI | 20 !20[41] | 825 GiB[24] | 740[27] | Apache 2.0 | based on the Megatron architecture |
Chinchilla | 000000002022-03-01-00002022年3月 | DeepMind | 70 !70[42] | 1400000000000 !1.4 trillion tokens[42][37] | 6805[38] | Proprietary | Reduced-parameter model trained on more data. Used in the Sparrow bot. Often cited for its neural scaling law. |
PaLM (Pathways Language Model) | 000000002022-04-01-00002022年4月 | 540 !540[43] | 768000000000 !768 billion tokens[42] | 29250 !29,250[38] | Proprietary | Trained for ~60 days on ~6000 TPU v4 chips.[38] 截至2024年10月 (2024-10)[update], it is the largest dense Transformer published. | |
OPT (Open Pretrained Transformer) | 000000002022-05-01-00002022年5月 | Meta | 175 !175[44] | 180000000000 !180 billion tokens[45] | 310[27] | Non-commercial research[d] | GPT-3 architecture with some adaptations from Megatron. Uniquely, the training logbook written by the team was published.[46] |
YaLM 100B | 000000002022-06-01-00002022年6月 | Yandex | 100 !100[47] | 1.7TB[47] | Apache 2.0 | English-Russian model based on Microsoft's Megatron-LM. | |
Minerva | 000000002022-06-01-00002022年6月 | 540 !540[48] | 38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server[48] | Proprietary | For solving "mathematical and scientific questions using step-by-step reasoning".[49] Initialized from PaLM models, then finetuned on mathematical and scientific data. | ||
BLOOM | 000000002022-07-01-00002022年7月 | Large collaboration led by Hugging Face | 175 !175[50] | 350000000000 !350 billion tokens (1.6TB)[51] | Responsible AI | Essentially GPT-3 but trained on a multi-lingual corpus (30% English excluding programming languages) | |
Galactica | 000000002022-11-01-00002022年11月 | Meta | 120 !120 | 350000000000 !106 billion tokens[52] | 未知 | CC-BY-NC-4.0 | Trained on scientific text and modalities. |
AlexaTM (Teacher Models) | 000000002022-11-01-00002022年11月 | Amazon | 20 !20[53] | 1300000000000 !1.3 trillion[54] | proprietary[55] | bidirectional sequence-to-sequence architecture | |
LLaMA (Large Language Model Meta AI) | 000000002023-02-01-00002023年2月 | Meta AI | 65 !65[56] | 1400000000000 !1.4 trillion[56] | 6300[57] | Non-commercial research[e] | Corpus has 20 languages. "Overtrained" (compared to Chinchilla scaling law) for better performance with fewer parameters.[56] |
GPT-4 | 000000002023-03-01-00002023年3月 | OpenAI | 未知[f] (According to rumors: 1760)[59] | 未知 | 未知 | proprietary | Available for ChatGPT Plus users and used in several products. |
Chameleon | 000000002024-06-01-00002024年6月 | Meta AI | 34 !34[60] | 4400000000000 !4.4 trillion | |||
Cerebras-GPT | 000000002023-03-01-00002023年3月 | Cerebras | 13 !13[61] | 270[27] | Apache 2.0 | Trained with Chinchilla formula. | |
Falcon | 000000002023-03-01-00002023年3月 | Technology Innovation Institute | 40 !40[62] | 1 trillion tokens, from RefinedWeb (filtered web text corpus)[63] plus some "curated corpora".[64] | 2800[57] | Apache 2.0[65] | |
BloombergGPT | 000000002023-03-01-00002023年3月 | Bloomberg L.P. | 50 !50 | 363 billion token dataset based on Bloomberg's data sources, plus 345 billion tokens from general purpose datasets[66] | Proprietary | Trained on financial data from proprietary sources, for financial tasks. | |
PanGu-Σ | 000000002023-03-01-00002023年3月 | Huawei | 1085 !1085 | 329 billion tokens[67] | Proprietary | ||
OpenAssistant[68] | 000000002023-03-01-00002023年3月 | LAION | 17 !17 | 1.5 trillion tokens | Apache 2.0 | Trained on crowdsourced open data | |
Jurassic-2[69] | 000000002023-03-01-00002023年3月 | AI21 Labs | 未知 | 未知 | Proprietary | Multilingual[70] | |
PaLM 2 (Pathways Language Model 2) | 000000002023-05-01-00002023年5月 | 340 !340[71] | 3600000000000 !3.6 trillion tokens[71] | 85000 !85,000[57] | Proprietary | Was used in Bard chatbot.[72] | |
Llama 2 | 000000002023-07-01-00002023年7月 | Meta AI | 70 !70[73] | 2000000000000 !2 trillion tokens[73] | 21000 !21,000 | Llama 2 license | 1.7 million A100-hours.[74] |
Claude 2 | 000000002023-07-01-00002023年7月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Used in Claude chatbot.[75] |
Granite 13b | 000000002023-07-01-00002023年7月 | IBM | 未知 | 未知 | 未知 | Proprietary | Used in IBM Watsonx.[76] |
Mistral 7B | 000000002023-09-01-00002023年9月 | Mistral AI | 7.3 !7.3[77] | 未知 | Apache 2.0 | ||
Claude 2.1 | 000000002023-11-01-00002023年11月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Used in Claude chatbot. Has a context window of 200,000 tokens, or ~500 pages.[78] |
Grok-1[79] | 000000002023-11-01-00002023年11月 | xAI | 314 | 未知 | 未知 | Apache 2.0 | Used in Grok chatbot. Grok-1 has a context length of 8,192 tokens and has access to X (Twitter).[80] |
Gemini 1.0 | 000000002023-12-01-00002023年12月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Multimodal model, comes in three sizes. Used in the chatbot of the same name.[81] |
Mixtral 8x7B | 000000002023-12-01-00002023年12月 | Mistral AI | 46.7 | 未知 | 未知 | Apache 2.0 | Outperforms GPT-3.5 and Llama 2 70B on many benchmarks.[82] Mixture of experts model, with 12.9 billion parameters activated per token.[83] |
Mixtral 8x22B | 000000002024-04-01-00002024年4月 | Mistral AI | 141 | 未知 | 未知 | Apache 2.0 | [84] |
DeepSeek LLM | 000000002023-11-29-00002023年11月29日 | DeepSeek | 67 | 2T tokens[85] | 12,000}} | DeepSeek License | Trained on English and Chinese text. 1e24 FLOPs for 67B. 1e23 FLOPs for 7B[85] |
Phi-2 | 000000002023-12-01-00002023年12月 | Microsoft | 2.7 | 1.4T tokens | 419[86] | MIT | Trained on real and synthetic "textbook-quality" data, for 14 days on 96 A100 GPUs.[86] |
Gemini 1.5 | 000000002024-02-01-00002024年2月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Multimodal model, based on a Mixture-of-Experts (MoE) architecture. Context window above 1 million tokens.[87] |
Gemini Ultra | 000000002024-02-01-00002024年2月 | Google DeepMind | 未知 | 未知 | 未知 | ||
Gemma | 000000002024-02-01-00002024年2月 | Google DeepMind | 7 | 6T tokens | 未知 | Gemma Terms of Use[88] | |
Claude 3 | 000000002024-03-01-00002024年3月 | Anthropic | 未知 | 未知 | 未知 | Proprietary | Includes three models, Haiku, Sonnet, and Opus.[89] |
Nova (页面存档备份,存于互联网档案馆) | 000000002024-10-01-00002024年10月 | Rubik's AI (页面存档备份,存于互联网档案馆) | 未知 | 未知 | 未知 | Proprietary | Includes three models, Nova-Instant, Nova-Air, and Nova-Pro. |
DBRX | 000000002024-03-01-00002024年3月 | Databricks and Mosaic ML | 136 !136 | 12T Tokens | Databricks Open Model License | Training cost 10 million USD. | |
Fugaku-LLM | 000000002024-05-01-00002024年5月 | Fujitsu, Tokyo Institute of Technology, etc. | 13 !13 | 380B Tokens | The largest model ever trained on CPU-only, on the Fugaku.[90] | ||
Phi-3 | 000000002024-04-01-00002024年4月 | Microsoft | 14[91] | 4.8T Tokens | MIT | Microsoft markets them as "small language model".[92] | |
Granite Code Models | 000000002024-05-01-00002024年5月 | IBM | 未知 | 未知 | 未知 | Apache 2.0 | |
Qwen2 | 000000002024-06-01-00002024年6月 | Alibaba Cloud | 72[93] | 3T Tokens | 未知 | Qwen License | Multiple sizes, the smallest being 0.5B. |
DeepSeek V2 | 000000002024-06-01-00002024年6月 | DeepSeek | 236 | 8.1T tokens | 28000 !28,000 | DeepSeek License | 1.4M hours on H800.[94] |
Nemotron-4 | 000000002024-06-01-00002024年6月 | Nvidia | 340 !340 | 9T Tokens | 200000 !200,000 | NVIDIA Open Model License | Trained for 1 epoch. Trained on 6144 H100 GPUs between December 2023 and May 2024.[95][96] |
Llama 3.1 | 000000002024-07-01-00002024年7月 | Meta AI | 405 | 15.6T tokens | 440000 !440,000 | Llama 3 license | 405B version took 31 million hours on H100-80GB, at 3.8E25 FLOPs.[97][98] |
DeepSeek V3 | 000000002024-12-01-00002024年12月 | DeepSeek | 671 | 14.8T tokens | 56000 !56,000 | DeepSeek License | 2.788M hours on H800 GPUs.[99] |
Amazon Nova | 000000002024-12-01-00002024年12月 | Amazon | 未知 | 未知 | 未知 | Proprietary | Includes three models, Nova Micro, Nova Lite, and Nova Pro[100] |
DeepSeek R1 | 000000002025-01-01-00002025年1月 | DeepSeek | 671 | 未知 | 未知 | MIT | No pretraining. Reinforcement-learned upon V3-Base.[101][102] |
Qwen2.5 | 000000002025-01-01-00002025年1月 | Alibaba | 72 | 18T tokens | 未知 | Qwen License | [103] |
MiniMax-Text-01 | January 2025 | Minimax | 456 | 4.7T tokens[104] | 未知 | Minimax Model license | [105][104] |
Gemini 2.0 | 000000002025-02-01-00002025年2月 | Google DeepMind | 未知 | 未知 | 未知 | Proprietary | Three models released: Flash, Flash-Lite and Pro[106][107][108] |
Mistral Large | 000000002024-11-01-00002024年11月 | Mistral AI | 123 | 未知 | 未知 | Mistral Research License | Upgraded over time. The latest version is 24.11.[109] |
Pixtral | 000000002024-11-01-00002024年11月 | Mistral AI | 123 | 未知 | 未知 | Mistral Research License | Multimodal. There is also a 12B version which is under Apache 2 license.[109] |
Grok 3 | 000000002025-02-01-00002025年2月 | xAI | 未知 | 未知 | 未知, estimated 5,800,000. |
专有 | Training cost claimed "10x the compute of previous state-of-the-art models".[110] |
Llama 4 | 000000002025-04-05-00002025年4月5日 | Meta AI | 400 !400 | 40000000000000 !40T tokens | Llama 4 license | [111][112] | |
Qwen3 | 000000002025-04-01-00002025年4月 | Alibaba Cloud | 235 | 36000000000000 !36T tokens | 未知 | Apache 2.0 | Multiple sizes, the smallest being 0.6B.[113] |
关闭
Remove ads
参见
注释
参考资料
Wikiwand - on
Seamless Wikipedia browsing. On steroids.
Remove ads