Generative pre-trained transformer
Type of large language model From Wikipedia, the free encyclopedia
Type of large language model From Wikipedia, the free encyclopedia
Generative pre-trained transformers (GPTs) are a type of large language model (LLM)[1][2][3] and a prominent framework for generative artificial intelligence.[4][5] They are artificial neural networks that are used in natural language processing tasks.[6] GPTs are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content.[2][3] As of 2023, most LLMs have these characteristics[7] and are sometimes referred to broadly as GPTs.[8]
The first GPT was introduced in 2018 by OpenAI.[9] OpenAI has released significant GPT foundation models that have been sequentially numbered, to comprise its "GPT-n" series.[10] Each of these was significantly more capable than the previous, due to increased size (number of trainable parameters) and training. The most recent of these, GPT-4, was released in March 2023.[11] Such models have been the basis for their more task-specific GPT systems, including models fine-tuned for instruction following—which in turn power the ChatGPT chatbot service.[1]
The term "GPT" is also used in the names and descriptions of such models developed by others. For example, other GPT foundation models include a series of models created by EleutherAI,[12] and seven models created by Cerebras in 2023.[13] Also, companies in different industries have developed task-specific GPTs in their respective fields, such as Salesforce's "EinsteinGPT" (for CRM)[14] and Bloomberg's "BloombergGPT" (for finance).[15]
Generative pretraining (GP) was a long-established concept in machine learning applications.[16][17][18] It was originally used as a form of semi-supervised learning, as the model is trained first on an unlabelled dataset (pretraining step) by learning to generate datapoints in the dataset, and then it is trained to classify a labelled dataset.[19]
While the unnormalized linear transformer dates back to 1992,[20][21][22] the modern transformer architecture was not available until 2017 when it was published by researchers at Google in a paper "Attention Is All You Need".[23] That development led to the emergence of large language models such as BERT in 2018[24] which was a pre-trained transformer (PT) but not designed to be generative (BERT was an "encoder-only" model).[25] Also around that time, in 2018, OpenAI published its article entitled "Improving Language Understanding by Generative Pre-Training", in which it introduced the first generative pre-trained transformer (GPT) system ("GPT-1").[26]
Prior to transformer-based architectures, the best-performing neural NLP (natural language processing) models commonly employed supervised learning from large amounts of manually-labeled data. The reliance on supervised learning limited their use on datasets that were not well-annotated, and also made it prohibitively expensive and time-consuming to train extremely large language models.[26]
The semi-supervised approach OpenAI employed to make a large-scale generative system—and was first to do with a transformer model—involved two stages: an unsupervised generative "pretraining" stage to set initial parameters using a language modeling objective, and a supervised discriminative "fine-tuning" stage to adapt these parameters to a target task.[26]
Regarding more recent GPT foundation models, OpenAI published its first versions of GPT-3 in July 2020. There were three models, with 1B, 6.7B, 175B parameters, respectively named babbage, curie, and davinci (giving initials B, C, and D).[citation needed]
In July 2021, OpenAI published Codex, a task-specific GPT model targeted for programming applications. This was developed by fine-tuning a 12B parameter version of GPT-3 (different from previous GPT-3 models) using code from GitHub.[27]
In March 2022, OpenAI published two versions of GPT-3 that were fine-tuned for instruction-following (instruction-tuned), named davinci-instruct-beta (175B) and text-davinci-001,[28] and then started beta testing code-davinci-002.[29] text-davinci-002 was instruction-tuned from code-davinci-002. Both text-davinci-003 and ChatGPT were released in November 2022, with both building upon text-davinci-002 via reinforcement learning from human feedback (RLHF). text-davinci-003 is trained for following instructions (like its predecessors), whereas ChatGPT is further trained for conversational interaction with a human user.[30][31]
OpenAI's most recent GPT foundation model, GPT-4, was released on March 14, 2023. It can be accessed directly by users via a premium version of ChatGPT, and is available to developers for incorporation into other products and services via OpenAI's API. Other producers of GPT foundation models include EleutherAI (with a series of models starting in March 2021)[12] and Cerebras (with seven models released in March 2023).[13]
A foundational model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks.[32] [33]
Thus far, the most notable GPT foundation models have been from OpenAI's GPT-n series. The most recent from that is GPT-4, for which OpenAI declined to publish the size or training details (citing "the competitive landscape and the safety implications of large-scale models").[34]
| Model | Architecture | Parameter count | Training data | Release date | Training cost |
|---|---|---|---|---|---|
| GPT-1 | 12-level, 12-headed Transformer decoder (no encoder), followed by linear-softmax. | 117 million | BookCorpus:[35] 4.5 GB of text, from 7000 unpublished books of various genres. | June 11, 2018[9] | 30 days on 8 P600 GPUs, or 1 petaFLOP/s-day.[9] |
| GPT-2 | GPT-1, but with modified normalization | 1.5 billion | WebText: 40 GB of text, 8 million documents, from 45 million webpages upvoted on Reddit. | February 14, 2019 (initial/limited version) and November 5, 2019 (full version)[36] | "tens of petaflop/s-day",[37] or 1.5e21 FLOP.[38] |
| GPT-3 | GPT-2, but with modification to allow larger scaling | 175 billion[39] | 499 billion tokens consisting of CommonCrawl (570 GB), WebText, English Wikipedia, and two books corpora (Books1 and Books2). | May 28, 2020[37] | 3640 petaflop/s-day (Table D.1[37]), or 3.1e23 FLOP.[38] |
| GPT-3.5 | Undisclosed | 175 billion[39] | Undisclosed | March 15, 2022 | Undisclosed |
| GPT-4 | Also trained with both text prediction and RLHF; accepts both text and images as input. Further details are not public.[34] | Undisclosed. Estimated 1.7 trillion.[40] | Undisclosed | March 14, 2023 | Undisclosed. Estimated 2.1 × 1025 FLOP.[38] |
Other such models include Google's PaLM, a broad foundation model that has been compared to GPT-3 and has recently been made available to developers via an API,[41][42] and Together's GPT-JT, which has been reported as the closest-performing open-source alternative to GPT-3 (and is derived from earlier open-source GPTs).[43] Meta AI (formerly Facebook) also has a generative transformer-based foundational large language model, known as LLaMA.[44]
Foundational GPTs can also employ modalities other than text, for input and/or output. GPT-4 is a multi-modal LLM that is capable of processing text and image input (though its output is limited to text).[45] Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion[46] and parallel decoding.[47] Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.[48]
A foundational GPT model can be further adapted to produce more targeted systems directed to specific tasks and/or subject-matter domains. Methods for such adaptation can include additional fine-tuning (beyond that done for the foundation model) as well as certain forms of prompt engineering.[49]
An important example of this is fine-tuning models to follow instructions, which is of course a fairly broad task but more targeted than a foundation model. In January 2022, OpenAI introduced "InstructGPT"—a series of models which were fine-tuned to follow instructions using a combination of supervised training and reinforcement learning from human feedback (RLHF) on base GPT-3 language models.[50][51] Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs. Hence, OpenAI began using this as the basis for its API service offerings.[52] Other instruction-tuned models have been released by others, including a fully open version.[53][54]
Another (related) kind of task-specific models are chatbots, which engage in human-like conversation. In November 2022, OpenAI launched ChatGPT—an online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT.[55] They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI, and mixed this new dialogue dataset with the InstructGPT dataset for a conversational format suitable for a chatbot. Other major chatbots currently include Microsoft's Bing Chat, which uses OpenAI's GPT-4 (as part of a broader close collaboration between OpenAI and Microsoft),[56] and Google's competing chatbot Bard (initially based on their LaMDA family of conversation-trained language models, with plans to switch to PaLM).[57]
Yet another kind of task that a GPT can be used for is the meta-task of generating its own instructions, like developing a series of prompts for 'itself' to be able to effectuate a more general goal given by a human user.[58] This is known as an AI agent, and more specifically a recursive one because it uses results from its previous self-instructions to help it form its subsequent prompts; the first major example of this was Auto-GPT (which uses OpenAI's GPT models), and others have since been developed as well.[59]
 | This section needs to be updated. The reason given is: Recent advancements in the field have emerged and warrant thorough documentation. (December 2023) |
Generative transformer-based systems can also be targeted for tasks involving modalities beyond text. For example, Microsoft's "Visual ChatGPT" combines ChatGPT with visual foundation models (VFMs) to enable input or output comprising images as well as text.[60] Also, advances in text-to-speech technology offer tools for audio content creation when used in conjunction with foundational GPT language models.[61]
 | This list contains entries that appear to advertise a subject. (May 2023) |
GPT systems can be directed toward particular fields or domains. Some reported examples of such models and apps are as follows:
Sometimes domain-specificity is accomplished via software plug-ins or add-ons. For example, several different companies have developed particular plugins that interact directly with OpenAI's ChatGPT interface,[69][70] and Google Workspace has available add-ons such as "GPT for Sheets and Docs"—which is reported to aid use of spreadsheet functionality in Google Sheets.[71][72]
In November 2023, OpenAI announced that it's enabling ChatGPT Plus subscribers to create custom versions of ChatGPT (being called GPTs).[73] These can be tailored for specific domains via prompt engineering, curated datasets, and/or targeted interaction with external tools. Users who register as verified builders are able to publish their custom GPTs for other users, with monetization potential. (This is notably distinct from OpenAI's API service, as this is based internally within OpenAI's platform.)
OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, has recently asserted that "GPT" should be regarded as a brand of OpenAI.[74] In April 2023, OpenAI revised the brand guidelines in its terms of service to indicate that other businesses using its API to run their artificial intelligence (AI) services would no longer be able to include "GPT" in such names or branding.[75] In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations of trademark infringement or demands to cease and desist).[74] As of November 2023, OpenAI still prohibits its API licensees from naming their own products with "GPT",[76] but it has begun enabling its ChatGPT Plus subscribers to make "custom versions of ChatGPT" that are being called GPTs on the OpenAI site.[77] OpenAI's terms of service says that its subscribers may use "GPT" in the names of these, although it's "discouraged".[76]
Relatedly, OpenAI has applied to the United States Patent and Trademark Office (USPTO) to seek domestic trademark registration for the term "GPT" in the field of AI.[74] OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023.[78] In May 2023, the USPTO responded to the application with a determination that "GPT" was both descriptive and generic.[79] As of November 2023, OpenAI continues to pursue its argument through the available processes. Regardless, failure to obtain a registered U.S. trademark does not preclude some level of common-law trademark rights in the U.S.,[80] and/or trademark rights in other countries.[81]
For any given type or scope of trademark protection in the U.S., OpenAI would need to establish that the term is actually "distinctive" to their specific offerings in addition to being a broader technical term for the kind of technology. Some media reports suggested that OpenAI may be able to obtain trademark registration based indirectly on the fame of its GPT-based chatbot product, ChatGPT,[78][82] for which OpenAI has separately sought protection (and which it has sought to enforce more strongly).[83] Other reports have indicated that registration for the bare term "GPT" seems unlikely to be granted,[74][84] as it is used frequently as a common term to refer simply to AI systems that involve generative pre-trained transformers.[3][85][86][87] In any event, to whatever extent exclusive rights in the term may occur the U.S., others would need to avoid using it for similar products or services in ways likely to cause confusion.[84][88] If such rights ever became broad enough to implicate other well-established uses in the field, the trademark doctrine of descriptive fair use could still continue non-brand-related usage.[89]
This section lists the main official publications from OpenAI and Microsoft on their GPT models.
Seamless Wikipedia browsing. On steroids.
Every time you click a link to Wikipedia, Wiktionary or Wikiquote in your browser's search results, it will show the modern Wikiwand interface.
Wikiwand extension is a five stars, simple, with minimum permission required to keep your browsing private, safe and transparent.
