Generative Pre-trained Transformers (GPTs) are a type of language-processing AI model. They use a machine learning technique known as transformers, which allows them to generate human-like text by predicting the likelihood of a word given the previous words used in the text.
The original GPT (Generative Pre-trained Transformer) model was created by researchers at OpenAI in 2018. GPT pioneered the approach of pre-training a transformer neural network model on a large text corpus using a language modeling objective, before fine-tuning on downstream NLP tasks. This enabled breakthrough performance gains in NLP.
GPTs are pre-trained on a large corpus of text from the internet, which allows them to generate coherent and contextually relevant sentences by leveraging the patterns and structures they learned during this pre-training phase. After pre-training, they can be fine-tuned on specific tasks like translation, summarization, question-answering, and more.
GPTs are important for several reasons:
Versatility: Because they’re pre-trained on a large corpus of text, they have a broad understanding of language and can be fine-tuned for a wide variety of tasks.
Efficiency: Because they use transformers, GPTs can process words in parallel, making them more efficient than previous types of language models that processed words sequentially.
Understanding Context: GPTs are good at understanding the context of a conversation or a piece of text, which makes them useful for tasks like chatbots or drafting emails.« Back to Glossary Index