How Generative AI Works
Demystifying the magic behind ChatGPT and other generative AI systems using first principles thinking
The Impossible Question
If I ask you a question you've never seen before, would you be able to answer it? Most people initially think, "No, that's impossible!" But that's not quite true. You actually can answer new questions by recognizing patterns and applying your understanding from similar situations.
For example, if you know how addition works and I give you two numbers you've never added before, you can still solve it by applying the pattern of addition. You don't need to have memorized every possible arithmetic problem to solve a new one.
This is precisely how AI systems like ChatGPT work! They're not just retrieving memorized answers—they're recognizing patterns in language and generating responses based on those patterns, even to questions they've never encountered. In this guide, we'll explore this fascinating capability using first principles and easy-to-understand examples.
First Principles Thinking
To understand ChatGPT, we need to break down our assumptions about how AI works. Consider this sequence: 2, 4, 6, 8... Even if you've never seen this exact sequence before, you can predict that the next number is 10.
You recognized a pattern (adding 2) and applied it to predict what comes next. This is essentially how ChatGPT works - it identifies patterns in language and uses them to predict what text should come next.
Predictive Power
When you ask ChatGPT a question like "Hello, how are you?", it doesn't retrieve a pre-written answer. Instead, it predicts what text should follow your input, one token (piece of text) at a time.
It recognizes that your greeting typically elicits responses like "I'm doing well, thank you for asking!" based on patterns it learned during training - not because it was programmed with specific answers to specific questions.
How ChatGPT Really Works
Tokenization
When you input text, ChatGPT first breaks it down into tokens (words or parts of words). For example, "Hello, how are you?" might become ["Hello", ",", "how", "are", "you", "?"].
/* Text broken into tokens */
Text: "hello how are you"
Tokens: ["hello", "how", "are", "you"]
/* These tokens are converted to numeric IDs */
Token IDs: [24912, 1495, 553, 481, 220]
Each model has its own unique tokenization method. GPT-3, GPT-3.5, and GPT-4 all use different tokenizers that split text in slightly different ways. This is why the same text might be broken into a different number of tokens depending on which model you're using.
A helpful rule of thumb is that one token generally corresponds to ~4 characters of English text, or roughly ¾ of a word (about 100 tokens = 75 words).
Prediction
The AI looks at your input tokens and predicts what should come next. For each position, it calculates probabilities for all possible next tokens. For example, after "Hello, how are", the word "you" has a high probability of being next.
Generation
The model selects tokens based on these probabilities, adds them to the response, and then uses all text so far to predict the next token. This process repeats until a complete response is generated - creating new text that wasn't in its training data.
Different Models, Different Tokenizers
Different AI models use different tokenization methods. Below is an example of how the same text could be tokenized differently depending on the model:
GPT-4
Input text: "hello how are you"
Tokens: 5
Characters: 18
[24912, 1495, 553, 481, 220]
GPT-3 (Legacy)
Input text: "hello how are you"
Tokens: 4
Characters: 18
[31373, 2200, 389, 345]
Notice how the same text produces different token counts and completely different token IDs depending on the model. This is important to understand when working with these systems, as it affects things like context length limits and processing costs.
Key Components of Generative AI
Large Language Models
Massive neural networks trained on diverse text datasets that can recognize and generate complex patterns in language.
Transformer Architecture
The breakthrough model design that enables AI to process and understand context in text through a mechanism called 'attention'.
Pattern Recognition
The ability to identify structure in data, allowing the AI to make predictions even for inputs it has never seen before.
Next-Token Prediction
The fundamental task of predicting what word or character comes next in a sequence based on previous context.
Context Understanding
How the model uses surrounding words to understand the meaning of text and generate appropriate responses.
Generative Capability
The ability to create new content rather than simply retrieving or classifying existing information.
Practical Example: The Code Fixer
One impressive capability of generative AI is fixing code errors without having seen that specific code before. Here's how it works:
/* Broken JavaScript code */
function calculateSum(a, b) {
return a + b
}
const result = calculatesum(5, 10);
console.log(result);
When shown this code, ChatGPT can identify that calculatesum should be calculateSum (case-sensitive) and that the function is missing a semicolon. It knows this despite never seeing this specific code before.
This works because the AI has learned patterns from millions of code examples - variable naming conventions, syntax rules, and common mistakes. It uses these patterns to recognize errors and predict the correct code, just like it predicts the next token in a conversation.
Key Takeaways
- Generative AI doesn't memorize answers - it predicts what should come next based on patterns
- It can handle questions it's never seen before by recognizing familiar patterns
- The same predictive capability works across text, code, images, and other formats
- Understanding these fundamentals helps us better utilize and interact with AI tools
Ready to dive deeper into how LLMs really work?