New to this topic? That is completely fine. It is completely fine if you have never studied some of these topics before. We want to see how fast you can learn something new, grasp it, and teach it from first principles.
This page is a quick primer, not a script. Use it to get your bearings, then read the links below and anything else you find: other blogs, papers, videos, code. We are judging the story you build, your examples and metaphors, your first-principles thinking, and any experiments you run yourself.
A quick primer
Almost every modern language model, from GPT to Llama to DeepSeek, is built on one idea from 2017: the transformer. Here is the shape of it, so you can find your bearings. Your job is to teach it far better than this page does.
The pieces to explain
- Tokens and embeddings. Text is split into tokens, and each token becomes a vector of numbers.
- Position. Attention by itself has no sense of order, so position information is added.
- Self-attention. Each token asks a question (query), every token offers a label (key) and some content (value). Similar query and key means more of that value flows in.
- The MLP. After tokens exchange information, each one is processed on its own by a small neural network.
- Residuals and normalisation. Why we add the input back after every sub-layer, and why that makes deep networks trainable.
- The output. A final projection and softmax give a probability for every token in the vocabulary.
Can you explain why we divide by ? Why the mask? Why many heads instead of one? Those are the questions that separate a good lesson from a great one.
Read
- Attention Is All You Need · Vaswani et al., 2017The original paper.
- The Illustrated Transformer · Jay AlammarThe classic visual walkthrough. A good model for whiteboard drawings.
- The Annotated Transformer · Harvard NLPThe paper, line by line, next to working PyTorch code.
- The Transformer Family, Version 2.0 · Lilian WengRigorous notation and the variants that came after.
- nanoGPT · Andrej Karpathy (GitHub)A tiny, readable GPT you can train yourself.
Watch
- ▶ Let's build GPT: from scratch, in code, spelled out · Andrej Karpathy
- ▶ Transformers, the tech behind LLMs (Chapter 5) · 3Blue1Brown
- ▶ Attention in transformers, step by step (Chapter 6) · 3Blue1Brown
Go beyond this list. These are starting points. The best submissions usually bring in something we did not link: an experiment, a paper, a story of their own.