C04 Stage B · Core AI 26 h (10 T / 16 P)
Transformers & Representation Learning
Explains and implements the architecture underlying modern language models.
Prerequisite
C03
Modules
3
NOS
—
Statutory mapping
—
Tech stack
PyTorchHugging Face Transformerstokenizers
Modules
3 modules · 26 hours
M1 · Attention
10 h TOPICS
- Query, key, value
- Scaled dot-product attention and the √d rationale
- Softmax weighting
- Multi-head attention
- Self versus cross attention
- Causal masking
Lab: Implement scaled dot-product and multi-head attention from scratch
Course material for this module is in production.
M2 · Transformer architecture
8 h TOPICS
- Encoder and decoder stacks
- Residual connections
- Layer normalisation
- Feed-forward blocks
- Positional encoding: absolute, learned, rotary
- Quadratic complexity in sequence length
Lab: Assemble a minimal transformer block and verify shapes and gradients
Course material for this module is in production.
M3 · Representation learning
8 h TOPICS
- Tokenisation: BPE, WordPiece, SentencePiece
- Vocabulary
- Embeddings
- Contextual versus static embeddings
- Embedding geometry
- Cosine similarity
- Pretraining objectives: causal and masked LM
Lab: Train a tokeniser; probe embedding similarity behaviour
Course material for this module is in production.
Assessment
From-scratch attention implementation · embedding probe report