G04 Stage C · Applied GenAI 44 h (16 T / 28 P)
LLM Evaluation, Observability & Cost
Proves whether an AI system is good enough to ship, with evidence.
Prerequisite
G02
Modules
6
NOS
SSC/N8121
Statutory mapping
—
Tech stack
RAGASDeepEvalLangSmithMLflow
Modules
6 modules · 44 hours
M1 · Evaluation strategy
8 h TOPICS
- What to measure
- Offline versus online
- Golden datasets
- Dataset curation and maintenance
- Annotation guidelines
- Inter-annotator agreement
- Test-set contamination
Lab: Construct a governed golden dataset with an annotation guide
Course material for this module is in production.
M2 · Retrieval metrics
8 h TOPICS
- recall@k
- precision@k
- MRR
- NDCG
- Hit rate
- Context precision and recall
- Evaluating retrieval independently of generation
Lab: Retrieval-only evaluation harness with dashboards
Course material for this module is in production.
M3 · Generation metrics
10 h TOPICS
- Faithfulness and groundedness
- Answer relevance
- Correctness and completeness
- Toxicity and safety
- Deterministic assertions
- LLM-as-a-judge design
- Judge bias and calibration
- Human agreement
Lab: Build and calibrate a judge against human labels
Course material for this module is in production.
M4 · Agent and multi-turn evaluation
6 h TOPICS
- Trajectory evaluation
- Tool-call correctness
- Task completion
- Conversation quality
- Regression suites
Lab: Trajectory evaluation for the G03 agent
Course material for this module is in production.
M5 · Observability
6 h TOPICS
- Tracing and spans
- Prompt and version logging
- Latency and token dashboards
- Error taxonomies
- Sampling live traffic
- User feedback capture
- Drift detection
Lab: Instrument a live pipeline with end-to-end tracing
Course material for this module is in production.
M6 · Cost and performance engineering
6 h TOPICS
- Token accounting
- Prompt compression
- Exact and semantic caching
- Model routing and tiering
- Batching
- Streaming
- Budget alarms
- Unit economics per request
Lab: Halve cost per query at constant measured quality
Course material for this module is in production.
Real-world work scenario
Leadership asks whether a prompt change may ship. Run the regression suite, separate retrieval from generation effects, quantify quality delta with confidence, report cost and latency impact, and issue a go/no-go recommendation with evidence.
Assessment
Evaluation harness · calibrated judge · cost-reduction practical