Model Evaluation & Performance Engineering
Measures, profiles and optimises a model to fit real system constraints, and documents the trade-off.
Tech stack
Modules
5 modules · 90 hours
M1 · Define model parameters
18 h · PC1, PC2- Technical specifications and limitations of the host system
- CPU/GPU architecture, cores, clock, cache hierarchy
- RAM and VRAM ceilings
- Storage I/O
- OS constraints
- Data flows and structures: tensors, batches, feature stores
- Serialisation formats
- Precision (fp32/fp16/int8)
- Interface contracts and schema definition
Lab: Author a formal specification sheet documenting host limits and the model's data-flow graph
Course material for this module is in production.
M2 · Runtime, speed and memory analysis
20 h · PC3- Algorithmic complexity and Big-O notation
- Time and space trade-offs
- Profiling tools
- Latency percentiles (p50/p95/p99)
- Throughput and cold start
- Memory leaks, fragmentation, garbage collection
- Batch-size versus memory curves
Lab: Profile a model end to end; produce a flame graph and a bottleneck report ranked by impact
Course material for this module is in production.
M3 · Parallel programming constraints
18 h · PC4- Flynn's taxonomy: SISD, SIMD, MISD, MIMD
- Threads versus processes
- The Python GIL
- Multiprocessing and vectorisation
- BLAS threading
- GPU execution model, kernels, occupancy, memory bandwidth
- Data versus model parallelism
- Amdahl's law
- Synchronisation, race conditions, deadlock
Lab: Parallelise an inference workload; measure scaling efficiency against Amdahl's prediction
Course material for this module is in production.
M4 · Optimise to fit constraints
20 h · PC5- Quantisation (post-training and quantisation-aware)
- Pruning
- Knowledge distillation
- Operator fusion and graph optimisation
- ONNX export
- Batching strategies
- Caching
- Mixed precision
- Accuracy-versus-resource trade-off analysis
- Regression testing after optimisation
Lab: Reduce model latency and memory to a stated budget while holding accuracy within tolerance; present the trade-off curve
Course material for this module is in production.
M5 · Document and feed back
14 h · PC6, PC7- Model cards
- Evaluation reports
- Versioned documentation
- Reproducibility metadata
- Anomaly identification and referral
- Structured performance feedback to the design team
- Organisational documentation and data-sharing policy
- Escalation paths
Lab: Publish a model card and deliver a formal performance-feedback review to the modelling team
Course material for this module is in production.