R5 Stage D · 160 h (48 T / 112 P)
Data & RAG Engineer
Builds and operates the data and retrieval layer that AI systems depend on.
Tech stack
PythonSQLAirflowdbtUnstructuredpgvectorQdrantOpenSearchsentence-transformersGreat Expectations
Modules
M1 · Ingestion at scale
40 h TOPICS
- Source connectors
- Batch versus streaming
- Change data capture
- Incremental and delta sync
- Complex document parsing: tables, forms, scans
- OCR quality
- Deduplication
- Idempotent loads
- Backfill strategy
Lab: Production ingestion pipeline over a heterogeneous, frequently changing corpus
Course material for this module is in production.
M2 · Transformation and modelling
40 h TOPICS
- Dimensional modelling
- Transformation frameworks
- Testing and data contracts
- Data quality dimensions
- Lineage and cataloguing
- Schema evolution
- PII handling in pipelines
Lab: Tested transformation layer with enforced data contracts and lineage
Course material for this module is in production.
M3 · Retrieval engineering
44 h TOPICS
- Embedding model selection and benchmarking
- Chunking experimentation
- Index type and parameter tuning (HNSW/IVF)
- Metadata filtering
- Hybrid search fusion
- Reranking
- Freshness and TTL
- Multi-tenancy and access control in retrieval
Lab: Tune an index to meet stated recall and p95 latency targets
Course material for this module is in production.
M4 · Operating the retrieval layer
36 h TOPICS
- Retrieval evaluation harness
- Monitoring recall drift
- Corpus growth and re-embedding strategy
- Cost of storage and embedding
- Capacity planning
- Disaster recovery
- Corpus poisoning defence
Lab: Retrieval evaluation and monitoring suite with scheduled refresh
Course material for this module is in production.
Track project
Production retrieval platform: ingestion of a multi-format changing corpus, tested transformation layer, tuned hybrid index with reranking, access-controlled multi-tenant retrieval, evaluation harness reporting recall@k, MRR and faithfulness, plus scheduled refresh and monitoring.
Job-ready exit standard
Builds and measures a retrieval layer; defends chunking, embedding and index choices with numbers.