AI Data Engineer Path¶
From LLM APIs and embeddings to retrieval, agents with safe data access, and the evaluation that shows whether any of it works.
Who it is for: you build or feed LLM applications with private data, and you need them to be measurable, observable and safe.
Labs in this path: 07 and 08, about 2 to 3 hours in total. Both run locally with Python only.
Stage 1 — Foundations¶
Goal: call a model reliably and control its output.
- Read: Prompt Engineering, then LLM APIs and SDKs.
- Also: Python for DE and SQL, if you are new to either.
Checkpoint: explain how you would enforce a JSON output shape, and what your code does when the model returns something else.
Stage 2 — Retrieval¶
Goal: give a model the right context from your own documents.
- Read: Embeddings, Vector Databases and RAG.
- Do: Lab 07, Capstone: Docs RAG (60–90 minutes): chunking, a BM25 index and a retrieval evaluation (hit rate and MRR) over this handbook's own guides.
Checkpoint: retrieval misses a document that contains the answer. List the causes, from chunking to ranking, and how you would tell them apart.
Stage 3 — Agents and data access¶
Goal: let a model use tools and query data without putting the data at risk.
- Read: AI Agents and Tool Use, then MCP and Text-to-SQL. LangChain and LlamaIndex cover orchestration libraries.
Checkpoint: an assistant can run SQL on your warehouse. List the controls that make that acceptable (read-only access, allowed tables, limits), and how you would test that each one holds.
Stage 4 — Evaluation and observability¶
Goal: measure quality before and after release.
- Read: Evals, then AI Observability.
- Do: Lab 08, Data Quality Gates (60–90 minutes). The same idea, checks with thresholds that block a release, applies to the data that feeds a model. Read Data Quality alongside it.
Checkpoint: design an evaluation set for a support chatbot: how many questions, how they are labelled, and what result would stop a release.
Stage 5 — Operating models¶
Goal: track experiments, and choose between prompting, retrieval, fine-tuning and local models.
- Read: MLflow, Fine-Tuning LLMs and Local LLMs.
- Also: Claude Code, for building with an AI coding assistant.
Checkpoint: for one task, say why you would prompt, retrieve, fine-tune or run a local model, and what evidence would change your mind.
Stage 6 — The data underneath¶
Goal: keep the data behind the application fresh, governed and secure.
You are done when you can explain, for an application of your choice, where each answer comes from, how you know retrieval and generation are good enough, and what data the system must never expose.
Going further¶
- Learn the pipelines that feed a retrieval index with the data platform engineer path.
- Extend Lab 07 with a vector index, and compare its hit rate with BM25.