AI 工程师笔记本：在 Colab 上免费、无需框架即可使用 RAG/智能体/评估工具

AI Engineer Notebooks

Learn the applied-LLM stack the way you'll actually be interviewed on it: framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark.

Runnable Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set. You build working systems on top of foundation models (model APIs, RAG, evals, agents, adaptation, serving) using raw APIs, not frameworks.

What makes this different

Framework-free, on purpose. You write the agent loop, RAG, and evals from raw API calls first, so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when not to). Patterns are durable; wrappers churn.
Evals are the spine. "Measure before you tune" is installed early and returns in every section. It's the habit that separates an engineer who shipped a system from one who built a demo.
Free to run, end to end. Everything runs on the free Groq API (no credit card). The two topics Groq can't host, LoRA fine-tuning (06) and self-hosted serving (09), are concept-first with optional, fenced Colab-GPU appendices that were verified on a real Colab T4.
Real case studies, not toy demos. Three end-to-end case studies show the skills combined under real constraints: a support assistant debugged in production, a pipeline-vs-agent cost showdown, and a red-team robustness benchmark.
OpenAI-compatible throughout, so every pattern transfers directly to OpenAI and (with small changes) Anthropic. Swap the base URL and the skills carry over.

Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer. The plan explains what to learn and why; these notebooks are where you run it.

Who this is for

Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles. Different titles, largely the same job. You can ship production code; you want the applied-model layer on top.

Learning order

Work top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises.

00 — Setup

Notebook	What you'll learn
Environment & cost hygiene	API keys via Colab secrets, spend guards, model picking

01 — Model APIs

Notebook	What you'll learn
Prompting fundamentals	Clear instructions, few-shot, output-format specs, step-by-step reasoning: the cheapest lever, each shown moving a number
Structured output	Getting reliable JSON out of a model, and where it breaks
Tool calling	Function/tool calling end to end, error paths included
Streaming	Streaming responses and what UIs need from them
Context & caching	Context-window budgeting, prompt caching, batch vs real-time pricing

02 — Evals I: measuring outputs

Notebook	What you'll learn
Measuring outputs	Golden sets and metrics on the section-01 task; install the "measure before you tune" habit before building anything you'd need to tune. Evals is the spine; it returns in every section after this

03 — RAG

Notebook	What you'll learn
What is RAG?	The retrieve → augment → generate loop, why RAG beats a plain LLM, and why RAG isn't the same as embeddings, with a 15-line working demo
Embeddings & retrieval	Embedding choice, vector search, similarity pitfalls. Get retrieval working first
Hybrid & reranking	Keyword + vector hybrid retrieval, rerankers, when each earns its cost
Chunking	Chunking strategies on a real messy corpus, revisited last once you can judge them against retrieval
Why RAG fails	Diagnosing bad answers: retrieval quality, not generation, is usually the bottleneck

04 — Evals II: the differentiator

Notebook	What you'll learn
Golden sets	Building a golden set for the RAG system from section 03
LLM as judge	Judge prompts, agreement with humans, and the judge's own failure modes
Regression evals	Evals as CI: catching quality regressions when you change a prompt or model

05 — Agents

Notebook	What you'll learn
Agent loop from scratch	A working agent loop in raw API calls, no framework
Tool design	Designing tools the model can actually use well
Guardrails & budgets	Stopping conditions, cost/latency budgets, when a pipeline beats an agent
MCP & the tool ecosystem	Concept: what the Model Context Protocol standardizes, how it maps to the raw tool loop, and when to reach for it
Skills & progressive disclosure	Concept: packaging reusable know-how an agent loads on demand. The `SKILL.md` pattern, the context-budget payoff, and Tools/MCP/Skills as one story
Harness engineering	Synthesis: the scaffold around the call: context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece

06 — Adapting the model

Notebook	What you'll learn
Fine-tune vs RAG vs prompt	When to change the model's weights vs its inputs; what LoRA/QLoRA are and cost; the argument you'll have in the room, plus an optional real LoRA fine-tune on a free GPU

07 — Security

Notebook	What you'll learn
Prompt injection & the trust boundary	Direct & indirect prompt injection, output handling, PII, excessive agency. The OWASP LLM Top 10 risks, failing live then defended

08 — Operations

Notebook	What you'll learn
Observability & LLMOps	Tracing every call, safe prompt logging, cost/latency/error metrics, drift detection, and the observe→eval feedback loop
Reliability & fallbacks	Retries with backoff, timeouts, fallback models, output validation, circuit breakers, graceful degradation
Experiment tracking & registry	MLflow end to end: log runs/params/metrics from the section-04 eval harness, register and version a model, and promote by stage: the tooling that turns "I ran an eval" into a tracked, reproducible workflow

09 — Serving & inference performance

Where the free Groq API can't run the topic (these frameworks need a GPU), the notebook teaches it concept-first and fences an optional Colab-GPU appendix, the same pattern as the section-06 LoRA appendix.

Notebook	What you'll learn
Serving frameworks	The serving stack an AI engineer actually picks between (vLLM, TGI, Triton, TensorRT-LLM): what each optimizes, how they map onto the raw API you've been calling, and when to reach for which
Inference performance	The levers behind throughput and latency: continuous batching, the KV cache, quantization, and the throughput-vs-latency trade, with the napkin math to size a deployment

10 — ML system design & performance

Notebook	What you'll learn
Designing an inference service	Concept: the ML system design interview, worked end to end: QPS/VRAM/latency/cost estimation, replica scaling, queueing, caching, and the SLA trade-offs, on a realistic LLM-serving prompt

11 — Customer craft (the FDE differentiator)

Notebook	What you'll learn
Scoping & discovery	Turn a vague customer ask into a scoped, evaluable system: discovery questions, a one-page scoping doc, the demo discipline: the customer-scenario interview round most engineers can't evidence

12 — Case Studies & Capstone

Where the skills come together into projects. First a case study (one realistic scenario worked end to end, runnable), then the capstone: the deployed repo you build yourself. (Section overview.)

Notebook	What you'll learn
Case study A — Customer-support assistant	One scenario scoped → built → served → debugged in production: a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11
Case study B — Contract extraction: pipeline vs agent	The judgment call interviewers love: build the same extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known
Case study C — Red-team robustness benchmark	A different kind of system, a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals

Capstone: the brief for the deployed project that goes on your resume, a real repo with a serving component and an eval report. Case studies are for learning; the capstone is for hiring.

Conventions

Raw model APIs, no frameworks. Patterns are durable; wrappers churn.
One shared corpus (data/) across RAG and eval sections, so evals measure the retrieval you actually built.
Self-contained notebooks. First cell installs, second cell calls from aien import setup; client, MODEL = setup() to load your key from Colab secrets (or a local env var). No hidden state between notebooks. aien is the tiny shared-setup package in this repo (one place to change credential loading), installed automatically by the first cell.
Every notebook ends with exercises. Do them before moving on.

Setup

Get a free API key at console.groq.com, no credit card required.
In Colab: the key icon in the left sidebar → add GROQ_API_KEY as a secret, and toggle notebook access on.
Open any notebook via its badge and run top to bottom.

Running locally instead: pip install -r requirements.txt && pip install -e . (the second installs the aien setup helper), export GROQ_API_KEY=..., open with Jupyter.