chapter 13 / capstone · the course's final hands-on project · estimated build time 1-2 weekends

The Capstone
Teaching a Small Model to Draw Pixel Art

This chapter has no new theory—it twists the knowledge of the previous 13 chapters into a single end-to-end project you can actually run on your own Mac: use LoRA+SFT to make a 4B-9B open-source small model reliably emit structured pixel-art JSON. Every step is tagged with the chapter it draws on. Finish it and you complete the leap from "having studied" to "having done."

Step one: task definition and schema design

Drawing on the first rule of task-design from Chapter 11: narrow the task surface. Don't build "AI draws pictures"; build "generate JSON conforming to this schema":

{
  "size": 8 | 16 | 32,                  // canvas side length
  "palette": ["#RRGGBB", ...],          // 2-16 hexadecimal colors
  "pixels": [[int, ...], ...]           // size×size matrix, values are palette indices
}

This representational choice alone is half the project's success (Chapter 8 argued this vs. diffusion models): every rule is programmatically decidable—the JSON parses, fields are complete, hex is valid, the matrix is rectangular, indices stay in bounds. So one and the same validate() function handles three jobs: scoring the evaluation, filtering the training data, and (advanced) supplying the RLVR reward (the terminus of Chapter 8's evolution axis of supervision signals). Verifiability isn't luck—it's designed in.

Interactive lab: the validator itself

Below is the web version of this validator. Required experiment: ① click the three error examples in turn—palette out of bounds (the model "wants to use" a color it never declared), uneven row length (miscounting a long sequence, the curse of tokenization, Chapter 7), and an illegal color value (treating a natural-language color name as hex)—these three are exactly the mistakes a small model makes most often before real training; ② edit the JSON directly in the text box and watch the checks turn red and green line by line as the canvas re-renders in real time (illegal cells flash bright red).

Step two: evaluation first—eval.py and the 0/12 baseline

The first rule of Chapter 8's methodology: have an evaluation before you train. Twelve stratified cases (4 easy: single color blocks; 4 medium: specified object + color scheme; 4 hard: style constraint + size 32), each running validate() + a task assertion:

import json, re, sys
from mlx_lm import load, generate

HEX = re.compile(r'^#[0-9a-fA-F]{6}$')

def validate(text: str) -> tuple[bool, str]:
    try: d = json.loads(extract_json(text))           # the model may wrap it in markdown, so extract first
    except Exception as e: return False, f'parse: {e}'
    if not all(k in d for k in ('size','palette','pixels')): return False, 'missing fields'
    if not (2 <= len(d['palette']) <= 16 and all(HEX.match(c) for c in d['palette'])):
        return False, 'bad palette'
    px, n = d['pixels'], d['size']
    if len(px) != n or any(len(r) != n for r in px): return False, 'not rectangular'
    if any(not (0 <= v < len(d['palette'])) for r in px for v in r): return False, 'index OOB'
    return True, 'ok'

model, tok = load(sys.argv[1])                         # base or adapter path
cases = [json.loads(l) for l in open('eval_cases.jsonl')]
passed = 0
for c in cases:
    out = generate(model, tok, prompt=render_chat(c['instruction']), max_tokens=2048)
    ok, why = validate(out)
    print(('✓' if ok else '✗'), c['id'], why)
    passed += ok
print(f'{passed}/{len(cases)} pass')

The expected result of running the base model (e.g., Qwen3-4B-Instruct): 0/12 to 2/12—it produces roughly plausible JSON, but with uneven row lengths and out-of-bounds indices (errors ②③ from the lab). This number is the frame of reference for all the work that follows.

Step three: dataset engineering—synthesis + filtering pipeline

Target: 3000-5000 "instruction → standard JSON" pairs (the sweet spot from MLX community experience). The pipeline (a miniature of Chapter 7's FineWeb philosophy):

1. Seed: hand-craft 30-50 examples covering the full difficulty range (Chapter 8's LIMA: quality > quantity, the seed sets the tone).
2. Programmatically generate half the data: the beauty of pixel art—you can generate it in reverse: a program randomly combines "object template × palette × size" to draw legal JSON, then has a strong model write a natural-language instruction for it ("draw a frog wearing a hat, retro green palette"). The instruction is generated, the answer is constructed and inherently 100% compliant.
3. Synthesize the other half with a strong model: a flagship model batch-produces "instruction + JSON" pairs in the seed style, each passing through validate(); anything non-compliant is discarded outright—the validator reports for duty a second time here.
4. Diversity audit: bucket and count by object category / color style / size, then fill in whichever category is missing (preventing a lopsided dataset → a lopsided model).

The format is chat JSONL (Chapter 8's template + loss mask), on the order of 13M tokens.

Step four: choosing a base model—a controlled experiment

Candidates: 4B dense / 8B dense / a same-tier hybrid architecture (linear-attention family) / a small MoE. Here are the real conclusions and the method straight from Chapter 7:

• Controlled experiment (change one variable at a time): same machine, same data subset (500 examples), same LoRA config, train each for 100 steps and record throughput and memory—a real case once measured dense being 15× faster and using 3/4 less memory than a delta-net-family hybrid architecture, the root cause being 100% the implementation maturity of mlx-lm for the newer operators. A 30-minute controlled experiment avoids a whole wasted night.
• Why MoE is out (the detailed account in Chapter 7): the cost of stuffing all the experts into unified memory + the router easily becoming imbalanced when fine-tuning on small data + uneven framework support.
• Conclusion: start with 4B dense—first get the loop running with the smallest viable model (the correctness of the pipeline is independent of model size); if quality isn't enough, step up to 8B/9B.

Step five: training—LoRA validation → overnight full run

# —— Stage A: LoRA pipeline validation (about 20 minutes, the Chapter 8 calculator did the memory math) ——
python -m mlx_lm lora --model Qwen/Qwen3-4B-Instruct \
  --train --data ./data --iters 600 --batch-size 4 \
  --num-layers 16 --learning-rate 1e-5            # 600 steps ≈ 3 epochs (small subset)
python eval.py ./adapters                          # expect: 0/12 → 6-9/12, loss decreasing smoothly
# the question this step answers is "is the pipeline right", not "is the quality good"

# —— Stage B: full data, overnight run (about 13M tokens, 8B is 3-4 hours per epoch / 4B is half that) ——
python -m mlx_lm lora --model Qwen/Qwen3-8B-Instruct \
  --train --data ./data_full --iters 2400 --batch-size 4 \
  --num-layers 32 --learning-rate 1e-5 \
  --steps-per-eval 200 --val-batches 25            # start before bed, check the val loss curve when you wake

The rhythm is exactly what Chapter 8 described: build data and evaluations by day, run training at night, check scores in the morning. Three signals worth watching: val loss falling together with train loss (no overfitting, Chapter 1), the row-length error rate of the generated samples (the most stubborn error type), and the situation where loss falls but the evaluation doesn't budge (which means the problem isn't fitting, it's the data—proceed to the next step).

Step six: failure-case analysis and data iteration

Suppose after the overnight run you're at 9/12. The 3 you got wrong aren't noise, they're the work orders for the next round of data engineering (Chapter 8's controlled-experiment discipline):

After two or three rounds of "evaluate → attribute → top up data → retrain" it stabilizes at 11-12/12, and this loop itself (dataset + evaluation + pipeline) is the asset you've accumulated—the model is merely its current output (the judgment of Chapters 7/8 closes the loop here).

Step seven: deployment, cost accounting, and Agent-ification

• Deployment: mlx_lm fuse to merge the adapter → 4-bit quantization (Chapter 9: 4-bit is the last stop before the cliff, nearly lossless for formatted-output tasks) → a local OpenAI-compatible service (mlx_lm server). 8B-int4 ≈ 4.5GB, serviceable from a single Mac mini.
• Cost accounting (the measured version of Chapter 9's estimator): at 2.4K in / 0.7K out per generation, the marginal cost of self-hosting is about 1/5-1/10 that of the flash API and 1/200 that of the flagship API—under a "1/100 remix rate" funnel, the monthly generation cost for 100K DAU drops from the flagship's thousands of dollars to a two-digit order of magnitude. More important is the qualitative shift: the cost is low enough that you can build generation into the core feature loop, rather than rationing it.
• Agent integration (Chapter 11): wrap the model as one tool of an Agent—generate_pixel_art(prompt) → validate() → on failure, retry once with the error message → if it still fails, escalate to the flagship model. The validation-failure rate is your production monitoring metric, and tiered fallbacks preserve tail quality.
• Advanced signpost (RLVR): validate() is a ready-made verifiable reward—run GRPO (Chapter 8) on top of the SFT model to push every programmatically decidable metric, except "aesthetic quality," toward 100%. You already possess all the knowledge to do this.

Graduation quiz