Generating a large question bank with AI without letting a single wrong answer into a product students trust to prepare them for an exam that determines their career. A subtly wrong physics answer, confidently presented, actively harms the student — and it's hard to detect, because a plausible wrong solution is exactly what a language model produces most readily.
What I did about it
A staged pipeline where AI does retrieval and drafting but never has final authority, with hard carve-outs where AI is banned entirely.
Provenance is modelled explicitly: QuestionSource distinguishes PYQ (a real past paper, authoritative) from BANK (AI-generated). Different classes of object in the schema, so a generated question can never be silently presented as a real past-year question.
Generation is grounded, not free — questions derive from ingested books with RAG retrieval over pgvector, and similarity thresholds reject drafts too close to existing items. A separate guard model checks the first model's work before a human adjudicates anything below the confidence band.
Figure and diagram questions are human-only — a model reading a circuit diagram is exactly where confident errors cluster. And every live-bank mutation snapshots to a backup table inside the transaction, so a bad batch is reversible rather than a restore-from-backup incident.
What I rejected, and why
Generate, spot-check a sample, ship is what most of the market does and is faster. Rejected because the failure mode is not a bad user experience — it is a student getting a question wrong in an exam because the practice material taught them something false, and there is no way to detect that from analytics.
The backend test script runs jest --passWithNoTests, which means backend coverage is effectively unverified while the Flutter app is the best-tested surface. For a platform handling payments and a question bank students rely on, that is the wrong way round.