Architecture, retrieval, evaluation, and the operational work of running an LLM-based product at scale.
Published Mar 28, 2026
ost AI assistant projects fail in the same way: a working demo, an enthusiastic stakeholder, and then a months-long slog through the unglamorous parts — retrieval that scales, evaluation that catches regressions, guardrails that don't break on edge cases.
The architecture
A three-tier system tends to be the right shape: a domain-tuned LLM at the core, a retrieval layer with vector embeddings over your internal docs, and a routing layer that classifies intents and directs traffic to specialized models. The exact tools matter less than the discipline of separating concerns.
Anyone can ship a demo. Shipping a system that doesn't regress on Friday afternoons is what separates AI products from AI experiments.
Evaluation is the moat
Build the eval harness before the product. Treat it as production code. Test adversarially. The teams that get this right ship steadily. The teams that don't ship demos that work in screenshots and fail in production.
Filed under
Engineering · AI
