Retrieval-Augmented Assistant Platform
LiveA question-answering assistant built end to end on Google Cloud, from document ingestion through retrieval, grounded generation, and a streaming interface — then operated in production across four environments. The interesting part was never the happy path. It was discovering, under real traffic, all the ways a system can report itself healthy while failing the people using it.
Architecture
Managed semantic retrieval with server-side scoping
Retrieval runs against a managed vector and semantic search service over a curated corpus rather than a self-hosted index, trading tuning control for operational simplicity. Scope filters — content set, version, release stage, confidence floor — are enforced server-side on every query, so a client can never widen its own access by editing a request.
Grounded generation with enforced citations
The generation step is constrained to the passages retrieval returned, and every response carries structured citations plus an explicit grounded flag. Answers that fail the grounding check are rejected rather than returned, which makes "I could not find that" a first-class outcome instead of a hallucinated near-miss.
Ingestion decoupled from serving
An offline pipeline normalizes source documents into clean text, a structured import format, and an audit manifest recording exactly what was published. It is batch-imported into the search store on its own schedule. Serving and ingestion share no request path and no queue — they meet only at the data store, so a bad ingest can never stall live traffic.
Streaming response path
Answers stream token by token over server-sent events through a server-side proxy, which keeps credentials off the browser and makes time-to-first-token the metric users actually feel. Caching sits in front of the retrieve-and-generate path, keyed on the normalized query, to keep repeat questions cheap.
Infrastructure
Serverless containers on a private network
Both services run as serverless containers with all egress routed through a private network connector and ingress restricted to internal traffic. Invocation is gated by IAM bindings rather than application-level checks, and each environment carries its own instance floor and ceiling.
Four environments from one Terraform definition
Development, sandbox, staging, and production are the same infrastructure module with per-environment variables and separate remote state. Shared building blocks — artifact registry, build triggers — come from private registry modules, so environment drift shows up as a plan diff instead of a production surprise.
Two-track delivery with guarded deploys
A managed build service on a private worker pool builds and pushes images on every push; deployment workflows authenticate through workload identity federation, so no long-lived cloud keys exist anywhere in CI. Promotions run as guarded deploys with a canary soak that has to stay healthy before traffic shifts.
Continuous evaluation as a pipeline stage
A golden-query set and a live-traffic evaluation run as their own workflows rather than as a manual pre-release ritual, so answer quality regressions surface as a failing job. This is also where the hardest lesson landed — see below.
Observability with an allow-listed log schema
Structured logs emit only explicitly allow-listed fields, so user content cannot leak into log sinks by default, with a separately gated diagnostic path for supervised debugging. Load testing runs on a dependency-free harness that drives the real streaming endpoint and gates on p95, p99, and time-to-first-token, with a mock mode that needs no cloud credentials.
Hurdles
Retrieval pollution: the corpus fought itself
Long financial-disclosure documents were winning roughly 80% of retrieval slots on ambiguous operational queries, purely through keyword overlap. They displaced the correct sources and produced fluent, confident answers built on the wrong sense of a word that meant one thing in a filing and something entirely different in operational documentation. Relevance thresholds and score floors barely moved the number — the fix was changing what retrieval returned, not how it was scored.
The retrieval tier capped answer quality, not the model
On the golden-query set, answers covered only about 17% of the expected facts. The cause was the search tier returning a single lexically-selected snippet per document, so the generation step was reasoning over fragments no model could complete. Moving to extractive segments and query expansion lifted the ceiling; more prompt engineering against the old return shape would not have.
Token budgets are one coupled envelope, not three knobs
Output limit, reasoning budget, and generation timeout turned out to be a single system. Raising the output cap to fix truncated answers roughly doubled reasoning-token usage and converted the truncation failures into timeout failures at the 45-second boundary. Separately, a fixed input cap combined with variable-length retrieved segments overflowed the prompt and hard-failed every request until the budget was made a function of what retrieval actually returned.
The evaluation harness measured a transport nobody used
The eval suite, deploy canary, and smoke tests all spoke plain JSON while every real user was served over streaming. A parser bug at the streaming boundary was failing roughly one in three real conversations while automated checks reported 96.4% healthy. A 24-hour response cache and unconfigured production logging hid it further. The lesson generalized: test the transport users are on, or the green check is measuring a system that does not exist.
Warm is not the same as initialized
Raising the instance floor to improve tail latency caused about a third of requests to fail instead. Readiness probes were passing while lazily-constructed retrieval and generation clients were still cold, so the platform routed traffic to instances that were up but not ready to serve. Compounding it, the platform provisioned only half the requested floor for several hours, capping real concurrency well below what the configuration claimed.
One error code for four different failures
A single capacity-exhausted code covered model quota limits, per-user rate limits, and two other causes. That collapsed the signal and led to a confident misdiagnosis — engineering time spent chasing model quota when the actual limit was a per-user cap of 30 requests per minute. Splitting the code by cause turned a recurring investigation into a log line.
Configuration coupling across model swaps
Changing model tiers surfaced hidden coupling: the new tier rejected a zero reasoning budget, so every canary request returned a gateway error until the paired setting was updated. Model choice and generation parameters had to be versioned and validated together rather than tuned independently.
Access control by inheritance is not access control
Content behind a release gate was protected only by inherited project-level IAM roles rather than an explicit runtime check, which made the boundary far broader than intended. It was caught in review, scoped, and moved to an enforced runtime gate — a reminder that "nobody has the URL" and "the system denies the request" are different security properties.