It Works on My Laptop: The Distance Between Running and Shipping

Post 5 of 7. Posts 1–4 built a playground and used it to make pre-training, post-training, and evaluation concrete. This post is about the last gap: the one between an application that runs and an application that ships, written from an actual deployment, not a plan for one.

---

For the entire project up to this point, "does it work?" had one meaning: I send a request, I get a sensible response. By that standard the playground was done. Then I deployed it publicly, and discovered "works" had been quietly borrowing four assumptions: I am the only user, I have infinite patience, restarts are free, and nothing costs money. Production revokes all four, and this post is the record of what that revocation actually required.

The reframe

That last row is the one I underestimated most. Locally, nothing distinguished a cheap decision from an expensive one, because nothing was expensive. Production is where every architectural choice in Posts 1–4 quietly acquires a price tag.

Sizing surprise: the model that was 10x its name

Before deploying anything, I'd considered adding Google's Gemma 4 to the model registry. The smallest variant is called E2B, which reads as "2 billion parameters." It's actually ~5.1B parameters and a ~9.3GB download, "E" stands for effective compute, a performance metric, not a parameter count. Locally this was a mild inconvenience: I checked the real numbers and left it out. On a cloud platform, where RAM directly sets your hosting tier and your bill, the gap between the marketing number and the actual footprint stops being trivia and becomes a budgeting error waiting to happen. Model names describe capability; deployment cares about footprint; they'd quietly stopped correlating.

The two-layer split, and the choice it forced

An LLM app is two workloads wearing one trench coat: model serving (GPU-shaped platforms, Modal, RunPod, Cloud Run with GPU) and the app around the model (ordinary web hosting, Railway, Vercel, Cloud Run without GPU). I deployed both halves as separate Cloud Run services, CPU-only, deliberately isolating one variable at a time: local CPU → cloud CPU, before any GPU question. In hindsight this was the right call, everything that went wrong below was about identity, networking, and lifecycle, not about needing more compute. Adding a GPU on top of an unsolved IAM problem would have just given me a more expensive unsolved IAM problem.

Containerization: three real findings

Dead dependencies were an archaeology site. pip freeze faithfully exported an environment that still contained streamlit and its charting stack, remnants of the very first architecture, abandoned in week one, never uninstalled. Locally, invisible. In a container, every unused package is image size and build time, paid on every single build.

The default torch package assumes you might want a GPU. One Dockerfile line fixed it, installed before the general requirements.txt:

RUN pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cpu

This ensures the CPU-only build gets used instead of whatever platform-ambiguous default pip freeze had captured. Confirmed working in the actual Cloud Build log: the installed package resolved to torch-2.13.0+cpu, the +cpu suffix being the tell that the fix actually took.

The frontend was not the app I thought I was containerizing. I wrote an nginx-based static-file Dockerfile, confidently, for what I assumed was a standard Vite SPA. It was structurally incapable of running the app: the Lovable-generated frontend is a TanStack Start SSR application, a running Node.js server, not a folder of static files, and its build tool defaults to a Cloudflare Workers target, not a dist/ folder at all. Nginx can serve files; it cannot run a Node server. The fix was a build-target switch (BUILD_TARGET=node-server) and a single-stage Node Dockerfile running the actual compiled server. npm run dev had made this distinction completely invisible for the entire development phase, dev servers make SPAs and SSR apps feel identical. Deployment is where the difference becomes load-bearing, and I discovered mine while trying to ship it.

The permissions maze: every automated step needs its own grant

This turned out to be the single biggest source of friction in the whole deployment, and also the most concentrated lesson in the "production adds security as an axis" claim from Post 1.

Locally, every command runs as me, I have blanket access to everything I own. In GCP, the moment anything runs automatically (a Cloud Build step, a Cloud Run service reading a secret), it runs as a service account, and that account starts with nothing. I hit this wall four separate times, each with a different specific fix:

1. Cloud Run couldn't read the Gemini API key from Secret Manager. The default compute service account had no grant on the secret at all. Fix: roles/secretmanager.secretAccessor, scoped to that one secret.
2. The CI/CD trigger's service account couldn't deploy to Cloud Run. Fix: roles/run.admin at the project level, plus roles/iam.serviceAccountUser on the runtime service account (Cloud Build needs permission both to issue the deploy and to act as the identity the deployed service will run as, two separate grants for what feels like one action).
3. The deploy step's container image itself was denied a pull. The google-cloud-sdk/slim image returned artifactregistry.repositories.downloadArtifacts denied, even after the run.admin grant, because pulling the tool needed to run gcloud deploy is a different permission from being allowed to deploy. Partial fix: roles/artifactregistry.reader.
4. That specific image still failed after the grant. Rather than debug further, I swapped gcr.io/google-cloud-sdk/slim for gcr.io/cloud-builders/gcloud, a builder image living in the same registry namespace my docker steps were already successfully pulling from. Sometimes the fix isn't more permissions, it's not fighting a specific resource's access path at all.

None of these are exotic, they're the standard shape of cloud IAM, but not one of them has a local equivalent. Locally there is no "who is allowed to do this," only "can I do this," and the answer is always yes. Every one of the four fixes above is a permission that has to be granted, individually, per action, per resource, a genuinely new category of debugging that only exists once a human stops being the one running the command.

The lazy-loading tax, found the hard way

Cloud Run logs surfaced something I hadn't designed for: Started server process appearing twice, fourteen seconds apart, each followed by a full model reload. Two separate container instances had spun up under light concurrent traffic (my own testing), and each one paid GPT-2's load cost independently, because GPT-2 was loaded as a module-level global at import time, a leftover from before the model registry existed, bypassing the get_model() cache that every other model already used. Every instance start reloaded it, unconditionally.

Fixing that surfaced the deeper issue: even after unifying GPT-2 into the registry and baking all four models' weights directly into the Docker image (eliminating first-request Hugging Face downloads entirely, TinyLlama had been stalling indefinitely trying to fetch ~2.2GB over the network from inside a request), /compare-models was still slow on a fresh instance. The image bake solved the network cost; it didn't solve the CPU cost of deserializing weights into memory, which happens on first from_pretrained() call regardless of whether the source is disk or network. Lazy loading had simply moved the tax from "download" to "deserialize", still paid by whichever user's request happened to be first to touch a given model on a given instance.

The fix was to stop deferring the cost at all:

@app.on_event("startup")
def preload_all_models():
    for key in MODEL_REGISTRY:
        get_model(key)

This trades a slower, no-one-sees-it startup for a guarantee that every user-facing request, from the very first one, is fast, because Cloud Run won't route traffic to an instance until it reports healthy, and the startup hook runs before that. The honest tradeoff, worth stating precisely: eager loading makes cost predictable and front-loaded; lazy loading makes it unpredictable and deferred. Under --max-instances 2, a second instance spinning up under real load now pays the full multi-model startup cost before it can serve anything, a real capacity consideration I hadn't had to think about once, in months of local development, because there was only ever one instance: my laptop.

Deploy mechanics: what actually shipped

• Backend: gcloud run deploy, 8Gi memory, 4 CPU, 600s timeout, concurrency 4, max-instances 2, --allow-unauthenticated, Gemini key via --set-secrets
• Frontend: same pattern, far lighter, 512Mi memory, 1 CPU (an SSR Node server serving pages, no ML inference)
• First backend Cloud Build (before model-baking): 3 minutes 5 seconds, upload, CPU-only torch install, requirements install, image build, push
• Backend Cloud Build after baking all four models into the image: roughly 10 minutes, the direct cost of trading network-at-request-time for weights-at-build-time
• CI/CD: GitHub → Cloud Build trigger, cloudbuild.yaml per repo, $SHORT_SHA image tags so every deploy is traceable to an exact commit, CLOUD_LOGGING_ONLY build option (required once a non-default service account runs the build)

[TO FILL IN: exact image sizes before/after the model-bake change, run gcloud artifacts docker images list on both the pre-bake and post-bake backend images and record the delta. This is the cleanest single number to illustrate "predictable cold start" cost.]

[TO FILL IN: cold-start duration measured directly from the startup logs, time from Started server process to All models preloaded and ready. on a genuinely fresh instance.]

[TO FILL IN: billing dashboard screenshot and total spend against the $300 credit, once there's been a real window of usage to measure against.]

What "real production" would still require

This deployment gets the playground to publicly usable, one small service, modest traffic, bounded cold starts. It is not production-grade, and the distance is worth naming precisely:

A real serving stack (vLLM/TGI-class, continuous batching) instead of per-request generate() calls. Request queuing and per-user rate limiting, nothing currently stops one visitor from exhausting the four-model preload capacity for everyone else. A database for history, which still evaporates on restart. Structured logging and alerting, replacing "I read gcloud run services logs read when something looks wrong." Authentication, the moment any endpoint costs real money per call, right now --allow-unauthenticated is a deliberate choice for a free demo, not a default I'd keep at any real scale.

Closing the series' technical arc

A language model is a next-token predictor, and half of what looks like model behavior is actually decoding (Post 2). Post-training doesn't teach facts, it gives the model something to want, and its failures trace back to that installed wanting (Post 3). Evaluation is bounded twice, by what metrics can measure and by what rubrics think to ask (Post 4). And an application that gets all of that right on a laptop is still separated from a shippable one by concerns, identity, concurrency, cost, that a single-user localhost is specifically built to hide, four of which (secret access, deploy permission, image-pull permission, and instance-lifecycle cost) I had to discover one IAM error at a time (this post).

The playground is live at llm-playground.gayathri.dev, the code and every experiment log referenced across this series is in github.com/lovable-api/llm-playground, and the exact commands used to deploy it are written up as a standalone runbook in the next post, for anyone who wants to reproduce this without hitting the same errors I did.

Caveats: all timings and costs are point-in-time measurements on one project's specific tier configuration, a data point, not a pricing guide. CPU-only serving is a pedagogical choice, not a recommendation for real traffic. Single region, no load testing, one developer's trial-and-error traffic pattern setting the observed instance-scaling behavior.