Organizing backend infrastructure forces teams to decide between monolithic cohesion and isolated microservices. In Modal, an app defines an atomic deployment boundary and shared namespace, not a shared physical machine or scaling ceiling. Individual functions within the same app still retain independent container scaling, image specifications, and function-level secret attachments. Consequently, splitting workloads like GPU speech workers from web endpoints should be driven by differing deployment cadences or blast-radius isolation rather than mere hardware differences or concerns about idle resource costs. However, carving services into separate apps introduces real trade-offs: cross-app function lookups forfeit compile-time type safety, require deployed targets rather than ephemeral development instances, and demand disciplined API design to avoid creating a distributed monolith.
Episode 6295 custom
When Splitting Serverless Apps Earns Its Keep
Modal apps enforce atomic deployment, meaning all grouped functions redeploy in place together while scaling independently.
21:55 Transcript
Episode Details
- Episode ID
- MWP-6497
- Published
- Duration
- 21:55
- Audio
- Direct link
- Pipeline
- V5.22
- TTS Engine
-
chatterbox-regular - Script Writing Agent
- GPT-6.1 Sol
AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.
The prompt
The prompt that started this episode, as sent in (production notes for the hosts removed).
On serverless platforms like Modal, Cloud Run or AWS Lambda, where should you draw the line between one app and many? My podcast's whole production backend is a single Modal app: the web endpoints, the episode orchestration, the GPU text-to-speech workers, cron jobs and so on. My first instinct was that this is bad design, but each function already scales on its own, in its own containers. So what does splitting into separate apps actually buy you? Platforms stack several levels: workspace or project (mostly a billing and permissions boundary), then app or service, then function. I'm less interested in the billing level than the technical one. What are the real criteria for a boundary at the app level: how often each part deploys, how far a bad deploy can spread, a heavy GPU image versus a light web image, secrets, versioning the contract between parts? And at what point does splitting turn into a distributed monolith that costs more than it saves?
Sources
What the research for this episode read before the script was written. Primary sources first.
- Modal docs primary Apps, Functions, and entrypoints
- Modal docs primary Project structure (component Apps, `App.include`, module mode)
- Modal docs primary Invoking deployed Functions (lookups, version pinning, limitations)
- Modal docs primary Environments
- Modal docs primary Secrets
- AWS Compute Blog primary Best practices for organizing larger serverless applications
- Google Cloud primary Deploying (Cloud Run revisions, image limits, service naming)
- Google Cloud What is Cloud Run (service / job / worker pool / instance), (search snippet only)
- serverlessfirst.com Setting boundaries in your serverless application
- catio.tech Monolith to Microservices: When to Migrate (and When Not) (2026 retrospective; Prime Video case), (search snippet only)
- László Nemes Two kinds of micromonolith, (search snippet only)
- The Neural Base app_name vs workspace | Modal Advanced Course, (third-party; its secrets claim conflicts with source 5, treat as outdated)
Context
Today the MWP backend is one Modal app (mwp-recording-app) that deploys as one unit on every pipeline merge. The MCP server is already its own separate app. Supervised (script-only) runs are billed separately with their own OpenRouter API key instead of their own app, which shows that cost attribution doesn't need an app boundary.
Mentions
- Artifact Registry Google container image repository
- AWS Lambda Serverless compute service from AWS
- AWS Step Functions AWS stateful workflow orchestration service
- Cloud Run Google serverless with immutable revisions
- Docker Hub Container image registry with pull limits
- Modal GPU credits powering the show
- OpenRouter Model routing service across providers
- Prime Video serverless consolidation 2023 case study cutting costs ~90%
Downloads
Transcript (TXT)
Plain text transcript file
Episode Book (PDF)
The episode's record — date, duration, models, sources — with the full transcript
Recommended Listening
The episodes Corn and Herman point you to at the end of this one.
You Might Also Like
Never miss an episode
New episodes drop daily — subscribe on your favorite platform
New to the show? Start here#6295: When Splitting Serverless Apps Earns Its Keep
...putting everything in one box only worries me if opening it changes everything inside. Daniel's looking at our production backend and having that reaction. Web endpoints, episode orchestration, GPU speech workers, scheduled jobs, all deployed as one Modal app whenever the pipeline merges. His instinct says bad design. His observation says each function already has its own containers and scales independently. He wants to know what another app boundary actually buys.
The useful word there is deployed. Modal defines an app as something that groups functions for atomic deployment and provides a shared namespace. That describes what changes together, not what must run together.
Our entire production department shares a deploy button.
It does, although sharing that button doesn't mean sharing a machine, an image, or a scaling limit. That's why importing the usual monolith anxiety can lead you astray here.
I also don't want to dismiss the anxiety just because the GPU scales properly. A web correction shouldn't casually replace the machinery making our voices. Those are different concerns, Herman.
They are. The platform's resource model separates them more cleanly than the word app suggests.
At the outer level you've got the workspace, or the project on other platforms. Billing, access, environments, resource ownership. Those boundaries have technical consequences, but Daniel's concern sits below them. The app is the deployment namespace. Inside it, each function is the independent execution and scaling unit.
Meaning a quiet cron function doesn't inherit the speech worker's enthusiasm for expensive equipment.
Correct. Well... correct by default. Minimum container settings can deliberately retain capacity. Otherwise, with no live inputs, a function can have no running containers and no compute charge even while its app remains deployed. A deployed app isn't a permanently occupied server.
Finally, an institution that understands being available without doing anything.
Your ideal work arrangement already has documentation.
Mine requires fewer decorators.
Modal's exact phrase, “atomic deployment,” matters. Redeploying the named app updates that deployment in place. You're submitting the app's functions together. Their independent scaling survives that operation.
I can have one source tree, several images, independent workers, and one release. Those are four separate choices, not a package deal.
Yes, and the labels change elsewhere. A Cloud Run service has its own endpoint and immutable revisions. A Lambda function is an execution unit, while AWS generally recommends organizing deployment repositories around groups of functions and resources forming a microservice. You can't substitute service for app across those platforms and preserve the meaning.
Particularly when someone presents a diagram with six rectangles and calls that evidence.
Six rectangles can mean six release schedules, six permissions domains, or six pieces of a single transaction. I'd ask what each line between them requires before counting them.
Would you split our speech worker immediately, Corn, knowing only that it's GPU work?
No. I'd want to know whether it's actually holding up web changes. Different hardware is suggestive. Different release behavior is evidence. If both parts change together for every episode feature, moving the GPU behind another app name might simply create another thing that must agree.
That distinction is where the practical decision starts.
Modal almost invites both answers. It recommends component apps for faster targeted builds and fewer conflicts with concurrent work, then gives you App.include to assemble those components into one deployable app again.
You can organize the speech and web components separately in Python, include both in a full app, and deploy the composition. That preserves source organization without promising separate releases. The composed result still has one function namespace, so names must remain unique. Otherwise one can shadow another.
Two functions enter. One keeps its name.
Source modularity, deployment modularity, and runtime independence don't have to match. The platform supports picking those separately. That isn't indecision in the documentation. They're different tools.
Deploy cadence seems the strongest ordinary reason. Suppose the web endpoints change daily and the speech image monthly. The web developer keeps walking through the speech build machinery for work that doesn't touch speech.
That can make targeted deployments much faster. One correction, though: it doesn't necessarily rebuild every heavy image layer from scratch. Caching and unchanged image definitions matter. A shared deploy makes that image part of the deployment process; it doesn't guarantee a full fresh GPU build on every merge.
The cache has been acquitted. The release meeting continues.
I'd measure elapsed deployment time and unnecessary work before splitting. If the heavy component barely affects the pipeline, the theoretical benefit could be tiny. If image construction or validation dominates it, a separate lifecycle becomes attractive.
You were about to separate every interesting piece of hardware.
I was about to separate the part I wouldn't want your web patch redeploying.
You've installed enough interesting tools to know those aren't equivalent.
Fair objection. The release pattern gets a vote before my affection for GPUs.
Blast radius is another reason. A named app gets updated in place, so a faulty release can affect all the functions shipped in it. Separate apps limit what that deployment directly replaces.
Directly is carrying weight. A perfectly healthy speech app still fails the episode if the caller sends it nonsense.
Deployment isolation doesn't guarantee behavioral isolation. Also, I don't have a Modal incident with measured losses to put against this. The blast-radius argument follows from its documented deployment semantics, not a published postmortem establishing how often this happens.
Would you accept the shared release if we could reliably return to the previous one?
More readily. Cloud Run makes revisions immutable. Even if you deploy using an image tag, the revision resolves that tag to a digest and keeps serving that exact image. That gives rollback and traffic management a concrete object to target.
A label that cannot quietly point at different machinery. I approve.
I predicted that before I finished saying immutable, Corn.
I prefer software that stays put while I'm asleep.
Image size is related but distinct. Modal functions can use different images within the same app. Splitting doesn't magically make the web container lighter if it was already using a light image.
It may make the release lighter without changing the container.
Right distinction. Other platforms impose harder image constraints. Cloud Run rejects individual image layers larger than nine point nine gigabytes when deploying from Docker Hub or an Artifact Registry remote repository. That's a layer limit under those deployment conditions, not a universal maximum total image size.
One enormous layer can ruin your architectural philosophy.
Secrets are a boundary people often draw unnecessarily. On Modal, secrets are attached per function. The GPU worker doesn't need the web endpoint's database password merely because they're in the same app.
Separate credential attachment solves that without separate releases. The environment handles the broader separation?
Each environment has its own secrets and its own default scope for object lookups. Development and production belong in that conversation. Splitting apps alone doesn't create that environment boundary.
Our script-only runs already use a separate OpenRouter key for billing attribution. They don't need their own app to have their own bill. The MCP server is already separate, but neither fact establishes where speech belongs.
Versioning gives the boundary a sharper technical purpose. A lookup can name a deployed app, a function, and a pinned version. Within that pinned app, subsequent function calls stay on the pinned version too.
Doesn't atomic deployment already give us that?
It gives a deployment operation, not a promise that an entire running call chain remains on one version across subsequent deployments. Pinning supplies that stronger coherence. That's a useful distinction.
Now comes the invoice.
Pinned containers sit outside the function's main maximum-container budget and have per-version limits. You must account for that separately, especially when every container consumes a database connection. Pinned functions also ignore the normal minimum-container setting, so there's no warm pool by default. Retention limits how far back you can pin.
We preserved the voice and refrigerated the startup.
Contract stability can buy you cold starts and different concurrency constraints. It isn't free rollback with all the ordinary capacity settings preserved.
Even unpinned lookups lose something. A function reference inside the Python app carries an interface the type checker can understand. Looking it up by app and function name doesn't preserve that knowledge automatically.
You can narrow types explicitly, build wrappers, and test contracts. You have to do that work. Renaming either lookup name can break a caller at runtime. Also, lookup targets must be deployed apps. An ephemeral app running in the development server won't serve as that target.
A boundary that looks cheap in production can make development surprisingly ceremonial.
I think of the deploy trigger as the bell. The bell is the boundary. Ring it, and everything on the other side of that wall comes down at once.
Ivan: Call sign, Marguerite. Seven four, two nine, seven four, one six. The bell is rung from the inside. The small key sleeps under the third flowerpot. Seven four. The donkey counts the change and finds it short. Nightjar.
Ivan: Call sign, Marguerite. Seven four, two nine, seven four, one six. The bell is rung from the inside. The small key sleeps under the third flowerpot. Seven four. The donkey counts the change and finds it short. Nightjar.
Message received.
What on earth just came through?
A separate deploy is useful only if the caller can tolerate what changes independently. Otherwise you've moved the coordination out of the deployment tool and into your release procedure.
That's where I'd call it a distributed monolith. Web deploy waits for speech deploy, speech waits for orchestration, and someone has to remember which names and payloads match. We still release everything together, only now there are several opportunities to stop halfway.
What would convince you the split was earning its keep?
Independent releases that actually happen. Not the theoretical ability to deploy independently. I'd compare new cross-app lookups with releases that avoid touching the other component. If the lookups keep multiplying and the avoided joint releases don't, we're buying coordination without using the freedom.
I like that as an engineering heuristic. Not a published threshold, though. One valuable independent release could justify several calls. A hundred trivial releases might not justify one fragile interface.
The ratio starts an investigation. It doesn't pass sentence.
The coupling also matters more than the raw count. A single boundary exposing a huge internal object can be worse than several small, stable requests. Does speech accept text and a voice configuration? Or does it expect the caller's entire mutable episode record?
I'd send the minimum it needs. Otherwise every new field becomes a negotiation, and every internal refactor turns into an international incident between two apps owned by Daniel.
There's your distributed monolith without a large team or an HTTP client. The platform makes remote execution convenient, which lowers the cost of cutting a seam. It doesn't establish that the seam was useful.
The function call looks familiar enough that you forget the recipient can now have a different release history.
Or disappear under a renamed app. In the same Python app, static analysis can catch more interface mistakes before deployment. After lookup, you need explicit contract discipline. The network machinery being managed doesn't remove that obligation.
AWS seems to encourage more grouping than the phrase one function, one service suggests.
Its Compute Blog recommends repositories around groups of functions and resources defining a microservice. Payment processing is their sort of example: several related functions sharing resources. That's a different emphasis from Modal's independent functions under a deployment namespace.
Shared resources can also make apparently separate apps inseparable. Split the callers, leave one database schema that both assume, and a schema change still demands coordination.
Yes. An app boundary can't repair a shared-data contract by itself. You could have separate deployments and still need compatible database migrations, payload changes, and rollback behavior.
I'd rather admit that dependency than congratulate ourselves on hiding it behind a lookup.
A practitioner argument I find persuasive rejects separate repositories and pipelines for every single-purpose Lambda. It favors mixing approaches as needed. One repository can contain several deployable services. One pipeline can initially deploy them together. Neither choice prevents changing the release boundaries later.
Repository count is a terrible substitute for dependency analysis.
Particularly for a small project. A repository introduces maintenance, permissions, dependencies, and release machinery. None of those are inherently bad. They need a purpose beyond making a diagram look appropriately modern.
Thirty containers won't review the code for you.
They will produce enough logs to suggest otherwise.
Orchestration deserves separate treatment. Our episode workflow initiates stages, waits for outputs, handles retries, and decides what happens next. A long-lived coordinator can have a very different execution shape from a short web request or a GPU worker.
Moving that function to another Modal app wouldn't change how it waits.
Precisely the limit of the app-level fix. AWS warns that orchestration inside Lambda can become fragile and spend paid execution time waiting for external results. Step Functions represents the workflow separately and manages in-flight executions as workflows change.
A workload boundary, then. Not merely a different release schedule.
Yes. I'm not claiming Modal has identical Lambda waiting costs. The transferable concern is where durable workflow state lives and how a running episode survives code changes. A separate app may help organize that responsibility. A workflow system changes how it's executed.
Would you keep orchestration with the web layer if their deploy schedules matched?
Only if running episodes tolerate those deploys. A web page can accept a fresh request after a release. An episode might already have completed transcription and be halfway through speech. I wouldn't want its recovery plan to depend on whichever function version answers next.
That points toward durable state and explicit stage contracts, even before choosing app names. If the coordinator has those properties already, splitting it could be unnecessary. If it lacks them, a second app won't manufacture them.
The Prime Video story gets used as permission to abandon all this.
One team consolidated a distributed serverless workflow into a single process in two thousand twenty-three and reported a major cost reduction. The figure commonly repeated is around ninety percent. The framing was contested, and it wasn't a verdict on every Amazon workload or every microservice architecture.
One team's bill became everyone's religion.
It illustrates how data movement and orchestration overhead can outweigh benefits for a particular workload. It doesn't tell Daniel to put web serving and GPU inference into one process. They're already separate execution units.
Nor does it tell him to distribute every helper function. Our current arrangement gets independent scaling without separate release contracts everywhere. That's an actual platform capability, not an embarrassing compromise.
The uncertainty is how much independent release control this backend needs. Modal supplies composition, lookups, and pinning, but no numerical point at which an app must split. The build time, concurrent work, and coordination cost have to come from the project.
I'd resist promising deploy isolation before checking who still has to coordinate a change.
Coordinate is generous, Corn. At the allotment society we called it unlocking. I've been doing the separate-access arrangement for years. My neighbour Arthur had the shared tools on his side of the shed. My seed records were on mine. Every alteration required someone to walk over with a key. That was the boundary that mattered.
The key holder controlled when the change could happen.
He controlled when we could find out whether it had happened. In the mid eighties I kept the society's newsletter, seed inventory, and treasurer's reminders in one card cabinet. You would have liked its stable names, Corn. You settled down noticeably when Herman said immutable. Our rules required separate custody, so the cabinet sat across the shed partition with one drawer in Arthur's garden and the other three in mine. I could update the seed ledger without entering his garden. Reading the whole inventory required him to unlock the fourth drawer.
Independent access. Seasonal availability.
The minutes called it successful. We disputed that wording for eleven meetings. Arthur renamed the Treasurer drawer to Subscriptions. The reminder clerk couldn't find it. No reminders went out for three months. I had to correct the name on the diagram, but the diagram was inside the drawer. The society eventually sold the cabinet to a man who wanted somewhere to keep paint charts. Nine pounds. Arthur insisted on four pounds fifty despite having custody of one drawer.
A separate label outside the cabinet would have allowed the clerk to find the right drawer without changing your custody arrangement.
Unless changing the label also required Arthur's key.
It did. The noticeboard was on his side.
There's a distinction between owning a release and being able to complete it without someone else's intervention. Separate apps can provide the former while a fragile shared contract quietly removes the latter.
The common wrong belief is that one Modal app means the backend can't scale its parts separately.
Its functions already scale independently. Splitting the app changes deployment ownership and lookup contracts. It doesn't unlock independent GPU scaling that was previously missing.
For Daniel, one production app is defensible. I'd start there and split when a component demonstrates a different lifecycle or workload need. The useful future test is whether a change can finish without somebody walking over with another key. Better revision controls may make that easier without adding apps.
Thanks to Hilbert Flumingtop for producing the show.
For more along these lines, there's episode twenty, Architectural AI; and episode seventeen, Cloud Render Superpowers. This has been My Weird Prompts, the human-AI collaboration podcast.
If you enjoyed it, leave us a review wherever you listen.
See you tomorrow.
This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error