A few thousand files land on a shared drive overnight. By morning, the ingestion pipeline has choked on partially copied files, duplicate jobs, and a scanner that cannot tell what has actually finished landing from what is still just halfway into the process. The knowledge base is now inconsistent, and nobody is sure which documents made it in cleanly.
That’s what happens when building a Retrieval-Augmented Generation (RAG) system for an environment where the data can never leave the building. No cloud, no external API calls, no exceptions. In a normal cloud-native stack, a managed ingestion service would absorb most of this. But in an air-gapped environment, there is no managed service to reach for. You have to solve the problem at the layer where it actually lives.
This piece walks through two problems that come up when RAG has to run air-gapped: what breaks when you try to bulk-ingest documents without a managed pipeline, and what breaks when the model doing the work isn’t a fixed choice but a moving target. Both problems forced a redesign, and both are the kind of thing you only learn by building it and watching it fail.
Why public cloud was off the table
The system this piece describes, an AI platform called NCP, is built for organisations that cannot use public cloud infrastructure for a specific class of data, no matter how mature or well-audited that infrastructure is. In government departments handling classified material, in law firms holding privileged client data, in banks operating under data residency rules, the constraint is structural rather than a preference.
The requirement is rarely “no cloud, ever.” It’s closer to: some workloads can run anywhere, and some must run entirely within a boundary with no external dependencies, and the system has to support both without treating them as two different ones.
NCP needed to run identically whether it was deployed to the public cloud, in a fully air-gapped environment, or somewhere in between as a hybrid estate. The environment had to be a configuration decision, not a different build.
Two parts of the system made that genuinely hard: how documents get ingested at scale without any external service doing the heavy lifting, and how model calls get routed when the model itself, local or cloud, is subject to change.
The first design: NFS, and why it did not survive contact with real data
The first version of the ingestion pipeline was built the way most people would build it if the priority was getting something working quickly inside a closed network. Files landed on shared NFS storage. A scanning process walked the filesystem, picked up anything new or changed, and handed it to the ingestion logic for chunking and embedding.
It worked cleanly in testing. But it broke under two conditions that only showed up with real usage.
The first was volume. In cloud and hybrid deployments, bulk loads occur when a client onboards an existing document estate from a cloud drive such as Google Drive or OneDrive rather than starting from an empty knowledge base, meaning thousands of files arrive in a short window.
In a fully air-gapped deployment, the same volume problem shows up during the initial migration of a document estate onto local storage, regardless of the transfer method. Either way, the scanning logic had no separation between discovering a file and being sure that file had actually finished landing. A large synchronisation or migration job doesn’t deliver files atomically. At any given moment during a bulk load, some files are complete, and some are still being written.
The second was a direct consequence of the first: partially copied files. The scanner would find a file, recognise that it existed, and hand it to the ingestion process, which would then try to chunk and embed a document still being written to disk.
This produced ingested content built from incomplete files, silently, with no clear signal that anything had gone wrong. Depending on exactly when the scanner caught it, the knowledge base would end up with either a duplicate, a corrupted chunk set, or nothing at all for a given document.
Neither issue was a problem in the strict sense of a bug. The design had no concept of “this file is still arriving.” It assumed that if a file existed on disk, it was ready. This is true of a few manually copied files. It does not hold for a bulk synchronisation job moving thousands of documents into an air-gapped store.
The redesign: Separating landing from processing
The fix was to stop treating file arrival and file readiness as the same event, and to build a system where they’re handled by completely independent components.
Cloud sources and the on-premises corporate NAS now converge on a dedicated file controller, which is solely responsible for syncing content in and writing it to shared storage. It then, separately, scans what has landed, writes file and directory metadata to a relational store, and places an ingestion job on a queue, currently Beanstalkd, the open-source work queue rather than the AWS service of a similar name. A pool of RAG process workers, independent of the
scanner and independent of each other, consumes that queue and performs the actual chunking and embedding.
Ingestion no longer starts because a file exists. It starts because a job was explicitly placed on a queue by a process that has already confirmed the file finished landing.
A worker never touches a file the sync layer hasn’t already accounted for in the metadata store. Workers themselves hold no state, so any worker can pick up any job, be added or removed under load, or restart after a failure without losing anything.
This is illustrative, not the production implementation, but it’s the whole architectural change in two lines. The first version asks the filesystem a question it can’t reliably answer: is this file done? The second version asks a process that actually knows, the one that tracked the file from arrival to completion, and only that process is allowed to say a file is ready to ingest.
Two details in the metadata model matter more than they look on a schema diagram. Every file record carries a content hash, which is what makes ingestion idempotent: if a bulk load is interrupted partway through and has to be retried, or a source is rescanned later,
already-ingested files are recognised by hash and skipped rather than reprocessed or duplicated.
Without that, every retry or rescan risks re-embedding content that’s already in the knowledge base, or worse, creating a second copy of it. And every knowledge base record carries the identifier of the embedding model used to build it.
That sounds like a minor audit field, but it’s what turns a future embedding model upgrade into a targeted, plannable operation, re-embed the knowledge bases still on the old model, run old and new side by side during validation, rather than an unplanned full re-crawl of every source the organisation has ever connected.
The queue is also what makes bulk loading survivable in the first place. A ten-thousand-document upload isn’t a spike the ingestion layer has to absorb all at once. It’s throttled by however many workers are running, and that number scales independently of everything else in the platform.
None of this is exotic distributed-systems design. It’s a fairly standard pattern: separate discovery from processing, make workers stateless, make the trigger explicit rather than inferred. What made it non-optional here was the environment. There was no managed ingestion service to absorb the mess, and in an air-gapped network, getting it wrong isn’t something a support ticket to a cloud provider fixes.
The second constraint: The model underneath won’t sit still
The ingestion problem was about volume and timing. Now, the second problem was about something less predictable: the model doing the actual language work doesn’t stay fixed, and in a system designed to run both locally and against commercial providers, that instability shows up from both directions at once.
The earliest version of NCP called local models directly. This was clean and fast to build. Model configuration lived in a .env file, and switching which local model was in use meant changing that file, which is honestly fine for a single, fixed local deployment.
It stopped being fine the moment the requirement changed from “run a local model” to “run a local model, or a cloud model, and be able to move between them.” An environment file isn’t an abstraction; it’s a setting one layer removed from a hard-coded value, and every place in the application that made a model call still needed to know, directly, which provider it was talking to and how to talk to it.
Adding cloud providers into that picture meant adding provider-specific logic throughout the application rather than in one place.
The instability wasn’t even only architectural. Working through the models needed for a course-generation module that includes audio and image processing for video output, the team moved across ElevenLabs, OpenAI and Google Gemini as the practical best option for a given capability shifted, sometimes because a provider changed pricing, sometimes because a newer model handled a specific task meaningfully better, and at least once because a model the system depended on for a specific request type was deprecated with limited notice.
None of that is unusual in the current AI market. What it exposes is whether the architecture treats the model as an external dependency you can swap, or as something quietly wired into the application logic.
On the local side, the honest answer is mixed. NCP currently runs about a dozen local models on a single on-premises server (16 CPU cores, a 10-core GPU, 128GB of RAM), with Llama 3.3 and Qwen 3.5 the strongest general performers of the set. For some tasks, local models still don’t fully match what OpenAI, Claude, or Gemini can do. For the RAG pipeline itself, ingestion and inference, they hold up well, and in an air-gapped deployment, they’re the only option regardless.
The redesign: One gateway, no direct calls
The fix mirrors the ingestion fix in shape, even though the problem was completely different: stop letting individual parts of the application talk to a model provider directly, and route every call through one component that owns that responsibility.
Every model interaction in the platform, from either the course generation module or the RAG knowledge base, now leaves through a single LLM Gateway. No application module holds a provider credential or a provider SDK.
The gateway presents locally hosted runtimes, currently Ollama and LM Studio, behind the same interface as the commercial providers, so from the rest of the application’s perspective, calling a local model and calling a cloud model look identical. Changing which model handles a given request, including switching between local and cloud entirely, is a configuration change at the gateway. It requires no code change and no redeployment of the modules that use it.
Again, illustrative rather than the exact call signature in production, and the change is not really the syntax. It’s that the module no longer knows or cares whether model resolves to a local runtime or a commercial API.
That decision, along with the credential it requires, now belongs entirely to the gateway. The abstraction only counts if it’s complete, though: it doesn’t hold if something downstream, prompt construction, output parsing, evaluation, still quietly assumes one provider’s response format or behaviour.
Wrapping the call is the easy 20 percent; making sure nothing past the wrapper depends on a specific provider is the part that actually determines whether switching is real.
This did more than solve the switching problem. It gave the platform one place to enforce access policy, monitor spend, and log every call for audit, which matters in regulated deployments in a way that’s easy to underestimate until an auditor asks a question you can’t
answer from a single point. It also meant the provider churn described above stopped being an application-wide problem. Replacing one provider with another for a given task is now a gateway-level decision, not a search-and-replace across the codebase.
What both fixes actually have in common
Neither redesign started from a diagram.
Both started from a system behaving badly under real conditions, and both come down to the same mistake stated two different ways.
That proxy didn’t hold up in either direction. In ingestion, a file existing was treated as though it meant the file was ready, but a file can exist and still be mid-write. In model routing, a model being configurable was treated as though it meant the model was actually abstracted, yet a model name can live in a config file while every call site still assumes one specific provider’s behaviour underneath it.
The two redesigns worked by introducing an explicit boundary where an implicit assumption used to sit, an explicit readiness check ahead of the queue, and an explicit gateway ahead of every model call.
The result is a platform that can deploy to public cloud, to a fully air-gapped setting, or to a hybrid estate, from the same set of components, with the deployment target as a configuration decision rather than a different build.
That wasn’t the starting design goal. It fell out of solving the ingestion and model-routing problems properly, because solving them properly meant removing every assumption that only held in one deployment mode.
The two lessons underneath both fixes are worth naming plainly. Don’t let a proxy signal, a file existing, a request coming in, stand in for the thing you actually need to know, whether it’s ready, whether it’s valid.
A queue or an explicit status check between the two is a small addition that prevents a whole class of silent failure. And treat an external dependency you expect to change, a provider, a model, a format, as infrastructure rather than something wired into application logic.
If swapping it touches more than one place, the abstraction isn’t finished yet, no matter how clean the initial integration looked.
Neither idea is exotic. What made them non-negotiable here was the environment: no managed service to fall back on, no cloud provider to quietly absorb the mess, and a real requirement that the same system run identically whether it can reach the internet or not.
Taking the managed infrastructure away didn’t introduce these problems. It removed the layer
that had been absorbing them without anyone having to think about it. Ingestion timing and provider coupling exist in cloud-native RAG systems too; they’re just usually somebody else’s problem to solve. Air-gapped deployment doesn’t create that complexity; it just leaves it with nowhere to hide.
About the build
This system was built for large-scale, mission-critical environments across government, defence, legal and financial services, with the ingestion and model-routing layers described here running across public cloud, hybrid and fully air-gapped deployments of the same platform.
Featured image credit
