Why Your Docker Build Cache Breaks Down When Your Registry Lives in Another Region

You've optimized your Dockerfile. You've ordered your layers carefully — dependencies before source code, rarely-changed base images at the top, the stuff that changes every commit at the bottom. You've read all the right guides. And your builds are still slow.

If your registry is sitting in us-west-2 and your CI runners are anywhere in the Treasure Valley, I'd start there.

This isn't a Docker problem. It's a physics problem.

---

What Layer Caching Actually Requires

Docker's build cache works by checking whether a layer already exists before rebuilding it. When you're building locally, that check is instant — the daemon looks at its local cache, finds the layer hash, and moves on. No network involved.

The moment you introduce a remote registry into that workflow, every cache check becomes a network round trip. Your CI runner has to reach out to the registry, ask "do you have this layer?", wait for a response, and then either pull the layer or proceed with the build.

That pull isn't optional if the layer isn't cached on the runner itself. And in most CI environments — especially ephemeral runners that spin up fresh for each job — the local cache is empty every single time.

So your "cached" build isn't cached at all. It's pulling gigabytes of layers from a registry in Oregon on every run.

---

The Math on Latency and Why It Compounds

Here's where the numbers get uncomfortable. From a CI runner in Boise to an AWS ECR endpoint in us-west-2, you're looking at 20-40ms round-trip latency under normal conditions. That sounds fine until you understand how Docker actually transfers layers.

Docker uses a chunked transfer protocol. A single layer pull isn't one HTTP request — it's a series of requests, each subject to that round-trip overhead. For a typical Node.js or Python application image, you might have 15-25 layers. Each one requires a manifest check, a blob request, and acknowledgment traffic. At 30ms per round trip, and multiple round trips per layer, you're adding real seconds to every single build.

Now multiply that by your team's commit frequency. If you're running 50 builds a day and each one loses 4 minutes to registry latency, that's over 3 hours of engineering time per day disappearing into a network pipe.

The latency from our Weiser data center to Boise metro is under 5ms. That's not a marketing number — it's what you'd expect from infrastructure that's 85 miles away with a direct path, not traffic routed through hyperscaler backbone networks designed for different priorities than your build pipeline.

---

Why --cache-from Doesn't Solve This the Way You Think

The standard advice for CI layer caching is to use --cache-from with your registry image. Pull the previous build, use it as a cache source, push the new image. Sounds reasonable.

docker pull registry.example.com/myapp:latest
docker build \
  --cache-from registry.example.com/myapp:latest \
  -t registry.example.com/myapp:latest \
  .
docker push registry.example.com/myapp:latest

The problem is that initial docker pull. If your registry is in another region, you're pulling the entire previous image before you can even start checking cache hits. For a 2GB image, that pull alone can take 3-5 minutes on a cross-region connection.

You've traded "slow builds due to cache misses" for "slow builds due to pulling a giant image to check the cache." The math only works in your favor if most of your layers actually hit the cache and you're saving more time than the pull costs. For applications that change frequently, that breakeven point is hard to reach.

BuildKit's inline cache and external cache backends (--cache-to type=registry) are better — they store cache metadata more efficiently and can do smarter partial pulls. But they still can't overcome the fundamental latency of a registry that's geographically far from your build infrastructure.

# syntax=docker/dockerfile:1

# BuildKit with registry cache — still subject to the same latency floor
DOCKER_BUILDKIT=1 docker build \
  --cache-to type=registry,ref=registry.example.com/myapp:cache,mode=max \
  --cache-from type=registry,ref=registry.example.com/myapp:cache \
  -t registry.example.com/myapp:latest \
  .

If your registry is close to your runners — same data center, same metro area — this works extremely well. The cache pulls are fast, the metadata checks are fast, and you're getting the full benefit of BuildKit's smarter caching logic. If your registry is in another region, you're just adding more network traffic to an already slow process.

---

What a Co-Located Registry Actually Changes

We had a customer — a healthcare SaaS company building HIPAA-compliant tooling — move their registry and CI infrastructure to our Idaho data center after spending months trying to tune their way out of slow builds on AWS. They had a moderately complex Python application, about 18 layers, with a base image that changed occasionally and application code that changed constantly.

On AWS ECR in us-west-2, their average build time was 11 minutes. About 4 of those minutes were pure registry I/O — layer pulls and pushes. They'd tried --cache-from, BuildKit caching, layer reordering, multi-stage builds. All of it helped at the margins.

After moving their registry to run on a VM in our data center and pointing their CI runners at the local endpoint, their average build time dropped to 6 minutes. That 4 minutes of registry I/O became under 30 seconds. The actual build work — the RUN npm install, the compilation, the test execution — that stayed the same. You can't optimize away real work. But the network overhead essentially disappeared.

The other thing they got was predictability. Cross-region network performance isn't constant. You'll have fast days and slow days depending on traffic, peering conditions, and factors you have zero visibility into. When your registry is 85 miles away on infrastructure you can actually call someone about, the variance goes away.

---

Setting Up a Local Registry That Actually Performs

If you want to run your own registry close to your build infrastructure, the operational overhead is lower than most people expect. Docker's official registry image is production-capable for most team sizes, and you can run it behind a reverse proxy with TLS termination without much ceremony.

# docker-compose.yml for a basic private registry
version: '3.8'
services:
  registry:
    image: registry:2
    restart: always
    ports:
      - "5000:5000"
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
      REGISTRY_HTTP_SECRET: your-secret-here
    volumes:
      - registry-data:/data

volumes:
  registry-data:

For teams doing serious volume, Harbor is worth the additional setup cost — it adds vulnerability scanning, role-based access control, and replication policies that the vanilla registry doesn't have. But for a team of 5-20 engineers running a few hundred builds a day, the official registry image is fine.

The key decisions are storage (fast local NVMe if you want pull speeds to actually be fast) and network placement (same LAN as your runners, or at minimum the same data center). If your registry is on the same network segment as your CI infrastructure, layer pulls become a local disk read problem, not a network problem.

---

When Regional Replication Makes Sense

Not every team should abandon cloud registries entirely. If you have distributed teams or need to deploy to multiple regions, a hybrid approach works: keep a local registry for builds and CI, replicate images to a cloud registry for production deployments.

Harbor and AWS ECR both support replication policies. You push once to your local registry, and it replicates to ECR on a schedule or trigger. Your builds stay fast because they're hitting local infrastructure. Your production deployments still pull from ECR in the region where your workloads run.

The point isn't that cloud registries are bad. It's that using a distant registry as your primary build cache source is a choice that has a real performance cost, and most teams don't realize they're making it.

---

If your CI build times have been stubbornly high despite all the right Docker optimizations, the registry location is worth looking at seriously before you spend more time tuning Dockerfiles. IDACORE runs virtual servers and bare metal in Weiser, Idaho with sub-5ms latency to Boise — purpose-built for exactly this kind of workload, not retrofitted from a hyperscaler's general-purpose infrastructure. If you want to talk through what a local registry setup would look like for your build pipeline, reach out and let's work through the architecture together.