When deploying large models in highly regulated industries such as finance, government, and energy, teams usually face two challenges:

1. Getting models into the private network: Public model hubs are inaccessible, and manually copying hundreds of gigabytes of weights can take days.
2. Managing model permissions: Models are critical assets, so there must be clear boundaries around who can download, modify, and manage them.

Both matter. A self-hosted, Hugging Face-compatible private model hub solves them together: it centralizes model storage and access control while allowing development teams to use familiar tools and workflows.

Starting a new inference worker involves two different data movements: model files must first reach a worker-local cache, then the weights must enter GPU memory. These stages are often described together as "model loading," but they have different bottlenecks and need different acceleration mechanisms.

MatrixHub accelerates the first stage by serving model repositories from a self-hosted, Hugging Face-compatible endpoint close to the cluster. ModelExpress GPU-to-GPU P2P addresses the second stage: after one source worker has loaded the model, a target worker can receive weights from that ready source over NIXL, UCX, and RDMA instead of downloading the weight files again.

This article records the runtime build, essential DGD configuration, and cross-node GPU-to-GPU P2P validation. It then uses four scenarios—Hugging Face/MatrixHub × direct/P2P—to explain the time spent in each stage.

llm-d is a Kubernetes-native distributed inference stack. It pairs vLLM with an Envoy routing layer and an Endpoint Picker that makes prefix-cache-aware and load-aware scheduling decisions across model server replicas. It solves orchestration and routing for inference at scale — but underneath it sits a more basic problem: how the model weights reach each serving replica in the first place.

Modern LLM weights routinely run to tens or hundreds of gigabytes. Every scale-out, every rescheduled Pod, and every rollout re-transfers a full copy of those weights to the node. When every replica pulls directly from a public model hub, startup time is bounded by public bandwidth, rate limits, and remote availability; in air-gapped or regulated networks, pulling from a public hub may not be possible at all. For production inference, model distribution tends to become a bottleneck earlier than model serving — the slow part of startup is usually not the inference engine but the weight download.

MatrixHub is an open-source, self-hosted AI model registry built for exactly this layer. It exposes a Hugging Face-compatible API: point HF_ENDPOINT at MatrixHub and clients such as vLLM and SGLang keep requesting models under their original repo names, while the files are served from a cache inside the cluster. The first request pulls from upstream and stores the files; subsequent requests for the same model hit the cache directly, without going back to the public network. MatrixHub also provides private model hosting, project-scoped access control and audit, and controlled distribution into offline environments.

Together they form a clean split. llm-d owns the inference control plane — deployment, routing, scheduling, and inference APIs. MatrixHub owns the model distribution layer beneath it — where models come from, how they are cached, and how they are served close to the workloads. Serving orchestration and artifact distribution have separate lifecycles; decoupling them lets platform teams keep a Hugging Face-style model experience without making production inference depend on public-hub availability or bandwidth.

This article quantifies what that distribution layer is worth. The same model was deployed on llm-d twice, with the weight source as the only variable — once directly from a public Hugging Face mirror, once through an in-cluster MatrixHub cache — and the time to reach a ready state was measured in each case.

When scaling an inference service to multiple workers, every new worker downloads the full model from the model registry. For a 3 GB model this adds 30–40 seconds per worker; for a 70B model it can be 10+ minutes each.

ModelExpress is a model distribution cache layer in NVIDIA Dynamo. It sits between the workers and the model source (MatrixHub or Hugging Face). The first worker triggers a download into the ModelExpress cache. Every subsequent worker gets the model from that cache — no second download.

In this test we deploy two Dynamo vLLM workers for Qwen/Qwen2.5-1.5B-Instruct (~3 GB) and compare the model acquisition time of the first worker (cache miss) versus the second worker (cache hit).

When starting an inference service locally or inside a private network, model download is often the slowest and least predictable step.

SGLang, Transformers, vLLM, and many other tools fetch model files through the Hugging Face Hub protocol. If every service pulls directly from public Hugging Face, startup time depends on public network bandwidth, rate limits, and remote availability.

In this test, we use Qwen/Qwen3-0.6B to compare two startup paths:

• SGLang pulls model files through MatrixHub's Hugging Face-compatible endpoint.
• SGLang pulls model files directly from Hugging Face.

We ran two experiments to measure how much an in-network MatrixHub speeds up the first model-weight download for a Dynamo inference service.

• Experiment 1: Deploy Dynamo on a GPU Kubernetes cluster and pull model weights from an internal MatrixHub. The result is an OpenAI-compatible inference service that can answer chat requests for the qwen3-0.6b model.
• Experiment 2: Repeat the same setup, but pull the weights from public Hugging Face instead, and compare the first-download time of the two runs.

Recently, DeepSeek released DeepSeek v4, and many teams rushed to integrate it.

But if you're operating in an enterprise environment, especially air-gapped or private deployments, you'll quickly realize one thing:

The model is not the biggest problem. Distribution is.

During our attempt to deploy DeepSeek v4 in an internal network, we ran into a lot of issues. In the end, they can all be boiled down to three fundamental problems.

Real-world examples of using MatrixHub.