Skip to main content

Deduplicating model downloads across Dynamo workers with ModelExpress

· 7 min read

When scaling an inference service to multiple workers, every new worker downloads the full model from the model registry. For a 3 GB model this adds 30–40 seconds per worker; for a 70B model it can be 10+ minutes each.

ModelExpress is a model distribution cache layer in NVIDIA Dynamo. It sits between the workers and the model source (MatrixHub or Hugging Face). The first worker triggers a download into the ModelExpress cache. Every subsequent worker gets the model from that cache — no second download.

In this test we deploy two Dynamo vLLM workers for Qwen/Qwen2.5-1.5B-Instruct (~3 GB) and compare the model acquisition time of the first worker (cache miss) versus the second worker (cache hit).

Speeding up SGLang model startup with MatrixHub cache

· 5 min read

When starting an inference service locally or inside a private network, model download is often the slowest and least predictable step.

SGLang, Transformers, vLLM, and many other tools fetch model files through the Hugging Face Hub protocol. If every service pulls directly from public Hugging Face, startup time depends on public network bandwidth, rate limits, and remote availability.

In this test, we use Qwen/Qwen3-0.6B to compare two startup paths:

  • SGLang pulls model files through MatrixHub's Hugging Face-compatible endpoint.
  • SGLang pulls model files directly from Hugging Face.

Dynamo + MatrixHub integration experiment

· 5 min read

We ran two experiments to measure how much an in-network MatrixHub speeds up the first model-weight download for a Dynamo inference service.

  • Experiment 1: Deploy Dynamo on a GPU Kubernetes cluster and pull model weights from an internal MatrixHub. The result is an OpenAI-compatible inference service that can answer chat requests for the qwen3-0.6b model.
  • Experiment 2: Repeat the same setup, but pull the weights from public Hugging Face instead, and compare the first-download time of the two runs.

DeepSeek v4 won't run? 99% of people get stuck at the distribution stage

· 6 min read

Recently, DeepSeek released DeepSeek v4, and many teams rushed to integrate it.

But if you're operating in an enterprise environment, especially air-gapped or private deployments, you'll quickly realize one thing:

The model is not the biggest problem. Distribution is.

During our attempt to deploy DeepSeek v4 in an internal network, we ran into a lot of issues. In the end, they can all be boiled down to three fundamental problems.