DNA Insider
繁中
Back
Coverage· Unknown· Published Sep 10, 2026

Amazon SageMaker HyperPod adds model caching to reduce inference cold starts

Amazon SageMaker HyperPod now supports optional caching for model weights and container images, preloading them onto local NVMe storage to reduce inference pod startup and scale-out delays. Nodes without caching retain network-download fallback behavior.

Coverage

DNA Brief

Signal Summary

AWS says deployments of large models can take 30 minutes or more without caching, with DeepSeek-R1 cited at more than 600 GB. With the weights cache enabled, the HyperPod Inference Operator downloads weights to each target node and waits for all nodes to become cache-ready before creating the deployment. Cached data can be read from local NVMe at about 7 GB/s, allowing pods to typically begin serving traffic in seconds. The image cache pre-pulls inference-server images and can save 5–7 minutes by skipping Amazon ECR pulls.

Why It Matters

The feature changes repeated large-model deployments from waiting on remote downloads to reusing data stored on each node, which can make inference scaling and recovery much faster. The first cache population still requires a remote download, and the number of models and images that can be cached is limited by per-node NVMe capacity.

Evidence

Start with the primary evidence, then review supporting sources and context.

1 items

Primary Evidence

1
  • OfficialAWS Artificial IntelligenceDocumentOriginal

    Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

    Kareem Syed-Mohammed

    “Amazon SageMaker HyperPod now supports model caching for inference, which pre-loads model weights and container images onto cluster nodes so pods read from local NVMe storage instead of downloading over the network. Learn how model caching cuts cold starts from tens of minutes to seconds, how it works, and how to enable it.”
Back to Live FeedExplore more signals