AWS publishes SageMaker GPU-partitioning benchmarks, with H100 slices outpacing a whole A100

On November 6, 2026, AWS published a technical guide to NVIDIA Multi-Instance GPU (MIG) partitioning in Amazon SageMaker Inference. MIG divides a physical GPU into hardware-isolated partitions with dedicated compute and memory resources. The feature is for endpoints using inference components; single-model, serverless, and heterogeneous instance-pool endpoints are not supported.

Users configure AcceleratorPartitionLayouts on a production variant and AcceleratorPartitionType on an inference component. AWS says the feature adds no API operations, resource types, or IAM actions, and endpoints that omit both fields continue to behave as before. The partition layout is selected when the endpoint is created, and models requiring more than one GPU still need whole devices. AWS measured roughly 17 seconds of additional provisioning time when carving accelerators.

The post compares one H100 mig-3g.40gb partition with one whole, unpartitioned A100-40GB, running Llama-3.1-8B-Instruct on SageMaker real-time endpoints. At the load where each scenario had its greatest cost benefit, AWS measured 53%–89% more output tokens per second from the H100 partition. Above a workload-dependent crossover, it was 18%–34% cheaper per million output tokens; below that point, the whole A100 was cheaper. These are results from a particular model and test setup, not a guarantee for other workloads.

AWS says Salesforce helped define requirements and validate the initial implementation, and was preparing to validate it with representative production workloads. AWS documentation lists P4d, P5, G7, and G7e instance families, with support in seven Regions spanning the United States, Singapore, Tokyo, Frankfurt, and Ireland. AWS first announced the feature on October 30, 2026; the November 6 post adds implementation details and benchmark data.