Kimi K3 on AWS: Production Deployment with HyperPod and EKS

Kimi K3 is not a model you casually squeeze onto the spare GPU in a development account. Moonshot AI released it on July 27, 2026 with 2.8 trillion total parameters, 896 experts, a one-million-token context window, and native text-and-image input. Only 16 experts are active for each token, but that still means roughly 104 billion active parameters per forward pass. The deployment problem starts at cluster design, not at the pip install command.

AWS published two supported patterns three days after the release: Amazon SageMaker HyperPod with its Inference Operator, and a self-managed Amazon EKS cluster. Both use a p6-b300.48xlarge host with eight NVIDIA B300 GPUs and a Kimi-specific vLLM image. They differ in who owns capacity, Kubernetes operations, endpoint exposure, and recovery.

This guide turns those launch instructions into a production plan. It covers the model’s unusual topology, the reservation decision, the inference manifest, model-weight custody, admission control, observability, security, rollout, and the licensing clause that can change the business decision. If you first need the general GPU platform mechanics, the AI on EKS operations guide explains device plugins, placement, node isolation, and autoscaling without tying them to one model.

Start with the model constraints

Kimi K3 uses a sparse mixture-of-experts architecture. Sparsity reduces the compute activated for a token; it does not make the full checkpoint disappear. The serving process still needs access to the expert weights, a high-bandwidth path among GPUs, enough memory for runtime state, and storage that can recover a cold pod without turning every restart into an internet-scale download.

Moonshot’s official model card provides the numbers that should appear in the design review:

The initial AWS recipe uses one ml.p6-b300.48xlarge and tensor parallelism across all eight GPUs. Don’t reinterpret that as a universal throughput benchmark. AWS and Moonshot publish the topology, but not one latency or requests-per-second figure that applies to every prompt distribution. A 2,000-token coding request and a 700,000-token multimodal request are different capacity products.

That distinction is easy to lose if your team recently deployed smaller models on the newer G7 family. The SageMaker G7 right-sizing guide is useful for 7B-to-30B models; Kimi K3 belongs in a different class.

HyperPod or EKS: make the ownership decision first

Both paths eventually create Kubernetes pods. The decision is about the control boundary.

Choose HyperPod when the business wants the model and does not want a new GPU platform project. Choose EKS when cluster policy, service mesh, network exposure, custom scheduling, or shared platform integration is a hard requirement. Don’t select EKS only because the YAML looks familiar. Operating eight premium GPUs badly is an expensive way to preserve familiarity.

For organizational context, SageMaker versus Bedrock inference explains when self-hosting is justified at all. Kimi K3’s open weights provide control, but Amazon Bedrock remains the simpler choice when a managed model meets the product requirement.

Reserve capacity before writing the manifest

The AWS deployment uses reserved capacity rather than assuming a B300 node will appear on demand. HyperPod consumes a Flexible Training Plan. EKS targets an EC2 Capacity Block. Capacity Blocks are prepaid reservations for a defined time window; supported accelerated instances run inside EC2 UltraClusters with the network locality expected by distributed ML workloads. Review the current EC2 Capacity Blocks documentation and pricing for the Region rather than pasting an hourly number into a long-lived design document.

Capacity is tied to time and Availability Zone. That creates three preflight questions:

1. Does the reservation’s instance type exactly match p6-b300.48xlarge or ml.p6-b300.48xlarge for the selected path?
2. Do the cluster subnets and node group target the reservation’s Availability Zone?
3. Does the reservation cover warm-up, load testing, the production window, and cleanup?

A reservation that starts at 09:00 is not a reason to schedule the customer launch at 09:01. Download, pod startup, model initialization, health checks, and cache warm-up all consume the window. Establish a readiness gate based on a real completion request, not only Ready=True on the pod.

Deploy with the HyperPod Inference Operator

Create an EKS-orchestrated HyperPod cluster and install the inference add-on. New clusters can include it during creation; existing clusters can follow the Inference Operator setup guide. Verify the controller before deploying a model:

kubectl get deployment hyperpod-inference-operator-controller-manager \
  -n hyperpod-inference-system

kubectl get crd inferenceendpointconfigs.inference.sagemaker.aws.amazon.com

The core manifest follows the launch configuration. Pin the image to a digest in production after validating it; a floating Kimi tag makes rollback ambiguous.

apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: kimi-k3
  namespace: model-serving
spec:
  modelName: Kimi-K3
  instanceType: ml.p6-b300.48xlarge
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    modelSourceType: huggingface
    huggingFaceModel:
      modelId: moonshotai/Kimi-K3
  worker:
    image: vllm/vllm-openai:kimi-k3
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      mountPath: /opt/ml/model
      name: model-weights
    resources:
      requests:
        nvidia.com/gpu: 8
      limits:
        nvidia.com/gpu: 8
    args:
      - --model
      - moonshotai/Kimi-K3
      - --trust-remote-code
      - --load-format
      - fastsafetensors
      - --enable-prefix-caching
      - --enable-auto-tool-choice
      - --tool-call-parser
      - kimi_k3
      - --reasoning-parser
      - kimi_k3
      - --served-model-name
      - Kimi-K3
      - --moe-backend
      - auto
      - --tensor-parallel-size
      - "8"
      - --no-enable-flashinfer-autotune
    environmentVariables:
      - name: VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION
        value: "1"

Apply it and watch the resource, pods, and events together:

kubectl apply -f kimi-k3.yaml
kubectl get inferenceendpointconfig kimi-k3 -n model-serving -w
kubectl get pods -n model-serving -o wide
kubectl get events -n model-serving --sort-by=.lastTimestamp

--trust-remote-code deserves an explicit security exception. It allows repository-provided Python code to run in the serving environment. Mirror and review the exact model revision, pin the commit, restrict egress after the artifacts are staged, and rebuild the serving image from reviewed inputs. A model name is not a supply-chain boundary.

Treat model weights as a release artifact

Pulling from Hugging Face at every restart is convenient for a demo. It couples production recovery to an external service, an unpinned revision, and a very large transfer. A stronger promotion path looks like this:

Moonshot release -> quarantined downloader -> checksum and license review
-> versioned S3 prefix -> immutable deployment manifest -> inference pods

Use an S3 bucket with Block Public Access, versioning, encryption, access logging, and a bucket policy that permits reads only from the inference role. Record the model repository commit, individual object checksums, container digest, vLLM arguments, and license version in one release manifest. That artifact is the rollback unit.

The HyperPod data-capture documentation supports several capture tiers. Don’t turn all of them on by reflex. Prompts can contain source code, credentials copied into tickets, customer data, and images. Capture metadata and sampled redacted payloads unless a reviewed business need justifies more.

Put admission control in front of the GPU

A one-million-token context window is a capability, not a free entitlement for every caller. Without limits, one request can consume the cache budget and distort tail latency for everyone else. Define a service contract before exposure:

HyperPod Inference Operator supports per-pod request limits. The request-limits guide shows maxConcurrentRequests, maxQueueSize, and overflowStatusCode. Start conservatively, then raise concurrency from measurements. Don’t copy a smaller model’s number.

Expose the endpoint privately by default. Use authenticated application identities, not a shared static API key embedded in notebooks. If a public path is unavoidable, place WAF, rate limiting, request-size limits, and an application authorization layer before the model. The model speaks an OpenAI-compatible API; compatibility does not provide tenancy or abuse protection.

Build an observability contract

HyperPod enables inference metrics by default and exposes Prometheus-compatible endpoints. The inference observability guide documents the operator settings. Capture at least:

• request count, error rate, queue time, and time to first token;
• input tokens, output tokens, and context-length distribution;
• active sequences, KV-cache utilization, and prefix-cache hit rate;
• GPU utilization, memory, power, and interconnect health;
• model-load duration, pod restarts, endpoint readiness, and reservation time remaining.

The alert that matters is not simply “GPU at 95%.” High utilization can be healthy. Page when queue delay and tail latency violate the service objective, when cache pressure causes rejects, when a node becomes unhealthy, or when the endpoint has no tested replacement before its reservation ends.

Use a fixed evaluation set in every promotion. It should include ordinary coding tasks, tool calls, structured output, long context, an image request, refusal cases, and an adversarial prompt. Compare answer quality and latency to the current production release. The AgentCore evaluation CI/CD pattern offers a useful release-gate structure even though this endpoint is not hosted on AgentCore.

Benchmark the workload you actually sell

Generic tokens-per-second numbers are not enough for a sparse, multimodal, long-context model. Build a request mix from product traces after redaction:

Warm the model, then run a steady interval and a burst interval. Record rejected requests; a benchmark that queues forever can show attractive throughput with unusable latency. Measure cold start separately, including S3 or Hugging Face weight transfer, container pull, deserialization, graph initialization, and first completion.

Increase concurrency until one service objective breaks: time to first token, p99 completion, queue depth, 429 rate, GPU memory, or correctness. Set the initial limit below that knee. Repeat after every vLLM, driver, model revision, or serving-argument change.

Prefix caching can transform repeated system prompts and shared repository prefixes. Report cache hit rate and run one test with cold or unique prefixes so the capacity plan does not depend on an unrealistically friendly benchmark.

Plan availability around scarce capacity

One eight-GPU node is a failure domain. Kubernetes can restart a pod, but it cannot create replacement B300 capacity outside the reservation. Decide the availability tier honestly.

A smaller fallback may be the best design. Route ordinary requests to the fallback when Kimi K3 is unavailable, and reject tasks that require its context or capability rather than returning silently lower-quality answers. Label the model used in every response and evaluation trace.

Test node loss, pod deletion, model-load failure, reservation expiry, upstream artifact outage, and a bad model release. Record recovery time. If the business objective is 99.9% availability, a single reserved node cannot meet it merely because the control plane is managed.

Secure the data plane and tool boundary

The OpenAI-compatible endpoint should accept requests only from a dedicated application layer. That layer authenticates users, enforces tenant quotas, redacts or blocks prohibited data, sets token ceilings, and records the model release. The inference pod should not receive end-user credentials.

Separate model inference from tool execution. The model returns a structured tool proposal. A tool gateway validates schema, caller, tenant, resource, and action before invoking anything. Never give the vLLM pod broad AWS credentials because the model supports tool calling.

Use network policies or security groups to limit model-pod egress to weight storage, telemetry, and required control-plane endpoints. If weights are mirrored to S3, remove public model-repository egress after initialization. Scan the serving image and pin the base CUDA, driver compatibility, Python packages, and vLLM commit.

For captured data, define:

• fields retained and fields redacted;
• sampling rate and maximum payload size;
• encryption and access role;
• retention and deletion SLA;
• whether images and tool results are stored;
• how customer opt-out is enforced.

Long context increases data concentration. One request can contain an entire private repository. Logging one “sample” can therefore capture far more sensitive material than a conventional API trace.

Run a capacity and license review at each scale step

Scaling from one internal endpoint to a customer-facing service can change both infrastructure and license obligations. Re-run legal review before exposing model capabilities to third parties, before crossing the Model-as-a-Service revenue threshold, and before a large product reaches the attribution thresholds described in the license.

Re-run capacity review when context limits, modalities, concurrency, or traffic mix changes. Eight GPUs remain eight GPUs; software improvements may raise throughput, but a new million-token use case can consume the gain immediately.

License review is part of deployment

“Open weight” is not the same as “unconditional commercial use.” The Kimi K3 license permits broad use, modification, and distribution, but it adds business thresholds. An organization operating a Model-as-a-Service business with aggregate revenue above $20 million over a consecutive 12-month period must enter a separate agreement with Moonshot before commercial use. Large products can also trigger attribution requirements.

Internal use is treated differently by the license. Have counsel classify the exact product before procurement. Store the reviewed license with the model release so a future operator doesn’t assume that a checkpoint mirror carries perpetual unchanged terms.

Roll out without betting the reservation

Use four gates:

1. Artifact gate. Verify weights, code revision, container digest, licenses, and scan results.
2. Infrastructure gate. Confirm capacity, node health, storage throughput, IAM, and private networking.
3. Model gate. Run correctness, safety, tool-call, multimodal, and long-context evaluations.
4. Traffic gate. Send a small internal cohort, then increase only while latency and quality remain inside bounds.

Rollback means restoring the complete release tuple, not only the model name. Keep the prior manifest, weight prefix, image digest, serving arguments, and evaluation report. If the new endpoint is separate, shift traffic at the load balancer. If capacity permits only one node, stop admitting new work, drain the queue, apply the last release, and execute the readiness suite before reopening.

For stateful agent workloads around the model, borrow the isolation principles in the Lambda MicroVM code-agent implementation. The inference server should not become the shell, secrets store, and tool executor merely because the model supports tool calling.

Production checklist

Kimi K3 makes frontier-scale open-weight inference possible on AWS, but it doesn’t make it ordinary. The sensible default is HyperPod, a pinned and mirrored release, bounded requests, and a measured rollout. Pick standalone EKS only when the additional control is worth owning every failure mode it exposes.

Sources

• AWS deployment guide for Kimi K3
• Moonshot Kimi K3 model card
• Kimi K3 license
• Set up HyperPod for model deployment
• HyperPod inference observability
• HyperPod inference request limits
• HyperPod inference data capture
• EC2 Capacity Blocks for ML