Skip to main content

Multi-Model Routing

info

This guide mirrors guides/multi-model-routing/README.md from the llm-d repository, including the manifests and overlays it links to. Edit it there.

Overview​

This guide deploys the Inference Payload Processor (IPP) to enable serving multiple LLMs behind a single Gateway endpoint. IPP extracts the model name from the request body and sets routing headers. HTTPRoutes then match these headers to direct traffic to the appropriate InferencePool.

Use this guide when you need to:

  • Serve multiple base models (e.g., Qwen for chatbots, DeepSeek for reasoning)
  • Provide a unified API endpoint following the OpenAI specification

For LoRA adapter routing, see Advanced: LoRA Adapter Routing after completing the base setup.

For simpler single-model deployments, see the Optimized Baseline guide instead.

Prerequisites​

  • Have the proper client tools installed on your local system to use this guide.

  • Checkout llm-d repo:

    export branch="main" # branch, tag, or commit hash
    git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout ${branch}

  • Set the following environment variables:

    export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
    source ${REPO_ROOT}/guides/env.sh
    export GUIDE_NAME="multi-model-routing"
    export NAMESPACE="llm-d-multi-model"

  • Install the Gateway API Inference Extension CRDs:

    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GAIE_VERSION}/v1-manifests.yaml

  • Create a target namespace for the installation:

    kubectl create namespace ${NAMESPACE} --dry-run=client -o yaml | kubectl apply -f -

  • Create the llm-d-hf-token secret in your target namespace with a valid HuggingFace token.

  • Multiple InferencePools deployed, each serving a different base model. Follow the Optimized Baseline guide for each pool, or Multi-Inference Pool Setup for adding pools to an existing deployment.

info

When deploying InferencePools for this guide, do not use --set httpRoute.create=true. This guide's HTTPRoutes (Step 3) handle routing based on model name headers. Pool-level catch-all routes would conflict with header-based routing.

Step 1: Deploy IPP​

Clone the IPP repository and deploy using Helm:

# Clone the IPP repository
git clone https://github.com/llm-d/llm-d-inference-payload-processor.git /tmp/ipp

# Install IPP (for Istio)
helm install ipp /tmp/ipp/config/charts/payload-processor \
    --set provider.name=istio \
    --set inferenceGateway.name=llm-d-inference-gateway \
    -n ${NAMESPACE}
note

For GKE, use --set provider.name=gke instead of istio. For standalone (no gateway), omit the provider.name flag.

Verify IPP is running:

kubectl get pods -n ${NAMESPACE} -l app=payload-processor

Step 2: Create Model Mapping ConfigMaps​

Create ConfigMaps that register each base model with IPP. Review and customize manifests/configmaps.yaml for your models, then apply:

kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/multi-model-routing/manifests/configmaps.yaml

Each ConfigMap must have the label inference.llm-d.ai/ipp-managed: "true" and specify a baseModel value matching the model name in API requests.

info

All model names must be globally unique across all InferencePools.

Step 3: Configure HTTPRoutes​

Create HTTPRoutes that match on the X-Gateway-Base-Model-Name header injected by IPP. Review and customize manifests/httproutes.yaml for your setup:

  • Update spec.parentRefs[0].name to match your Gateway name
  • Update backendRefs[0].name to match your InferencePool names
  • Ensure the header value matches the baseModel in the corresponding ConfigMap
kubectl apply -n ${NAMESPACE} -f ${REPO_ROOT}/guides/multi-model-routing/manifests/httproutes.yaml

Step 4: Test the Deployment​

Get the Gateway IP and send test requests:

export GATEWAY_IP=$(kubectl get gateway llm-d-inference-gateway -n ${NAMESPACE} -o jsonpath='{.status.addresses[0].value}')

# Request to Qwen base model
curl -X POST "http://${GATEWAY_IP}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-32B", "messages": [{"role": "user", "content": "Hello"}]}'

# Request to DeepSeek base model
curl -X POST "http://${GATEWAY_IP}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek/DeepSeek-r1", "messages": [{"role": "user", "content": "Solve this problem"}]}'

Troubleshooting​

Check IPP Logs​

kubectl logs -n ${NAMESPACE} -l app=payload-processor --tail=100

Verify ConfigMap Discovery​

IPP should log discovered model mappings at startup. Check that your ConfigMaps have the required label:

kubectl get configmap -l inference.llm-d.ai/ipp-managed=true -n ${NAMESPACE}

Verify Routing​

Check EPP logs to confirm requests reach the correct pool (replace <pool-name> with your InferencePool name):

kubectl logs -n ${NAMESPACE} -l llm-d-router-gateway=<pool-name>-epp --tail=20

Cleanup​

# Remove HTTPRoutes and ConfigMaps
kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/multi-model-routing/manifests/httproutes.yaml
kubectl delete -n ${NAMESPACE} -f ${REPO_ROOT}/guides/multi-model-routing/manifests/configmaps.yaml

# Remove IPP
helm uninstall ipp -n ${NAMESPACE}

# Clean up cloned IPP repo
rm -rf /tmp/ipp

# Remove namespace (if no longer needed)
kubectl delete namespace ${NAMESPACE}

Advanced: LoRA Adapter Routing​

Once base model routing is working, you can extend ConfigMaps to route LoRA adapter requests to their base model's InferencePool.

Add the adapters field to your ConfigMaps listing the LoRA adapter names:

kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: qwen-model-mapping
  labels:
    inference.llm-d.ai/ipp-managed: "true"
data:
  baseModel: "Qwen/Qwen3-32B"
  adapters: |
    - food-review-1
    - travel-assistant
EOF

When a request comes in with "model": "food-review-1", IPP looks up which base model owns that adapter and sets X-Gateway-Base-Model-Name: Qwen/Qwen3-32B. The HTTPRoute then routes the request to the Qwen pool.

Test LoRA routing:

curl -X POST "http://${GATEWAY_IP}/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model": "food-review-1", "messages": [{"role": "user", "content": "Review this restaurant"}]}'
info

All adapter names must be globally unique across all ConfigMaps.

Further Reading​