Skip to main content

Deploy a Private LLM Endpoint with Auto Scaling

Goal: deploy a model as a dedicated endpoint and configure auto scaling so the service adds and removes replicas with load, scaling down to zero off-peak to save cost.

Cost: billed by usage hours for the resource specification chosen at deployment. The deployment panel shows the estimated cost; with Scale-to-Zero enabled, the replica count drops to 0 during no-traffic periods.

Prerequisites​

  • You have an account with available credits
  • You have created an API key

Step 1: Deploy the endpoint​

  1. Go to Hub > Models Hub, click the model you want to deploy, and choose Private Endpoint.
  2. Enter an endpoint name (lowercase letters, digits, -; this name is also the model name for API calls).
  3. Choose a resource specification, check the estimated cost in the bottom-left corner, and click Save.
  4. In My Endpoints, wait for the status to change from Pending to Running.

Step 2: Enable auto scaling​

  1. Click the endpoint to open its details, and click Enable in the Auto Scaling section.
  2. Replica Limits: set the minimum replica count to 0 (enabling Scale-to-Zero) and the maximum based on your expected peak traffic.
  3. Scaling Configuration: set the GPU utilization threshold; leave the scale-down cooldown (default 600 seconds) and polling interval (default 15 seconds) at their defaults for now.
  4. Click Save.
tip

If your production service is sensitive to cold-start latency, set the minimum replica count to 1 rather than 0 — Scale-to-Zero saves money, but the first request has to wait for a replica to start.

Step 3: Call the endpoint​

In the Private API Information section of the endpoint details, click Show Code to get the Base URL, model name, and a sample. Replace the API key and you can call the endpoint — the same way as public endpoints (OpenAI-compatible).

Step 4: Verify scaling behavior​

  1. Send sustained requests to the endpoint and watch GPU utilization rise in the Monitoring tab.
  2. In the replica status section of the details, watch the ready/desired replica counts change.
  3. Stop the requests, wait out the scale-down cooldown, and confirm the replica count gradually drops.

Clean up​

When testing is done and you no longer need the endpoint, delete it from the My Endpoints list to stop billing.

Next steps​