Skip to main content

Private Endpoints (My Endpoints)

A private endpoint deploys a model as an inference service dedicated to you, billed by the usage hours of the GPU/CPU resources in the deployment. It is ideal for production applications with latency, throughput, or data-isolation requirements.

Deploy a private endpoint​

  1. In Hub > Models Hub, click the model you want to deploy and choose Private Endpoint to open the deployment panel.
  2. Fill in the settings:
    • Endpoint Name: lowercase letters, digits, and - only; it must start with a letter and end with a letter or digit. This name is also the model name used in API calls.
    • Description (optional)
    • Resource Specification: choose the compute resources to use
  3. Check the estimated cost (Est. Cost) in the bottom-left corner of the panel, then click Save.
  4. The system redirects to the My Endpoints page; deployment is complete once the status changes from Pending to Running.

View endpoints​

The My Endpoints list shows each endpoint's name, status, replicas (ready/desired), uptime, usage duration, and creation time. Click an endpoint to view four tabs:

Details​

  • Private API Information: Base URL, model name, endpoint type, and API path. Click Show Code for a call sample (remember to replace the API key).
  • Specification: model name, type, CPU, GPU, and memory size.
  • Replica status: endpoint health, ready/desired replica counts, and available/unavailable replica counts.
  • Auto Scaling: see the Auto Scaling section below.
  • Resource cost: estimated hourly/monthly cost and accumulated cost over the usage period.

Monitoring​

Real-time monitoring of GPU core utilization, GPU memory utilization, CPU utilization, memory utilization, and network I/O. You can query the past 30 minutes, 1 hour, 1 day, 7 days, or 30 days, or a custom range (up to 30 days).

Logs​

The endpoint's access records and inference output, used for debugging and performance analysis. Logs can be copied or downloaded.

Events​

Records status changes and error events during endpoint startup and external access. Events can be downloaded for retention.

Auto Scaling​

Private endpoints can automatically scale the replica count up and down based on GPU utilization, including Scale-to-Zero, so the service scales up at traffic peaks and releases resources off-peak to save cost.

Enable auto scaling​

  1. In the endpoint's Details, click Enable in the Auto Scaling section to open the settings panel.
  2. Replica Limits: enter the minimum and maximum replica counts. Setting the minimum to 0 enables Scale-to-Zero.
  3. Scaling Configuration: set the GPU utilization threshold, and adjust the scale-down cooldown (default 600 seconds) and polling interval (default 15 seconds) as needed.
  4. Click Save to confirm; the settings then appear in the Auto Scaling section.

You can later click the Edit icon in the top-right corner of the section to adjust the settings, or Delete to disable the feature.

tip

If your production service is sensitive to cold-start latency, set the minimum replica count to 1 rather than 0 — Scale-to-Zero saves money, but the first request has to wait for a replica to start.

Advanced: Scale Up Policy (optional)​

Fine-tune how quickly replicas are added when demand increases:

  • Stabilization Window: wait time before scaling up (default 0 seconds)
  • Max Percent per Step: maximum percentage of Pods added in a single scale-up (default 100%)
  • Percent Policy Period: evaluation period for the percentage-based scaling policy (default 15 seconds)
  • Max Pods per Step: maximum number of Pods added in a single scale-up (default 4)
  • Pods Policy Period: evaluation period for the Pod-count-based scaling policy (default 15 seconds)
note

Scale-up uses a maximizing strategy: the system computes both the percentage-based and Pod-count-based results and applies whichever adds more replicas, ensuring fast response to load.

Advanced: Scale Down Policy (optional)​

Fine-tune how quickly replicas are removed when demand decreases:

  • Stabilization Window: wait time before scaling down (default 60 seconds)
  • Max Percent per Step: maximum percentage of Pods removed in a single scale-down (default 50%)
  • Percent Policy Period: evaluation period for the percentage-based scale-down policy (default 60 seconds)

Related tutorial: Deploy a private LLM endpoint with auto scaling