Providers Guide

HyperInfer's provider system is modular and extensible. Every LLM backend (OpenAI, Anthropic, Azure, local models) is implemented as a pluggable provider that conforms to a common interface. This guide covers the built-in providers, how to configure them, and how to add your own.

1. What is a Provider?

A provider is a Rust trait implementation that translates HyperInfer's generic ChatRequest into a vendor-specific API call and back. The router doesn't care which provider handles a request — it just calls the standard interface.

1.1. The Provider Abstraction

┌──────────────┐       ┌──────────────┐       ┌──────────────┐
│  ChatRequest │──────▶│  LlmProvider │──────▶│  Vendor API  │
└──────────────┘       └──────────────┘       └──────────────┘
                              │
                              ▼
                       ┌──────────────┐
                       │ ChatResponse │
                       └──────────────┘

1.2. The LlmProvider Trait (Summary)

The full trait is documented in the Custom LLM Providers guide. At a glance:

2. Built-in Providers

HyperInfer ships with first-class support for the most popular LLM providers.

2.1. Supported Providers

The following providers are currently implemented in the core:

2.2. Configuring a Provider

Providers are configured at the deployment level via the control plane. Each deployment specifies which provider to use, the model name, and the API endpoint.

# Create an OpenAI deployment
curl -X POST http://localhost:8080/v1/deployments \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gpt-4o-mini-prod",
    "provider": "openai",
    "model": "gpt-4o-mini",
    "base_url": "https://api.openai.com/v1",
    "api_key_ref": "openai_prod",
    "weight": 1
  }'

# Create an Anthropic deployment
curl -X POST http://localhost:8080/v1/deployments \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "claude-sonnet-prod",
    "provider": "anthropic",
    "model": "claude-3-5-sonnet-20241022",
    "base_url": "https://api.anthropic.com/v1",
    "api_key_ref": "anthropic_prod",
    "weight": 1
  }'

2.3. Deployment Fields

The CreateDeploymentRequest struct accepts the following fields:

3. Provider Registry

The ProviderRegistry is a thread-safe, in-memory store of all available providers. The router consults it to find the correct provider for each deployment.

3.1. Built-in Registration

The built-in providers are registered automatically when the data plane starts. You don't need to do anything special.

3.2. Custom Provider Registration (Rust)

If you've implemented a custom provider, you need to register it with the data plane at startup.

use hyperinfer_providers::ProviderRegistry;

fn main() {
    let registry = ProviderRegistry::new();

    // Register your custom provider (synchronous)
    registry.register(MyCustomProvider::new());

    // ... start the data plane with this registry ...
}

4. Health Checks

Each provider implements a health_check method that validates the API key and connectivity to the upstream service. This is used by:

• Liveness probes: The data plane periodically checks provider health.
• Routing decisions: Unhealthy providers are excluded from routing pools.

How It Works

The LlmProvider trait has a default health_check implementation that sends a minimal probe request ("ping" with max_tokens=1) to the upstream API. If the call succeeds, the provider is considered healthy.

// From the LlmProvider trait:
async fn health_check(&self, api_key: &str) -> Result<(), HyperInferError> {
    let request = ChatRequest {
        model: "health-check-probe".to_string(),
        messages: vec![/* ping message */],
        max_tokens: Some(1),
        // ...
    };
    self.chat(&request, api_key).await?;
    Ok(())
}

5. Custom Providers

7. Comparison Table: Provider Concepts