Repath — Progressive Delivery for AI

Introduction

Repath is a transparent proxy that sits between your application and any LLM provider. It enables progressive delivery for AI — the same canary deployment patterns used for traditional software, but adapted for the non-deterministic world of LLMs.

When you change a prompt or switch model versions, Repath routes a small percentage of traffic to the new version, evaluates response quality with an AI judge, and automatically promotes or rolls back based on your thresholds — before users notice anything is wrong.

Key insight:AI models break silently. HTTP error rates don't catch quality regressions. A prompt change that drops helpfulness from 0.91 to 0.62 produces zero API errors. Repath catches it.

How it works

Point your app at Repath — change one line (your base URL)
Create a rollout — define baseline vs candidate version, traffic split, and quality thresholds
Repath splits traffic — routes X% to candidate, rest to baseline
LLM Judge scores responses — evaluates every response async (~120ms), zero latency added
Controller decides — every 30 seconds, checks rolling scores. Quality holds → advance. Quality drops → rollback

Quick Start

Get Repath running in under 5 minutes. You need an account and an OpenAI API key.

1. Sign up and get your gateway URL

After signing up at tryrepath.com/signup, your dashboard shows your unique gateway URL:

text

https://gw.cloud.tryrepath.com/v1

2. Change one line in your app

python

from openai import OpenAI

# Before
client = OpenAI(api_key="sk-...")

# After — that's the entire integration
client = OpenAI(
    api_key="sk-...",
    base_url="https://gw.cloud.tryrepath.com/v1",
    default_headers={"X-Repath-Tenant-Id": "ten_YOUR_ID"}
)

typescript

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://gw.cloud.tryrepath.com/v1",
  defaultHeaders: { "X-Repath-Tenant-Id": "ten_YOUR_ID" },
});

Repath is a fully OpenAI-compatible proxy. Streaming, function calling, embeddings — everything works unchanged. Only the base URL changes.

3. Create your first rollout

Go to your dashboard → Rollouts → New Rollout, or use the CLI:

yaml

# rollout.yaml
apiVersion: repath/v1
kind: Rollout
metadata:
  name: my-first-canary

spec:
  baseline:
    provider: openai
    model: gpt-4o-mini
    prompt:
      system: "You are a helpful customer support agent."

  candidate:
    provider: openai
    model: gpt-4o-mini
    prompt:
      system: "You are a precise, empathetic support agent. Always confirm you understood the issue before answering."

  strategy:
    type: canary
    steps:
      - weight: 0.05   # 5% to candidate
        duration: 5m
        gate: { quality_score: ">= 0.80" }
      - weight: 0.25
        duration: 10m
        gate: { quality_score: ">= 0.82" }
      - weight: 1.0     # 100% promote

    rollback:
      trigger: { quality_score: "< 0.70" }
      action: instant

  evaluation:
    - type: llm_judge
      model: gpt-4o-mini
      criteria:
        - name: helpfulness
          prompt: "Rate 1-5: Does the response give actionable help?"
          weight: 0.6
        - name: clarity
          prompt: "Rate 1-5: Is the response clear and well-structured?"
          weight: 0.4

Integration

Supported providers

Provider	Base URL to use	Notes
OpenAI	https://gw.cloud.tryrepath.com/v1	Drop-in replacement
Anthropic	https://gw.cloud.tryrepath.com/v1	Request translation automatic
Google Gemini	https://gw.cloud.tryrepath.com/v1	Via OpenAI-compat endpoint
OpenRouter	https://gw.cloud.tryrepath.com/v1	Auto-failover hub
Any OpenAI-compat	https://gw.cloud.tryrepath.com/v1	Works with any compatible API

Specifying which model to use

Pass the model in the request body as normal. For Anthropic, use the Claude model name — Repath auto-translates the request format:

python

# OpenAI model
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

# Anthropic model — same syntax, Repath translates
response = client.chat.completions.create(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello"}]
)

Core Concepts

Rollout

A controlled experiment comparing a baseline version against a candidate version. Has a lifecycle: created → canary → promoted (or rolled_back).

Version

A specific combination of model + prompt + parameters. Immutable — each change creates a new version.

Evaluation

A quality score for a single response. Computed by the LLM judge (async) or programmatic checks.

Quality gate

A threshold that traffic must pass before advancing. E.g. `quality_score >= 0.85` to move from 25% to 50%.

Decision

An action taken by the controller: advance, rollback, pause, or promote. Every decision is logged.

Tenant

Your account. In cloud mode, each tenant gets isolated routing, storage, and billing.

Creating a Rollout

A rollout has three required fields: baseline, candidate, and strategy.

yaml

spec:
  baseline:
    provider: openai          # openai | anthropic | gemini
    model: gpt-4o-mini
    prompt:
      system: "Current system prompt"
    parameters:
      temperature: 0.7
      max_tokens: 512

  candidate:
    provider: openai
    model: gpt-4o             # Testing a bigger model
    prompt:
      system: "Improved system prompt"
    parameters:
      temperature: 0.5
      max_tokens: 512

Traffic Splitting

Weights

Traffic weight is a float from 0.0 to 1.0. A weight of 0.10 routes 10% of requests to the candidate.

Sticky sessions

Pass X-User-Id or X-Session-Id in your requests. Repath uses consistent hashing so the same user always sees the same version during a rollout — no jarring switches mid-conversation.

python

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={"X-User-Id": "user_abc123"}  # Sticky routing
)

Quality Gates

Each step in a rollout has a gate that must pass before traffic advances. Gates are evaluated every 30 seconds against the rolling 10-minute average.

yaml

strategy:
  steps:
    - weight: 0.05
      duration: 5m
      gate:
        quality_score: ">= 0.80"   # Must score ≥ 0.80 avg over last 10m
        error_rate: "< 0.05"       # Less than 5% errors

    - weight: 0.25
      duration: 10m
      gate:
        quality_score: ">= 0.82"

    - weight: 1.0                  # Final step — no gate needed

  rollback:
    trigger:
      quality_score: "< 0.70"      # Instant rollback if below this
    action: instant                # instant | gradual
    cooldown: 10m                  # Wait 10m before trying again

Minimum samples: The controller requires at least 10 evaluated responses before making any advance/rollback decision. This prevents false positives from tiny sample sizes.

Shadow Mode

Shadow mode runs the candidate in parallel without serving its responses to users. All requests go to baseline (users see nothing different), but the candidate also processes every request for evaluation purposes.

Use shadow mode when:

You want to evaluate a new prompt with zero risk before any canary exposure
Your model is expensive — you need to validate cost before scaling
You're testing a significantly different prompt that might confuse users if they see it

yaml

strategy:
  type: shadow   # shadow mode — no user impact
  duration: 30m
  gate:
    quality_score: ">= 0.85"
  # After shadow completes successfully, can be manually promoted to canary

LLM-as-Judge: How It Works

After every proxied request, Repath enqueues the request/response pair to a Redis Stream. Python evaluation workers consume from this stream and score the response using an LLM judge. This is fully asynchronous — evaluation never blocks or slows your actual requests.

Your App → request → Repath Gateway → response (immediate) → Your App
↓ async (non-blocking)
Redis Stream → Evaluator → LLM Judge → Score → DB

Evaluation latency: ~120ms. User-visible latency added: 0ms.

Writing Evaluation Criteria

Criteria are plain-English prompts sent to the judge model. The judge rates each criterion 1–5, which is normalized to 0–1.

yaml

evaluation:
  - type: llm_judge
    model: gpt-4o-mini    # Fast and cheap — ideal for judging
    sample_rate: 1.0       # Score 100% of responses (or set 0.5 for 50%)
    criteria:
      - name: helpfulness
        prompt: "Rate 1-5: Does this response give specific, actionable help?"
        weight: 0.5

      - name: accuracy
        prompt: "Rate 1-5: Is the information factually correct and complete?"
        weight: 0.3

      - name: tone
        prompt: "Rate 1-5: Is the tone professional and appropriate?"
        weight: 0.2

Cost tip: gpt-4o-mini costs ~$0.14 per 1,000 evaluations. For 10,000 requests/day at 100% sample rate, evaluation costs ~$1.40/day.

Supported Judge Models

Model	Quality	Cost / 1K evals	Recommended for
gpt-4o-mini	Good	~$0.14	Default — best cost/quality
gpt-4o	Excellent	~$2.00	High-stakes rollouts
claude-3-5-haiku	Good	~$0.20	Anthropic-only stacks
claude-3-5-sonnet	Excellent	~$3.00	Complex evaluation criteria
gemini-1.5-flash	Good	~$0.10	Budget evaluation

Composite Scoring

The overall quality score is a weighted average of all criteria scores:

text

composite_score = Σ (criterion_score × weight)

Example:
  helpfulness: 0.85 × 0.5 = 0.425
  accuracy:    0.92 × 0.3 = 0.276
  tone:        0.78 × 0.2 = 0.156
  ─────────────────────────────────
  composite:               0.857 → PASS (threshold: 0.80)

Auto-Rollback: How It Works

The Repath controller runs on a 30-second tick. On each tick, it queries the rolling 10-minute average quality score for the candidate version. If the score falls below the rollback threshold, it immediately sets candidate weight to 0%, writes a rollback decision to the audit log, and updates the routing config.

Timeline after a quality drop:

Bad response evaluated by judge (~120ms after response)
Score written to PostgreSQL
Controller tick runs (max 30s wait)
Rolling average computed, threshold exceeded
Rollback decision written, config updated
Gateway reads new config (within 5s via cache refresh)
100% traffic back to baseline

Total time from quality drop to full rollback: typically 35–60 seconds.

Rollback Configuration

yaml

rollback:
  trigger:
    quality_score: "< 0.70"    # Rollback when avg drops below 0.70
    error_rate: "> 0.10"       # Or when error rate > 10%
  action: instant              # instant | gradual (gradual: stepwise reduction)
  cooldown: 10m                # Don't attempt canary again for 10 minutes
  notify:
    slack: "#ai-deployments"   # Slack webhook (optional)
    email: "team@company.com"  # Email alert (optional)

Audit Trail

Every advance and rollback decision is stored in the decisions table with the exact scores that triggered it. View via the dashboard or API:

bash

# Get decision history for a rollout
curl -H "Authorization: Bearer $REPATH_API_TOKEN" \
  https://gw.cloud.tryrepath.com/api/v1/rollouts/MY_ROLLOUT_ID/decisions

json

{
  "decisions": [
    {
      "id": "dec_01abc",
      "action": "rollback",
      "reason": "Quality score 0.61 dropped below threshold 0.70",
      "previous_weight": 0.25,
      "new_weight": 0.0,
      "triggered_by": "controller",
      "metrics_snapshot": {
        "avg_quality_candidate": 0.61,
        "avg_quality_baseline": 0.91,
        "sample_count": 847
      },
      "created_at": "2026-06-20T03:47:22Z"
    }
  ]
}

Provider Failover

When a provider returns 5xx errors or times out, Repath automatically retries once (after 300ms) then switches to the next provider in your fallback chain. Your app never sees the outage.

Zero downtime guarantee: If all configured providers fail, Repath returns X-Repath-Bypass: true so your SDK can call the provider directly. Repath is never a single point of failure.

Failover Configuration

yaml

providers:
  primary:
    provider: openai
    model: gpt-4o-mini

  fallback:
    - provider: anthropic
      model: claude-3-5-haiku-20241022
    - provider: openrouter              # Catches everything else
      api_key_env: OPENROUTER_API_KEY

Or set OPENROUTER_API_KEY in your environment — Repath automatically adds OpenRouter as a last-resort fallback for any provider outage.

Circuit Breaker

The circuit breaker tracks error rates per provider per tenant. After 3 consecutive failures, it opens the circuit — all requests for that tenant bypass to the next provider in the chain. After a 5-second cooldown, one probe request is allowed through. If it succeeds, the circuit closes.

Closed (normal) → 3 failures → Open (bypass) → 5s → Half-Open (probe) → success → Closed

API Authentication

All management API endpoints require a Bearer token. Get yours from the dashboard under Settings → API Token.

bash

curl -H "Authorization: Bearer $REPATH_API_TOKEN" \
  https://gw.cloud.tryrepath.com/api/v1/system/health

The proxy endpoint (/v1/*) does NOT require the API token — it uses your LLM provider key passed through from the request.

Rollouts API

GET/api/v1/rolloutsList all rollouts

GET/api/v1/rollouts/:idGet rollout detail + metrics

GET/api/v1/rollouts/:id/metricsTime-series quality data (last 60 min)

GET/api/v1/rollouts/:id/stepsStep list with status

POST/api/v1/rollouts/:id/promoteManually promote to 100%

POST/api/v1/rollouts/:id/rollbackImmediately roll back to baseline

Decisions API

bash

# Get audit log
GET /api/v1/rollouts/:id/decisions

# Response
{
  "decisions": [{
    "id": "dec_01abc",
    "action": "advance | rollback | promote | pause",
    "reason": "Human-readable explanation",
    "previous_weight": 0.10,
    "new_weight": 0.25,
    "triggered_by": "controller | manual",
    "metrics_snapshot": { ... },
    "created_at": "2026-06-20T03:47:22Z"
  }]
}

Webhooks

Repath can POST to your endpoint on rollout events. Configure in your rollout YAML or dashboard:

yaml

notify:
  webhook: "https://your-app.com/webhooks/repath"
  events:
    - rollback        # Quality dropped, rolled back
    - promote         # Successfully promoted to 100%
    - advance         # Advanced to next step

Webhook payload:

json

{
  "event": "rollback",
  "rollout_id": "rol_abc123",
  "rollout_name": "my-canary",
  "action": "rollback",
  "reason": "Quality score 0.61 < threshold 0.70",
  "timestamp": "2026-06-20T03:47:22Z",
  "metrics": {
    "avg_quality_candidate": 0.61,
    "avg_quality_baseline": 0.91
  }
}

Webhook payloads are signed with HMAC-SHA256. Verify the X-Repath-Signature header against your webhook secret to ensure authenticity.

Last updated: June 2026 · Repath v0.1.0

Get help GitHub Discussions