Skip to main content

​
Overview

Ragas can evaluate TokenLab-backed applications by passing an OpenAI-compatible AsyncOpenAI client into llm_factory.
Type: Evaluation frameworkPrimary Path: OpenAI-compatible Chat CompletionsSupport Confidence: Supported OpenAI-compatible path

​
Environment

export TOKENLAB_API_KEY="sk-your-tokenlab-key"

​
Example Evaluator

import os

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics import DiscreteMetric

client = AsyncOpenAI(
    api_key=os.environ["TOKENLAB_API_KEY"],
    base_url="https://api.tokenlab.sh/v1",
)
llm = llm_factory("claude-sonnet-5", client=client)

metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="Evaluate whether the response is accurate. Answer only accurate or inaccurate.\n\nResponse: {response}",
)
Use llm in Ragas metrics and testsets the same way you would use an OpenAI SDK-backed model.

​
Endpoint Notes

Ragas uses the OpenAI SDK client path here. Native TokenLab Responses, Anthropic Messages, and Gemini routes are best used through evaluation runners that support those request shapes directly.