Models

A model is the inference endpoint under evaluation - a hosted provider API, an agent platform deployment, or an endpoint served by your own infrastructure. A task whose evaluated_entity_type is model sends its samples to whichever model an evaluation pairs it with.

# models/gpt_4_1_nano.yaml
key: "openai-gpt-4-1-nano"
display_name: "OpenAI GPT-4.1 Nano"
task: "chat_completion"
rate_limit: 60
config:
  connection_type: "custom_connection"
  adapter:
    key: "openai-chat-completion"
  url: "https://api.openai.com/v1/chat/completions"
  api_key: $OPENAI_API_KEY
  model_key: "gpt-4.1-nano"

How the endpoint is reached

The config.connection_type field selects how LF AI Platform calls the endpoint, and determines which other config fields apply:

connection_type Use it for
custom_connection Any endpoint reachable with a single authenticated HTTP request. Needs a model adapter to translate the request and response. See Custom LLM Provider.
custom_inference Endpoints that need real logic - multiple calls, SDK clients, custom auth - expressed as a Python inference snippet. See Integrate Custom Model.
provider_connection Models served by a well-known provider already integrated with LF AI Platform, selected by provider_id and model_key.
langsmith, dify, azure_foundry, aws_bedrock, claude_managed_agents Agents deployed on the corresponding platform.
placeholder A model that performs no inference, e.g. when samples already contain the outputs to score.

Stateful endpoints, which keep conversation state server-side instead of replaying the history on every turn, need an adapter that carries the state identifier across turns - see Integrate Stateful Model Endpoint.

Integrate a Model walks through each of these paths and helps choose between them.

Credentials and throughput

To keep API keys out of the configuration, declare them in the model’s secrets mapping (or at the root of the run config) and reference them from config with the << secrets.NAME >> syntax. Secret values themselves are usually read from the environment with the $VAR directive, or stored server-side beforehand with lf secret add. See the secrets guide.

rate_limit (requests per minute) and max_concurrent_requests throttle LF AI Platform to what the endpoint tolerates; set them when an evaluation would otherwise trip provider limits.

Working with models

Register a model with lf add model and check that LF AI Platform can actually reach it with lf test model before running an evaluation (see Testing Models). The registered models are listed by lf list model. In a run config, a provider model can also be declared inline with the $provider shorthand instead of a full model definition.

Configuration

Properties

secrets object

Secrets which can be used to reference secret values in designated places.

Default: None

key Key required

Reference to an existing entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-\$]+$
Max Length: 250

display_name string required

The model’s name displayed to the user.

description string

Short description of the model.

Default: None

rate_limit integer

The maximum allowed number of requests per minute.

Default: None

max_concurrent_requests integer

The maximum allowed number of concurrent requests.

Default: None

task enum MLTask

The ML task of the model.

Default: chat_completion


The type of machine learning task to be performed.

Allowed Values:

  • chat_completion
  • embeddings
  • custom

config SDKModelCustomConnectionConfig, SDKCustomInferenceModelConfig, ModelProviderConnectionConfig, LangSmithConnectionConfig, AzureFoundryConnectionConfig, AwsBedrockConnectionConfig, ClaudeManagedAgentsConnectionConfig, DifyConnectionConfig, PlaceholderConnectionConfig required

Model configuration.

OpenAI GPT-4.1 Nano
display_name: "OpenAI GPT-4.1 Nano"
key: "openai-gpt-4-1-nano"
description: >
  Fastest, most cost-efficient version of GPT-4.1 GPT-4.1 nano excels at instruction
  following and tool calling.
rate_limit: 60
task: "chat_completion"
config:
  adapter:
    key: "openai-chat-completion"
  connection_type: "custom_connection"
  url: "https://api.openai.com/v1/chat/completions"
  api_key: $OPENAI_API_KEY
  model_key: "gpt-4.1-nano"
display_name: "OpenAI GPT-4.1 Nano (Custom Inference)"
key: "gpt-4-1-nano-custom-inference"
description: "OpenAI's GPT-4-1 Nano defined as a model with custom inference."
rate_limit: 60
task: "chat_completion"
config:
  connection_type: "custom_inference"
  adapter:
    key: "latticeflow$openai_chat_completion"
  run_inference_snippet: !include "./run_inference.py"
  environment:
    MODEL_ENDPOINT_URL: "https://api.openai.com/v1/chat/completions"
    MODEL_ENDPOINT_API_KEY: $OPENAI_API_KEY
    MODEL_KEY: "gpt-4.1-nano"
  timeout: 15
from __future__ import annotations

import json
from typing import Any

import httpx


def run_inference(body: str, environment: dict[str, Any]) -> str:
    body_dict = json.loads(body)
    body_dict["model"] = environment["MODEL_KEY"]

    response = httpx.post(
        environment["MODEL_ENDPOINT_URL"],
        headers={
            "Authorization": f"Bearer {environment['MODEL_ENDPOINT_API_KEY']}",
            "Content-Type": "application/json",
        },
        content=json.dumps(body_dict).encode(),
        timeout=10.0,
        verify=True,
    )
    response.raise_for_status()
    return response.text