Integrate a Model

A model is the inference endpoint under evaluation - a hosted provider API, an agent deployed on a platform, or an endpoint served by your own infrastructure. A task whose evaluated_entity_type is model sends its samples to whichever model an evaluation pairs it with.

What differs between integrations is a single field, config.connection_type: it decides how LF AI Platform reaches the endpoint, and which other config fields apply. There are four paths to choose from.

Pick an Integration Path

Built-in provider
Register OpenAI, Anthropic, Gemini, Fireworks, Together, SambaNova or Novita once - or an agent hosted on LangSmith, Dify, Azure AI Foundry, AWS Bedrock or Claude Managed Agents.
URL and API key
Point at any OpenAI-compatible HTTP endpoint of your own with a custom connection.
Stateful endpoint
Thread a server-side conversation identifier across turns with a model adapter.
Custom inference logic
Chained calls, an SDK client or a protocol of your own, written as a Python inference snippet.

A fifth connection type, placeholder, performs no inference at all. Use it when the outputs you want to score were produced elsewhere and are already in the samples - see Import traces.

The cards do not separate the two middle paths cleanly, because both start from a single HTTP request. Pick by what the endpoint actually needs:

Your endpoint Path connection_type
Is served by a provider LF AI Platform already integrates with. Register the provider once, then add models by provider-qualified key. provider_connection
Is an agent deployed on LangSmith, Dify, Azure AI Foundry, AWS Bedrock or Claude Managed Agents. Use that platform’s connection - it creates the session and round-trips its identifier for you. one per platform
Speaks the OpenAI chat completion format, wherever it runs. Custom connection with the built-in latticeflow$openai_chat_completion adapter. custom_connection
Answers in one POST, but with its own field names or payload shape. Custom connection plus a model adapter you write. custom_connection
Keeps conversation state server-side, and neither latticeflow$openai_responses nor an agent provider covers it. Custom connection plus a stateful adapter. custom_connection
Needs several chained calls, an SDK, a confirmation step, or auth that templates cannot express. Custom inference - a Python snippet you supply. custom_inference

Prefer the highest row that fits. A provider connection is one command, a custom connection is configuration, and a custom inference snippet makes you responsible for HTTP calls, authentication and error handling yourself.

What Every Model Has in Common

Whichever path you take, a model is declared with the same identity fields and registered the same way. Only config changes.

model.yaml
key: "airline-assistant"
display_name: "Airline Assistant"
description: "Customer-facing assistant behind the booking flow."
task: "chat_completion"
config:
  connection_type: "custom_connection"
  # … fields specific to the connection type
secrets:
  ASSISTANT_API_KEY: $ASSISTANT_API_KEY
  • key is how everything else refers to the model - tasks, run configs, lf test model. It must be unique within the AI app.
  • task declares the model I/O shape the endpoint speaks, for example chat_completion, and therefore what solvers and scorers can assume.
  • secrets uploads values to server-side storage so config can reference them as << secrets.NAME >> instead of holding a key in plaintext. See Manage secrets.

Register it, then confirm the LF AI Platform can reach it:

lf add model -f model.yaml
lf test model airline-assistant

For a provider model there is no file to write at all:

lf integration add --provider openai --api-key $OPENAI_API_KEY
lf add model -p openai/gpt-4.1-nano

Every field of every connection type is documented in the Models reference.

After Integrating

  • Test it. lf test model walks one sample through the model’s full request and response pipeline and prints the result of each phase, which is the fastest way to find a broken adapter or a rejected credential.
  • Reference it from an evaluation. A run config pairs a task with a model by key under task_specifications, and a provider model can be declared inline with $provider: "openai/gpt-4.1-nano" instead of a full definition. A provider model is keyed by provider and sanitized model name - lf add model -p openai/gpt-4.1-nano yields the key openai$gpt-4-1-nano - so run lf list model rather than guessing.
  • Track cost. Have the adapter surface the endpoint’s usage numbers so token counts and cost appear in results - see Token usage tracking.