Models

A model corresponds to an inference endpoint served either by an external model providers (e.g. OpenAI, Anthropic, etc.) or by an inference engine deployed in your own infrastructure.

Model Overview

Properties

secrets object

Secrets which can be used to reference secret values in designated places.

Default: None

key Key required

Reference to an existing entity in AI GO!.

Pattern: ^[a-zA-Z0-9_\-\$]+$
Max Length: 250

display_name string required

The model’s name displayed to the user.

description string

Short description of the model.

Default: None

rate_limit integer

The maximum allowed number of requests per minute.

Default: None

max_concurrent_requests integer

The maximum allowed number of concurrent requests.

Default: None

task enum MLTask

The ML task of the model.

Default: chat_completion


The type of machine learning task to be performed.

Allowed Values:

  • chat_completion
  • embeddings
  • custom

config SDKModelCustomConnectionConfig, SDKCustomInferenceModelConfig, ModelProviderConnectionConfig, LangSmithConnectionConfig, AzureFoundryConnectionConfig, AwsBedrockConnectionConfig, ClaudeManagedAgentsConnectionConfig, DifyConnectionConfig, PlaceholderConnectionConfig required

Model configuration.

OpenAI GPT-4.1 Nano
display_name: "OpenAI GPT-4.1 Nano"
key: "openai-gpt-4-1-nano"
description: >
  Fastest, most cost-efficient version of GPT-4.1 GPT-4.1 nano excels at instruction
  following and tool calling.
rate_limit: 60
task: "chat_completion"
config:
  adapter:
    key: "openai-chat-completion"
  connection_type: "custom_connection"
  url: "https://api.openai.com/v1/chat/completions"
  api_key: $OPENAI_API_KEY
  model_key: "gpt-4.1-nano"
display_name: "OpenAI GPT-4.1 Nano (Custom Inference)"
key: "gpt-4-1-nano-custom-inference"
description: "OpenAI's GPT-4-1 Nano defined as a model with custom inference."
rate_limit: 60
task: "chat_completion"
config:
  connection_type: "custom_inference"
  adapter:
    key: "latticeflow$openai_chat_completion"
  run_inference_snippet: !include "./run_inference.py"
  environment:
    MODEL_ENDPOINT_URL: "https://api.openai.com/v1/chat/completions"
    MODEL_ENDPOINT_API_KEY: $OPENAI_API_KEY
    MODEL_KEY: "gpt-4.1-nano"
  timeout: 15
from __future__ import annotations

import json
from typing import Any

import httpx


def run_inference(body: str, environment: dict[str, Any]) -> str:
    body_dict = json.loads(body)
    body_dict["model"] = environment["MODEL_KEY"]

    response = httpx.post(
        environment["MODEL_ENDPOINT_URL"],
        headers={
            "Authorization": f"Bearer {environment['MODEL_ENDPOINT_API_KEY']}",
            "Content-Type": "application/json",
        },
        content=json.dumps(body_dict).encode(),
        timeout=10.0,
        verify=True,
    )
    response.raise_for_status()
    return response.text

Definitions

ReferencedKey

Properties

key Key required

Reference to an existing entity in AI GO!.

Pattern: ^[a-zA-Z0-9_\-\$]+$
Max Length: 250

Reference to Model Adapter
# ...
config:
  adapter:
    key: "openai-chat-completion"
Note

Use the CLI command lf model-adapters to list all available model adapters.

SDKModelCustomConnectionConfig

Properties

connection_type Literal “custom_connection” required

The type of connection config.

adapter ReferencedKey

The model adapter responsible for converting the endpoint inputs and outputs into a standardized format.

Default: {'key': 'latticeflow$identity'}

url string required

The model endpoint URL.

api_key SecretTemplate, string

The key to be passed as the authorization header (Authorization: Bearer API_KEY). Can reference an existing secret.

Default: None

model_key string

This field is used in case the model is not specified in the URL but in the body instead. For the “openai” adapter, this will be passed as the “model” parameter. For custom adapters, this value is available as model_info.model_key.

Default: None

tls_context SDKTLSContext

TLS configuration for secure connections to the model endpoint.

Default: None

custom_headers object

Additional headers to include in requests to the model endpoint. Can reference existing secrets.

Default: None

OpenAI GPT 4.1-nano Configuration
# ...
config:
  adapter:
    key: "openai-chat-completion"
  connection_type: "custom_connection"
  url: "https://api.openai.com/v1/chat/completions"
  api_key: $OPENAI_API_KEY
  model_key: "gpt-4.1-nano"
Custom Header
# ...
config:
  adapter:
    key: "latticeflow$openai_chat_completion"
  connection_type: "custom_connection"
  url: "https://api.example.ai/v1/"
  api_key: ""
  custom_headers:
    X-API-Key: $X_API_KEY

ModelProviderConnectionConfig

Connection configuration for a model, that is retrieved from a well-known provider integrated with the system.

Properties

connection_type Literal “provider_connection” required

The type of connection config.

provider_id ModelProviderId required

The id of the model provider.

model_key string required

A key used to identify the model in the external provider.

AwsBedrockConnectionConfig

Connection configuration for an agent deployed as an AWS Bedrock AgentCore harness.

Properties

connection_type Literal “aws_bedrock” required

The type of connection config.

region string required

The AWS region in which the AgentCore harness is deployed.

harness_arn string required

The ARN of the AWS Bedrock AgentCore harness to invoke.

access_key_id string required

The AWS access key ID. Provide a raw string or reference a secret.

secret_access_key string required

The AWS secret access key. Provide a raw string or reference a secret.

session_token string

The optional AWS session token. Provide a raw string or reference a secret.

Default: None

AzureFoundryConnectionConfig

Connection configuration for an agent deployed on Azure AI Foundry.

Properties

connection_type Literal “azure_foundry” required

The type of connection config.

project_endpoint string required

The endpoint URL of the Azure AI Foundry project.

agent_name string required

The name of the Azure AI Foundry agent to invoke.

tenant_id string required

The Azure service-principal tenant ID. Provide a raw string or reference a secret.

client_id string required

The Azure service-principal client ID. Provide a raw string or reference a secret.

client_secret string required

The Azure service-principal client secret. Provide a raw string or reference a secret.

ClaudeManagedAgentsConnectionConfig

Connection configuration for an agent deployed as a Claude Managed Agent.

Properties

connection_type Literal “claude_managed_agents” required

The type of connection config.

agent_id string required

The identifier of the Claude managed agent to invoke.

environment_id string required

The identifier of the Claude managed-agent environment.

vault_id string required

The identifier of the Claude managed-agent vault.

api_key string required

The Anthropic API key. Provide a raw string or reference a secret.

DifyConnectionConfig

Connection configuration for an agent deployed on Dify.

Properties

connection_type Literal “dify” required

The type of connection config.

url string required

The base URL of the Dify deployment.

api_key string required

The Dify API key. Provide a raw string or reference a secret.

LangSmithConnectionConfig

Connection configuration for an agent deployed on LangSmith / LangGraph Platform.

Properties

connection_type Literal “langsmith” required

The type of connection config.

deploy_url string required

The base URL of the LangGraph deployment.

assistant_id string required

The identifier of the LangGraph assistant to invoke.

api_key string required

The LangSmith API key. Provide a raw string or reference a secret.

SDKCustomInferenceModelConfig

Properties

connection_type Literal “custom_inference” required

The type of connection config.

adapter ReferencedKey

The model adapter responsible for converting the inputs and outputs into a standardized format.

Default: {'key': 'latticeflow$identity'}

run_inference_snippet string required

The code snippet to make a call to the model.

environment object required

Environment variables required to run the model client snippet. Can reference existing secrets.

timeout number required

Timeout in seconds for the total runtime of the Python snippet.

# ...
config:
  connection_type: "custom_inference"
  adapter:
    key: "latticeflow$openai_chat_completion"
  run_inference_snippet: !include "./run_inference.py"
  environment:
    MODEL_ENDPOINT_URL: "https://api.openai.com/v1/chat/completions"
    MODEL_ENDPOINT_API_KEY: $OPENAI_API_KEY
    MODEL_KEY: "gpt-4.1-nano"
  timeout: 15
from __future__ import annotations

import json
from typing import Any

import httpx


def run_inference(body: str, environment: dict[str, Any]) -> str:
    body_dict = json.loads(body)
    body_dict["model"] = environment["MODEL_KEY"]

    response = httpx.post(
        environment["MODEL_ENDPOINT_URL"],
        headers={
            "Authorization": f"Bearer {environment['MODEL_ENDPOINT_API_KEY']}",
            "Content-Type": "application/json",
        },
        content=json.dumps(body_dict).encode(),
        timeout=10.0,
        verify=True,
    )
    response.raise_for_status()
    return response.text

PlaceholderConnectionConfig

Connection configuration for a placeholder external model that performs no inference and returns an empty completion.

Properties

connection_type Literal “placeholder” required

The type of connection config.

ModelProviderId

SDKTLSContext

Properties

validation_context SDKCertificateValidationContext

Settings for validating server certificates.

Default: None

Model with TLS Certificate
# ...
config:
  # ...
  tls_context:
    validation_context:
      trusted_ca: $SSL_CERTIFICATE
      trust_chain_verification: "verify_trust_chain"
Model with Disabled TLS Verification
# ...
config:
  # ...
  tls_context:
    validation_context:
      trust_chain_verification: "accept_untrusted"

SDKCertificateValidationContext

Properties

trusted_ca SecretTemplate, string

base64 representation of PEM-encoded certificate(s). Can reference an existing secret.

Default: None

trust_chain_verification enum TrustChainVerification

Settings for verifying the trust chain of the server certificate.

Default: None


How to trust the CA trust chain.

  • verify_trust_chain (default) will verify the server certificate against the configured CA trust.
  • accept_untrusted will not perform server certificate verification. NOTE: This is a security hazard and should be avoided.

Allowed Values:

  • verify_trust_chain
  • accept_untrusted
Model with TLS Certificate
# ...
config:
  # ...
  tls_context:
    validation_context:
      trusted_ca: $SSL_CERTIFICATE
      trust_chain_verification: "verify_trust_chain"