Evaluations and Evaluation Runs

An evaluation pairs tasks with the entity they are run against. An evaluation run is the execution of an evaluation. Each entry in task_specifications names one task, the model or dataset to run it on and the configuration to run it with, and produces one task result log. Evaluations are normally declared inside a run config together with the entities they reference. By providing the evaluation run config section, we complete the data necessary for a successful evaluation run.

# excerpt of run.yaml
# definition of an evaluation
evaluation:
  key: "hp-trivia-evaluation"
  display_name: "Harry Potter Trivia Evaluation"
  config_spec:
    - type: "model"
      key: "model_key"
      display_name: "Model"
      description: "The model to use for the evaluation."
  task_specifications:
    - key: "hp-trivia-gpt-4-1-nano"
      task_key: "hp-trivia"
      model_key: "<< config.model_key >>"

# configuration of an evaluation to obtain an evaluation run
config:
  model_key: "openai$gpt-4-1-nano"
run_config:
  num_samples: 50
  cache_policy: "reuse"

Task specifications

A task specification is what makes a task concrete:

  • task_key selects the task, and model_key or dataset_key the evaluated entity - which of the two applies follows from the task’s evaluated_entity_type.
  • task_config supplies values for the parameters the task declares in its config specification. The same task can appear several times in one evaluation with different configurations, e.g. to compare prompt variants, or across several models to compare them under identical conditions.
  • task_result_log_path uploads an existing task result log instead of running the task, which reproduces results computed elsewhere (see Download and Upload Evidence).

Sampling, caching and repeated runs

Field Effect
num_samples, subsampling Cap how many samples each dataset contributes, taking either the first N (head, the default) or a random N (random). Set on the evaluation it applies to every specification that does not set its own.
cache_policy Whether earlier task results are reused and whether new ones are cached: reuse, update or no-cache. See the caching guide.
repeatability Runs a specification num_runs times and compares the results, showing how stable the reported numbers are. See the repeatability guide.
trials Repeats every individual sample num_trials times within one run. See the per-sample trials guide.

Working with evaluation runs

Start an evaluation run with lf run, which creates the referenced entities and then executes every task specification. While it runs and afterwards, lf list eval-run shows the evaluation runs of the current app and lf overview eval-run reports the computed metrics. lf export eval-run downloads the evidence, lf cancel eval-run stops a running evaluation run, and lf delete eval-run removes it.

An evaluation definition that can be used to run evaluations with a given config.

Configuration

Properties

key string required

Unique identifier assigned to the entity in AI Platform.

Pattern: ^[a-zA-Z0-9_\-]+$
Max Length: 250

display_name string required

The evaluation’s name displayed to the user.

description string required

Short description of the evaluation.

long_description string

Long description of the evaluation. Supports Markdown formatting.

Default: None

config_spec array[FloatParameterSpec, IntParameterSpec, BooleanParameterSpec, StringParameterSpec, ModelParameterSpec, DatasetParameterSpec, DatasetColumnParameterSpec, ListParameterSpec, DictParameterSpec, CategoricalParameterSpec]

Configuration specification of the evaluation.

Default: []

task_specifications array[SDKTaskSpecificationTemplate, SDKRepeatabilityTaskSpecificationTemplate]

List of task specification templates for the evaluation.

Default: []

tags array[string]

Tags associated with the evaluation.

Default: []

Model Evaluation Definition
# ...
evaluation:
  key: "hp-trivia"
  display_name: "Harry Potter Trivia"
  description: >-
    This evaluation checks that the model can answer trivia questions about
    the Harry Potter universe.
  config_spec:
    - key: "model_key"
      type: "model"
      display_name: "Model"
      description: "Model to be used for the task."
    - key: "judge_model_key"
      type: "model"
      display_name: "Judge Model"
      description: "Judge model to be used for the task."
  task_specifications:
    - key: "hp-trivia"
      task_key: "hp-trivia"
      display_name: "Harry Potter Trivia"
      task_config:
        judge_model_key: "<< config.judge_model_key >>"
      model_key: "<< config.model_key >>"
Dataset Evaluation Definition
# ...
evaluation:
  display_name: "Data Quality"
  key: "data-quality"
  tags: ["Data Quality", "Uniqueness", "Completeness"]
  description: >-
    This evaluation checks the quality of a dataset by evaluating the uniqueness
    and completeness of a specified field across samples in the dataset.
  config_spec:
    - key: "dataset_key"
      type: "dataset"
      display_name: "Dataset"
      description: "Dataset to be used for the task."
  task_specifications:
    - key: "uniqueness"
      task_key: "uniqueness"
      task_config:
        field: "content"
      dataset_key: "<< config.dataset_key >>"
      display_name: "Sample Uniqueness"
    - key: "completeness"
      task_key: "completeness"
      task_config:
        field: "content"
      dataset_key: "<< config.dataset_key >>"
      display_name: "Sample Completeness"
Note

The task_config allows instantiating the same Task template in different ways. Always refer to the Task definition for the available configuration options.