v3.11.0

Token usage is now tracked and displayed across models, evaluations, and task results, giving you full visibility into inference costs. The dataset data table is paginated, filterable, and sortable, making it practical to work with large datasets directly in the UI.

What’s new

  • Token usage tracking: input and output token counts are now recorded per inference call and surfaced on the model overview, evaluation overview, and task result pages, including a breakdown by entity (evaluation, dataset generator, or judge).
  • Dataset generators can now reference secrets directly in their YAML definition and in Python synthesizer snippets.
  • Load a directory of files as a dataset using the new folder source type — useful when building RAG datasets from local documents.
  • RAG checker: new mode field (all_claims, required_only, required_and_optional) controls which claims are used when computing metrics. Built-in claim extraction and labeling synthesizers are now packaged and available without custom code.
  • Multi-turn solver: new include_conversation option in generate_message passes the full conversation history to the model on each turn.

Improvements

  • The dataset data table is now paginated, filterable, and sortable, making it practical to browse large datasets in the UI.
  • JSON values in the dataset sample sidebar can now be expanded and collapsed.
  • Model adapter name on the model overview page is now a direct link to the adapter.
  • Improved error message when a model request hits the retry limit — the response body from the last attempt is now included.
  • Curated model lists now show stable model aliases (e.g. gpt-5.5).
  • Dataset and task result log uploads now accept files up to 1 GB (previously 100 MB).
  • Most backend request handling has been moved to async, improving throughput under concurrent load.

Bug fixes

  • Plain HTTP requests on the HTTPS port are now redirected to HTTPS instead of failing with a TLS handshake error.
  • Fixed lf test dataset crashing with a KeyError after the dataset generator preview API was updated.
  • Fixed API key validation — invalid characters in API keys or custom header values are now caught and reported clearly without leaking the key value in error messages.
  • Fixed custom inference model failing to save due to an incorrect adapter_id type.
  • Fixed test connection for custom ML task models.
  • Fixed broken up/down arrow navigation in the evidence table.
  • Fixed RAG checker precision/recall metric key conflict and switched aggregation from micro to macro.