Python Snippets
A core capability of LF AI Platform is the ability to tailor evaluations and integrations to your specific use case and infrastructure - for advanced scenarios, custom logic can be expressed as Python snippets.
These snippets are executed server-side inside a fixed Python runtime (Python 3.11).
Where Snippets Are Used
Every extension point takes the snippet as a string, so keep the code in a real .py file and pull it in with !include "./my_snippet.py" - your editor, type checker and tests then work on it normally.
| Extension point | Field | Function to define | Guide |
|---|---|---|---|
| Synthesizer | synthesize_snippet |
synthesize(source) |
Dataset generation |
| Scorer | compute_scores_snippet |
compute_scores(sample, solver_output) |
Benchmark tasks |
| Metric | compute_metrics_snippet |
compute_metrics(scores) |
Benchmark tasks |
| Solver | run_solver_snippet |
run_solver(sample, model, trace) |
Benchmark tasks |
| Solver postprocessor | postprocess_snippet |
postprocess(sample, solver_output) |
Solver postprocessor |
| System task | compute_evidence_snippet |
compute_evidence() |
System tasks |
| Custom model inference | run_inference_snippet |
run_inference(body, environment) |
Integrate a custom model |
Both def and async def are accepted everywhere. The exact argument and return types of each function are documented on the corresponding reference page - see Synthesizers, Scorers, Metrics and Solvers.
The Runtime
Only the libraries pinned in the requirements.txt below are available to import; any other dependency is not resolvable at execution time. You can install the same requirements.txt locally (e.g. uv pip install -r requirements.txt) to reproduce the server environment and debug your snippets.
requirements.txt
anthropic==0.104.0
boto3==1.43.19
botocore==1.43.19
httpcore==1.0.9
httptools==0.7.1
httpx==0.28.1
Jinja2==3.1.6
jiter==0.14.0
latticeflow-assessment
latticeflow-core
openai==2.8.1
orjson==3.11.6
pandas==2.2.3
pikepdf==9.11.0
pydantic==2.13.4
pydantic_core==2.46.4
pymupdf==1.27.2
PyYAML==6.0.3
requests==2.33.1
requests-toolbelt==1.0.0
scikit-learn==1.8.0
scipy==1.12.0
numpy>=1.19.0,<2.0
tenacity==9.1.2
together==2.9.0
urllib3==2.7.0
websockets==16.0
xxhash==3.6.0