Client

PrompticClient and AsyncPrompticClient cover AI Applications, components, prompt experiments, traces, and datasets. Agent authoring, external execution, submissions, and benchmark results use the dedicated AgentGymClient and AsyncAgentGymClient described below. Async methods follow the same contracts and are awaited.

Constructor

from promptic_sdk import PrompticClient, AsyncPrompticClient

# Sync
client = PrompticClient(
    api_key=None,           # str | None
    access_token=None,      # str | None — session token from device login
    ai_application_id=None,      # str | None — required with access_token
    endpoint=None,          # str | None — default: https://promptic.eu
    timeout=30.0,           # float — request timeout in seconds
)

# Async
client = AsyncPrompticClient(...)  # Same parameters

Both support the context manager protocol:

with PrompticClient() as client:
    ai_application = client.get_ai_application()
    print(ai_application["id"])

async with AsyncPrompticClient() as client:
    ai_application = await client.get_ai_application()
    print(ai_application["id"])

AI Application

client.get_ai_application() -> AIApplication

Returns the current AI application details.

Components

client.list_components() -> ComponentList
client.create_component(name, *, description=None) -> ComponentCreated
client.get_component(component_id) -> Component
client.delete_component(component_id) -> None

Experiments

client.list_experiments(
    *, component_id=None, status=None, limit=50, offset=0
) -> ExperimentList

client.create_experiment(
    ai_component_id,
    target_model,
    *,
    task_type="classification",     # "classification" | "textGeneration" | "structuredOutput"
    initial_prompt=None,
    name=None,
    description=None,
    provider="openai",              # "openai" | "openrouter" | "custom" | "google"
    optimizer="prompticV2",         # "prompticV2" | "miproV2" | "bootstrapFewShot" | "gepa"
    hyperparameters=None,           # dict with epochs, trainSplitRatio, numFewShots, enableCot
    initial_prediction_model_schema=None,
) -> Experiment

client.create_tool_selection_experiment(
    ai_component_id,
    *,
    tools,                           # [{"name", "description", "input_schema"?}]
    test_cases,                      # [{"query", "expected_tool"}]
    target_model=None,
    tool_source="manual",           # "manual" | "mcp"
    system_prompt=None,
    optimize_system_prompt=False,
    epochs=None,
    train_split_ratio=None,
    name=None,
    description=None,
) -> Experiment

client.get_experiment(experiment_id) -> Experiment
client.update_experiment(experiment_id, **updates) -> Experiment
client.delete_experiment(experiment_id) -> None
client.start_experiment(experiment_id) -> ExperimentStarted
client.cancel_experiment(experiment_id) -> ExperimentCancellation

client.duplicate_experiment(
    experiment_id,
    *,
    continue_from_optimized=False,  # seed initial prompt from the source's best iteration
    initial_prompt_override=None,   # explicit initial prompt; ignored if continue_from_optimized=True
) -> Experiment

create_tool_selection_experiment atomically creates the tool-selection experiment, managed dataset, tools, canonical cases, system-prompt settings, and required evaluator. Use an empty expected-tool value or a supported no-tool alias for cases that should not call a tool.

duplicate_experiment clones a source experiment under the same AI component, copying its dataset cases and evaluators. By default the new experiment starts from the source's initial prompt. Pass continue_from_optimized=True to seed it from the source's best optimized iteration instead — useful for chaining optimization runs after promising results.

The response includes a modelUnavailable flag, set when the source's target model is no longer available in the AI application; update the model before calling start_experiment.

Cancellation

Requires an SDK and server release with experiment-cancellation support. cancel_experiment is available on both sync and async clients. It returns {"status": "canceled"} for an immediately canceled ownerless pending/scheduled run, or {"status": "canceling"} while a running worker finishes in-flight work. The method does not wait for completion. Poll get_experiment with a bounded timeout until experimentStatus is canceled, completed, or failed, and report that actual outcome. Repeated cancellation requests are idempotent.

In-flight work may finish and incur usage charges; saved results remain. Cancel and wait for a terminal state before deleting an active experiment. A local timeout or Ctrl-C does not cancel remote work. Completed/failed runs return a 409 API error; authentication and not-found errors remain 401/404.

Evaluators

client.list_evaluators(experiment_id) -> EvaluatorList

client.create_evaluators(
    experiment_id,
    evaluators,         # list[dict] — each with "name", "type", "weight", optional "config"
) -> EvaluatorList

client.update_evaluator(experiment_id, evaluator_id, **data) -> Evaluator
client.delete_evaluator(experiment_id, evaluator_id) -> None

Iterations

client.list_iterations(experiment_id) -> IterationList
client.get_iteration(experiment_id, iteration_id) -> IterationWithScores
client.get_best_iteration(experiment_id) -> IterationWithScores

Iterations return both overallNormalizedScore (train split) and evalNormalizedScore (held-out eval split, null when no trainSplitRatio is configured on the experiment). get_best_iteration ranks by the eval score when available, otherwise by the overall score.

Iterations also expose avgPredictionLatencyMs — the mean wall-clock duration (milliseconds) of the target-model prediction calls in that iteration, across train and eval predictions. It excludes retries, rate-limit backoff, and failed attempts. The field is absent (or null) on runs completed before per-prediction latency was tracked, so use .get("avgPredictionLatencyMs") rather than direct subscript when handling legacy iterations.

Deployments

client.get_deployment(component_id) -> Deployment | None
client.deploy(component_id, experiment_id) -> DeploymentCreated
client.undeploy(component_id) -> None
client.get_deployed_prompt(component_id) -> DeployedPrompt | None

Traces

client.list_traces(
    *, limit=50, offset=0, status=None, start_after=None, start_before=None
) -> TraceList

client.get_trace(trace_id) -> Trace
client.list_trace_artifacts(trace_id) -> TraceArtifactList
client.get_artifact(artifact_id) -> TraceArtifact
client.get_artifact_content(artifact_id) -> bytes
client.download_artifact(artifact_id, path) -> None
client.get_stats(*, days_back=30) -> TracingStats

Service and environment correspond to the OpenTelemetry service.name and deployment.environment.name resource attributes. The current Python SDK methods do not expose these filters; use the service and environment query parameters through the REST API when filtering programmatically.

Trace responses contain lightweight artifact references by default. Use the artifact methods when you need the original bytes for images, PDFs, markdown files, or other large payloads. Artifact metadata includes an optional name that is used as the default download filename.

Trace detail pages use the database trace id in dashboard URLs. The OTLP traceId is still the identifier accepted by the REST API methods above.

Datasets

client.create_dataset(
    component_id, name, *, description=None, trace_ids=None
) -> Dataset

client.list_datasets(component_id) -> DatasetList
client.get_dataset(component_id, dataset_id) -> DatasetWithCases
client.delete_dataset(component_id, dataset_id) -> None

client.list_dataset_cases(component_id, dataset_id) -> DatasetCaseList
client.get_dataset_case(component_id, dataset_id, case_id) -> DatasetCase

client.create_dataset_cases(
    component_id,
    dataset_id,
    cases,  # DatasetCaseInput or list[DatasetCaseInput]
) -> DatasetCaseList

client.update_dataset_case(
    component_id,
    dataset_id,
    case_id,
    updates,  # DatasetCaseInput
) -> DatasetCase

client.delete_dataset_case(component_id, dataset_id, case_id) -> None

Experiments include their dedicated datasetId. Use it together with the experiment's aiComponentId when adding training cases. Case payloads are canonical JSON objects; idx, split, and metadata are optional.

Agent Optimization

Use the dedicated Agent Optimization client when the unit being tested is a complete Agent rather than a prompt alone. It uses the same API key, interactive login, endpoint, and AI Application selection as the core client.

Constructor

from promptic_sdk import AgentGymClient, AsyncAgentGymClient

gym = AgentGymClient(
    api_key=None,
    access_token=None,
    ai_application_id=None,
    endpoint=None,
    timeout=30.0,
)

async_gym = AsyncAgentGymClient(...)  # Same parameters

Both clients support their corresponding context-manager protocol. Environment variables and a saved CLI login are used when explicit credentials are omitted.

Author Agents and benchmark versions

agent = gym.benchmarks.create(
    name="Invoice classifier",
    goal="Classify the supplied invoice and return the requested fields.",
    description=None,
    input_schema=None,
    output_schema=None,
    evaluators=(),
)

gym.benchmarks.list() -> list[AgentGymBenchmark]
gym.benchmarks.get(benchmark_id) -> AgentGymBenchmark

agent.configure(
    name=None,
    goal=None,
    description=None,
    input_schema=None,
    output_schema=None,
    evaluators=None,
) -> AgentGymBenchmark

agent.cases.list() -> list[CreatedBenchmarkCase]
agent.cases.add(input=..., output=None, expected_behavior=None, title=None)
agent.cases.update(case_id, input=..., output=None, expected_behavior=None, title=None)
agent.cases.add_many(cases)

agent.revisions()
agent.revision_draft()
agent.resolve_revision_draft(field, kind=..., value=...)
agent.abandon_revision_draft()
agent.publish()
agent.delete()

An Agent owns a versioned benchmark: its goal, input/output contract, representative cases, and evaluation plan. Once an active version exists, configuration and case changes are staged in a draft so existing runs remain reproducible. Publishing activates an immutable version.

Evaluator configuration uses typed models such as ClassificationF1, FieldLevelJudge, VerifierAgent, and ExpectedBehaviorJudge. Case files use BenchmarkFile; the SDK uploads and binds them to schema file fields.

If you explicitly set a benchmark judge model, use a PrompticClient configured for the same AI Application as gym to call client.models.list(). Choose an id whose judgeEligible value is true. Other available models cannot be used for benchmark judging. Omit the evaluator's model to use its default. An invalid explicit choice returns HTTP 400 with code invalid_judge_model and the current valid model IDs.

Execute and submit an Agent

For a trusted local callback, use the high-level operation:

result = gym.run_and_submit(
    benchmark_id,
    executor,
    name="invoice-agent",
    version="1.0.0",
    architecture_description="Extract, validate, and return the requested invoice fields.",
)
print(result.run_id)

For isolated or custom runtimes, use a resumable submission session:

submission = gym.start_submission(
    benchmark_id,
    idempotency_key="invoice-agent-1.0.0",
    variant_identity={
        "name": "invoice-agent",
        "version": "1.0.0",
        "architecture_description": "Extract and validate invoice fields.",
    },
)

manifest = submission.get_manifest()
materialized = submission.materialize_manifest("./agent-inputs")
submission.add_prediction(case_id, result)
finalized = submission.finalize(
    idempotency_key="invoice-agent-1.0.0-submit",
)

submission.status()
submission.wait(max_wait=600, poll_interval=2)
submission.cancel()          # Only before finalization
submission.retry_scoring()   # Recover original scoring with its pinned evaluator revision

Use resume_submission(benchmark_id, submission_id) to reconnect to an existing session. The session API also uploads generated artifacts and resolves submitted trace IDs. Each completed prediction is persisted immediately; finalization verifies complete frozen-case coverage and then requests scoring.

Inspect results

Verifier agents define one to eight stable metrics with VerifierMetric. Set each metric's aggregation weight and optional threshold directly on that object; the SDK builds the API's binding map internally. Use EvidencePolicy with EvidenceKind to select the evidence the verifier may inspect. Verifier evaluators do not have an evaluator-level weight. ExpectedBehaviorJudge uses the fixed behavior_compliance metric and accepts an optional model and binding for that metric.

gym.get_run_results(benchmark_id, run_id)
gym.list_case_results(benchmark_id, run_id, sort="score", limit=100, cursor=None)
gym.iter_case_results(benchmark_id, run_id, page_size=100, sort="score")
gym.get_case_result(benchmark_id, run_id, case_id)

gym.compare_runs(
    benchmark_id,
    parent_run_id=baseline_run_id,
    candidate_run_id=candidate_run_id,
)

gym.download_prediction_artifact(artifact, destination, overwrite=False)

Re-evaluate or retry scoring

Both operations reuse saved predictions without executing the Agent again. Choose based on whether you want to apply the active evaluator revision or recover the original scoring job:

Re-evaluate creates a new evaluation with the active evaluator revision. Publish evaluator changes before requesting it. The saved predictions must still be compatible with the active revision's execution inputs; if those inputs changed, execute the Agent again instead.

Retry scoring recovers blocked or interrupted original scoring for an external run, for example after a queue-delivery failure or after correcting billing. It does not adopt evaluator changes and is unavailable once official scoring has succeeded. submission.retry_scoring() provides the same recovery action through a submission session.

For a failed run, get_run_results() includes failure.code and failure.message. For example, verifier_tool_call_budget means the verifier reached its tool-call limit before scoring finished. The same safe failure summary appears as run.failure in submission status and as run_failure in case-result responses. These fields do not contain raw verifier exception text.

Benchmark overall score

Use get_run_results()["aggregates"][i]["overall_score"] for each variant's overall score. The leaderboard, variant details, and run export use this same value. Scores are normalized from 0 to 1; for example, 0.8333 displays as 83.33%.

The overall score is the weighted average of that variant's available evaluator aggregates: sum(score × weight) / sum(weights). Each Agent Verifier metric participates separately with its configured metric weight; its verifier's parent weight is not applied again. Configured thresholds are already applied to metric aggregates. Variants and evaluation attempts are never mixed in this score.

If some evaluations fail, available scores remain visible with the yellow warning. The warning does not by itself remove the overall score or disqualify the variant. If no metric aggregate is available, overall_score is null. score_status_counts reports the states of recorded case/evaluator pairs. Each variant's evaluation_coverage also reports expected and successful evaluations and fully evaluated cases. For example, evaluations.succeeded: 5, evaluations.expected: 6, cases.fully_evaluated: 2, and cases.total: 3 means 5/6 evaluations succeeded and 2/3 cases were fully evaluated. An evaluation succeeds only when all its configured metric slots succeed. Missing rows count as missing, rather than disappearing from coverage. Coverage is null for historical attempts without a usable frozen evaluator plan. These counts describe case executions present in the run; run integrity checks separately track absent case executions.

Additional statistics have different aggregation rules:

• evaluation.evaluators[].score is the aggregate for one evaluator or verifier metric on one variant. Ordinary evaluators average their successful case scores. Verifier metrics count missing or unsuccessful slots as zero, except explicit missing-required-evidence failures, which are excluded from the denominator.
• mean_per_field_scores averages successful raw field results only, without filling missing results with zero. Its basis is also stated in mean_per_field_scores_basis.

Each variant exposes one overall score. The raw run-detail, export, and optimization-loop endpoints name this field overallScore. Run comparisons expose summary.overall_score_delta. Evaluator, field, and case scores remain available as breakdowns; there is no alternative case-average or run-wide overall total.

compare_runs() requires compatible runs from the same immutable benchmark and evaluator setup. Check score_status_counts before interpreting aggregate scores; it separates succeeded scores from failed, skipped, and insufficient-evidence outcomes. Verifier evaluator results expose source_evaluator_id and metric_key, so clients do not need to parse their opaque result IDs. Artifact downloads are authenticated, size-bounded, verified against available integrity metadata, and installed atomically. See External Agent Submissions for the end-to-end workflow and CLI for equivalent interactive commands.

Error handling

All methods raise PrompticAPIError on API errors:

from promptic_sdk import PrompticClient, PrompticAPIError

with PrompticClient() as client:
    try:
        exp = client.get_experiment("invalid-id")
    except PrompticAPIError as e:
        print(e.status_code)  # 404
        print(e.message)      # "Not found"

start_experiment uses platform-managed models and raises PrompticAPIError with status_code == 402 when the organization has no active subscription and payment method, or is blocked by the free-tier limit. See the Errors guide for the full status-code reference.

Agent Optimization methods raise AgentGymAPIError. It exposes the HTTP status and platform error message while preserving validation errors for malformed local IDs, files, schemas, and options.