External Agent Submissions

External submissions let you run any agent in your own environment while Promptic owns the immutable benchmark snapshot, hidden evaluation evidence, scoring, insights, and leaderboard.

Workflow

The dashboard is therefore optional for Agent configuration. A coding agent can authenticate, create the complete definition, upload cases and files, publish a revision, execute the candidate, and submit the run for scoring as one automated workflow. Teams can still configure the Agent in the dashboard and use external automation only for execution and submission.

The manifest contains task instructions, public success criteria, case inputs, and short-lived input-file URLs. It never exposes expected outputs, reference files, private rubrics, judge prompts, or evaluator configuration.

Configure authentication and environment

If the SDK is not installed or authenticated yet, start with SDK installation and authentication. The setup below adds the benchmark-specific values needed by an external Agent runner.

Create an AI Application API key

Open the AI Application that owns the Agent component, go to Settings → API Keys, and create a key for the external runner. Copy it immediately; the complete value is shown only once. API keys are scoped to one AI Application, so no separate AI Application ID is required.

Copy the benchmark ID

Open the Agent component and use Copy benchmark ID in the header. This control remains available whether the leaderboard is empty or already contains runs. It identifies the benchmark whose cases the runner will receive. When the runner creates a submission session, Promptic automatically creates or reuses an immutable snapshot of the current task, cases, and scoring setup.

Export the variables

For Promptic-hosted production:

export PROMPTIC_ENDPOINT="https://promptic.eu"
export PROMPTIC_API_KEY="ptc_..."
export BENCHMARK_ID="<benchmark-uuid>"

For another Promptic environment, also set the dashboard origin. Do not append /api or /api/v1:

export PROMPTIC_ENDPOINT="https://your-promptic-host"
export PROMPTIC_API_KEY="ptc_..."
export BENCHMARK_ID="<benchmark-uuid>"

BENCHMARK_ID is a shell variable used by the examples on this page. The SDK reads PROMPTIC_API_KEY and PROMPTIC_ENDPOINT automatically. The SDK defaults to https://promptic.eu when PROMPTIC_ENDPOINT is absent, but the curl examples require it.

You can use an interactive CLI login instead of an API key:

export PROMPTIC_ENDPOINT="https://promptic.eu" # or your other Promptic environment
promptic login
export BENCHMARK_ID="<benchmark-uuid>"

The login flow stores the access token and selected AI Application in ~/.promptic/config.toml. Keep PROMPTIC_ENDPOINT set when the target is not https://promptic.eu.

Configure the candidate's provider credentials

Promptic authentication and model-provider authentication are separate. If the candidate calls OpenAI, Anthropic, or another provider directly, configure that provider's normal environment variable in the external runtime:

export OPENAI_API_KEY="..."

Provider credentials are for the candidate process only. Do not put them in PROMPTIC_API_KEY, prediction data, traces, or uploaded artifacts.

Configure and run with the CLI or Python SDK

Configure the complete Agent component

After authenticating, coding agents and CI can configure the complete Agent through either interface. They do not need to create it in the dashboard first.

promptic agent-gym create agent.json

from pathlib import Path

from promptic_sdk import AgentGymClient, BenchmarkFile, ClassificationF1

with AgentGymClient(ai_application_id="<ai-application-id>") as gym:
    agent = gym.benchmarks.create(
        name="Invoice classifier",
        goal="Classify the supplied invoice and return the requested fields.",
        input_schema={
            "type": "object",
            "properties": {
                "document": {"type": "array", "x-promptic-type": "file"},
            },
            "required": ["document"],
        },
        output_schema={
            "type": "object",
            "properties": {
                "label": {"type": "string", "enum": ["invoice", "other"]},
            },
            "required": ["label"],
        },
        evaluators=[ClassificationF1(field_paths=("label",))],
    )
    agent.cases.add(
        input={"document": [BenchmarkFile(Path("fixtures/invoice.pdf"))]},
        output={"label": "invoice"},
        expected_behavior="Inspect the complete document before classifying it.",
    )
    agent.refresh()
    if not agent.ready_for_submission:
        raise RuntimeError(agent.data["configuration"])
    revision = agent.publish()

The definition response contains readiness, the current evaluation plan, and evaluator recommendations. Publishing snapshots the goal, schemas, cases, and evaluators together; external runs are always evaluated against that immutable revision.

Execute the external Agent

Both interfaces execute the same trusted callback. Save this example as my_agent.py; the callback receives one AgentGymCase at a time:

from pathlib import Path

from promptic_sdk import AgentGymCase, AgentGymCaseResult, AgentGymOutputArtifact


def run(case: AgentGymCase) -> AgentGymCaseResult:
    deck_path = build_deck(case.instructions, case.input)
    return AgentGymCaseResult.artifact(
        AgentGymOutputArtifact(Path(deck_path), field_path="deck"),
    )

Choose the invocation that fits the external process:

promptic agent-gym run "$BENCHMARK_ID" my_agent:run \
  --name pitch-deck-agent \
  --version 1.2.0 \
  --architecture architecture.md

import os

from my_agent import run
from promptic_sdk import AgentGymClient


with AgentGymClient() as gym:
    result = gym.run_and_submit(
        benchmark_id=os.environ["BENCHMARK_ID"],
        executor=run,
        name="pitch-deck-agent",
        version="1.2.0",
        architecture_description="""
## Workflow
1. Extracts the brief and source evidence into a slide outline.
2. Generates each slide from the approved outline and renders a preview.
3. Checks overflow, missing citations, and visual consistency before export.

## Models and tools
Uses one planning call, bounded per-slide generation, and the local presentation renderer.

## Validation
Retries only failed slides, then exports the reviewed deck as a PPTX artifact.
""".strip(),
    )

print(result.run_id)

architecture_description accepts Markdown. When a processing flow is easier to understand visually, add a focused Mermaid diagram with a fenced mermaid block. Keep a short written explanation as well, and prefer several small diagrams over one oversized diagram.

## Processing flow

```mermaid
flowchart LR
    Brief --> Plan
    Plan --> Render
    Render --> Validate
    Validate --> Export
```

Validation retries only failed slides before the reviewed deck is exported.

AgentGymClient uses the saved CLI login by default. It also accepts a normal Promptic API key through PROMPTIC_API_KEY or the api_key argument. The SDK uploads each completed case prediction immediately, retries transient upload failures idempotently, and dynamically bounds lower-level upload batches by serialized size and case count.

The high-level callback API executes the callback in the authenticated process and is therefore intended for code you trust. Do not pass platform credentials into generated or untrusted code. Run that code in an isolated environment, collect its outputs in a trusted runner, and use the lower-level session API from that runner to upload artifacts and submit the completed session for scoring.

Inspect a scored run

Use the returned run ID to retrieve aggregate metrics and the weakest cases.

promptic agent-gym results "$BENCHMARK_ID" "$RUN_ID"
promptic agent-gym case-results "$BENCHMARK_ID" "$RUN_ID" \
  --sort score \
  --limit 5
promptic agent-gym case-result "$BENCHMARK_ID" "$RUN_ID" 42
promptic agent-gym artifact-download \
  "$BENCHMARK_ID" "$RUN_ID" 42 0 \
  --output review/report.pdf

with AgentGymClient() as gym:
    summary = gym.get_run_results(os.environ["BENCHMARK_ID"], result.run_id)
    weakest = gym.list_case_results(
        os.environ["BENCHMARK_ID"],
        result.run_id,
        sort="score",
        limit=5,
    )

    print(summary["aggregates"])
    for case_result in weakest["data"]:
        print(case_result["case_id"], case_result["overall_score"])
        print(case_result["judgements"])
        print(case_result["traces"])

        for artifact in case_result["artifacts"]:
            destination = Path("review") / str(case_result["case_id"]) / (
                artifact["path"] or "artifact.bin"
            )
            gym.download_prediction_artifact(artifact, destination)

Each case result contains the submitted output, official scores, normalized judge rationales, output artifact metadata, linked execution traces, metrics, and generated insights. Failed evaluator scores include a safe error message; a recognized verifier tool limit or timeout is named explicitly. Raw provider errors and internal diagnostics are withheld from API responses. Optional artifacts and traces are empty lists when the external runtime did not submit them.

sort="score" lists failed executions first and then successful cases from lowest to highest score. Use get_case_result() for one case or iter_case_results() to scan every case.

Compare a child architecture with its baseline:

promptic agent-gym compare-runs \
  "$BENCHMARK_ID" \
  "$BASELINE_RUN_ID" \
  "$CANDIDATE_RUN_ID"

with AgentGymClient() as gym:
    comparison = gym.compare_runs(
        os.environ["BENCHMARK_ID"],
        parent_run_id="<baseline-run-uuid>",
        candidate_run_id="<candidate-run-uuid>",
    )

print(comparison["summary"])
print([case for case in comparison["cases"] if case["classification"] == "regressed"])

Paired comparison requires the same immutable benchmark snapshot, scorer contract, evaluator configuration, and case set. Review case regressions, execution failures, latency, and token usage instead of selecting a variant from the mean score alone.

Monitor and recover an external submission

Custom runners that retain a submission ID can inspect or wait for its state from either interface:

promptic agent-gym submission-status "$BENCHMARK_ID" "$SUBMISSION_ID"
promptic agent-gym submission-wait \
  "$BENCHMARK_ID" "$SUBMISSION_ID" \
  --max-wait 600 \
  --poll-interval 2

promptic agent-gym retry-scoring "$BENCHMARK_ID" "$RUN_ID"

promptic agent-gym submission-cancel "$BENCHMARK_ID" "$SUBMISSION_ID" --yes

with AgentGymClient() as gym:
    status = gym.wait_for_submission(
        os.environ["BENCHMARK_ID"],
        submission_id,
        max_wait=600,
        poll_interval=2,
    )
    if status["status"] == "dispatch_failed" and status["run"] is not None:
        gym.retry_scoring(os.environ["BENCHMARK_ID"], status["run"]["id"])

Retrying scoring is idempotent and reuses the existing persisted predictions and run. It does not execute the Agent again or create a replacement leaderboard entry.

Submission protocol

The SDK handles the protocol below. Use it directly only when implementing a custom or isolated runner.

The curl examples use PROMPTIC_ENDPOINT, PROMPTIC_API_KEY, and BENCHMARK_ID as configured above.

curl -X POST "$PROMPTIC_ENDPOINT/api/v1/benchmarks/$BENCHMARK_ID/submissions" \
  -H "Authorization: Bearer $PROMPTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: run-2026-07-17-001" \
  -d '{
    "ttl_seconds": 86400,
    "variant_identity": {
      "name": "pitch-deck-agent",
      "version": "1.2.0",
      "repository_url": "https://github.com/acme/pitch-deck-agent",
      "commit_hash": "6f1ed002ab5595859014ebf0951522d9d5f25a73",
      "architecture_description": "Extract evidence, generate and render the deck, then validate overflow and citations."
    }
  }'

curl "$PROMPTIC_ENDPOINT/api/v1/benchmarks/$BENCHMARK_ID/submissions/$SUBMISSION_ID/manifest?limit=50" \
  -H "Authorization: Bearer $PROMPTIC_API_KEY"

curl -X POST "$PROMPTIC_ENDPOINT/api/v1/benchmarks/$BENCHMARK_ID/submissions/$SUBMISSION_ID/artifacts" \
  -H "Authorization: Bearer $PROMPTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "path": "deck.pptx",
    "role": "deliverable",
    "mime_type": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
    "size_bytes": 42891,
    "sha256": "<64 lowercase hex characters>"
  }'

curl "$PROMPTIC_ENDPOINT/api/v1/benchmarks/$BENCHMARK_ID/submissions/$SUBMISSION_ID/traces?trace_id=$OTEL_TRACE_ID" \
  -H "Authorization: Bearer $PROMPTIC_API_KEY"

{
  "predictions": [
    {
      "dataset_case_id": 1042,
      "status": "succeeded",
      "output": {
        "deck": ["promptic-artifact://<verified-artifact-id>"]
      },
      "artifact_ids": ["<verified-artifact-id>"],
      "execution_refs": {
        "trace_ids": ["<trace-id>"]
      },
      "token_usage": { "prompt": 1800, "completion": 950, "total": 2750 },
      "latency_ms": 18240
    },
    {
      "dataset_case_id": 1043,
      "status": "failed",
      "artifact_ids": [],
      "error_code": "render_failed",
      "error_category": "execution",
      "retryable": true,
      "error": "The renderer exited before producing a valid deck."
    }
  ]
}

{
  "metadata": { "executor_region": "eu" }
}

For a child architecture, add parent_name and parent_version to the variant_identity supplied when creating the session. When the name stays unchanged, parent_version alone is sufficient. Use rationale for the observed weakness and intent for the expected behavioral effect. Add repository_url and commit_hash to record the exact source revision used by the variant.

If an evaluator cannot run because required evidence is missing, its evaluation fails only for that prediction. Those unavailable evaluations are excluded from the score; other evaluated cases and metrics remain visible. The leaderboard marks the score as partially evaluated and shows how many cases lack required evidence. If no score is available, it shows “—” with a warning instead of 0%; the warning explains that the variant was not evaluated.

Other failed, skipped, and cancelled cases still count as zero. A technical failure of a required evaluator still makes the evaluation ineligible. Only eligible runs for the current benchmark snapshot appear on the official leaderboard.

Retries

• Retry scoring reuses the immutable predictions only when no official scoring attempt completed.
• Rerun agent creates a new empty submission session for the same benchmark snapshot.
• Repeating session creation or submit with the same idempotency key and payload is safe. Reusing a key with different content returns 409.

Generated artifacts, trace links, evaluator reasoning, eligibility reasons, and deterministic failure insights remain available from the run drill-down. If scoring fails, run results include a failure object with a stable code and a readable message. For example, verifier_tool_call_budget means the verifier reached its tool-call limit before finishing. Submission status returns the same object under run.failure; case-result responses include it as run_failure. Raw verifier exception text is not included in these fields.

The same evidence is available to authenticated external runners:

• GET /api/v1/benchmarks/{benchmark_id}/runs/{run_id}/results
• GET /api/v1/benchmarks/{benchmark_id}/runs/{run_id}/case-results
• GET /api/v1/benchmarks/{benchmark_id}/runs/{run_id}/case-results/{case_id}
• GET /api/v1/benchmarks/{benchmark_id}/runs/compare?parent_run_id=...&candidate_run_id=...