Create an ephemeral evaluation sandbox

curl --request POST \
  --url https://davinci-app.com/api/v2/evaluation-runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "name": "<string>",
  "ttlSeconds": 900
}
'

import requests

url = "https://davinci-app.com/api/v2/evaluation-runs"

payload = {
    "name": "<string>",
    "ttlSeconds": 900
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({name: '<string>', ttlSeconds: 900})
};

fetch('https://davinci-app.com/api/v2/evaluation-runs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({name: '<string>', ttlSeconds: 900})
};

fetch('https://davinci-app.com/api/v2/evaluation-runs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

{
  "runId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "sourceProjectId": "<string>",
  "status": "provisioning",
  "sandbox": {
    "projectId": "<string>",
    "branchName": "<string>",
    "ephemeral": true,
    "name": "<string>",
    "deleteAfter": "2023-11-07T05:31:56Z"
  },
  "cleanup": {
    "status": "not_started",
    "requested": true,
    "requestedAt": "2023-11-07T05:31:56Z",
    "completedAt": "2023-11-07T05:31:56Z",
    "grantsRevoked": 1,
    "projectDeleted": true,
    "failure": {
      "code": "EVALUATION_SCOPE_MISSING",
      "message": "<string>",
      "infrastructure": true,
      "details": {}
    }
  },
  "createdAt": "2023-11-07T05:31:56Z",
  "expiresAt": "2023-11-07T05:31:56Z",
  "testId": "<string>",
  "testRunId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "codeExecutionId": "<string>",
  "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "dataPlane": {
    "url": "<string>",
    "token": "<string>",
    "expiresAt": "2023-11-07T05:31:56Z",
    "connectionId": "<string>",
    "ownerId": "<string>",
    "userId": "<string>",
    "projectKey": "<string>",
    "operations": [
      "<string>"
    ],
    "purpose": "evaluation"
  },
  "evidence": {
    "capturedAt": "2023-11-07T05:31:56Z",
    "transcript": {
      "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
      "taskId": "<string>",
      "source": "live",
      "status": "no_session",
      "entryCount": 1,
      "messages": [
        {
          "index": 1,
          "role": "user",
          "kind": "<string>",
          "text": "<string>",
          "timestamp": 123,
          "messageId": "<string>",
          "tool": {
            "name": "<string>",
            "status": "<string>",
            "params": "<unknown>",
            "result": "<unknown>",
            "error": "<string>"
          }
        }
      ]
    },
    "tree": {
      "objectCount": 1,
      "tree": [
        {
          "id": "<string>",
          "name": "<string>",
          "type": "<string>",
          "parent": "<string>",
          "childrenCount": 1
        }
      ]
    },
    "usage": {
      "scope": "account",
      "asOf": "2023-11-07T05:31:56Z",
      "totals": {
        "credits": 123,
        "promptTokens": 123,
        "completionTokens": 123,
        "reasoningTokens": 123,
        "cacheWriteTokens": 123,
        "cacheReadTokens": 123,
        "eventCount": 123
      },
      "projectId": "<string>",
      "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
      "groupBy": "model",
      "groups": [
        {
          "credits": 123,
          "promptTokens": 123,
          "completionTokens": 123,
          "reasoningTokens": 123,
          "cacheWriteTokens": 123,
          "cacheReadTokens": 123,
          "eventCount": 123,
          "provider": "<string>",
          "model": "<string>",
          "projectId": "<string>",
          "sessionId": "<string>",
          "day": "2023-11-07T05:31:56Z"
        }
      ]
    },
    "seededObjectIds": [
      "<string>"
    ],
    "artifacts": [
      {
        "name": "<string>",
        "mediaType": "<string>",
        "fileId": "<string>",
        "sha256": "<string>"
      }
    ],
    "summary": {}
  },
  "quality": {
    "value": 123,
    "unit": "<string>",
    "rubric": "<string>",
    "metrics": {}
  },
  "failure": {
    "code": "EVALUATION_SCOPE_MISSING",
    "message": "<string>",
    "infrastructure": true,
    "details": {}
  },
  "updatedAt": "2023-11-07T05:31:56Z"
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

Evaluations

Create an ephemeral evaluation sandbox

Creates a separate ephemeral project to evaluate against. The caller must hold the purpose=automation grant issued to a running Code execution, which is what binds the sandbox to code rather than to a person’s key. The source project and execution are taken from the grant rather than accepted from the body.

The response contains sandbox metadata and a second, purpose=evaluation grant/data-plane block bound to that sandbox. This target grant can seed and inspect the sandbox and drive its agent, but cannot access another project, account resources, or git routes.

POST

api

evaluation-runs

Create an ephemeral evaluation sandbox

curl --request POST \
  --url https://davinci-app.com/api/v2/evaluation-runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "name": "<string>",
  "ttlSeconds": 900
}
'

import requests

url = "https://davinci-app.com/api/v2/evaluation-runs"

payload = {
    "name": "<string>",
    "ttlSeconds": 900
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.text)

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({name: '<string>', ttlSeconds: 900})
};

fetch('https://davinci-app.com/api/v2/evaluation-runs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: JSON.stringify({name: '<string>', ttlSeconds: 900})
};

fetch('https://davinci-app.com/api/v2/evaluation-runs', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

{
  "runId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "sourceProjectId": "<string>",
  "status": "provisioning",
  "sandbox": {
    "projectId": "<string>",
    "branchName": "<string>",
    "ephemeral": true,
    "name": "<string>",
    "deleteAfter": "2023-11-07T05:31:56Z"
  },
  "cleanup": {
    "status": "not_started",
    "requested": true,
    "requestedAt": "2023-11-07T05:31:56Z",
    "completedAt": "2023-11-07T05:31:56Z",
    "grantsRevoked": 1,
    "projectDeleted": true,
    "failure": {
      "code": "EVALUATION_SCOPE_MISSING",
      "message": "<string>",
      "infrastructure": true,
      "details": {}
    }
  },
  "createdAt": "2023-11-07T05:31:56Z",
  "expiresAt": "2023-11-07T05:31:56Z",
  "testId": "<string>",
  "testRunId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "codeExecutionId": "<string>",
  "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
  "dataPlane": {
    "url": "<string>",
    "token": "<string>",
    "expiresAt": "2023-11-07T05:31:56Z",
    "connectionId": "<string>",
    "ownerId": "<string>",
    "userId": "<string>",
    "projectKey": "<string>",
    "operations": [
      "<string>"
    ],
    "purpose": "evaluation"
  },
  "evidence": {
    "capturedAt": "2023-11-07T05:31:56Z",
    "transcript": {
      "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
      "taskId": "<string>",
      "source": "live",
      "status": "no_session",
      "entryCount": 1,
      "messages": [
        {
          "index": 1,
          "role": "user",
          "kind": "<string>",
          "text": "<string>",
          "timestamp": 123,
          "messageId": "<string>",
          "tool": {
            "name": "<string>",
            "status": "<string>",
            "params": "<unknown>",
            "result": "<unknown>",
            "error": "<string>"
          }
        }
      ]
    },
    "tree": {
      "objectCount": 1,
      "tree": [
        {
          "id": "<string>",
          "name": "<string>",
          "type": "<string>",
          "parent": "<string>",
          "childrenCount": 1
        }
      ]
    },
    "usage": {
      "scope": "account",
      "asOf": "2023-11-07T05:31:56Z",
      "totals": {
        "credits": 123,
        "promptTokens": 123,
        "completionTokens": 123,
        "reasoningTokens": 123,
        "cacheWriteTokens": 123,
        "cacheReadTokens": 123,
        "eventCount": 123
      },
      "projectId": "<string>",
      "sessionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
      "groupBy": "model",
      "groups": [
        {
          "credits": 123,
          "promptTokens": 123,
          "completionTokens": 123,
          "reasoningTokens": 123,
          "cacheWriteTokens": 123,
          "cacheReadTokens": 123,
          "eventCount": 123,
          "provider": "<string>",
          "model": "<string>",
          "projectId": "<string>",
          "sessionId": "<string>",
          "day": "2023-11-07T05:31:56Z"
        }
      ]
    },
    "seededObjectIds": [
      "<string>"
    ],
    "artifacts": [
      {
        "name": "<string>",
        "mediaType": "<string>",
        "fileId": "<string>",
        "sha256": "<string>"
      }
    ],
    "summary": {}
  },
  "quality": {
    "value": 123,
    "unit": "<string>",
    "rubric": "<string>",
    "metrics": {}
  },
  "failure": {
    "code": "EVALUATION_SCOPE_MISSING",
    "message": "<string>",
    "infrastructure": true,
    "details": {}
  },
  "updatedAt": "2023-11-07T05:31:56Z"
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

{
  "success": false,
  "error": {
    "code": "<string>",
    "message": "<string>",
    "details": {}
  }
}

Authorizations

Authorization

string

header

default:dav_gr_execution_grant

required

The grant injected into a running Code object's container. It delegates the reach of whoever started the execution over the one project the Code object lives in, and it expires with the container.

On this mount it can create, inspect, and clean up the evaluation sandboxes that execution owns. It cannot select an existing project as a target, and a sandbox-bound purpose=evaluation grant is refused here so an evaluation cannot start another one.

Body

application/json

name

string

Human-readable sandbox name used in logs.

Maximum string length: 120

ttlSeconds

integer

default:900

Hard cleanup deadline, independent of Code execution timeout.

Required range: 60 <= x <= 3600

Response

The target sandbox is ready and its grant has been attached.

runId

string<uuid>

required

sourceProjectId

string

required

status

enum<string>

required

ready means the target is pinned and its grant can be used. failed is an infrastructure failure, not a low quality score. cleaned and expired are terminal and carry no usable grant.

Available options:

provisioning,

ready,

running,

cleaning,

cleaned,

failed,

expired

sandbox

object

required

Hide child attributes

sandbox.projectId

string

required

Server-chosen target project. It is never accepted from the create body.

sandbox.branchName

string

required

sandbox.ephemeral

boolean

required

Always true for an evaluation-created target.

sandbox.name

string

sandbox.deleteAfter

string<date-time>

cleanup

object

required

Hide child attributes

cleanup.status

enum<string>

required

Available options:

not_started,

pending,

completed,

failed

cleanup.requested

boolean

required

cleanup.requestedAt

string<date-time>

cleanup.completedAt

string<date-time>

cleanup.grantsRevoked

integer

Required range: x >= 0

cleanup.projectDeleted

boolean

cleanup.failure

object

Hide child attributes

cleanup.failure.code

enum<string>

required

Available options:

EVALUATION_SCOPE_MISSING,

EVALUATION_AGENT_FAILED,

EVALUATION_TIMEOUT,

EVALUATION_EVIDENCE_MISSING,

EVALUATION_OBSERVATION_INVALID,

EVALUATION_CLEANUP_FAILED,

CODE_EXECUTION_FAILED,

TEST_CANCELLED,

TEST_RUN_FAILED,

TEST_RUN_TIMED_OUT

cleanup.failure.message

string

required

cleanup.failure.infrastructure

boolean

required

True when no quality verdict may be inferred from this failure.

cleanup.failure.details

object

createdAt

string<date-time>

required

expiresAt

string<date-time>

required

testId

string | null

The Test whose Results Value started the Code, when a Test did. Absent for a Code object run directly.

testRunId

string<uuid> | null

The Test run that started the Code, when one did.

codeExecutionId

string

sessionId

string<uuid>

Headless target session attached to the evaluation data-plane grant.

dataPlane

object

Everything needed to address the data plane: where to send, what to send with, and the path parameters that are not obvious from the project id alone.

Hide child attributes

dataPlane.url

string<uri>

required

Base URL for data-plane operations. Names no specific machine, so it stays valid if the project moves.

dataPlane.token

string

required

The grant, dav_gr_…. Returned only here and only once — it is not stored in recoverable form, so a lost grant is replaced by refreshing the session rather than looked up.

dataPlane.expiresAt

string<date-time>

required

dataPlane.connectionId

string

required

Pass as the connectionId path parameter on upload.

dataPlane.ownerId

string

required

Pass as the ownerId path parameter.

dataPlane.userId

string

required

Pass as the userId path parameter on upload.

dataPlane.projectKey

string

required

Pass as the projectId path parameter. Compound form {projectId}--{branchName}, since the data plane addresses a branch.

dataPlane.operations

string[]

required

Data-plane operations this grant authorizes.

dataPlane.purpose

string

required

Target-bound evaluation authority. Ordinary data-plane/execution grants remain denied from agent and git routes; this purpose admits only the target agent, and still denies git.

Allowed value: "evaluation"

evidence

object

Hide child attributes

evidence.capturedAt

string<date-time>

required

evidence.transcript

object

required

Hide child attributes

evidence.transcript.sessionId

string<uuid>

required

evidence.transcript.taskId

string

required

evidence.transcript.source

enum<string>

required

Where the answer came from. live is current; persisted is the copy saved when a run ended, and so lags a run in progress.

Available options:

live,

persisted,

none

evidence.transcript.status

enum<string>

required

no_session — the machine has no record of this session; it lapsed or the project restarted. Create a new one.
project_not_running — the session is known but its project has unloaded.
idle — the session is live and no run has been started.
starting — a run is being set up.
running — a run is in progress.
paused — a run was stopped and can be continued with a new prompt.
completed — the run finished.
failed — the run ended in an error. The transcript says why.
unknown — the machine could not be reached for this field. Retry; anything else in the same response is still valid.

paused, completed, and failed are terminal for a run.

Available options:

no_session,

project_not_running,

idle,

starting,

running,

paused,

completed,

failed,

unknown

evidence.transcript.entryCount

integer

required

Entries in the whole conversation, not just the returned window.

Required range: x >= 0

evidence.transcript.messages

object[]

required

Oldest first. Bounded by limit, taken from the end.

Hide child attributes

evidence.transcript.messages.index

integer

required

Position in the whole conversation, so a window can be placed.

Required range: x >= 0

evidence.transcript.messages.role

enum<string>

required

Available options:

user,

agent

evidence.transcript.messages.kind

string

required

text for what was said, thought for the agent's reasoning, tool for a tool it used, error for a failure it hit. Treat an unrecognized value as informational rather than as a problem — new kinds can appear.

evidence.transcript.messages.text

string

The content. For a tool entry, the tool's name.

evidence.transcript.messages.timestamp

integer | null

Milliseconds since the epoch.

evidence.transcript.messages.messageId

string

Present on your own prompts, matching what you sent.

evidence.transcript.messages.tool

object

Present only on entries whose kind is tool.

Hide child attributes

evidence.transcript.messages.tool.name

string | null

evidence.transcript.messages.tool.status

string | null

working while it runs, complete when it succeeded, otherwise the severity of what went wrong.

evidence.transcript.messages.tool.params

any

The arguments the agent chose. Null when the tool takes none.

evidence.transcript.messages.tool.result

any

What the tool returned. Shape is the tool's own.

evidence.transcript.messages.tool.error

string

Present when the tool failed.

evidence.tree

object

required

Flat list of project objects excluding the project object itself. New projects contain reserved Model and Library packages whose parent is the project id; callers must not assume a parentless node exists.

Hide child attributes

evidence.tree.objectCount

integer

required

Total objects in the project, including the project object itself.

Required range: x >= 0

evidence.tree.tree

object[]

required

Hide child attributes

evidence.tree.tree.id

string

required

evidence.tree.tree.name

string

required

evidence.tree.tree.type

string

required

evidence.tree.tree.parent

string | null

evidence.tree.tree.childrenCount

integer

Required range: x >= 0

evidence.usage

object

required

Hide child attributes

evidence.usage.scope

enum<string>

required

What the numbers cover, determined by the path you called.

Available options:

account,

project,

session

evidence.usage.asOf

string<date-time> | null

required

When the newest counted event was recorded, or null when the scope holds none. This is the honest bound on freshness: the engine reports usage on an interval, so a run's last few seconds may not be included. Null means "nothing here", not "nothing yet" — a scope with no usage looks the same either way, and inventing a timestamp would hide that.

evidence.usage.totals

object

required

Hide child attributes

evidence.usage.totals.credits

number

required

Credits charged, to six decimal places.

evidence.usage.totals.promptTokens

integer

required

evidence.usage.totals.completionTokens

integer

required

evidence.usage.totals.reasoningTokens

integer

required

Thinking tokens, charged at the completion rate.

evidence.usage.totals.cacheWriteTokens

integer

required

evidence.usage.totals.cacheReadTokens

integer

required

evidence.usage.totals.eventCount

integer

required

Charged calls counted. Not all of them are token-priced — image generation and web search are charged per call, so a non-zero credits with zero tokens is normal rather than a gap.

evidence.usage.projectId

string

Present on the project and session reads. Always the bare id.

evidence.usage.sessionId

string<uuid>

Present on the session read.

evidence.usage.groupBy

enum<string>

Echoed back when you asked for a breakdown.

Available options:

model,

project,

session,

day

evidence.usage.groups

object[]

One entry per group, largest spend first — except day, which is chronological. Absent unless you passed groupBy.

Hide child attributes

evidence.usage.groups.credits

number

required

Credits charged, to six decimal places.

evidence.usage.groups.promptTokens

integer

required

evidence.usage.groups.completionTokens

integer

required

evidence.usage.groups.reasoningTokens

integer

required

Thinking tokens, charged at the completion rate.

evidence.usage.groups.cacheWriteTokens

integer

required

evidence.usage.groups.cacheReadTokens

integer

required

evidence.usage.groups.eventCount

integer

required

Charged calls counted. Not all of them are token-priced — image generation and web search are charged per call, so a non-zero credits with zero tokens is normal rather than a gap.

evidence.usage.groups.provider

string | null

Present when grouping by model.

evidence.usage.groups.model

string | null

Present when grouping by model.

evidence.usage.groups.projectId

string | null

Present when grouping by project.

evidence.usage.groups.sessionId

string | null

Present when grouping by session.

evidence.usage.groups.day

string<date-time> | null

Midnight UTC of the day. Present when grouping by day.

evidence.seededObjectIds

string[]

evidence.artifacts

object[]

Hide child attributes

evidence.artifacts.name

string

required

evidence.artifacts.mediaType

string

evidence.artifacts.fileId

string

evidence.artifacts.sha256

string

Pattern: ^[0-9a-f]{64}$

evidence.summary

object

quality

object

Hide child attributes

quality.value

number

required

Finite numeric observation returned to native Test evaluation criteria.

quality.unit

string

quality.rubric

string

quality.metrics

object

Hide child attributes

quality.metrics.{key}

number

failure

object

Hide child attributes

failure.code

enum<string>

required

Available options:

EVALUATION_SCOPE_MISSING,

EVALUATION_AGENT_FAILED,

EVALUATION_TIMEOUT,

EVALUATION_EVIDENCE_MISSING,

EVALUATION_OBSERVATION_INVALID,

EVALUATION_CLEANUP_FAILED,

CODE_EXECUTION_FAILED,

TEST_CANCELLED,

TEST_RUN_FAILED,

TEST_RUN_TIMED_OUT

failure.message

string

required

failure.infrastructure

boolean

required

True when no quality verdict may be inferred from this failure.

failure.details

object

updatedAt

string<date-time>

Getting Started

Davinci Browser

Davinci Editor

Integrations

Modeling Guide

Developer API

Examples

Release Notes

Create an ephemeral evaluation sandbox

Authorizations

Body

Response