> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.vellum.ai/developers/client-sdk/test-suites/runs/create/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.vellum.ai/_mcp/server.

# Create Test Suite Run

POST https://api.vellum.ai/v1/test-suite-runs
Content-Type: application/json

Trigger a Test Suite and create a new Test Suite Run

Reference: https://docs.vellum.ai/developers/client-sdk/test-suites/runs/create

## Authentication

- `X-API-KEY` header (required) — API Key authentication via header

## Request

### Body (application/json)

This endpoint expects a TestSuiteRunCreateRequest.

- `exec_config` (TestSuiteRunExecConfigRequest, required) — Configuration that defines how the Test Suite should be run
- `test_suite_id` (string, optional) — The ID of the Test Suite to run. Must provide either this or test_suite_id.
- `test_suite_name` (string, optional) — The name of the Test Suite to run. Must provide either this or test_suite_id.

## Response

### 201

- `id` (string, required)
- `created` (string, required)
- `test_suite` (TestSuiteRunTestSuite, required)
- `state` (enum, required) — The current state of this run * `QUEUED` - Queued * `RUNNING` - Running * `COMPLETE` - Complete * `FAILED` - Failed * `CANCELLED` - Cancelled
  - Allowed values: `QUEUED`, `RUNNING`, `COMPLETE`, `FAILED`, `CANCELLED`
- `exec_config` (TestSuiteRunExecConfig, optional, nullable) — Configuration that defines how the Test Suite should be run
- `progress` (TestSuiteRunProgress, optional)

## Types

### TestSuiteRunExecConfigRequest

### TestSuiteRunTestSuite

- `id` (string, required)
- `history_item_id` (string, required)
- `label` (string, required)

### TestSuiteRunExecConfig

### TestSuiteRunProgress

- `number_of_requested_test_cases` (integer, required)
- `number_of_completed_test_cases` (integer, required)

### TestSuiteRunDeploymentReleaseTagExecConfigRequest

Execution configuration for running a Test Suite against a Prompt Deployment

- `data` (TestSuiteRunDeploymentReleaseTagExecConfigDataRequest, required)
- `type` (enum, optional) — * `DEPLOYMENT_RELEASE_TAG` - DEPLOYMENT_RELEASE_TAG
  - Allowed values: `DEPLOYMENT_RELEASE_TAG`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunPromptSandboxExecConfigRequest

Execution configuration for running a Test Suite against a Prompt Sandbox

- `data` (TestSuiteRunPromptSandboxExecConfigDataRequest, required)
- `type` (enum, optional) — * `PROMPT_SANDBOX` - PROMPT_SANDBOX
  - Allowed values: `PROMPT_SANDBOX`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunPromptSandboxHistoryItemExecConfigRequest

Execution configuration for running a Test Suite against a Prompt Sandbox History Item

- `data` (TestSuiteRunPromptSandboxHistoryItemExecConfigDataRequest, required)
- `type` (enum, optional) — * `PROMPT_SANDBOX_HISTORY_ITEM` - PROMPT_SANDBOX_HISTORY_ITEM
  - Allowed values: `PROMPT_SANDBOX_HISTORY_ITEM`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunWorkflowReleaseTagExecConfigRequest

Execution configuration for running a Test Suite against a Workflow Deployment

- `data` (TestSuiteRunWorkflowReleaseTagExecConfigDataRequest, required)
- `type` (enum, optional) — * `WORKFLOW_RELEASE_TAG` - WORKFLOW_RELEASE_TAG
  - Allowed values: `WORKFLOW_RELEASE_TAG`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunWorkflowSandboxExecConfigRequest

Execution configuration for running a Test Suite against a Workflow Sandbox

- `data` (TestSuiteRunWorkflowSandboxExecConfigDataRequest, required)
- `type` (enum, optional) — * `WORKFLOW_SANDBOX` - WORKFLOW_SANDBOX
  - Allowed values: `WORKFLOW_SANDBOX`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunWorkflowSandboxHistoryItemExecConfigRequest

Execution configuration for running a Test Suite against a Workflow Sandbox History Item

- `data` (TestSuiteRunWorkflowSandboxHistoryItemExecConfigDataRequest, required)
- `type` (enum, optional) — * `WORKFLOW_SANDBOX_HISTORY_ITEM` - WORKFLOW_SANDBOX_HISTORY_ITEM
  - Allowed values: `WORKFLOW_SANDBOX_HISTORY_ITEM`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunExternalExecConfigRequest

Execution configuration for running a Vellum Test Suite against an external callable

- `data` (TestSuiteRunExternalExecConfigDataRequest, required)
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.
- `type` (enum, optional) — * `EXTERNAL` - EXTERNAL
  - Allowed values: `EXTERNAL`

### TestSuiteRunDeploymentReleaseTagExecConfig

Execution configuration for running a Test Suite against a Prompt Deployment

- `data` (TestSuiteRunDeploymentReleaseTagExecConfigData, required)
- `type` (enum, optional) — * `DEPLOYMENT_RELEASE_TAG` - DEPLOYMENT_RELEASE_TAG
  - Allowed values: `DEPLOYMENT_RELEASE_TAG`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunPromptSandboxHistoryItemExecConfig

Execution configuration for running a Test Suite against a Prompt Sandbox History Item

- `data` (TestSuiteRunPromptSandboxHistoryItemExecConfigData, required)
- `type` (enum, optional) — * `PROMPT_SANDBOX_HISTORY_ITEM` - PROMPT_SANDBOX_HISTORY_ITEM
  - Allowed values: `PROMPT_SANDBOX_HISTORY_ITEM`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunWorkflowReleaseTagExecConfig

Execution configuration for running a Test Suite against a Workflow Deployment

- `data` (TestSuiteRunWorkflowReleaseTagExecConfigData, required)
- `type` (enum, optional) — * `WORKFLOW_RELEASE_TAG` - WORKFLOW_RELEASE_TAG
  - Allowed values: `WORKFLOW_RELEASE_TAG`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunWorkflowSandboxHistoryItemExecConfig

Execution configuration for running a Test Suite against a Workflow Sandbox History Item

- `data` (TestSuiteRunWorkflowSandboxHistoryItemExecConfigData, required)
- `type` (enum, optional) — * `WORKFLOW_SANDBOX_HISTORY_ITEM` - WORKFLOW_SANDBOX_HISTORY_ITEM
  - Allowed values: `WORKFLOW_SANDBOX_HISTORY_ITEM`
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.

### TestSuiteRunExternalExecConfig

Execution configuration for running a Vellum Test Suite against an external callable

- `data` (TestSuiteRunExternalExecConfigData, required)
- `test_case_ids` (list of string, optional, nullable) — Optionally specify a subset of test case ids to run. If not provided, all test cases within the test suite will be run by default.
- `type` (enum, optional) — * `EXTERNAL` - EXTERNAL
  - Allowed values: `EXTERNAL`

### TestSuiteRunDeploymentReleaseTagExecConfigDataRequest

- `deployment_id` (string, required) — The ID of the Prompt Deployment to run the Test Suite against.
- `tag` (string, optional, default: LATEST) — A tag identifying which release of the Prompt Deployment to run the Test Suite against. Useful for testing past versions of the Prompt Deployment

### TestSuiteRunPromptSandboxExecConfigDataRequest

- `prompt_sandbox_id` (string, required) — The ID of the Prompt Sandbox to run the Test Suite against.
- `prompt_variant_id` (string, required) — The ID of the Prompt Variant within the Prompt Sandbox that you'd like to run the Test Suite against.

### TestSuiteRunPromptSandboxHistoryItemExecConfigDataRequest

- `history_item_id` (string, required) — The ID of the Prompt Sandbox History Item that the Test Suite will run against.
- `prompt_variant_id` (string, required) — The ID of the Prompt Variant within the Prompt Sandbox History Item that you'd like to run the Test Suite against.

### TestSuiteRunWorkflowReleaseTagExecConfigDataRequest

- `workflow_deployment_id` (string, required) — The ID of the Workflow Deployment to run the Test Suite against.
- `tag` (string, optional, default: LATEST) — A tag identifying which release of the Workflow Deployment to run the Test Suite against. Useful for testing past versions of the Workflow Deployment

### TestSuiteRunWorkflowSandboxExecConfigDataRequest

- `workflow_sandbox_id` (string, required) — The ID of the Workflow Sandbox to run the Test Suite against.

### TestSuiteRunWorkflowSandboxHistoryItemExecConfigDataRequest

- `history_item_id` (string, required) — The ID of the Workflow Sandbox History Item that the Test Suite will run against.
- `workflow_variant_id` (string, required) — The ID of the Workflow Variant within the Workflow Sandbox History Item that you'd like to run the Test Suite against.

### TestSuiteRunExternalExecConfigDataRequest

- `executions` (list of ExternalTestCaseExecutionRequest, required) — The executions of some callable external to Vellum whose outputs you would like to evaluate.

### TestSuiteRunDeploymentReleaseTagExecConfigData

- `deployment_id` (string, required) — The ID of the Prompt Deployment to run the Test Suite against.
- `tag` (string, optional, default: LATEST) — A tag identifying which release of the Prompt Deployment to run the Test Suite against. Useful for testing past versions of the Prompt Deployment

### TestSuiteRunPromptSandboxHistoryItemExecConfigData

- `history_item_id` (string, required) — The ID of the Prompt Sandbox History Item that the Test Suite will run against.
- `prompt_variant_id` (string, required) — The ID of the Prompt Variant within the Prompt Sandbox History Item that you'd like to run the Test Suite against.

### TestSuiteRunWorkflowReleaseTagExecConfigData

- `workflow_deployment_id` (string, required) — The ID of the Workflow Deployment to run the Test Suite against.
- `tag` (string, optional, default: LATEST) — A tag identifying which release of the Workflow Deployment to run the Test Suite against. Useful for testing past versions of the Workflow Deployment

### TestSuiteRunWorkflowSandboxHistoryItemExecConfigData

- `history_item_id` (string, required) — The ID of the Workflow Sandbox History Item that the Test Suite will run against.
- `workflow_variant_id` (string, required) — The ID of the Workflow Variant within the Workflow Sandbox History Item that you'd like to run the Test Suite against.

### TestSuiteRunExternalExecConfigData

- `executions` (list of ExternalTestCaseExecution, required) — The executions of some callable external to Vellum whose outputs you would like to evaluate.

### ExternalTestCaseExecutionRequest

- `outputs` (list of NamedTestCaseVariableValueRequest, required) — The output values of a callable that was executed against a Test Case outside of Vellum
- `test_case_id` (string, required)

### ExternalTestCaseExecution

- `outputs` (list of NamedTestCaseVariableValue, required) — The output values of a callable that was executed against a Test Case outside of Vellum
- `test_case_id` (string, required)

### NamedTestCaseVariableValueRequest

### NamedTestCaseVariableValue

### NamedTestCaseStringVariableValueRequest

Named Test Case value that is of type STRING

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required, nullable)
- `name` (string, required)

### NamedTestCaseNumberVariableValueRequest

Named Test Case value that is of type NUMBER

- `type` (enum, required)
  - Allowed values: `NUMBER`
- `value` (double, required, nullable)
- `name` (string, required)

### NamedTestCaseJsonVariableValueRequest

Named Test Case value that is of type JSON

- `type` (enum, required)
  - Allowed values: `JSON`
- `value` (any, required, nullable)
- `name` (string, required)

### NamedTestCaseChatHistoryVariableValueRequest

Named Test Case value that is of type CHAT_HISTORY

- `type` (enum, required)
  - Allowed values: `CHAT_HISTORY`
- `value` (list of ChatMessageRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseSearchResultsVariableValueRequest

Named Test Case value that is of type SEARCH_RESULTS

- `type` (enum, required)
  - Allowed values: `SEARCH_RESULTS`
- `value` (list of SearchResultRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseErrorVariableValueRequest

Named Test Case value that is of type ERROR

- `type` (enum, required)
  - Allowed values: `ERROR`
- `value` (VellumErrorRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseFunctionCallVariableValueRequest

Named Test Case value that is of type FUNCTION_CALL

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCallRequest, required, nullable) — The final resolved function call value.
- `name` (string, required)

### NamedTestCaseArrayVariableValueRequest

Named Test Case value that is of type ARRAY

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of VellumValueRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseAudioVariableValueRequest

Named Test Case value that is of type AUDIO

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudioRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseVideoVariableValueRequest

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideoRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseImageVariableValueRequest

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImageRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseDocumentVariableValueRequest

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocumentRequest, required, nullable)
- `name` (string, required)

### NamedTestCaseStringVariableValue

Named Test Case value that is of type STRING

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required, nullable)
- `name` (string, required)

### NamedTestCaseNumberVariableValue

Named Test Case value that is of type NUMBER

- `type` (enum, required)
  - Allowed values: `NUMBER`
- `value` (double, required, nullable)
- `name` (string, required)

### NamedTestCaseJsonVariableValue

Named Test Case value that is of type JSON

- `type` (enum, required)
  - Allowed values: `JSON`
- `value` (any, required, nullable)
- `name` (string, required)

### NamedTestCaseChatHistoryVariableValue

Named Test Case value that is of type CHAT_HISTORY

- `type` (enum, required)
  - Allowed values: `CHAT_HISTORY`
- `value` (list of ChatMessage, required, nullable)
- `name` (string, required)

### NamedTestCaseSearchResultsVariableValue

Named Test Case value that is of type SEARCH_RESULTS

- `type` (enum, required)
  - Allowed values: `SEARCH_RESULTS`
- `value` (list of SearchResult, required, nullable)
- `name` (string, required)

### NamedTestCaseErrorVariableValue

Named Test Case value that is of type ERROR

- `type` (enum, required)
  - Allowed values: `ERROR`
- `value` (VellumError, required, nullable)
- `name` (string, required)

### NamedTestCaseFunctionCallVariableValue

Named Test Case value that is of type FUNCTION_CALL

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCall, required, nullable) — The final resolved function call value.
- `name` (string, required)

### NamedTestCaseArrayVariableValue

Named Test Case value that is of type ARRAY

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of VellumValue, required, nullable)
- `name` (string, required)

### NamedTestCaseAudioVariableValue

Named Test Case value that is of type AUDIO

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudio, required, nullable)
- `name` (string, required)

### NamedTestCaseVideoVariableValue

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideo, required, nullable)
- `name` (string, required)

### NamedTestCaseImageVariableValue

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImage, required, nullable)
- `name` (string, required)

### NamedTestCaseDocumentVariableValue

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocument, required, nullable)
- `name` (string, required)

### ChatMessageRequest

- `role` (enum, required) — * `SYSTEM` - System * `ASSISTANT` - Assistant * `USER` - User * `FUNCTION` - Function
  - Allowed values: `SYSTEM`, `ASSISTANT`, `USER`, `FUNCTION`
- `text` (string, optional, nullable)
- `content` (ChatMessageContentRequest, optional, nullable)
- `source` (string, optional, nullable) — An optional identifier representing who or what generated this message.
- `metadata` (map from string to any, optional, nullable)

### SearchResultRequest

- `text` (string, required) — The text of the chunk that matched the search query.
- `score` (double, required) — A score representing how well the chunk matches the search query.
- `keywords` (list of string, required)
- `document` (SearchResultDocumentRequest, required) — The document that contains the chunk that matched the search query.
- `meta` (SearchResultMetaRequest, optional, nullable) — Additional information about the search result.

### VellumErrorRequest

- `code` (enum, required) — * `INVALID_REQUEST` - INVALID_REQUEST * `INVALID_INPUTS` - INVALID_INPUTS * `PROVIDER_ERROR` - PROVIDER_ERROR * `PROVIDER_CREDENTIALS_UNAVAILABLE` - PROVIDER_CREDENTIALS_UNAVAILABLE * `INTEGRATION_CREDENTIALS_UNAVAILABLE` - INTEGRATION_CREDENTIALS_UNAVAILABLE * `REQUEST_TIMEOUT` - REQUEST_TIMEOUT * `INTERNAL_SERVER_ERROR` - INTERNAL_SERVER_ERROR * `USER_DEFINED_ERROR` - USER_DEFINED_ERROR * `WORKFLOW_CANCELLED` - WORKFLOW_CANCELLED * `NODE_CANCELLED` - NODE_CANCELLED * `PROVIDER_QUOTA_EXCEEDED` - PROVIDER_QUOTA_EXCEEDED * `CHAT_QUOTA_EXCEEDED` - CHAT_QUOTA_EXCEEDED
  - Allowed values: `INVALID_REQUEST`, `INVALID_INPUTS`, `PROVIDER_ERROR`, `PROVIDER_CREDENTIALS_UNAVAILABLE`, `INTEGRATION_CREDENTIALS_UNAVAILABLE`, `REQUEST_TIMEOUT`, `INTERNAL_SERVER_ERROR`, `USER_DEFINED_ERROR`, `WORKFLOW_CANCELLED`, `NODE_CANCELLED`, `PROVIDER_QUOTA_EXCEEDED`, `CHAT_QUOTA_EXCEEDED`
- `message` (string, required)
- `raw_data` (map from string to any, optional, nullable)

### FunctionCallRequest

The final resolved function call value.

- `arguments` (map from string to any, required)
- `name` (string, required)
- `id` (string, optional, nullable)

### VellumValueRequest

### VellumAudioRequest

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumVideoRequest

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumImageRequest

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumDocumentRequest

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### ChatMessage

- `role` (enum, required) — * `SYSTEM` - System * `ASSISTANT` - Assistant * `USER` - User * `FUNCTION` - Function
  - Allowed values: `SYSTEM`, `ASSISTANT`, `USER`, `FUNCTION`
- `text` (string, optional, nullable)
- `content` (ChatMessageContent, optional, nullable)
- `source` (string, optional, nullable) — An optional identifier representing who or what generated this message.
- `metadata` (map from string to any, optional, nullable)

### SearchResult

- `text` (string, required) — The text of the chunk that matched the search query.
- `score` (double, required) — A score representing how well the chunk matches the search query.
- `keywords` (list of string, required)
- `document` (SearchResultDocument, required) — The document that contains the chunk that matched the search query.
- `meta` (SearchResultMeta, optional, nullable) — Additional information about the search result.

### VellumError

- `code` (enum, required) — * `INVALID_REQUEST` - INVALID_REQUEST * `INVALID_INPUTS` - INVALID_INPUTS * `PROVIDER_ERROR` - PROVIDER_ERROR * `PROVIDER_CREDENTIALS_UNAVAILABLE` - PROVIDER_CREDENTIALS_UNAVAILABLE * `INTEGRATION_CREDENTIALS_UNAVAILABLE` - INTEGRATION_CREDENTIALS_UNAVAILABLE * `REQUEST_TIMEOUT` - REQUEST_TIMEOUT * `INTERNAL_SERVER_ERROR` - INTERNAL_SERVER_ERROR * `USER_DEFINED_ERROR` - USER_DEFINED_ERROR * `WORKFLOW_CANCELLED` - WORKFLOW_CANCELLED * `NODE_CANCELLED` - NODE_CANCELLED * `PROVIDER_QUOTA_EXCEEDED` - PROVIDER_QUOTA_EXCEEDED * `CHAT_QUOTA_EXCEEDED` - CHAT_QUOTA_EXCEEDED
  - Allowed values: `INVALID_REQUEST`, `INVALID_INPUTS`, `PROVIDER_ERROR`, `PROVIDER_CREDENTIALS_UNAVAILABLE`, `INTEGRATION_CREDENTIALS_UNAVAILABLE`, `REQUEST_TIMEOUT`, `INTERNAL_SERVER_ERROR`, `USER_DEFINED_ERROR`, `WORKFLOW_CANCELLED`, `NODE_CANCELLED`, `PROVIDER_QUOTA_EXCEEDED`, `CHAT_QUOTA_EXCEEDED`
- `message` (string, required)
- `raw_data` (map from string to any, optional, nullable)

### FunctionCall

The final resolved function call value.

- `arguments` (map from string to any, required)
- `name` (string, required)
- `id` (string, optional, nullable)

### VellumValue

### VellumAudio

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumVideo

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumImage

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### VellumDocument

- `src` (string, required) — The reference to the source data. This can be one of the following formats: 1. A base64-encoded data URL (e.g., 'data:application/pdf;base64,JVBERi0xLjQKJcfs...'). 2. A url pointing to a file accessible over HTTP or HTTPS. 3. A reference to a file that's been previously uploaded to Vellum, in the form of 'vellum:uploaded-file:\{uploaded\_file\_id}'. You can use the Uploaded Files API to retrieve the url of the uploaded file given its ID. See [https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve](https://docs.vellum.ai/developers/client-sdk/uploaded-files/retrieve) for details.
- `metadata` (map from string to any, optional, nullable)

### ChatMessageContentRequest

### SearchResultDocumentRequest

- `label` (string, required) — The human-readable name for the document.
- `id` (string, optional, nullable) — The ID of the document.
- `external_id` (string, optional, nullable) — The unique ID of the document as represented in an external system and specified when it was originally uploaded.
- `metadata` (map from string to any, optional, nullable) — A previously supplied JSON object containing metadata that can be filtered on when searching.

### SearchResultMetaRequest

- `source` (SearchResultMetaSourceRequest, optional, nullable)

### StringVellumValueRequest

A value representing a string.

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required, nullable)

### NumberVellumValueRequest

A value representing a number.

- `type` (enum, required)
  - Allowed values: `NUMBER`
- `value` (double, required, nullable)

### JsonVellumValueRequest

A value representing a JSON object.

- `type` (enum, required)
  - Allowed values: `JSON`
- `value` (any, required, nullable)

### AudioVellumValueRequest

A base Vellum primitive value representing audio.

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudioRequest, required, nullable)

### VideoVellumValueRequest

A base Vellum primitive value representing a video.

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideoRequest, required, nullable)

### ImageVellumValueRequest

A base Vellum primitive value representing an image.

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImageRequest, required, nullable)

### DocumentVellumValueRequest

A base Vellum primitive value representing a document.

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocumentRequest, required, nullable)

### FunctionCallVellumValueRequest

A value representing a Function Call.

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCallRequest, required, nullable) — The final resolved function call value.

### ErrorVellumValueRequest

A value representing an Error.

- `type` (enum, required)
  - Allowed values: `ERROR`
- `value` (VellumErrorRequest, required, nullable)

### ArrayVellumValueRequest

A value representing an array of Vellum variable values.

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of VellumValueRequest, required, nullable)

### ChatHistoryVellumValueRequest

A value representing Chat History.

- `type` (enum, required)
  - Allowed values: `CHAT_HISTORY`
- `value` (list of ChatMessageRequest, required, nullable)

### SearchResultsVellumValueRequest

A value representing Search Results.

- `type` (enum, required)
  - Allowed values: `SEARCH_RESULTS`
- `value` (list of SearchResultRequest, required, nullable)

### ThinkingVellumValueRequest

A value representing Thinking mode output.

- `type` (enum, required)
  - Allowed values: `THINKING`
- `value` (StringVellumValueRequest, required, nullable) — A value representing a string.

### ChatMessageContent

### SearchResultDocument

- `label` (string, required) — The human-readable name for the document.
- `id` (string, optional, nullable) — The ID of the document.
- `external_id` (string, optional, nullable) — The unique ID of the document as represented in an external system and specified when it was originally uploaded.
- `metadata` (map from string to any, optional, nullable) — A previously supplied JSON object containing metadata that can be filtered on when searching.

### SearchResultMeta

- `source` (SearchResultMetaSource, optional, nullable)

### StringVellumValue

A value representing a string.

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required, nullable)

### NumberVellumValue

A value representing a number.

- `type` (enum, required)
  - Allowed values: `NUMBER`
- `value` (double, required, nullable)

### JsonVellumValue

A value representing a JSON object.

- `type` (enum, required)
  - Allowed values: `JSON`
- `value` (any, required, nullable)

### AudioVellumValue

A base Vellum primitive value representing audio.

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudio, required, nullable)

### VideoVellumValue

A base Vellum primitive value representing a video.

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideo, required, nullable)

### ImageVellumValue

A base Vellum primitive value representing an image.

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImage, required, nullable)

### DocumentVellumValue

A base Vellum primitive value representing a document.

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocument, required, nullable)

### FunctionCallVellumValue

A value representing a Function Call.

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCall, required, nullable) — The final resolved function call value.

### ErrorVellumValue

A value representing an Error.

- `type` (enum, required)
  - Allowed values: `ERROR`
- `value` (VellumError, required, nullable)

### ArrayVellumValue

A value representing an array of Vellum variable values.

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of VellumValue, required, nullable)

### ChatHistoryVellumValue

A value representing Chat History.

- `type` (enum, required)
  - Allowed values: `CHAT_HISTORY`
- `value` (list of ChatMessage, required, nullable)

### SearchResultsVellumValue

A value representing Search Results.

- `type` (enum, required)
  - Allowed values: `SEARCH_RESULTS`
- `value` (list of SearchResult, required, nullable)

### ThinkingVellumValue

A value representing Thinking mode output.

- `type` (enum, required)
  - Allowed values: `THINKING`
- `value` (StringVellumValue, required, nullable) — A value representing a string.

### StringChatMessageContentRequest

A string value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required)

### FunctionCallChatMessageContentRequest

A function call value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCallChatMessageContentValueRequest, required) — The final resolved function call value.

### ArrayChatMessageContentRequest

A list of chat message content items.

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of ArrayChatMessageContentItemRequest, required)

### AudioChatMessageContentRequest

An audio value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudioRequest, required)

### VideoChatMessageContentRequest

A video value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideoRequest, required)

### ImageChatMessageContentRequest

An image value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImageRequest, required)

### DocumentChatMessageContentRequest

A document value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocumentRequest, required)

### SearchResultMetaSourceRequest

### StringChatMessageContent

A string value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `STRING`
- `value` (string, required)

### FunctionCallChatMessageContent

A function call value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `FUNCTION_CALL`
- `value` (FunctionCallChatMessageContentValue, required) — The final resolved function call value.

### ArrayChatMessageContent

A list of chat message content items.

- `type` (enum, required)
  - Allowed values: `ARRAY`
- `value` (list of ArrayChatMessageContentItem, required)

### AudioChatMessageContent

An audio value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `AUDIO`
- `value` (VellumAudio, required)

### VideoChatMessageContent

A video value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `VIDEO`
- `value` (VellumVideo, required)

### ImageChatMessageContent

An image value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `IMAGE`
- `value` (VellumImage, required)

### DocumentChatMessageContent

A document value that is used in a chat message.

- `type` (enum, required)
  - Allowed values: `DOCUMENT`
- `value` (VellumDocument, required)

### SearchResultMetaSource

### FunctionCallChatMessageContentValueRequest

The final resolved function call value.

- `name` (string, required)
- `arguments` (map from string to any, required)
- `id` (string, optional, nullable)

### ArrayChatMessageContentItemRequest

### PdfSearchResultMetaSourceRequest

The source of a search result from a PDF document.

- `document_type` (enum, required)
  - Allowed values: `PDF`
- `start_page_num` (integer, required, nullable) — The 1-indexed page number where the chunk starts in the document. Only available for supported chunking strategies and document types.
- `end_page_num` (integer, required, nullable) — The 1-indexed page number where the chunk ends in the document. Only available for supported chunking strategies and document types.

### FunctionCallChatMessageContentValue

The final resolved function call value.

- `name` (string, required)
- `arguments` (map from string to any, required)
- `id` (string, optional, nullable)

### ArrayChatMessageContentItem

### PdfSearchResultMetaSource

The source of a search result from a PDF document.

- `document_type` (enum, required)
  - Allowed values: `PDF`
- `start_page_num` (integer, required, nullable) — The 1-indexed page number where the chunk starts in the document. Only available for supported chunking strategies and document types.
- `end_page_num` (integer, required, nullable) — The 1-indexed page number where the chunk ends in the document. Only available for supported chunking strategies and document types.

## Examples

**Request**

```json
{
  "exec_config": {
    "data": {
      "deployment_id": "string"
    }
  }
}
```

**Response**

```json
{
  "id": "string",
  "created": "2024-01-15T09:30:00Z",
  "test_suite": {
    "id": "string",
    "history_item_id": "string",
    "label": "string"
  },
  "state": "QUEUED",
  "exec_config": {
    "type": "DEPLOYMENT_RELEASE_TAG",
    "data": {
      "deployment_id": "string",
      "tag": "LATEST"
    },
    "test_case_ids": [
      "string"
    ]
  },
  "progress": {
    "number_of_requested_test_cases": 1,
    "number_of_completed_test_cases": 1
  }
}
```

**SDK Code**

```python
import requests

url = "https://api.vellum.ai/v1/test-suite-runs"

payload = { "exec_config": { "data": { "deployment_id": "string" } } }
headers = {
    "X-API-KEY": "<apiKey>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```typescript
import { VellumClient } from "vellum-ai";

const client = new VellumClient({ apiKey: "YOUR_API_KEY", apiVersion: "YOUR_API_VERSION" });
await client.testSuiteRuns.create({
    execConfig: {
        type: "DEPLOYMENT_RELEASE_TAG",
        data: {
            deploymentId: "deployment_id"
        }
    }
});

```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.vellum.ai/v1/test-suite-runs"

	payload := strings.NewReader("{\n  \"exec_config\": {\n    \"data\": {\n      \"deployment_id\": \"string\"\n    }\n  }\n}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("X-API-KEY", "<apiKey>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.vellum.ai/v1/test-suite-runs")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["X-API-KEY"] = '<apiKey>'
request["Content-Type"] = 'application/json'
request.body = "{\n  \"exec_config\": {\n    \"data\": {\n      \"deployment_id\": \"string\"\n    }\n  }\n}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.vellum.ai/v1/test-suite-runs")
  .header("X-API-KEY", "<apiKey>")
  .header("Content-Type", "application/json")
  .body("{\n  \"exec_config\": {\n    \"data\": {\n      \"deployment_id\": \"string\"\n    }\n  }\n}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.vellum.ai/v1/test-suite-runs', [
  'body' => '{
  "exec_config": {
    "data": {
      "deployment_id": "string"
    }
  }
}',
  'headers' => [
    'Content-Type' => 'application/json',
    'X-API-KEY' => '<apiKey>',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.vellum.ai/v1/test-suite-runs");
var request = new RestRequest(Method.POST);
request.AddHeader("X-API-KEY", "<apiKey>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"exec_config\": {\n    \"data\": {\n      \"deployment_id\": \"string\"\n    }\n  }\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "X-API-KEY": "<apiKey>",
  "Content-Type": "application/json"
]
let parameters = ["exec_config": ["data": ["deployment_id": "string"]]] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.vellum.ai/v1/test-suite-runs")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```