Monitoring

Webhook Integration

Vellum supports real-time monitoring of platform events through Webhook integration. This allows organizations to stream event data to external systems for monitoring, alerting, and custom integrations.

Configuration

To set up Webhook integration:

  1. Navigate to your organization settings page in Vellum
  2. Scroll down to the “Monitoring Integration” section
  3. Click on “Webhook Integration” or the “Edit” button if it’s already configured
  4. Configure the integration with your desired settings

Webhook Integration Settings

Authentication

Webhook integration supports two authentication methods to ensure your data is securely transmitted:

  1. API Key Header: Send an API key in a custom header with each webhook request
  2. Bearer Token: Use a bearer token for authentication

Webhook Authentication Configuration

For additional security, you can also implement HMAC Authentication to verify that webhook requests are genuinely from Vellum.

Supported Events

You can configure which events are sent to your webhook endpoint. The following events are available:

Webhook Event Configuration

CategoryEventDescription
Workflow Execution Eventsworkflow.execution.initiatedTriggered when a workflow starts execution
workflow.execution.fulfilledTriggered when a workflow successfully completes
workflow.execution.rejectedTriggered when a workflow execution fails
Workflow Usage Calculation Eventsworkflow_execution.usage_calculation.fulfilledTriggered when workflow usage is successfully calculated
workflow_execution.usage_calculation.rejectedTriggered when workflow usage calculation fails
Metric Execution Eventsmetric.execution.fulfilledTriggered when a metric execution completes successfully

Event Schema

Webhook events follow a consistent schema that includes essential information about the event:

1{
2    "id": "UUID",
3    "timestamp": "ISO-8601 datetime",
4    "api_version": "2024-10-25",
5    "trace_id": "UUID",
6    "span_id": "UUID",
7    "parent": {
8        // Parent context information
9    },
10    "name": "event.name.format",
11    "body": {
12        // Event-specific data
13    }
14}

Workflow Execution Events

Workflow Execution Initiated

1{
2    "id": "UUID",
3    "timestamp": "ISO-8601 datetime",
4    "api_version": "2024-10-25",
5    "trace_id": "UUID",
6    "span_id": "UUID",
7    "parent": {
8        // Parent context information
9    },
10    "name": "workflow.execution.initiated",
11    "body": {
12        "workflow_class": {
13            "name": "string",
14            "module": "string",
15            "bases": []
16        },
17        "inputs": {
18            // Input variables
19        }
20    }
21}

Workflow Execution Fulfilled

1{
2    "id": "UUID",
3    "timestamp": "ISO-8601 datetime",
4    "api_version": "2024-10-25",
5    "trace_id": "UUID",
6    "span_id": "UUID",
7    "parent": {
8        // Parent context information
9    },
10    "name": "workflow.execution.fulfilled",
11    "body": {
12        "workflow_class": {
13            "name": "string",
14            "module": "string",
15            "bases": []
16        },
17        "outputs": {
18            // Output variables
19        }
20    }
21}

Workflow Execution Rejected

1{
2    "id": "UUID",
3    "timestamp": "ISO-8601 datetime",
4    "api_version": "2024-10-25",
5    "trace_id": "UUID",
6    "span_id": "UUID",
7    "parent": {
8        // Parent context information
9    },
10    "name": "workflow.execution.rejected",
11    "body": {
12        "workflow_class": {
13            "name": "string",
14            "module": "string",
15            "bases": []
16        },
17        "error": {
18            "code": "ERROR_CODE",
19            "message": "Error message"
20        }
21    }
22}

Workflow Usage Calculation Events

1{
2    "id": "UUID",
3    "timestamp": "ISO-8601 datetime",
4    "api_version": "2024-10-25",
5    "trace_id": "UUID",
6    "span_id": "UUID",
7    "parent": {
8        // Parent context information
9    },
10    "name": "workflow.usage_calculation.fulfilled",
11    "body": {
12        "usage": {
13             "input_token_count": 123,
14             "output_token_count": 456,
15             "input_char_count": 789,
16             "output_char_count": 101,
17             "cache_creation_input_tokens": 0,
18             "cache_read_input_tokens": 0,
19             "compute_nanos": 1000000000,
20        },
21        "cost": {
22            "value": 0.0123,
23            "currency": "USD",
24        }
25    }
26}

Parent Context

The parent field provides context about what triggered the event, which can include:

1{
2    "span_id": "UUID",
3    "parent": null,
4    "deployment_id": "UUID",
5    "deployment_name": "my-deployment",
6    "deployment_history_item_id": "UUID",
7    "workflow_version_id": "UUID",
8    "release_tag_id": "UUID",
9    "release_tag_name": "PRODUCTION",
10    "external_id": "optional-external-id"
11}

Use Cases

The Webhook integration enables several key monitoring and integration capabilities:

  1. Real-time Monitoring: Track workflow executions as they happen
  2. Custom Alerting: Build custom alerting systems based on workflow failures or performance metrics
  3. Cost Tracking: Monitor resource consumption and costs
  4. Integration with External Systems: Connect Vellum events to your existing monitoring infrastructure
  5. Audit Logging: Maintain a record of all executions for compliance and debugging
  6. Custom Analytics: Build custom dashboards and reports using your Vellum data

Best Practices

When setting up your Webhook integration:

  1. Start with Critical Events: Begin by monitoring the most important events like execution failures
  2. Implement Proper Error Handling: Ensure your webhook endpoint can handle occasional failures or retries
  3. Use HMAC Authentication: For production systems, implement HMAC verification to ensure webhook authenticity
  4. Monitor Webhook Performance: Ensure your webhook endpoint can handle the volume of events without introducing latency
  5. Implement Idempotency: Design your webhook handler to be idempotent in case of duplicate events