Changelog | March, 2025
New Workflow Deployment Executions APIs
March 26th, 2025
We now have two new apis List Workflow Deployment Executions and Retrieve Workflow Deployment Execution for listing your Executions for a specific Workflow Deployment, and retrieving any specific Workflow Deployment Execution.
Support for Gemini 2.5 Pro Model
March 25th, 2025
We’ve added support for Gemini 2.5 Pro Experiemental (Version 03-25) with a whopping 1M input token context window and 64k output tokens via Google’s Gemini API
Select all rows in Evaluation Reports
March 25th, 2025
You can now select all rows on a page in the Evaluation Report by clicking the checkbox in the header. This selection persists across pages, with the checkbox applying only to the current page.
Workflow Execution Detail Cost Breakdown
March 24th, 2025
We are now displaying the aggregated cost for a Workflow Execution’s inside of our Workflow Execution’s Detail Page. This aggregation is available for all Workflow spans within your Workflow Execution, giving a holistic view of your total execution cost.
Deployment Release Reviews
Inspired by Github PR reviews, you can now provide reviews on a Prompt or Workflow Deployment Release after it has been deployed. This is helpful if your company mandates a review process for all changes that make it to production.
To see Release Reviews in action, check out the demo video below:
Support for Llama 3.3 70B via Cerebras
March 21st, 2025
We’ve added support for Llama 3.3 70B via Cerebras AI
New Webhook and Datadog Events
March 20th, 2025
We are releasing 2 new events for Webhooks and Datadog integrations:
workflow.execution.initiatedworkflow.execution.fulfilled
These events can be used for better visibility in the execution of a Workflow, such as retrieving the input values used to kick it off, as well as the output values that it produced. You might also use these events to track latency, data drift, and more.
Retry and Try Node adornments
March 22nd, 2025
Error handling and retry logic have both been traditionally quite cumbersome in Vellum. Often, you want to catch errors around a single node and “adorn” it with different error handling behavior. Today we are releasing Retry and Try Node Adornments.
Adornments are accessible in the Node sidepanel after clicking on a Node. They act as Nodes themselves, that reference the node it wraps as a single node Subworkflow:
- Retry Node Adornments repeatedly invokes the Node until it either succeeds or the max number of attempts are hit.
- Try Node Adornments attempts to invoke the Node, and continues with an
Erroroutput if it fails.
Monitoring views for Node Adornment invocations show as if the targeted Node was invoked as a single node Subworkflow:
Monitoring View Overhaul
March 22nd, 2025
We’re excited to introduce completely revamped Monitoring interfaces for Prompts and Workflows Deployments that bring significant improvements to how you visualize and analyze your AI system performance.
Our new Grafana-based implementation loads data faster and handles metric processing more efficiently, making it easier than ever to monitor your AI applications at scale.
Key improvements include:
- Faster dashboard load times
- Improved date range and Release Tag selection
- Zoom in on specific time ranges
Copy/Paste Prompt Variants
March 20th, 2025
It is now possible to copy a JSON representation of a Vellum prompt to your clipboard and then paste over another Prompt Variant. This is especially useful if you want to go back to a prior history state, copy just one Prompt, and then paste it in your current live draft.
Updated Workflow Node Inputs
March 20th, 2025
Workflow Nodes used to use a chunky UI for configuring their inputs. We’ve now updated this UI to be simpler by making use of what we call “Expression Inputs.” These Node inputs are functionally the same as before, but tease soon-to-come functionality where you’ll be able to define more complex expressions for what gets passed to a Node.
Metric Setup Improvements
March 20th, 2025
We’ve significantly simplified the process of configuring and editing Evaluation Metrics in Vellum. You can now manage your evaluation metrics directly within an evaluation report – adding, editing, and removing metrics – all without leaving the Prompt/Workflow that you’re evaluating.
You’ll also find a new Metric Settings button that opens a modal where you can configure your metrics directly
Datadog Integration
March 20th, 2025
It’s now possible to receive real-time updates about actions taking place in Vellum using Datadog.
From the organization settings page, you can configure a Datadog integration with a custom list of events you care about.
This is useful if your organization already uses Datadog, or you’d like to leverage Datadog’s monitoring, alerting, and BI capabilities using your Vellum data.
All New Workflows are SDK-Enabled
March 20th, 2025
About two months ago, we began the Beta period of SDK-enabled Workflows. These Workflows use the new Workflows SDK as the underlying engine and run these Workflows in a, secure, more performant, isolated environment. The Workflows SDK enables exciting new functionality including custom nodes, custom docker runtimes, new expression inputs, Node adornments, and so much more.
As of today, all new Workflows going forward will be SDK-enabled by default. We expect that all features from the old Workflows engine are supported, with the exception of “Run from Node”, which we hope to reenable later this month. To revert a Workflow that is SDK-enabled back to the legacy Workflows engine, simply toggle off this checkbox from the Workflow settings:
If you have an existing Workflow and would like to make it SDK-enabled, you can use the same toggle.
JSON Outputs for Prompt Nodes
March 19th, 2025
Until now in SDK enabled workflows, you only had string or array outputs for Prompt Nodes. With this update,
you will now be able to reference a Prompt Node’s JSON output if the json_mode or json_schema parameter is enabled.
This is useful as it enables you to not have to do any additional casting in a Templating Node or Code Execution Node.
PDF Support for Gemini 2.0 Flash Models
March 19th, 2025
The following Gemini 2.0 Flash models now allow for drag-and-drop PDF documents to be used within your Prompts.
- Gemini 2.0 Flash Experimental
- Gemini 2.0 Flash Experimental Thinking Mode
- Gemini 2.0 Flash
Automatic Evaluations Setup
March 14th, 2025
Now, when navigating to the Evaluations tab of a Prompt/Workflow for the first time, Vellum will auto-generate an initial Test Suite for you. We’ll automatically create one Test Case per Scenario found in the Sandbox and prepare everything needed for you to add Metrics and Ground Truth data.
File Hosting for Images and PDFs
March 11th, 2025
Until now, when you provided an image to a Prompt in Vellum, it’d either have to be a public URL or the base64 encoded representation of the image. With this update, we now support secure file hosting, such that when you drag-and-drop an image (and also now PDFs!) into a Prompt, Vellum securely hosts the file on your behalf.
The end result is that you can now provide much larger images and PDFs to LLMs within Vellum without worrying about a decrease in performance or page load times.
PDFs as a Prompt Input
March 11th, 2025
Recently, certain model providers like Anthropic have begun supporting PDFs as native LLM inputs using a special
content type called document (check out their docs for details here.
This is similar to how you might provide a multi-modal model with an image as an input, but now you can provide a PDF as well.
Vellum now also supports passing PDFs as inputs to a Prompt for models that support it. You can do this by drag-and-dropping a PDF file into a Chat History variable in a Prompt. The mechanics are very similar to how you might work with images (see details here).
This is particularly useful for data extraction tasks, where you might want to extract structured data from a PDF and then use that data to power some downstream process.
Support for Qwen QwQ Models via Groq
March 11th, 2025
We’ve added support for a variety of Qwen’s newest QwQ 32B models via Groq’s preview models.
We’ve added the following models:
- QwQ 32B
- QwQ 2.5 Coder 32B
- QWQ 2.5 32B
Support for Qwen QwQ 32B via Fireworks AI
March 11th, 2025
We’ve added support for Qwen’s newest QwQ 32B model via Fireworks AI.
Webhooks
March 10th, 2025
It’s now possible to receive real-time updates about actions taking place in Vellum using Webhooks.
From the organization settings page, you can configure a webhook endpoint with a custom list of events you care about. You can further customize it with your own auth configurations.
This is useful if you’d like to store Vellum monitoring data in your own external data stores. For example, you might save events to a Data Warehouse to power a custom health dashboard.
Keep an eye out, as more event types will be added soon!
Workflow Deployment Executions - Cost Column
March 7th, 2025
You can now see the total cost per Workflow Execution for a given Workflow Deployment in its Executions table. This toggle can be shown/hidden via the “columns” menu.
This is useful for getting a sense of how much a given AI use-case costs to support. This column will be populated for new Workflow Executions going forward and sums the costs associated with all Prompt invocations within the Workflow’s execution (included nested invocations within Subworkflow Nodes, Map Nodes, etc.).
We’ll be exposing more cost metrics throughout Vellum in the coming weeks. Stay tuned!
Prompt Sandbox Pagination
March 5th, 2025
We’ve added pagination to the Prompt Sandbox page. Now, when you have a large number of Scenarios in a Prompt Sandbox, they’ll be split across multiple pages. You can navigate between pages using the pagination controls at the bottom
This should result in performance improvements for those with large Prompt Sandboxes.
Global Search
March 3rd, 2025
We’ve added an eagerly-awaited-for and long-overdue feature to Vellum – Global Search 🎉 You can now search across all your Prompts, Workflows, Document Indexes, and more using the new Search bar in the Vellum side nav.
Doing so will pull up a search bar where you can search for any resource in your Workspace. You can directly navigate to the resource from the search results.
You can also access Global Search from any page through the keyboard shortcut Cmd/Ctrl + K. Give it a try and let us
know what you think!