Changelog | March, 2024

Configurable Chunk Settings for Document Indexes

March 26th, 2024

We’ve added the ability to configure the chunk size and the overlap between consecutive chunks for Document Indexes. You can find it under the “Advanced” section when creating or cloning a Document Index.

Workflow Template Node Debugging

March 26th, 2024

There’s a new debugging feature for iterating on Workflow Template Nodes. You can click the new “Test” button in the full-screen editor and test your template without having to run the whole Workflow. Then you can further iterate on your template by modifying your test data in the “Test Data” tab.

Workflow Node Search

March 26th, 2024

We have added a new Workflow node search feature to help you find your way in large and complex Workflows. Click the new search icon in the top right to quickly find the node you are looking for, or use the ⌘ + shift + F shortcut (ctrl + shift + F on windows).

Workflow Node Mocking

March 26th, 2024

While iteratively developing a Workflow in Vellum, you often want to focus on improving a specific branch or node. It can be cumbersome to re-run the entire Workflow just to test the part you’re iterating on, especially if you already know what the upstream nodes are going to output.

To help address this, Vellum now supports node mocking. You can now mock out a Prompt or Subworkflow Node such that its execution is skipped and a hard-coded value is returned.

This can help you dramatically speed up your Workflow development since you no longer have to wait for early Prompt Nodes to complete. This has the added benefit of saving the expense of tokens with LLM providers!

For more information on Workflow Node Mocking, visit our new help center page.

Claude 3 and Mistral on Bedrock

March 23rd, 2024

We now support both of the Claude 3 and both of the Mistral models on AWS Bedrock. Add these models to your workspace by heading to the models page and searching for the one you need from the search bar.

Navigation Updates

March 22nd, 2024

We’ve made some significant changes to Vellum’s navigation UI.

The app sidebar has been reorganized with the goal of making it easier to navigate between a Prompt/Workflow’s Sandbox, Evaluations, and Deployments. You’ll find that after you’ve clicked on a “Prompt” or “Workflow,” there’s an integrated submenu within the navigation sidebar that shows “Sandbox,” “Evaluations,” and “Deployments.”

Additionally, you’ll find that some nav items, such as “Deployments”, “Models,” “API Keys,” “Organization,” and “Profile” have been grouped into the new “More” and “Settings” nav items.

Read-only Workflow Diagrams

March 22th, 2024

You can now see a read-only view of workflow diagrams for Workflow Deployment Executions, Workflow Test Case Executions, and Workflow Releases. You can access the diagram by clicking the “Graph View” icon tab on the top right.

This is particularly helpful if you want to visualize what your Workflow looked like at that time, as well as visualize the execution path your Workflow took.

Test Suite Table Updates

March 21th, 2024

The Test Cases table on the Test Suites page has been updated to use the same new styling and functionality as the Test Cases table that you’ll find when viewing a Prompt/Workflow Evaluation Report. With this, adding, editing, and deleting Test Cases is generally more reliable. Additionally, special variables types, like Chat History, have an improved display are are no longer displayed as raw JSON.

Additional Headers on API Nodes

March 20th, 2024

Previously, API Nodes only accepted one configurable header, defined on the Authorization section on the node. You can now configure additional headers in the new advanced Settings section. Header values could be regular STRING values or Secrets, and any headers defined here would override the Authorization header.

Indicators for Deployed Prompt/Workflow Sandboxes

March 19th, 2024

You can now tell at a glance whether a given Prompt/Workflow Sandbox has been deployed. You can also hover over the tag to see when it was last deployed.

Cancellable Workflow Deployment Executions

March 18th, 2024

You can now cancel running Workflow Deployment Executions. Simply click the cancel button on the Workflow Execution details page.

Code Execution Eval Metric Debugging

March 18th, 2024

There’s a new debugging feature for iterating on custom Code Evaluation Metrics. You can click the new “Test” button and test your code without having to run the whole test suite. You can update the example data that’s passed into your Code Metric by going to the “Test Data” tab.

Workflow Details for Workflow Evaluations

March 18th, 2024

You can now view Workflow Execution details from the Workflow Evaluations table! To view the details, click on the new “View Workflow Details” button located within a test case’s value cell.

Subworkflow Nodes

March 14th, 2024

Are your Workflows becoming giant and unwieldy? Wish you could define composable groups of nodes to be used across Workflows?

We’re excited to introduce the latest node type in the Workflows node picker - Subworkflow Nodes! With Subworkflow Nodes, you can now link directly to deployed Workflows to reuse commonly grouped nodes and execution logic. Subworkflow Nodes also supports release tagging, giving users the flexibility to either pin to a specific version (say, production) or always automatically update with LATEST.

For more details, check out our new help center doc.

Image Support in the UI

March 13th, 2024

Image support is LIVE in the Vellum UI for OpenAI’s GPT-4 Turbo with Vision! Vellum API’s have had image support for a while and now you can add images in your Prompt and Workflow Sandbox scenarios!

For more details on supported image formats and working with OpenAI’s vision models in Vellum, check out our new help center doc.

Workflow Node Input Value Display

March 11th, 2024

You can now view a Node’s input values directly from the Workflow Editor! This makes it easier to understand what data is being passed into a Node and to debug issues.

Inline Editing for Evaluations

March 11th, 2024

You can now edit test cases directly from the “Evaluations” tab in Workflows and Prompts!

The new editing interface makes it easier than ever to make changes to test cases with long variable values, allows you to edit Chat History values with the same drag-and-drop editor you use elsewhere in the app, and adds support for formatted editing of JSON.

We’re continuing to add support for more variable types and will soon be applying this new edit flow to other tables throughout the app.

Workflow Node ‘Reject on Error’ Toggle

March 9th, 2024

Previously, if a Node in a Workflow errored, the Workflow would proceed to execute until another downstream Node tried to use the output of the Node that errored and would only then terminate. This made Workflows hard to debug and put the onus on you to implement error handling.

Going forward, by default, Workflows will immediately terminate if a Node errors. There are still cases in which you might want to continue despite a Node error (e.g. implementing your own error handling or retry logic). In this cases, you can disable the new “Reject on Error” toggle.

Historical Workflow Nodes have this toggle disabled so that there’s no change in behavior. New Nodes going forward will have this toggle enabled by default.

Workflow Code Execution Node Debugging

March 8th, 2024

We have introduced a new debugging feature for workflow code execution nodes! You can click the new “Test” button in the full-screen editor and test your code without having to run the whole workflow! Then you can further iterate on your code by modifying your test data in the “Test Data” tab.

List Document Indexes API

March 7th, 2024

We’ve exposed a new API endpoint to list all the Document Indexes in a Workspace. You can find the details of the API here.

In-Progress Workflows Executions Now Available in Monitoring

March 6th, 2024

You previously had to wait for a workflow to fully resolve before seeing it in the Workflow Executions table. We now start publishing executions as soon as Workflows are initiated! This allows users building complex Workflows to see any that are still in progress:

We also updated the Workflow Execution Details page to also reflect in progress workflows:

Expand Scenario in Prompt Sandbox

March 6th, 2024

Looking for more room to edit your scenarios in the prompt sandbox? We’ve just added an expand scenario modal! You can now easily make changes to scenarios with longer inputs.

Code Execution Logs

March 6th, 2024

You can now use print or console.log statements in code execution nodes and view the logs by looking at a node’s result and clicking the logs tab.

We’ve also added logs for eval metrics. You can view them by enabling the logs column in the table columns settings.

Code Execution Improvements

March 6th, 2024

We’ve made some huge improvements to code execution! You can now include custom packages for code execution workflow nodes and code execution eval metrics. On top of this, we have added support for TypeScript. You can select the programming language you want from the new “Runtime” dropdown.

We have also introduced a few smaller improvements:

• The maximum size for code input values has been increased to 10mb, a significant leap from the previous cap of 128k characters
• The layout of the workflow code execution node editor has been revamped with a new side by side layout
• All Vellum input types are now supported for code execution node input variables
• Line numbers in the code editor will no longer be squished together

Claude 3

March 5th, 2024

Anthropic’s two newest models, Claude 3 Opus and Claude 3 Sonnet, are now both available in Vellum! These models have been added to all workspaces so they should be selectable from prompt sandboxes upon refresh.

In-App Support Now Accessed via “Get Help” Button

March 5th, 2024

It used to be that the In-App Support Widget we showed in the bottom right corner of the screen would get in the way of other actions like Save buttons. Now, that widget is hidden by default and you can open it by clicking the “Get Help” button in the side navigation. Once opened, we also now display bookmarked links to useful resources like the Vellum Help Docs.

Workflow Error Nodes

March 4th, 2024

It’s now possible to terminal a Workflow and raise an error through the use of Error Nodes. You can either re-raise an error from an upstream node, or construct and raise a custom error message.

Retrieve Workflow Deployment API

March 1st, 2024

We’ve exposed a new API endpoint to retrieve details of a Workflow Deployment. This is useful if you want to do things like programmatically detect if a Workflow Deployment with a specific name exists, or has the inputs/outputs you expect. You can find the details of the API here.