Changelog | July, 2024

Metadata Filtering in Search Nodes

July 31st, 2024

For a while now you’ve been able to supply structured JSON metadata alongside Documents and then filtering on that metadata when making an API call to search across Documents in a Document index (see here for more info).

However, Search Nodes within Workflows didn’t offer this same functionality through the UI. The workaround has been to use a Code Node or API Node and invoke Vellum’s Search API manually.

We’re happy to share that the UI has reached parity with the API and you can now filter on metadata natively in Search Nodes. You’ll be able to construct arbitrarily complex boolean logic using the new Metadata Filters section of the Search Node’s Advanced settings.

Test Suite Test Case External IDs

July 30th, 2024

We’ve added a new feature to Test Suites that allows you to optionally assign an external ID to each Test Case. This is useful if you track your Test Cases in an external system and you want to periodically sync them with Vellum. You assign an external ID to each Test Case upon creation and then later upsert Test Cases to that Test Suite, keying off of the external ID.

Index Page Sorting

July 30th, 2024

We’ve added another quality-of-life improvement for the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. You’ll now see a “Sort by” dropdown next to the other page-level controls. You can now sort both folders’ and files’ by created date, modified date, and label. If there are other sort fields that you’d find useful, please let us know!

Auto-Conversion to Variable Chips on Paste

July 30, 2024

Building on our recent update that introduced Prompt Variable Chips, we’ve improved the experience by adding support for copy/pasting variables across blocks of different types. Now, when you copy text that includes a {{ my_var }} variable reference from a Jinja block and paste it into a Rich Text block, it’s seamlessly converted into a variable chip.

Google Vertex AI Support

July 29th, 2024

We now support Google Vertex AI models. Previously you could only use Google AI Studio for using Google’s models. You can add them to your workspace from the models page.

Expandable Meta Params in Retrieve Provider Payload Endpoint

July 26th, 2024

For a while now we’ve had an API for compiling a Prompt and retrieving the exact payload that Vellum would send to a model provider on your behalf. We now support a new parameter in this API – expand_meta. With expand_meta, you can opt-in to return additional metadata relating to the compiled prompt payload. Learn more about which fields are expandable in our API docs here.

This new field is available in our SDKs starting v0.7.3.

Prompt Node Usage in Workflows

July 25th, 2024

You can now see token counts and other usage metrics appear in Prompt Node results when invoking Workflows in the Workflow Sandbox:

This setting is now on by default, but can be toggled off in the Workflow Builder Settings.

You can also now return usage data when invoking a Workflow Deployment via API, by passing in True to the expand_meta.usage parameter on either Execute Workflow endpoints.

1
stream = client.execute_workflow_stream(
2
  workflow_deployment_name="demo",
3
  inputs=[
4
    WorkflowRequestInputRequest_String(
5
      type="STRING",
6
      name="foo",
7
      value="bar",
8
    ),
9
  ],
10
  event_types=["WORKFLOW", "NODE"],
11
  expand_meta=WorkflowExpandMetaRequest(
12
    usage=True
13
  )
14
)
15
16
for event in stream:
17
  if event.type == "NODE" and event.data.state == "FULFILLED":
18
    node_result_data = event.data.data
19
    if node_result_data and node_result_data.type == "PROMPT":
20
      print(node_result_data.data.execution_meta.usage)

Enable/Disable All Workflow Node Mocks

July 25th, 2024

Mocking Prompt Nodes helps to save token usage and time when developing the later stages of your Workflow. However, once the Workflow is in a good state, it’s often useful to run the full Workflow end-to-end without mocks to make sure it all comes together. Previously, you had to enable/disable each mock individually. Now, beneath the scenario inputs there is a toggle that allows you to enable/disable all mocks in a workflow at once.

Support for Bulk Upserting Test Suite Test Cases via API

July 24th, 2024

For a while now we’ve had an API for creating, replacing, and deleting Test Cases in a Test Suite in bulk. We now support a fourth operation in this API – upsert. With upsert, you can provide an external_id and a Test Case payload. If there is already a Test Case with that external_id, it’ll be replaced. Otherwise, it’ll be created.

This new operation is available in our SDKs starting v0.6.12.

Llama 3.1 on Groq

July 23rd, 2024

Meta’s newest Llama 3.1 models are now available in Vellum through our Groq integration!

Deployed Prompt Variant Display

July 19th, 2024

When on the Prompt Deployment Overview page, you can now see the name of the Prompt Variant that’s been deployed. This is useful if your Prompt Sandbox has multiple Prompt Variants that you were comparing against one another and you’re not sure which one is currently deployed.

Improvements to Prompt Chat History Variables

July 18th, 2024

It used to be that Prompts that accepted a dynamic Chat History required an input variable whose name was specifically $chat_history. This nomenclature caused frequent confusion and was a bit cumbersome to work with.

Now, you can name Chat History input variables whatever you want and even rename them after-the-fact. As part of this, we’ve also centralized input variable definitions so that whether you want to create a String variable or a Chat History variable, you can do so via the “Add” button in the “Input Variables” section of the Prompt Editor.

Copyable Text to Clipboard

July 18th, 2024

We’ve introduced the ability to copy Prompt Variant IDs, Document Indexes, Models, Workflow Deployment Names and IDs, Document Keys, and Prompt Deployment Names and IDs to clipboard. This feature comes with an enhanced UI with intuitive indicators and tooltips for copyable fields.

GPT-4o Mini

July 18th, 2024

OpenAI’s newest GPT-4o Mini models gpt-4o-mini & gpt-4o-mini-2024-07-18 are now available in Vellum and have been added to all workspaces!

Prompt Variable Chips

July 18th, 2024

It used to be that any time you wanted to reference a variable in a Prompt, you did so using {{ myVariable }} syntax. While powerful if you need to use more complex Jinja templating syntax, using double-curlies for simple variable substitution can be a bit cumbersome.

1. They are harder to visually parse from the rest of your Prompt
2. They can get confusing when dealing with json, which also uses double-curly brackets
3. Whenever you rename a variable, you need to hunt down its usages.

To make this easier, we’ve introduced a new way to reference variables in Prompts: Variable Chips. Variable Chips are small, clickable chips that you can reference in your Prompt text. You can add them by beginning to type {{ or by typing /. Renaming a variable automatically renames all of its references.

Variable chips can be used in the new “Rich Text” block type. New Prompt blocks will default to Rich Text, but you can change existing blocks to Rich Text by clicking the block type dropdown in the block’s toolbar and converting from Jinja to Rich Text and vice versa.

Check out a full video demo here:

New Layout for Sandbox Evaluations

July 17th, 2024

Previously, when a Prompt/Workflow had multiple Test Suites associated with it, we’d shown them all on the page at once. This made navigation difficult (you had to scroll up and down to see each) and could also lead to performance issues. We’ve addressed these issues updating the page layout to display just one Test Suite at a time with a searchable select input that allows you to easily load and view each table one at a time.

New “Add Document to Document Index” API

July 16th, 2024

We’ve introduced a new API for adding previously uploaded Documents to a Document Index. This API is useful when you have a Document that had previously been added to one Document Index and you want to add it to another without having to re-upload its contents altogether. It’s available in our SDKs beginning version 0.6.10. You can find docs for this new API here.

Prompt Deployment Executions Table Improvements

July 12th, 2024

We’ve made several quality-of-life enhancements to the Prompt Deployment Executions table, simplifying the process of adding and editing ‘Desired Output’ values. The entire table has been updated to align with the design of our other tables, such as Evaluations, ensuring a familiar editing experience. Additionally, it is now easier than ever to expand/collapse and copy values.

Moreover, we’ve significantly improved the consistency and usability of the ‘Quality’ column. You can now edit quality ratings with a single click, and the ‘Desired Output’ column will automatically update to reflect your rating where applicable.

Constant Values in Workflow Node Inputs

July 11th, 2024

It’s often the case that you might want to specify a constant value as a Workflow Node Input, either as the input’s primary value or as its fallback value. The solution up until now was to specify a Templating Node, have it output a constant value, and then feed its output to the downstream Node.

Today, we are releasing the ability to inline constant values directly within Workflow Node inputs! First, start typing in the Node Input until the no options modal shows:

A modal will appear to specify your value:

Upon confirming, Vellum will use an icon to denote that the input value represents a constant. As part of this work, we also added icons for all other Node Input types:

Note that constant values will always drop to the last fallback option of a given Node input, and there can only be maximum one constant defined per input. This is due to the nature fallback values – fallbacks are only used if other values aren’t available (i.e. the node that produced the value hadn’t executed yet). In the case of constants, their values are always present.

Test Case CSV Upload in Evaluation Reports

July 9th, 2024

We’ve introduced the ability to upload Test Cases to a Test Suite directly from within the Evaluations tab of a Prompt or Workflow. Now, you’ll find an “Upload Test Cases” button in the table header of every Evaluations table, for both Workflows and Prompt Sandboxes whereas previously, you needed to first navigate to the Test Suite itself and upload from there.

Index Page List View

July 3rd, 2024

We’ve introduced a list-view toggle to the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. Your preferred view will be saved automatically by entity type, allowing you to, for instance, default to list view for Prompts and grid view for Documents.

Collapsible Index Page Sections

July 2nd, 2024

You can now collapse sections on the index/file browser pages for Prompts, Documents, Test Suites, and Workflows. Simply click the heading of any section to toggle the visibility of all folders and items within that section.