Accessing Salesforce Data with Direct Data in Recipes

In Winter ’22, we introduced Direct Data for recipes, a new way to pull Salesforce object data into data prep recipes. The first thing to do when building a recipe is to bring in data. Before Winter ’22, you could either use a connected object or an existing dataset as an input data source.

• A connected object is created by extracting data from an input connection. This data sync can be configured and scheduled from the Connect page.
• A dataset is typically the output of a recipe, or the result of a manual upload.

With Direct Data, your recipe has access to all your Salesforce objects, not just the ones that you have configured for data sync. You can now use these objects directly as input data sources. When the recipe runs, it pulls the latest data directly from the object.

This comes in handy in certain situations:

• Freshness: It’s important that you see the most recent data when building or running a recipe.
• Convenience: Direct Data is a convenient way to try out an object inside a recipe before getting into the nitty-gritty of configuring data sync for it. You can always come back later to set up data sync after you’ve finalized your recipe.

Alright then! Let us see how you can use this feature in your recipes.

Getting Started

In this section, I’ll walk you through how you can pull Direct Data in your recipes, and how you can add filters to the data being pulled.

Add Direct Data

In the Add Input Data screen, there is an additional option for displaying direct data objects.

After selecting Direct Data, the list shows objects that you can bring into your recipe. The icon with a lightning bolt () indicates that it is a direct data object and will pull the data directly from Salesforce. Direct data objects can be used just as any other input data object is used. For example, you can click on the node to view the data preview for it.

Direct Data Filters

If you don’t need all of a direct data object’s records pulled into your recipe, you can speed up execution by using a direct data filter. A direct data filter reduces the number of records pulled into the recipe. Instead of pulling in all records for an object and then filtering them, you can use a direct data filter to avoid the time it takes to transfer the records you do not need.

To create a direct data filter, switch to the Filter tab and click the “Filter” button. If you add more than one filter condition, they all must be satisfied for a record to be included in the recipe.

When clicking Replace to select a new input data source, all direct data filters are removed.

Note: You can click Undo to bring back the filters in case you didn’t mean to remove them.

Direct Data – Connected Object Dynamics

In this section, I’ll answer the important question – “When to use what?”, some key considerations, and how you can make the switch between the two.

Direct Data

If you decide that you want to have the freshest data for an object, you can use Direct Data for this. Now, a particular object can either be used as Direct Data object or as a connected object. So, remove the relevant object(s) from data sync. When you disconnect an object from data sync, it automatically becomes a data direct object in recipes that use it. Future recipe runs will pull data directly for the object.

If the object has a filter node following it, it still appears as a separate filter node, but it will be converted into a direct data filter when the recipe executes (as if it appeared under the Filter tab of the input node).

Connected Object

Each time a recipe using direct data executes, it pulls data from a direct data object. If you have multiple recipes doing the same, or the size of the object is large, then you may see some increase in the execution time. In such scenarios, you can use data sync so that you pull data only one time for use in your recipes.

You can convert a direct data object into a connected object by following the steps:

1. Go to the Connect page
2. Click Connect to Data
3. Navigate to the object you want to convert
4. Select the columns you want to include in the connected object. 

After running data sync on this object, it will automatically appear as a connected object in the recipe. You do not need to replace the input source in the recipe. Future recipe runs will use the replicated data for that object. Always remember to schedule data sync to pull fresh data before your recipes need it.

Note: If you missed adding a column to sync, and a recipe references that field, it will be resolved the next time the recipe executes. The missing field will be added to data sync, data sync runs, and then the recipe executes.

After converting an object with a direct data filter, and you return to the recipe, you’ll see that the filter is converted into a standard filter node in the recipe. It no longer appears in the Filter tab of the input node.

Connected Objects: All Fields

When you select a Salesforce connected object in the Add Input Data screen, you now have access to select all its fields, both synced and unsynced. Unsynced fields are marked with an unlinked icon (). When you bring an unsynced field into the recipe, the data previews are populated by pulling data directly from Salesforce. When the recipe runs, the unsynced fields are added to data sync, data sync runs, and then the recipe executes.

Wrap Up

Direct Data simplifies the way you bring Salesforce data into your recipes. You have access to all your Salesforce objects and fields right away. It’s great for accessing small objects and ensuring that you always have the latest data. You can convert to a connected object in situations where the object is large or is used in multiple recipes. You now have the flexibility to decide the best way to bring in your data.

Visit the docs for more on Direct Data.

Written by