Mastering Second Generation Packaging in Data Cloud: A Comprehensive Guide

Are you ready to dive into the world of packaging on the Salesforce platform? In this comprehensive guide, we’ll walk you through the process of creating a second-generation package (2GP) in Data Cloud, step by step. But before we delve into the nitty-gritty details, let’s first understand what second-generation packaging is and why it’s considered a game-changer in the world of Salesforce development.

Understanding Second Generation Packaging (2GP)

Second-generation packaging, or 2GP, represents a significant evolution in the way packages are built and managed on the Salesforce platform. Unlike its predecessor, first-generation packaging (1GP), 2GP introduces a more modern and modular approach to packaging applications.

Advantages of Second Generation Packaging

So, what makes 2GP stand out from 1GP? Here are some key advantages:

• Enhanced Modularity: 2GP allows developers to package modular components of their applications, making it easier to organize and distribute functionality. This modular approach promotes code reuse and simplifies the management of dependencies.
• Improved Version Control: With 2GP, versioning becomes more flexible and granular. Developers can create multiple versions of their packages and easily manage dependencies between them, ensuring smooth upgrades and rollbacks.
• Streamlined Development Workflow: 2GP integrates seamlessly with Salesforce DX (SFDX), enabling a modern development workflow with features such as scratch orgs and version control systems. This streamlined workflow accelerates development cycles and improves collaboration among team members.
• Better Performance and Governance: By leveraging the latest Salesforce infrastructure and best practices, 2GP packages offer improved performance and governance compared to 1GP packages. This ensures that your applications meet the scalability and security requirements of enterprise-grade deployments.

Understanding Data Kits in Salesforce Data Cloud

In the realm of Salesforce Data Cloud, Data Kits play a crucial role in organizing and managing metadata & configuration for various applications and use cases. Let’s explore what Data Kits are and how they can empower partners, developers and administrators to harness the power of the data cloud platform, seamlessly package metadata & customizations and create end to end solutions/apps. 

What are Data Kits?

A Data Kit is “A portable & customized bundle of independently package-able Data Cloud metadata in the form of  templates, which can be created easily from the UI without writing any line of code and can be deployed into a subscriber org multiple times for different dataspaces”

Datakits do not store the actual metadata but packages the metadata in the form of templates. No feature in Data Cloud should be packaged directly but only via Data Kits.

Benefits of Data Kits

Datakits present a streamlined approach for partners or developers in packaging data cloud metadata, providing the following advantages:

• Reusability of schema through templates.
• Support for deployment to multiple data spaces.
• Enhanced flexibility: Template updates occur during package upgrades instead of altering the actual runtime metadata, allowing customers to retain unchanged elements if not immediately required.

With Datakits, ISVs gain the capability to develop exhaustive end-to-end Data Cloud solutions and selectively deploy metadata into a customer’s org.

Features supported in Data Kits

• Data streams and Bundles
  • CRM datastreams
  • S3 Datastreams
  • Web/Mobile sdk 
  • Ingestion API
• Data Model, Mappings and Relationships
• Data Lake objects
• Streaming and Batch Transforms
• Calculated Insights
• Connector Framework.

Steps to Create a 2GP Package in Data Cloud

Now that we’ve covered the basics of 2GP, its advantages over 1GP, and Data Kits, let’s dive into the practical steps involved in creating a second-generation package in Data Cloud.

1. Setting Up Your Developer Environment

The first step in creating a second-generation package is setting up your developer environment. You’ll need to obtain a Developer Edition org where you can define a unique namespace for your package. This namespace will be used for all managed packages created using second-generation packaging.

To define a namespace, go to Setup > Quick Find Box -> Search for “Package Manager”, Click on “Edit” under namespace settings, and enter a unique namespace.

2. Configuring Your DEV Hub Org

After you have defined your namespace in the developer edition org, you need one more org (eg. Partner Business Org), which you will designate as your DEV Hub org. A DEV hub org lets you create & manage your scratch orgs. 

A scratch org is a source-driven and disposable deployment of Salesforce code and metadata, made for developers and automation (CI/CD). A scratch org is fully configurable, allowing developers to emulate different Salesforce editions with different features and preferences. 

You need to request via the salesforce partner community to enable data cloud scratch orgs for your Dev Hub Org. 

In the Developer Edition org, go to Setup -> DevHub and set “Enable Deb Hub” to True.

On the same page, set the “Enable Unlocked Packages and Second-Generation Managed Packages” to True.

3. Linking Your Namespace

Next, you’ll need to link the namespace you defined in your first Developer Edition org to your DEV Hub org. This can be done through the App Launcher by accessing Namespace Registries and clicking on the “Link Namespace” button.

After you click on “Link Namespace”, you need to login to your first developer edition org where you had defined your namespace. After successful authentication, the namespace will be linked to your dev hub org. 

4. Creating an SFDX Project

With your environment configured, it’s time to create an SFDX project. Before creating the SFDX project, ensure the following prerequisites are met:

1. Download and install VScode
2. Install Salesforce DX CLI
3. Install Salesforce Extension Pack for Visual Studio Code

You can then create a project with the standard SFDX folder structure, ensuring to specify the linked namespace and DEV Hub org in the SFDX project configuration files.

In the sfdx-project json file, specify the namespace which you had defined in your first developer edition org. 

Also, make sure the sfdcLoginURL is set to https://login.salesforce.com.

In the .forceignore file, You need to add the following: **ssot**. SSOT namespace is managed internally by Salesforce and this should be mentioned in the forceignore file to ensure unnecessary files are not retrieved.

Create a package.xml file with the following configuration, if you wish to add datastreams, data models, mappings and calculated insights to your package. 

Since we plan to support more features to Data Kit in the future, always refer to the salesforce documentation of second generation packaging for Data Cloud for the exact metadata components that should be in your package.xml file.

Note: The documentation of second generation packaging for Data Cloud will be published soon.

<?xml version="1.0" encoding="UTF-8"?>
<Package xmlns="http://soap.sforce.com/2006/04/metadata">   
   <types>
       <members>*</members>
       <name>DataSource</name>
   </types>
   <types>
       <members>*</members>
       <name>DataSrcDataModelFieldMap</name>
   </types>
   <types>
       <members>*</members>
       <name>DataSourceBundleDefinition</name>
   </types>
   <types>
       <members>*</members>
       <name>DataSourceObject</name>
   </types>
   <types>
       <members>*</members>
       <name>DataStreamTemplate</name>
   </types>
   <types>
       <members>*</members>
       <name>DataPackageKitObject</name>
   </types>
   <types>
       <members>*</members>
       <name>DataPackageKitDefinition</name>
   </types>
   <types>
       <members>*</members>
       <name>DataCalcInsightTemplate</name>
   </types>
   <version>59.0</version>
</Package>

5. Preparing the proj-scratch-def.json file

In order to create a scratch org with Data cloud, you need to use the feature “CustomerDataPlatform” in the proj-scratch-def.json file.

Below is a sample “proj-scratch-def.json file that can be used.

{
  "orgName": "jhaydraude company",
  "edition": "Developer",
  "features": ["EnableSetPasswordInApi","CustomerDataPlatform","MarketingUser"],
  "settings": {
    "lightningExperienceSettings": {
      "enableS1DesktopEnabled": true
    },
    "mobileSettings": {
      "enableS1EncryptedStoragePref2": false
    },
    "customerDataPlatformSettings": {
      "enableCustomerDataPlatform": true
      }      
  }
}

Authorize your DEV Hub org by pressing  (Command+Shift+P) on mac and you can see an option to “Authorize a Dev Hub” org.

You will need to log in to your Dev Hub org to successfully authenticate your DEV Hub org. 

Also, please make sure that the Dev Hub org is set as the default org in Visual Studio. The benefit of setting an org as a default org is that you will not have to mention the org alias in your CLI commands. All your commands will run against your default org.

6. Creating Scratch Orgs

Create scratch orgs for your project developers and ensure that Data Cloud is provisioned in these scratch orgs to enable package development.

You can choose the “Create a scratch org” option from the list by typing in “Command+Shift+P” if you are using MacBook, then give an org alias, select the source file as “project-scratch-def.json” file, and enter the duration of your scratch org, which can be between 7 to 30 days only.

Alternatively, you can also run the CLI command to create a scratch org. 

sf org create scratch --definition-file config/project-scratch-def.json --alias MyScratchOrg --set-default --target-dev-hub MyHub

After you have created your scratch org(s), you need to manually provision Data Cloud by going to Data Cloud Setup. If you do not see “Data Cloud Setup”, assign the “Data Cloud Admin” perm set to the logged-in user.

7. Working with Data Kits

Developers can work on individual scratch orgs while collaborating on the same data kit. Ensure that data kit metadata is properly managed and pushed/pulled from scratch orgs to maintain a single source of truth.

After you create data streams, data models, mappings, CI’s, and other metadata in Data Cloud Scratch org. Add it to a datakit.

Then you can retrieve the data kit metadata in your SFDX project.

You need to run the following command to retrieve the Datakit metadata from scratch org. Make sure the scratch org is set as your default org.

Sf project retrieve start –manifest package.xml

After you run the retrieve command, all data kit metadata files will be retrieved in your SFDX project, which can then be uploaded to Github.

Let us assume, there is another developer who is working on the same data kit. He can download the SFDX project from the github repository and then run the following command to deploy the datakit to his scratch org, which is different from the scratch org where the datakit was created. Make sure the scratch org to which the data kit needs to be deployed is set as your default org.

Sf project deploy start –manifest package.xml

After the command is successfully run, you can log in to the scratch org of the second developer and you can see the data kit is available. 

This developer can then continue updating the data kit or add new metadata to the same data kit in his scratch org. If he wants to update a datastream by adding an additional mapping, then he will first need to deploy the datastream from the data kit in his org. Add the mapping. And then update the data kit with the changes made to the datastream.

The below screen shows the process of deploying the data stream from the data kit:

Let us assume that a new datastream is created and added to the data kit as shown in the screens below:

Then the metadata can be retrieved in the SFDX project so that the SFDX project has the latest changes made to the datakit. Then the project can be uploaded back to github.

8. Packaging Your Application

After the data kit is ready to be packaged, you can then create a second-generation package using the following SFDX CLI command.

sf package create –name <PackageAlias> –package-type Managed \ –path “force-app” –target-dev-hub <DevHubAlias>

After the package is successfully created, specify dependency to SSOT package version ID. The SSOT package is managed internally by Salesforce. Without specifying this dependency, you will not be able to create a package version. 

For the latest version ID of the SSOT package, refer to salesforce data cloud 2GP documentation, which will be published soon. 

In order to create a package version, type the following command. Do not forget to include the code coverage parameter, without which you will not be able to promote your package to the released version. The minimum requirement for code coverage for a 2GP package version is 75%.

After you create a package version, it gets created as a beta package which can only be installed on another scratch org for testing purposes. If you want to install the package to a production org, you will need to promote the package to a released version.

After you have created a package version, you will get a package version ID and an installation URL. Your sfdx-project.json file will be updated with the package version ID and package alias. 

You can use the installation URL to install the package to a scratch org. After you have tested the package, you can then promote the package to a released version.

9. Promoting Your Package

Once your package is created, you can promote it to a released version, allowing installation in production orgs.

sf package version promote --package awesome_package_alias --target-dev-hub devhub@example.com

10. Conclusion

Congratulations! You’ve successfully created a second-generation package in Data Cloud. By following these steps, you can streamline your package development process and deliver powerful solutions on the Salesforce Data Cloud platform.

Written by

Jitin Mehndiratta

See author's posts