By Pablo Gonzalez in CICD — Mar 23, 2023

Salesforce Continuous Integration—Concepts, components, and challenges

This site is currently dedicated to discussing Salesforce CI/CD (Continous Integration and Continous Delivery), yet ironically, I've never written about what CI actually is, so let's fix that.

Concepts

Let's first discuss some basic concepts of CI, and then we'll see how they apply to Salesforce DevOps

Origin of the term

The first thing to know is that this is not a Salesforce-specific term; it's a general software development concept.

Some time ago, software developers used to work in isolation; each building features in their computers without much knowledge about what their colleagues were building.

Towards the end of the project, there would be an integration phase where everyone's code would be integrated and tested. As you can imagine, this almost never went well, and more often than not, the integration phase was longer than any other phase in the project.

This problem can easily be replicated in Salesforce.

Imagine you are working on a bunch of flows for 2 months while your colleague is working on a data validation framework based on validation rules, required fields, etc.

And let's imagine you are doing this in separate sandboxes. And now, let's imagine that 2 months later, you deploy both frameworks to a UAT sandbox.

The most likely outcome is that many of those flows you built won't work because they'll hit many of the validation rules your colleague created. Now you both need to reverse engineer everything and make sure it all works together. This is the dreaded integration phase that has plagued software teams for many years.

So in the 90s, a prominent programmer called Kent Beck wrote a book called Extreme Programming, and in this book, he coined the term Continuous Integration:

Code is integrated and tested after a few hours—a day of development at most

So basically, he was advocating the idea of not waiting 2 months to integrate and test your changes; instead, you do it every day, multiple times a day if possible.

So this leads us to the next concept.

What is Continous Integration

Now that we understand the problem CI aims to solve, we can define it as follows:

✅

A software development practice in which developers integrate and test their changes as frequently as possible. This helps catch conflicts between separate streams of work as quickly as possible.

Before we explain how CI actually works, let's explore a high-level example, going back to our scenario earlier.

With CI, you and your colleague wouldn't wait 2 months before deploying the flows and data validation frameworks to UAT. Instead, you'd deploy your changes as you go, as many times as possible, to an intermediate sandbox, and there, you would ensure that your logic still works.

You do this multiple times a day until the features are complete.

So, it's called Continuous Integration because you are integrating your changes continuously, as opposed to at the end of the project.

It's called Continuous Integration because you are integrating your changes continuously, as opposed to at the end of the project.

Continuous Integration Flow

Before I explain the individual components of CI, it's helpful to see a high-level flow; otherwise, you could get lost in the details.

⚠️

The sections below assume some knowledge about Git and Git branches.

Let's look at this image:

Here's how CI works:

1- You make changes to your sandbox in relation to 1 user story. No more than 1.

2- You commit those changes to a shared Git repository.

3- A special piece of software known as a CI server picks up the change and starts an sfdx deployment against another developer sandbox called Integration.

4- If the deployment succeeds, great! The first part of the continuous integration phase is done:

✅

You've confirmed your changes are deployable

5- You log in to the sandbox and test that your flow works as expected. If it does, the 2nd part of the continuous integration phase is done:

✅

You've confirmed your changes work as expected in a production-like environment, which has other people's changes in it.

Why does it have other people's changes in it?

Because your colleague is doing the exact same thing! That is key to CI.

CI only works if everyone is following the same process. If step 4 or 5 fails, you would know that something about your changes doesn't play along with your colleagues' changes, and you need to work together to fix that right now, not before deploying to production.

You can also check out this video where I show a similar flow:

Components

Now that we know what CI looks like, let's explore the individual components in a bit more detail.

Version Control (Git Repository)

For CI to be possible, you have to keep your sfdx project in version control with Git (there is other version control software, but Git is by far the most widely used). Here's an example of what it looks like to have an sfdx project in GitHub

This repository was created by downloading all the metadata of my production org into an sfdx project and uploading it to GitHub

And so, this is where things start to get a little confusing.

When you look at tutorials on using Git, it's all about tracking changes in your codebase and pushing changes to GitHub. It's not straightforward how this is connected to CI.

To see how Git fits into CI, let's go back to our previous scenario and see what it would look like with CI.

1- Before you start working on a new Flow, you git clone the sfdx project into VS Code. You then use this project to authenticate against your developer sandbox.

2- You create the flow and make other changes in the Salesforce UI.

3- Back to VS Code, you create a feature branch called feature/flow-changes. Remember that a feature branch is a clone of the original Git repository, but now you will add your changes to it.

4- You use the sfdx force:source:retrieve (or pull if using source tracking) command to retrieve the flow and other changes.

5- The downloaded files change your sfdx project; it has more files now. So you commit this change with git commit -m "added a new flow for sales" , and you push this feature branch to GitHub using git push

6- When the your feature branch is in GitHub, you can create a merge request (also known as a pull request or PR) to merge your feature branch into the integration branch

And when that PR is created, the CI server starts doing its magic. Which takes us to the next section.

Continuous Integration Server

A CI server is specialized software that is connected to your Git repo. It listens to events that are happening in the repo, such as push, merge, etc.

Typical CI servers are GitHub Actions, BitBucket Pipelines, Circle CI, etc.

And we can automate actions in response to those events.

So, when the PR is created, the CI server is going to do the following:

1- Create a virtual machine on demand in the cloud

2- Install the Salesforce CLI and other dependencies

3- Do a git clone of your feature branch

4- Deploy the branch to the Integration sandbox using sfdx force:source:deploy

5- Run all tests

If there are any errors during steps 4 and 5, you'll know that either

🛑

Your change is not deployable

🛑

Your logic does not work alongside changes made by other colleagues

Build Script

Let's revisit the image above

The build script is basically what I described earlier, the CI server runs a script, and in that script, we define what should happen.

And here, we have an opportunity to automate anything we want, not just a deployment against the integration sandbox.

The script is typically written in YAML, here's what an example script looks like:

- name: "Scan code"
        run: |
          cd changed-sources
          sfdx scanner:run --format sarif --target './**/*.cls' --category "Design,Best Practices,Performance" --outfile 'apexScanResults.sarif'  
          cd ..
      # Now we upload the .sarif file as explained in the previous step
      - name: Upload SARIF file
        uses: github/codeql-action/upload-sarif@v1
        with:
          sarif_file: changed-sources/apexScanResults.sarif

      # We do a check-only deploy and we only run the tests specified in the PR
      # If the env variable does not equal 'all', we know that there is a list of
      # tests that can be run

      - name: "Check-only deploy delta changes - run specified tests"
        if: ${{ env.APEX_TESTS != 'all' }}
        run: |
          echo ${{env.APEX_TESTS}}
          sfdx force:source:deploy -p "changed-sources/force-app" --checkonly --testlevel RunSpecifiedTests --runtests ${{env.APEX_TESTS}} --json

In the build script, we can automate things such as:

Static code analysis
Check-only deployments to another sandbox
Run all tests
Any other custom logic

Tests

The final component of CI is tests. Remember we said that when your changes are deployed to integration, you are supposed to test that they work alongside the other changes your colleagues have deployed in the meantime.

How do you test? In any way you can, such as:

Apex tests
Manual tests done by a QA person
Automated tests through specialised software such as Provar Testing

Whatever way you test, know this:

🛑

If you can't test your changes, you are not practicing CI

Why not? Because the whole purpose of CI is to confirm that your changes work alongside other changes that are in progress.

If you cannot test, then you are not doing CI; you are just automatically deploying to another sandbox.

This then takes us to the final section of this article.

Challenges

We now understand that CI in Salesforce is about testing that your changes work alongside other changes, and you do this during development, not one day before deploying to production :)

This sounds all good and well, but it's not without its challenges.

Continuous Integration can be slow

This paradigm is basically asking us to deploy changes multiple times a day and run tests!

I've worked in orgs where running all tests can take half a day. In such an org, I can't imagine trying to deploy multiple times a day and running all tests. I would probably do it only when I know my changes are done, and not while they are in progress.

The slowness could be due to Salesforce being slow, another deployment already in the queue, or your apex tests are slow to complete.

You need really good tests

As I said earlier, your tests should confirm that the functionality works as expected. If your tests are not really testing the right things, you are getting a false sense of confidence.

You think your changes work fine alongside other changes, but that's only because your testing is not testing the right things.

And you'll likely only find out once you deploy to production and people start using your feature.

Not everything can be tested

When Kent Beck coined this term, he wasn't thinking about admins creating flows, validation rules, or sharing rules via the UI. He was writing code all day.

So what about things that cannot be natively tested? How do you test that a validation rule works, or how do you test that a user has only access to the reports they should have access to?

For this kind of thing, you'll need specialized software that can test areas that Apex tests won't cover. Two products that I know can help are Provar Testing and Testim.io

Where to learn more

Continuous Integration in Salesforce is a large topic, and this article didn't cover half the things we could talk about.

Here are some resources you can use to learn more.

Dreamforce '22 Design Patterns for Salesforce CI/CD

CI/CD Checklist for Salesforce Architects

Build your own CI/CD Pipeline from Scratch