Parallel Salesforce metadata retrieval with Promise.all(), array.map() and the Composite API

When creating fullstack Salesforce apps that work with Salesforce metadata, it's common to want to retrieve the metadata of certain records to do some processing, display them in the UI, etc.

For example, I may want to inspect the metadata of a FlexiPage, to see which custom fields are being used by it (since this is not supported by the Dependency API).

Metadata attribute in Tooling API

The Tooling API provides a metadata attribute that is present in most objects. Let's see it in action:

SELECT Metadata FROM FlexiPage LIMIT 1

[
  {
    "attributes": {
      "type": "FlexiPage",
      "url": "/services/data/v55.0/tooling/sobjects/FlexiPage/0M03h000003bedLCAQ"
    },
    "Id": "0M03h000003bedLCAQ",
    "Metadata": {
      "description": null,
      "events": [],
      "flexiPageRegions": [
        ...// a lot of stuff
                "componentName": "flexipage:richText",
                "flexipageDataSources": null,
                "identifier": "flexipage_richText",
                "visibilityRule": {
                  "booleanFilter": null,
                  "criteria": [
                    {
                      "leftValue": "{!Record.CustomerPriority__c}",
                      "operator": "EQUAL",
                      "rightValue": "High"
                    }
                  ]
                }
              },
              "fieldInstance": null
            }
         //more stuff
      ],
      "masterLabel": "Account Record Page",
      //more stuff here
    }
  }
]

This is great. Now, I can see that the CustomerPriority__c field is being used in a visibility rule

"criteria": [
    {
    "leftValue": "{!Record.CustomerPriority__c}",
    "operator": "EQUAL",
    "rightValue": "High"
    }
 ]

Limitations

The problem is this metadata attribute is only available if you limit your SOQL query to return only one result. So, the below query fails

SELECT Id, metadata FROM FlexiPage

MALFORMED_QUERY: When retrieving results with Metadata or FullName fields, the query qualificatio​ns must specify no more than one row for retrieval. Result size: 11

This renders this functionality almost useless. If an org has 50 FlexiPages, I need to issue 50 distinct queries, which means 50 calls to the Tooling API.

So, how can we optimize this?

Optimization with Composite API

Turns out, that if you query any field of a tooling API object, you also get back a URL that points to that record

"attributes": {
      "type": "FlexiPage",
      "url": "/services/data/v55.0/tooling/sobjects/FlexiPage/0M03h000000fopJCAQ"
}

You can take this URL and hit it with a GET request on workbench or postman, and you get the entire metadata of the record, just as if you had queried the metadata attribute.

So what we can do is use this URL in a Composite Request of the Tooling API. I'll spare you the details because you can read the docs, but basically, you are able to pass 25 URLs in a single request object, and make one API call to get the information of those URLs.

So if we have 100 FlexiPages, we can query the Id of all of them with a simple query. The result will have 100 URLs.

 let query = `SELECT Id FROM FlexiPage WHERE EntityDefinitionId = '${edf.entityDefinitionId}'`
 let soql = {query,filterById:false,useToolingApi:true};

let rawResults = await restApi.query(soql);

We can then take those 100 URLs, group them in batches of 25, and make each batch a composite API request.

 let endpoint = `${connection.url}${restEndpoint}tooling`;

let subRequests = metadata.map(md => { 

    let request = { 
        method:'GET',
        url:`${restEndpoint}tooling/sobjects/${md.type}/${md.id}`, 
        referenceId:md.id 
    }
    return request;
});

//max number of subrequest per composite request
let batches = splitInBatchesOf(subRequests,25);

Finally, we can hit the composite endpoint with a single API call, where each API call will return the metadata of 25 records.

let data = await Promise.all(

    compositeRequests.map(async (request) => {

        let fetchOptions = getFetchOptions(connection.token,'POST');
        fetchOptions.body = JSON.stringify(request);

        try {

            let res = await fetch(compositeEndpoint,fetchOptions);
            let json = await res.json();

            return json;

        } catch (error) {
            //do nothing, we'll process the other requests, this is a partial success operation
        }
    })
)

Promise.all

Note that I'm using Promise.all to send the requests in parallel, rather than one by one. This means, we send 4 API calls to salesforce at the same time, where each call has 25 URLs, resulting in one hundred metadata objects.

Now, in reality, it wouldn't matter if I make 4 API calls one by one because the Promise.all call is inside an await function, which means no processing will be done until all the API calls return. Still, this results in easier-to-read code I think.

Array.map()

Very important to note that inside Promise.all, I'm using Array.map() as follows

compositeRequests.map(async (request) => {

This is because it at the time of me writing the code, it wasn't possible to use async inside a for loop. The correct pattern was to use Array.map() to return a new array of Promises, and then execute all of them inside Promise.all.

There's is an await...of iterator now that may resolve this, but I'll cover that in a future issue.

Recap

To retrieve large sets of metadata using the Tooling API

1. Query the Ids of the records in question (which will return a URL)
2. Group those URLs in batches of 25, where each batch is a Composite Request
3. Send all the composite requests in a single API call, using Promise.all() and Array.map()

Much better than hitting the Org API limit, no? ;)