Restructure payload with a preSavePage hook

Stephen Brandt

Updated September 30, 2024 14:36

Your flow will generate an error if any page of records to be processed exceeds the 5 MB limit; needless to say, a single record must be smaller than 5 MB.

The example script below demonstrates how to restructure a large record to reduce the exported payload size and successfully send it downstream to the next flow step.

Real-world data with a massive record

Picture the following record, extracted from an endpoint, in which each order contains a large amount of serial information. The payload size could be so large that a single order exceeds 5 MB because the serials array may contain up to 1,000 objects.

{
  "orderNumber": "100001",
  "shipItems": [
    {
      "lineItemNumber": 1,
      "customerLineId": "",
      "partNumber": "S120006",
      "quantity": 100000,
      "serials": [
        {
          "serialNumber1": "391000",
          "serialNumber2": "6214M1"
        },
        {
          "serialNumber1": "391100",
          "serialNumber2": "6214M1"
        },

        . . .
        {
          "serialNumber1": "390900",
          "serialNumber2": "6214M13"
        }
      ],
      "ssccSerials": []
    }
  ]
}

Sample record

For illustration purposes, consider the following input data, which contains a single record with multiple items in serials[]:

{
  "order": "12345",
  "serials": [
    {
      "_id": 47050,
      "role": "TA",
      "firstname": "Karolina",
      "lastname": "Jacqui",
      "age": 81
    },
    {
      "_id": 85855,
      "role": "Student",
      "firstname": "Frank",
      "lastname": "Holbrook",
      "age": 97
    },
    {
      "_id": 26805,
      "role": "TA",
      "firstname": "Gabriellia",
      "lastname": "Freddi",
      "age": 56
    },
    {
      "_id": 10182,
      "role": "Student",
      "firstname": "Helena",
      "lastname": "Pearse",
      "age": 56
    }
  ]
}

Restructure the record in a preSavePage hook

In Flow Builder , click the Define options (+) button on the export and then click the Hooks () button to add JavaScript code. The Hooks panel opens.

In the Pre save page section, click the Create script (+) button. Name the new script and add the following function:

function preSavePage (options) {
  let tempRecords =[];
  let eachRecord;
  
  for (var i=0 ; i < options.data[0].serials.length; i++){
    eachRecord = {
      "record1": options.data[0].order,
      "_id": options.data[0].serials[i]._id,
      "role": options.data[0].serials[i].role,
      "firstname": options.data[0].serials[i].firstname,
      "lastname" : options.data[0].serials[i].lastname,
      "age" : options.data[0].serials[i].age
    }
    tempRecords.push(eachRecord);
  }
  options.data = tempRecords;
  return {
    data: options.data,
    errors: options.errors,
    abort: false,
    newErrorsAndRetryData: []
  }
}

Click Save, and enter the Function name, preSavePage , into the Hooks panel as the Pre save entry point.
Save & close the import Hooks.

In essence, this script “flattens” the single serials[] object into multiple smaller records containing each child item as a new record. It might be useful, for example, if you’re importing each serial information into a custom record in NetSuite.

Try it out

Return to the export’s hooks, and click the Edit () button to open the preSavePage function in the Script editor.
Paste the sample record, above, into the Function input pane, as shown below:
Click Preview to see the restructured records.

Adjust the settings and run the flow

For this example, set the Page size setting of the export to 100, which tells integrator.io to group 100 records into one page of data to be processed by the next step.

Note that this page size is a maximum value. integrator.io will automatically ensure that the page to be created is less than 5 MB. Suppose that the 81^st record would make the page larger than 5 MB, then that page would contain only 80 records.

When you run the flow, the Run console shows that the flow has already exported 38,900 records chunked into 389 pages (100 records/page):