Import hooks

Hooks are custom code that can run at different stages during the execution of a flow to modify the behavior of export and import processes. A flow in integrator.io consists of a source (export from an app, database, or file server; or a webhook listener) and a destination (import, file transfer, or lookup). In simple terms, an export retrieves the data of the source application and breaks it into smaller pages. An import processes those pages and stores the data in the destination application. You can use different types of hooks at each stage of the integration process to perform advanced functions. 

Note: Hooks typically modify the JSON records in your flow’s data, including any records parsed from a file exported from a server or uploaded in Data Loader. Hooks cannot make changes to a “blob” (file download passed intact within a flow). 

The following table lists the script formats available for each import hook type.

In NetSuite integrations, the script format used for hooks should be the same as the NetSuite API version used in the import.

Contents

• postResponseMap hook
  • postResponseMap hook options argument fields
  • postResponseMap hook examples
  • PostResponseMap scripts on lookups will run per page of data
• Pre map hook
  • Pre map hook options argument fields
  • Pre map hook callback arguments
  • Pre map hook example
• Post map hook
  • Post map hook options argument fields
  • Post map hook callback arguments
  • Post map hook example
• Post submit hook
  • Post submit hook options argument fields
  • Post submit hook callback arguments
  • Post submit hook example
• postAggregate hook
  • postAggregate hook options fields
  • postAggregate hook callback arguments

postResponseMap hook

This hook runs immediately after response mappings and regular mappings. Use this hook to process records after response mapping before they are passed on to downstream applications in your flow. Any changes that you made to records with the postResponseMap hook will affect all downstream applications. You do not need to define a response/results mapping to use this hook. You could also use this hook instead to perform the same logic. For more detailed information about how 'postResponseMap' works, see the help text provided in the default function stub.

Common use cases:

• Error Management: Ignore errors, trigger errors, modify error messages
• Data Logging: Use script logs to record data and events
• Data Processing: Perform calculations and transformations on the raw data returned by the endpoint

postResponseMap hook options argument fields

The function will be passed one ‘options’ argument that has the following fields:

Note: The function needs to return the responseData array provided by options.responseData. The length of the responseData array must remain unchanged. Elements within the responseData array can be modified to enhance error messages, modify the complete _json response data, etc...
Throwing an exception will fail the entire page of records.

postResponseMap hook examples

Example 1

Let’s say you're bringing in customer-generated quotes from an eCommerce website to your ERP system. You'd prefer to automatically create customer records in your ERP as you pull in the quotes, but a lot of the customer data entered in the eCommerce website can be faulty. Instead of failing to import the quotes in such a situation, you could ignore errors related to creating the customer and instead tie the order to a generic customer. In your ERP, perhaps your sales representatives would receive an alert when a quote is tied to this generic customer, and they would reach out to correct the details. You could accomplish this with the following script on your customer import:

function postSubmit (options) {

  options.responseData.map( responseData => 
    processErrors(responseData, options.settings)
  );
  
  return options.responseData;
}

function processErrors(responseData, settings){
  if(hasErrors(responseData)){
    responseData.errors = filterUserErrors(responseData);
    if(!hasErrors(responseData))
      setDefaults(responseData, settings);
  }
}

function hasErrors(responseData){
  return !!responseData.errors.length;
}

function filterUserErrors(responseData){
  return responseData.errors.reduce( (errors, error) =>  {
    if(error.code !== "user_error")
      errors.push(error);
    return errors;
  }, []);  
}

function setDefaults(responseData, settings){
  const id = settings.flow.defaultCustomerId.toString();
  responseData.id = id;
  responseData._json = {id};
  responseData.statusCode = 200;
}

Additional scenarios

If you are importing records into an application, and the application returns complex response data, you can use this hook to get just the fields you want from the response, and then put them in a very simple set of fields in your source record.

If you are merging the results of a lookup back into your record, you can use this hook to only merge specific fields, or filter entries that do not meet a complex criteria.

You can use this hook to perform calculations on values that span all of the records returned by a lookup, and then place the results in a simple field in your record.

You can use this hook to sort the results of a lookup, or group the results of a lookup into one or more new structures in your record.

PostResponseMap scripts on lookups will run per page of data

The behavior of the postResponseMap scripts was different between lookups and imports. The postResponseMap scripts on lookups ran for every record if the one-to-many path was not set and every child record if a one-to-many path was set. With the 2023.9.1 fix, the postResponseMap scripts run per page of data. The below table shows the existing and new behavior of the postResponseMap scripts.

Pre map hook

This hook gets invoked before the fields are mapped to their respective fields in the objects to be imported. This hook can be used to reformat the record's fields before they get mapped.

You can use this hook to apply logic to the data within an import prior to going through the mapping process. Any manipulation of the data in this step only applies to the import that it is running on.

Common use cases:

• Insert default values
• Format data needed for mapping

preMapFunction: The name of the function can be changed to anything you like.The function will be passed an 'options' argument and a callback argument.

Pre map hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "data":[],
    "settings":{},
    "configuration":{}
}

Pre map hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{ 
     data: {}/[], 
     errors: [{
          code: '',
          message: '',
          source: ''
     }] 
}

{
    code: '',
    message: '',
    source: ''
}

{}, 
{
    data: {}
},
{
    data: []
}, 
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}, 
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.preMapFunction= function (options, callback) {
    return callback (err, responseData)
}

Pre map hook examples

Example 1: In this use case, sometimes the export will send the items list across as either an array or an object. The mappings require it to be an array so the hook will convert it to an array containing a single object if the data is presented as an object.

function convertObjToArray(options) {
  var elementName = 'items';  //Change item to be the name of the property that holds the items
  for(var i=0; i<options.data.length; i++){
	if(Array.isArray(options.data[i][elementName]))
  	continue;
    
	var arr = [];
	arr[0] = options.data[i][elementName];
	options.data[i][elementName] = arr;
  }
  return options.data.map((d) => {
	return {
  	data: d
	}
  })
}

Example 2: Use Pre map hook to reformat phone numbers in this format:

5555555555

to this format:

(555) 555-5555

The following code iterates over each record in the data[ ] array and concatenates the pieces of the number with the additional characters:

function convertPhoneNumber (options) {
    options.data.forEach(function(i) {
        let tmp = i["Store Phone Number"] 
        i.cleanNumber = "(" + tmp.substring(0,3) + ") " + tmp.substring(3,6) + 
        "-" + tmp.substring(6,10)
 })
    return {
        data: options.data,
        errors: options.errors
    }
}

The following sample:

{
    "errors": [],
    "data": [{
        "Store Phone Number": "3334445555"
    },
    {
        "Store Phone Number": "1234567890"
    }]
}

Yields:

{
    "data": [
        {
            "Store Phone Number": "3334445555",
            "cleanNumber": "(333) 444-5555"
        },
        {
            "Store Phone Number": "1234567890",
            "cleanNumber": "(123) 456-7890"
        }
    ],
    "errors": []
}

Post map hook

This hook gets invoked after the fields in the source objects have been mapped to their respective fields in object to be imported. This hook can be used to further modify the mapped data.

These hooks run on the mapped data in the import before it is submitted to the target system.

Common use cases:

• Set default values based on mapped data
• Run complex calculations

postMapFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

Post map hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "settings":{},
    "configuration":{}
}

Post map hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{
    data: {}/[],
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

{
    code: '',
    message: '',
    source: ''
}

{},
{
    data: {}
},
{
    data: []
},
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
},
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.postMapFunction = function (options, callback) {
    return callback(err, responseData)
}

Post map hook example

This hook sums up the discounts on each line and sets it into a field on the mainline of the JSON object.

function postMap (options) {
  for(var i=0; i<options.postMapData.length; i++){
	var totalDiscount = 0.0;
	for(var j=0; j<options.postMapData[i].items.length; j++){
  	if(j<options.postMapData[i].items[j].discount){
    	totalDiscount += options.postMapData[i].items[j].discount;
  	}
	}
	options.postMapData[i].totalDiscount = totalDiscount;
  }
  return options.postMapData.map((d) => {
	return {
  	data: d
	}
  })
}

Post submit hook

This hook gets invoked after the records are processed by the import. You can use this hook to further process imported objects and modify the response data received from import for success and error cases. This can also be used to invoke some other process which need to be done at the end of the import.

These hooks run after the record has been successfully synced to the target system, and prior to the flow moving on to the next step of the integration.

Common use cases:

• Format the response to be used in your response mappings
• Validate the response from the target system
• Provide details around error messages

postSubmitFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

Post submit hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "responseData":[],
    "settings":{},
    "configuration":{}
}

[{code: '', message: '', source: ''}]

Post submit hook callback arguments

The function needs to call back with the following arguments:

module.hooks.postSubmitFunction = function (options, callback) {
    return callback(err, returnResponseData)
}

Post submit hook examples

Example 1: By default integrator.io will write the error returned by the target system, but in this case the error message isn’t very useful for individuals maintaining the integration. In this hook we will provide more details around the error message to make it clear of what the error actually means.

function postSubmitFunction(options) {
  for(var rec = 0; rec < options.responseData.length; rec++){
	if(!options.responseData[rec].errors)
  	continue;

	for( var i = 0; i < options.responseData[rec].errors.length; i++){
  	if(options.responseData[rec].errors[i].source != 'netsuite'){
    	continue;
  	}
  	if(options.responseData[rec].errors[i].message.endsWith('the order is already closed.')){
    	//Use this option to extend the error message
    	options.responseData[rec].errors[i].message = "Original Error: " + options.responseData[rec].errors[i].message + " This error usually means that the sales order has already been fulfilled or cancelled in NetSuite.";
  	}
	} 	 
  }
  return options.responseData;
}

Example 2: You can use the Post submit hook to ignore known errors in custom flows. In some circumstances, records may generate errors that do not impact the the business needs met by the flow's performance. Known errors on import have to be manually handled on a case-by-case basis, but you can use the Post submit hook to handle such errors automatically. 

For example purposes the hook function is named createIOUsersPostSubmitHook. You can use any name you like, but set up the hook function in the destination application's UI (or in the import application that you want to ignore or process the error message). You can do this with the following code:

function createIOUsersPostSubmitHook(options) {
 let responseData = processPostSubmitErrors(options, ignoreIOUserExistsErrorFn)
 return responseData
}

The createIOUsersPostSubmitHook function will be passed one options argument that has the following structure:

{ code: '', message: '', source: '', ignored: true/false, preMapDataRecord: {}/[]}

• code - the error code.
• message - the error message.
• source - the error source.
• ignored - has the error already been tagged as ignored?
• preMapDataRecord - the full preMapData record.

The function returns an object with the following structure:

 {ignored: true/false, betterMessage: ''}

• ignored - should the error be ignored?
• betterMessage - enhanced error message to replace the original message.

function ignoreIOUserExistsErrorFn(options) {
  let toReturn = {ignored: options.ignored, betterMessage: options.message}
  if (options.source === 'api.integrator.io' && options.code === 'UserExistsError') {
  	toReturn.ignored = true
    toReturn.betterMessage = 'The email ' + options.preMapDataRecord.Email + ' already has an integrator.io account.' 
  }
  return toReturn
}
function processPostSubmitErrors(postSubmitOptions, myProcessErrorFunction) {
  postSubmitOptions.responseData.forEach(function(rd, i) {
  	if (rd.statusCode !== 422) return
    let processedErrors = []
    rd.errors.forEach(function(re) {
      let errsHelper = []
      let parsedMessage = re.message
      try {
        parsedMessage = JSON.parse(re.message)
      } catch (ex) {}
      // Modify the following for your needs.
      if (typeof parsedMessage === 'object') {
        if (Array.isArray(parsedMessage)) {
          errsHelper = parsedMessage    
        } else if (Array.isArray(parsedMessage.errors)) {
          errsHelper = parsedMessage.errors
        } else {
          errsHelper.push({code: re.code, message: re.message})
        }
      } else {
        errsHelper.push({code: re.code, message: re.message})
      }
      errsHelper.forEach(function(e) {
        let processedErrorMessage = myProcessErrorFunction({
          code: e.code || re.code,
          message: e.message || re.message,
          source: re.source,
          ignored: re.ignored,
          preMapDataRecord: postSubmitOptions.preMapData[i]
        })
        processedErrors.push({
          code: e.code || re.code,
          message: processedErrorMessage.betterMessage || e.message || re.message,
          source: re.source,
          ignored: processedErrorMessage.ignored || re.ignored          
        })
      })
    })
    rd.errors = processedErrors
    let allErrorsIgnored = true
    for (let j = 0; j < processedErrors.length; j++) {
      if (!processedErrors[j].ignored) {
        allErrorsIgnored = false
        break
      }
    }
    if (allErrorsIgnored) {
      rd.ignored = true
      rd.errors = []
      if (!rd._json) rd._json = {}
      rd._json.ignoredErrors = processedErrors
    }
  })
  return postSubmitOptions.responseData
}

Note: For a more detailed example use case for the postSubmit hook, see Use postSubmit hook to ignore specific errors.

postAggregate hook

This hook gets invoked after the final aggregated file is uploaded to the destination service. Note that this hook only works when the 'skipAggregation' property is 'false'. This hook is passed a read only object.

postAggregrateFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

postAggregate hook options fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "postAggregateData":{},
    "settings":{},
    "configuration":{}
}

{
    success: true/false,
    _json: {}
}

postAggregate hook callback arguments

The function calls back with the following argument:

module.hooks.postAggregateFunction = function (options, callback) {
    return callback(err)
}