A flow in integrator.io consists of a source (export from an app, database, or file server; or a webhook listener) and a destination (import, file transfer, or lookup). In simple terms, an export retrieves the data of the source application and breaks it into smaller pages. An import processes those pages and stores the data in the destination application. Hooks are custom code that can run at different stages during the execution of a flow to modify the behavior of export and import processes.

Contents

• Export hooks
  • Pre save hook
    • Pre save hook options argument fields
    • Pre save callback arguments
    • Pre save hook examples
• Import hooks
  • postResponseMap hook
    • postResponseMap hook examples
  • Pre map hook
    • Pre map hook options argument fields
    • Pre map hook callback arguments
    • Pre map hook example
  • Post map hook
    • Post map hook options argument fields
    • Post map hook callback arguments
    • Post map hook example
  • Post submit hook
    • Post submit hook options argument fields
    • Post submit hook callback arguments
    • Post submit hook example
  • postAggregate hook
    • postAggregate hook options fields
    • postAggregate hook callback arguments

Export hooks

Export hooks run during the export stage of a flow. Currently integrator supports only one type of export hook: Pre save.

Hook Types:

• Script: Integrator.io manages and executes your code.
• Stack: Your own server or AWS lambda hosts your code.

See the following articles for specific hook implementations:

• Create a hook for your import record
• Create a hook for your lookup data
• Create a hook for your export data record

Pre save hook

You can use this hook to format, filter, and perform logic on the data coming from your export before it moves on to the rest of your data flow. Logic that applies to the data at all steps of the integration should be done at this stage of the flow.

Common use cases:

• Filter out data at a line level.
• Apply calculations to the data (examples: sum of lines, calculate discounts).
• Complex data formatting (example: turn comma-separated text into an array).
• Perform data validation.

The Pre save hook is invoked after the export of a page is complete but before the page is sent to the destination application (import or lookup) for processing. You can use this hook to add or delete records or modify the existing records present in the data.

The Pre save hook function is passed an options argument and a callback argument.

Pre save hook options argument fields

The first argument (options) has the following fields:

{
    code: '',
    message: '',
    source: ''
}

Pre save hook callback arguments

The function can use the following callback arguments:

{
    "data":[],
    "errors":[{
        "code":"",
        "message":"",
        "source":""
    }],
    "abort":"true|false"
}

module.hooks.preSavePageFunction = function (options, callback) { 
    return callback(err, responseData)
}

Pre save hook examples

Example 1: 

This example runs on a set of JSON data that returns the customer’s email address in an object within an array called identity-profiles that then has objects containing of a type and value property. This hook will find the object with the type of EMAIL and set the corresponding value to the main level of the object.

function preSavePageFunction (options) {
  for(var i=0; i < options.data.length; i++){
	if(!options.data[i]['identity-profiles'] || options.data[i]['identity-profiles'].length < 1){
  	break;
	}
    
	var emailFound = false;
	for(var j=0; j < options.data[i]['identity-profiles'].length; j++){
  	if(options.data[i]['identity-profiles'][j].identities){
    	for(var k=0; k < options.data[i]['identity-profiles'][j].identities.length; k++){
      	if(options.data[i]['identity-profiles'][j].identities[k].type != "EMAIL")
        	continue;
     	 
      	options.data[i].email = options.data[i]['identity-profiles'][j].identities[k].value;
      	emailFound = true;
      	break;
    	}
  	}
  	if(emailFound)
    	break;
	}
  }
  return {
	data: options.data,
	errors: options.errors
  }
}

Example 2: You are exporting records out of an API that does NOT natively support a last modified search filter, but there is a last modified or created date field in the actual records being exported.

This use case uses the following arguments:

• lastExportDateTime
• currentExportDateTime
• abort

Using lastExportDateTime and currentExportDateTime in your Pre save hook, you can programmatically exclude export records with date fields that fall outside of a specified range. If, during an export, you determine that all subsequent records returned will fall outside of the specified range, then you can return an abort argument at any time to stop the export from returning more records.

Note: You can both abort and return records in the same response. This instructs the export to stop retrieving more pages of records, but the flow should still process the records returned in the response.

Example 3: This use case uses the following arguments:

• pageIndex
• abort

You are exporting records out of an API where the first page of records contains information about the full set of records, and then based on info in the first page you decide to continue with the export, or return abort right away.

Import hooks

Import hooks are the hooks that are executed during the import process of a flow run.

postResponseMap hook

This hook runs immediately after response mappings and regular mappings. Use this hook to process records after response mapping before they are passed on to downstream applications in your flow. Any changes that you made to records with the postResponseMap hook will affect ALL downstream applications. You do NOT need to define a response/results mapping to use this hook. You could also use this hook instead to perform the same logic. For more detailed information about how 'postResponseMap' works, please read the help text provided in the default function stub.

postResponseMap hook examples

If you are importing records into an application, and the application returns complex response data, you can use this hook to get just the fields you want from the response, and then put them in a very simple set of fields in your source record.

If you are merging the results of a lookup back into your record, you can use this hook to only merge specific fields, or filter entries that do not meet a complex criteria.

You can use this hook to perform calculations on values that span all of the records returned by a lookup, and then place the results in a simple field in your record.

You can use this hook to sort the results of a lookup, or group the results of a lookup into one or more new structures in your record.

Pre map hook

This hook gets invoked before the fields are mapped to their respective fields in the objects to be imported. This hook can be used to reformat the record's fields before they get mapped.

You can use this hook to apply logic to the data within an import prior to going through the mapping process. Any manipulation of the data in this step only applies to the import that it is running on.

Common use cases:

• Insert default values
• Format data needed for mapping

preMapFunction: The name of the function can be changed to anything you like.The function will be passed an 'options' argument and a callback argument.

Pre map hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "data":[],
    "settings":{},
    "configuration":{}
}

Pre map hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{ 
     data: {}/[], 
     errors: [{
          code: '',
          message: '',
          source: ''
     }] 
}

{
    code: '',
    message: '',
    source: ''
}

{}, 
{
    data: {}
},
{
    data: []
}, 
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}, 
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.preMapFunction= function (options, callback) {
    return callback (err, responseData)
}

Pre map hook examples

Example 1: In this use case, sometimes the export will send the items list across as either an array or an object. The mappings require it to be an array so the hook will convert it to an array containing a single object if the data is presented as an object.

function convertObjToArray(options) {
  var elementName = 'items';  //Change item to be the name of the property that holds the items
  for(var i=0; i<options.data.length; i++){
	if(Array.isArray(options.data[i][elementName]))
  	continue;
    
	var arr = [];
	arr[0] = options.data[i][elementName];
	options.data[i][elementName] = arr;
  }
  return options.data.map((d) => {
	return {
  	data: d
	}
  })
}

Example 2: Use Pre map hook to reformat phone numbers in this format:

5555555555

to this format:

(555) 555-5555

The following code iterates over each record in the data[ ] array and concatenates the pieces of the number with the additional characters:

function convertPhoneNumber (options) {
    options.data.forEach(function(i) {
        let tmp = i["Store Phone Number"] 
        i.cleanNumber = "(" + tmp.substring(0,3) + ") " + tmp.substring(3,6) + 
        "-" + tmp.substring(6,10)
 })
    return {
        data: options.data,
        errors: options.errors
    }
}

The following sample:

{
    "errors": [],
    "data": [{
        "Store Phone Number": "3334445555"
    },
    {
        "Store Phone Number": "1234567890"
    }]
}

Yields:

{
    "data": [
        {
            "Store Phone Number": "3334445555",
            "cleanNumber": "(333) 444-5555"
        },
        {
            "Store Phone Number": "1234567890",
            "cleanNumber": "(123) 456-7890"
        }
    ],
    "errors": []
}

Post map hook

This hook gets invoked after the fields in the source objects have been mapped to their respective fields in object to be imported. This hook can be used to further modify the mapped data.

These hooks run on the mapped data in the import before it is submitted to the target system.

Common use cases:

• Set default values based on mapped data
• Run complex calculations

postMapFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

Post map hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "settings":{},
    "configuration":{}
}

Post map hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{
    data: {}/[],
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

{
    code: '',
    message: '',
    source: ''
}

{},
{
    data: {}
},
{
    data: []
},
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
},
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.postMapFunction = function (options, callback) {
    return callback(err, responseData)
}

Post map hook example

This hook sums up the discounts on each line and sets it into a field on the mainline of the JSON object.

function postMap (options) {
  for(var i=0; i<options.postMapData.length; i++){
	var totalDiscount = 0.0;
	for(var j=0; j<options.postMapData[i].items.length; j++){
  	if(j<options.postMapData[i].items[j].discount){
    	totalDiscount += options.postMapData[i].items[j].discount;
  	}
	}
	options.postMapData[i].totalDiscount = totalDiscount;
  }
  return options.postMapData.map((d) => {
	return {
  	data: d
	}
  })
}

Post submit hook

This hook gets invoked after the records are processed by the import. You can use this hook to further process imported objects and modify the response data received from import for success and error cases. This can also be used to invoke some other process which need to be done at the end of the import.

These hooks run after the record has been successfully synced to the target system, and prior to the flow moving on to the next step of the integration.

Common use cases:

• Format the response to be used in your response mappings
• Validate the response from the target system
• Provide details around error messages

postSubmitFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

Post submit hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "responseData":[],
    "settings":{},
    "configuration":{}
}

[{code: '', message: '', source: ''}]

Post submit hook callback arguments

The function needs to call back with the following arguments:

module.hooks.postSubmitFunction = function (options, callback) {
    return callback(err, returnResponseData)
}

Post submit hook examples

Example 1: By default integrator.io will write the error returned by the target system, but in this case the error message isn’t very useful for individuals maintaining the integration. In this hook we will provide more details around the error message to make it clear of what the error actually means.

function postSubmitFunction(options) {
  for(var rec = 0; rec < options.responseData.length; rec++){
	if(!options.responseData[rec].errors)
  	continue;

	for( var i = 0; i < options.responseData[rec].errors.length; i++){
  	if(options.responseData[rec].errors[i].source != 'netsuite'){
    	continue;
  	}
  	if(options.responseData[rec].errors[i].message.endsWith('the order is already closed.')){
    	//Use this option to extend the error message
    	options.responseData[rec].errors[i].message = "Original Error: " + options.responseData[rec].errors[i].message + " This error usually means that the sales order has already been fulfilled or cancelled in NetSuite.";
  	}
	} 	 
  }
  return options.responseData;
}

Example 2: You can use the Post submit hook to ignore known errors in custom flows. In some circumstances, records may generate errors that do not impact the the business needs met by the flow's performance. Known errors on import have to be manually handled on a case-by-case basis, but you can use the Post submit hook to handle such errors automatically. 

For example purposes the hook function is named createIOUsersPostSubmitHook. You can use any name you like, but set up the hook function in the destination application's UI (or in the import application that you want to ignore or process the error message). You can do this with the following code:

function createIOUsersPostSubmitHook(options) {
 let responseData = processPostSubmitErrors(options, ignoreIOUserExistsErrorFn)
 return responseData
}

The createIOUsersPostSubmitHook function will be passed one options argument that has the following structure:

{ code: '', message: '', source: '', ignored: true/false, preMapDataRecord: {}/[]}

• code - the error code.
• message - the error message.
• source - the error source.
• ignored - has the error already been tagged as ignored?
• preMapDataRecord - the full preMapData record.

The function returns an object with the following structure:

 {ignored: true/false, betterMessage: ''}

• ignored - should the error be ignored?
• betterMessage - enhanced error message to replace the original message.

function ignoreIOUserExistsErrorFn(options) {
  let toReturn = {ignored: options.ignored, betterMessage: options.message}
  if (options.source === 'api.integrator.io' && options.code === 'UserExistsError') {
  	toReturn.ignored = true
    toReturn.betterMessage = 'The email ' + options.preMapDataRecord.Email + ' already has an integrator.io account.' 
  }
  return toReturn
}
function processPostSubmitErrors(postSubmitOptions, myProcessErrorFunction) {
  postSubmitOptions.responseData.forEach(function(rd, i) {
  	if (rd.statusCode !== 422) return
    let processedErrors = []
    rd.errors.forEach(function(re) {
      let errsHelper = []
      let parsedMessage = re.message
      try {
        parsedMessage = JSON.parse(re.message)
      } catch (ex) {}
      // Modify the following for your needs.
      if (typeof parsedMessage === 'object') {
        if (Array.isArray(parsedMessage)) {
          errsHelper = parsedMessage    
        } else if (Array.isArray(parsedMessage.errors)) {
          errsHelper = parsedMessage.errors
        } else {
          errsHelper.push({code: re.code, message: re.message})
        }
      } else {
        errsHelper.push({code: re.code, message: re.message})
      }
      errsHelper.forEach(function(e) {
        let processedErrorMessage = myProcessErrorFunction({
          code: e.code || re.code,
          message: e.message || re.message,
          source: re.source,
          ignored: re.ignored,
          preMapDataRecord: postSubmitOptions.preMapData[i]
        })
        processedErrors.push({
          code: e.code || re.code,
          message: processedErrorMessage.betterMessage || e.message || re.message,
          source: re.source,
          ignored: processedErrorMessage.ignored || re.ignored          
        })
      })
    })
    rd.errors = processedErrors
    let allErrorsIgnored = true
    for (let j = 0; j < processedErrors.length; j++) {
      if (!processedErrors[j].ignored) {
        allErrorsIgnored = false
        break
      }
    }
    if (allErrorsIgnored) {
      rd.ignored = true
      rd.errors = []
      if (!rd._json) rd._json = {}
      rd._json.ignoredErrors = processedErrors
    }
  })
  return postSubmitOptions.responseData
}

Note: For a more detailed example use case for the postSubmit hook, see Use postSubmit hook to ignore specific errors.

postAggregate hook

This hook gets invoked after the final aggregated file is uploaded to the destination service. Note that this hook only works when the 'skipAggregation' property is 'false'. This hook is passed a read only object.

postAggregrateFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

postAggregate hook options fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "postAggregateData":{},
    "settings":{},
    "configuration":{}
}

{
    success: true/false,
    _json: {}
}

postAggregate hook callback arguments

The function calls back with the following argument:

module.hooks.postAggregateFunction = function (options, callback) {
    return callback(err)
}