A flow in integrator.io consists of an export and an import. The export retrieves the data of the source application and breaks it into smaller pages. The import processes those pages and stores the data in the destination application. Hooks are custom code that can run at different stages during the execution of a flow to modify the behavior of export and import processes.

Contents

• Export hooks
  • preSavePage hook
    • preSavePage hook options argument fields
    • preSavePage callback arguments
    • preSavePage hook examples
• Import hooks
  • postResponseMap hook
    • postResponseMap hook examples
  • preMap hook
    • preMap hook options argument fields
    • preMap hook callback arguments
    • preMap hook example
  • postMap hook
    • postMap hook options argument fields
    • postMap hook callback arguments
  • postSubmit hook
    • postSubmit hook options argument fields
    • postSubmit hook callback arguments
    • postSubmit hook example
  • postAggregate hook
    • postAggregate hook options fields
    • postAggregate hook callback arguments

Export hooks

Export hooks run during the export stage of a flow. Currently integrator supports only one type of export hook: preSavePage.

Hook Types:

• Script: Integrator.io manages and executes your code.
• Stack: Your own server or AWS lambda hosts your code.

See the following articles for specific hook implementations:

• Create a hook for your import record
• Create a hook for your lookup data
• Create a hook for your export data record

preSavePage hook

The preSavePage hook is invoked after the export of a page is complete but before the page is sent to the destination application (import or lookup) for processing. You can use this hook to add or delete records or modify the existing records present in the data.

The preSavePage hook function is passed an options argument and a callback argument.

preSavePage hook options argument fields

The first argument (options) has the following fields:

{
    code: '',
    message: '',
    source: ''
}

preSavePage hook callback arguments

The function can use the following callback arguments:

{
    "data":[],
    "errors":[{
        "code":"",
        "message":"",
        "source":""
    }],
    "abort":"true|false"
}

module.hooks.preSavePageFunction = function (options, callback) { 
    return callback(err, responseData)
}

preSavePage hook examples

Example 1: You are exporting records out of an API that does NOT natively support a last modified search filter, but there is a last modified or created date field in the actual records being exported.

This use case uses the following arguments:

• lastExportDateTime
• currentExportDateTime
• abort

Using lastExportDateTime and currentExportDateTime in your preSavePage hook, you can programmatically exclude export records with date fields that fall outside of a specified range. If, during an export, you determine that all subsequent records returned will fall outside of the specified range, then you can return an abort argument at any time to stop the export from returning more records.

Note: You can both abort and return records in the same response. This instructs the export to stop retrieving more pages of records, but the flow should still process the records returned in the response.

Example 2: This use case uses the following arguments:

• pageIndex
• abort

You are exporting records out of an API where the first page of records contains information about the full set of records, and then based on info in the first page you decide to continue with the export, or return abort right away.

Import hooks

Import hooks are the hooks that are executed during the import process of a flow run.

postResponseMap hook

Note: The postResponseMap hook is available only in the beta release of integrator.io.

This hook runs immediately after response mappings and regular mappings. Use this hook to process records after response mapping before they are passed on to downstream applications in your flow. Any changes that you made to records with the postResponseMap hook will affect ALL downstream applications. You do NOT need to define a response/results mapping to use this hook. You could also use this hook instead to perform the same mapping typelogic. For more detailed information about how 'postResponseMap' works, please read the help text provided in the default function stub.

postResponseMap hook examples

If you are importing records into an application, and the application returns complex response data, you can use this hook to get just the fields you want from the response, and then put them in a very simple set of fields in your source record.

If you are merging the results of a lookup back into your record, you can use this hook to only merge specific fields, or filter entries that do not meet a complex criteria.

You can use this hook to perform calculations on values that span all of the records returned by a lookup, and then place the results in a simple field in your record.

You can use this hook to sort the results of a lookup, or group the results of a lookup into one or more new structures in your record.

preMap hook

This hook gets invoked before the fields are mapped to their respective fields in the objects to be imported. This hook can be used to reformat the record's fields before they get mapped.

preMapFunction: The name of the function can be changed to anything you like.The function will be passed an 'options' argument and a callback argument.

preMap hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "data":[],
    "settings":{},
    "configuration":{}
}

preMap hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{ 
     data: {}/[], 
     errors: [{
          code: '',
          message: '',
          source: ''
     }] 
}

{
    code: '',
    message: '',
    source: ''
}

{}, 
{
    data: {}
},
{
    data: []
}, 
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}, 
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.preMapFunction= function (options, callback) {
    return callback (err, responseData)
}

preMap hook example

Use preMap hook to reformat phone numbers in this format:

5555555555

to this format:

(555) 555-5555

The following code iterates over each record in the data[ ] array and concatenates the pieces of the number with the additional characters:

function convertPhoneNumber (options) {
    options.data.forEach(function(i) {
        let tmp = i["Store Phone Number"] 
        i.cleanNumber = "(" + tmp.substring(0,3) + ") " + tmp.substring(3,6) + 
        "-" + tmp.substring(6,10)
 })
    return {
        data: options.data,
        errors: options.errors
    }
}

The following sample:

{
    "errors": [],
    "data": [{
        "Store Phone Number": "3334445555"
    },
    {
        "Store Phone Number": "1234567890"
    }]
}

Yields:

{
    "data": [
        {
            "Store Phone Number": "3334445555",
            "cleanNumber": "(333) 444-5555"
        },
        {
            "Store Phone Number": "1234567890",
            "cleanNumber": "(123) 456-7890"
        }
    ],
    "errors": []
}

postMap hook

This hook gets invoked after the fields in the source objects have been mapped to their respective fields in object to be imported. This hook can be used to further modify the mapped data.

postMapFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

postMap hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "settings":{},
    "configuration":{}
}

postMap hook callback arguments

The function calls back with the following arguments:

[ { }, { }, ... ]

{
    data: {}/[],
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

{
    code: '',
    message: '',
    source: ''
}

{},
{
    data: {}
},
{
    data: []
},
{
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
},
{
    data: {},
    errors: [{
        code: '',
        message: '',
        source: ''
    }]
}

module.hooks.postMapFunction = function (options, callback) {
    return callback(err, responseData)
}

postSubmit hook

This hook gets invoked after the records are processed by the import. You can use this hook to further process imported objects and modify the response data received from import for success and error cases. This can also be used to invoke some other process which need to be done at the end of the import.

postSubmitFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

postSubmit hook options argument fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "preMapData":[],
    "postMapData":[],
    "responseData":[],
    "settings":{},
    "configuration":{}
}

[{code: '', message: '', source: ''}]

postSubmit hook callback arguments

The function needs to call back with the following arguments:

module.hooks.postSubmitFunction = function (options, callback) {
    return callback(err, returnResponseData)
}

postSubmit hook example

You can use the postSubmit hook to ignore known errors in custom flows. In some circumstances, records may generate errors that do not impact the the business needs met by the flow's performance. Known errors on import have to be manually handled on a case-by-case basis, but you can use the postSubmit hook to handle such errors automatically. 

For example purposes the hook function is named createIOUsersPostSubmitHook. You can use any name you like, but set up the hook function in the destination application's UI (or in the import application that you want to ignore or process the error message). You can do this with the following code:

function createIOUsersPostSubmitHook(options) {
 let responseData = processPostSubmitErrors(options, ignoreIOUserExistsErrorFn)
 return responseData
}

The createIOUsersPostSubmitHook function will be passed one options argument that has the following structure:

{ code: '', message: '', source: '', ignored: true/false, preMapDataRecord: {}/[]}

• code - the error code.
• message - the error message.
• source - the error source.
• ignored - has the error already been tagged as ignored?
• preMapDataRecord - the full preMapData record.

The function returns an object with the following structure:

 {ignored: true/false, betterMessage: ''}

• ignored - should the error be ignored?
• betterMessage - enhanced error message to replace the original message.

function ignoreIOUserExistsErrorFn(options) {
  let toReturn = {ignored: options.ignored, betterMessage: options.message}
  if (options.source === 'api.integrator.io' && options.code === 'UserExistsError') {
  	toReturn.ignored = true
    toReturn.betterMessage = 'The email ' + options.preMapDataRecord.Email + ' already has an integrator.io account.' 
  }
  return toReturn
}
function processPostSubmitErrors(postSubmitOptions, myProcessErrorFunction) {
  postSubmitOptions.responseData.forEach(function(rd, i) {
  	if (rd.statusCode !== 422) return
    let processedErrors = []
    rd.errors.forEach(function(re) {
      let errsHelper = []
      let parsedMessage = re.message
      try {
        parsedMessage = JSON.parse(re.message)
      } catch (ex) {}
      // Modify the following for your needs.
      if (typeof parsedMessage === 'object') {
        if (Array.isArray(parsedMessage)) {
          errsHelper = parsedMessage    
        } else if (Array.isArray(parsedMessage.errors)) {
          errsHelper = parsedMessage.errors
        } else {
          errsHelper.push({code: re.code, message: re.message})
        }
      } else {
        errsHelper.push({code: re.code, message: re.message})
      }
      errsHelper.forEach(function(e) {
        let processedErrorMessage = myProcessErrorFunction({
          code: e.code || re.code,
          message: e.message || re.message,
          source: re.source,
          ignored: re.ignored,
          preMapDataRecord: postSubmitOptions.preMapData[i]
        })
        processedErrors.push({
          code: e.code || re.code,
          message: processedErrorMessage.betterMessage || e.message || re.message,
          source: re.source,
          ignored: processedErrorMessage.ignored || re.ignored          
        })
      })
    })
    rd.errors = processedErrors
    let allErrorsIgnored = true
    for (let j = 0; j < processedErrors.length; j++) {
      if (!processedErrors[j].ignored) {
        allErrorsIgnored = false
        break
      }
    }
    if (allErrorsIgnored) {
      rd.ignored = true
      rd.errors = []
      if (!rd._json) rd._json = {}
      rd._json.ignoredErrors = processedErrors
    }
  })
  return postSubmitOptions.responseData
}

Note: For a more detailed example use case for the postSubmit hook, see Use postSubmit hook to ignore specific errors.

postAggregate hook

This hook gets invoked after the final aggregated file is uploaded to the destination service. Note that this hook only works when the 'skipAggregation' property is 'false'. This hook is passed a read only object.

postAggregrateFunction: The name of the function can be changed to anything you like.

The function will be passed an 'options' argument and a callback argument.

postAggregate hook options fields

The first argument 'options' has the following structure:

{
    "bearerToken":"",
    "_importId":"",
    "_connectionId":"",
    "_integrationId":"",
    "_flowId":"",
    "postAggregateData":{},
    "settings":{},
    "configuration":{}
}

{
    success: true/false,
    _json: {}
}

postAggregate hook callback arguments

The function calls back with the following argument:

module.hooks.postAggregateFunction = function (options, callback) {
    return callback(err)
}