A flow in integrator.io consists of a source (export from an app, database, or file server; or a webhook listener) and a destination (import, file transfer, or lookup). In simple terms, an export retrieves the data of the source application and breaks it into smaller pages. An import processes those pages and stores the data in the destination application. Hooks are custom code that can run at different stages during the execution of a flow to modify the behavior of export and import processes.
Contents
- Export hooks
- Import hooks
Export hooks
Export hooks run during the export stage of a flow. Currently integrator supports only one type of export hook: Pre save.
Hook Types:
- Script: Integrator.io manages and executes your code.
- Stack: Your own server or AWS lambda hosts your code.
See the following articles for specific hook implementations:
- Create a hook for your import record
- Create a hook for your lookup data
- Create a hook for your export data record
Pre save hook
You can use this hook to format, filter, and perform logic on the data coming from your export before it moves on to the rest of your data flow. Logic that applies to the data at all steps of the integration should be done at this stage of the flow.
Common use cases:
- Filter out data at a line level.
- Apply calculations to the data (examples: sum of lines, calculate discounts).
- Complex data formatting (example: turn comma-separated text into an array).
- Perform data validation.
The Pre save hook is invoked after the export of a page is complete but before the page is sent to the destination application (import or lookup) for processing. You can use this hook to add or delete records or modify the existing records present in the data.
The Pre save hook function is passed an options argument and a callback argument.
Pre save hook options argument fields
The first argument (options) has the following fields:
Field Name | Description |
bearerToken | A one-time bearer token which can be used to invoke selected integrator.io API routes. |
preview | A Boolean flag used to indicate that this export is being used by the integrator.io UI to get a sample of the data being exported. |
data | An array of records representing one page of data. An individual record can be an object {}, or an array [] depending on the data source. |
errors | An array of errors where each error has the following structure:
{ |
_exportId | The _exportId currently running. |
_connectionId | The _connectionId currently running. |
_flowId | The _flowId currently running. |
_integrationId | The _integrationId currently running. |
pageIndex | 0 based. context is the batch export currently running. This is sent to Pre save hooks so that developers know which page is being processed. |
lastExportDateTime | This variable is sent to Pre save hooks for delta flows so that developers can programmatically exclude records using these dynamic date values. |
currentExportDateTime | This variable is sent to Pre save hooks for delta flows so that developers can programmatically exclude records using these dynamic date values. |
settings | A container object for all the SmartConnector settings associated with the integration (applicable to SmartConnectors only). |
configuration | An optional configuration object that can be set directly on the export resource (to further customize the hooks behavior). |
Pre save hook callback arguments
The function can use the following callback arguments:
Argument Name | Description |
err | An error object to signal a fatal exception and will stop the flow. |
responseData | An object that has the following structure:
{ "data":[], "errors":[{ "code":"", "message":"", "source":"" }], "abort":"true|false" } |
data | Your modified data. |
errors | Your modified errors. |
abort | Instruct the batch export currently running to stop generating new pages of data.
module.hooks.preSavePageFunction = function (options, callback) { This function is invoked at the end of the data record after the transformations and the filters. It’s the last step before the export record is passed to the destination app. The functions are available in a drop-down menu at the top of your script editor. |
Pre save hook examples
Example 1:
This example runs on a set of JSON data that returns the customer’s email address in an object within an array called identity-profiles that then has objects containing of a type and value property. This hook will find the object with the type of EMAIL and set the corresponding value to the main level of the object.
function preSavePageFunction (options) { for(var i=0; i < options.data.length; i++){ if(!options.data[i]['identity-profiles'] || options.data[i]['identity-profiles'].length < 1){ break; } var emailFound = false; for(var j=0; j < options.data[i]['identity-profiles'].length; j++){ if(options.data[i]['identity-profiles'][j].identities){ for(var k=0; k < options.data[i]['identity-profiles'][j].identities.length; k++){ if(options.data[i]['identity-profiles'][j].identities[k].type != "EMAIL") continue; options.data[i].email = options.data[i]['identity-profiles'][j].identities[k].value; emailFound = true; break; } } if(emailFound) break; } } return { data: options.data, errors: options.errors } }
Example 2: You are exporting records out of an API that does NOT natively support a last modified search filter, but there is a last modified or created date field in the actual records being exported.
This use case uses the following arguments:
- lastExportDateTime
- currentExportDateTime
- abort
Using lastExportDateTime and currentExportDateTime in your Pre save hook, you can programmatically exclude export records with date fields that fall outside of a specified range. If, during an export, you determine that all subsequent records returned will fall outside of the specified range, then you can return an abort argument at any time to stop the export from returning more records.
Note: You can both abort and return records in the same response. This instructs the export to stop retrieving more pages of records, but the flow should still process the records returned in the response.
Example 3: This use case uses the following arguments:
- pageIndex
- abort
You are exporting records out of an API where the first page of records contains information about the full set of records, and then based on info in the first page you decide to continue with the export, or return abort right away.
Import hooks
Import hooks are the hooks that are executed during the import process of a flow run.
postResponseMap hook
This hook runs immediately after response mappings and regular mappings. Use this hook to process records after response mapping before they are passed on to downstream applications in your flow. Any changes that you made to records with the postResponseMap hook will affect ALL downstream applications. You do NOT need to define a response/results mapping to use this hook. You could also use this hook instead to perform the same mapping typelogic. For more detailed information about how 'postResponseMap' works, please read the help text provided in the default function stub.
postResponseMap hook examples
If you are importing records into an application, and the application returns complex response data, you can use this hook to get just the fields you want from the response, and then put them in a very simple set of fields in your source record.
If you are merging the results of a lookup back into your record, you can use this hook to only merge specific fields, or filter entries that do not meet a complex criteria.
You can use this hook to perform calculations on values that span all of the records returned by a lookup, and then place the results in a simple field in your record.
You can use this hook to sort the results of a lookup, or group the results of a lookup into one or more new structures in your record.
Pre map hook
This hook gets invoked before the fields are mapped to their respective fields in the objects to be imported. This hook can be used to reformat the record's fields before they get mapped.
You can use this hook to apply logic to the data within an import prior to going through the mapping process. Any manipulation of the data in this step only applies to the import that it is running on.
Common use cases:
- Insert default values
- Format data needed for mapping
preMapFunction: The name of the function can be changed to anything you like.The function will be passed an 'options' argument and a callback argument.
Pre map hook options argument fields
The first argument 'options' has the following structure:
{ "bearerToken":"", "_importId":"", "_connectionId":"", "_integrationId":"", "_flowId":"", "data":[], "settings":{}, "configuration":{} }
Field Name | Description |
bearerToken | A one-time bearer token which can be used to invoke selected integrator.io API routes. |
_importId | The _importId of the import for which the hook is defined. |
_connectionId | The _id of the connection linked to the import for which the hook is defined. |
_integrationId | The _id of the integration linked to the import for which the hook is defined. |
_flowId | the _id of the flow linked to the import for which the hook is defined. |
data | An array of records representing the page of data before it has been mapped. An individual record can be an object {}, or an array [] depending on the data source. |
settings | A container object for all the SmartConnector settings associated with the integration (applicable to SmartConnectors only). |
configuration | An optional configuration object that can be set directly on the import resource (to further customize the hooks behavior). |
Pre map hook callback arguments
The function calls back with the following arguments:
Argument name | Description |
err | An error object to signal a fatal exception and will fail the entire page of records. |
responseData |
An array that has the following structure: [ { }, { }, ... ] The array length MUST match the options.data array length. Each element in the array represents the actions that should be taken on the record at that index. Each element in the array should have the following structure: { data: {}/[], errors: [{ code: '', message: '', source: '' }] } |
data | The modified (or unmodified) record that should be passed along for processing. An individual record can be an object {} or an array [] depending on the data source. |
errors |
Used to report one or more errors for the specific record. Each error must have the following structure: { code: '', message: '', source: '' } Returning an empty object {} for a specific record will indicate to integrator.io that the record should be ignored. Returning both data and errors for a specific record will indicate to integrator.io that the record should be processed but errors should also be logged on the job. Examples: {}, { data: {} }, module.hooks.preMapFunction= function (options, callback) { return callback (err, responseData) } |
Pre map hook examples
Example 1: In this use case, sometimes the export will send the items list across as either an array or an object. The mappings require it to be an array so the hook will convert it to an array containing a single object if the data is presented as an object.
function convertObjToArray(options) { var elementName = 'items'; //Change item to be the name of the property that holds the items for(var i=0; i<options.data.length; i++){ if(Array.isArray(options.data[i][elementName])) continue; var arr = []; arr[0] = options.data[i][elementName]; options.data[i][elementName] = arr; } return options.data.map((d) => { return { data: d } }) }
Example 2: Use Pre map hook to reformat phone numbers in this format:
5555555555
to this format:
(555) 555-5555
The following code iterates over each record in the data[ ] array and concatenates the pieces of the number with the additional characters:
function convertPhoneNumber (options) { options.data.forEach(function(i) { let tmp = i["Store Phone Number"] i.cleanNumber = "(" + tmp.substring(0,3) + ") " + tmp.substring(3,6) + "-" + tmp.substring(6,10)
}) return { data: options.data, errors: options.errors } }
The following sample:
{ "errors": [], "data": [{ "Store Phone Number": "3334445555" }, { "Store Phone Number": "1234567890" }] }
Yields:
{ "data": [ { "Store Phone Number": "3334445555", "cleanNumber": "(333) 444-5555" }, { "Store Phone Number": "1234567890", "cleanNumber": "(123) 456-7890" } ], "errors": [] }
Post map hook
This hook gets invoked after the fields in the source objects have been mapped to their respective fields in object to be imported. This hook can be used to further modify the mapped data.
These hooks run on the mapped data in the import before it is submitted to the target system.
Common use cases:
- Set default values based on mapped data
- Run complex calculations
postMapFunction: The name of the function can be changed to anything you like.
The function will be passed an 'options' argument and a callback argument.
Post map hook options argument fields
The first argument 'options' has the following structure:
{ "bearerToken":"", "_importId":"", "_connectionId":"", "_integrationId":"", "_flowId":"", "preMapData":[], "postMapData":[], "settings":{}, "configuration":{} }
Field Name | Description |
bearerToken | a one-time bearer token which can be used to invoke selected integrator.io API routes. |
_importId | the _importId of the import for which the hook is defined. |
_connectionId | the _id of the connection linked to the import for which the hook is defined. |
_integrationId | the _id of the integration linked to the import for which the hook is defined. |
_flowId | the _id of the flow linked to the import for which the hook is defined. |
preMapData | an array of records representing the page of data before it was mapped. An individual record can be an object {}, or an array [] depending on the data source. |
postMapData | an array of records representing the page of data after it was mapped. An individual record can be an object {}, or an array [] depending on the data source. |
settings | a container object for all the SmartConnector settings associated with the integration (applicable to SmartConnectors only). |
configuration | an optional configuration object that can be set directly on the import resource (to further customize the hooks behavior). |
Post map hook callback arguments
The function calls back with the following arguments:
Argument Name | Description |
err | An error object to signal a fatal exception and will fail the entire page of records. |
responseData |
An array that has the following structure: [ { }, { }, ... ] The returned array length MUST match the options.data array length. Each element in the array represents the actions that should be taken on the record at that index. Each element in the array should have the following structure: { data: {}/[], errors: [{ code: '', message: '', source: '' }] } |
data | The modified (or unmodified) record that should be passed along for processing. An individual record can be an object {} or an array [] depending on the data source. |
errors |
Used to report one or more errors for the specific record. Each error must have the following structure: { code: '', message: '', source: '' } Returning an empty object {} for a specific record will indicate to integrator.io that the record should be ignored. Returning both 'data' and 'errors' for a specific record will indicate to integrator.io that the record should be processed but errors should also be logged on the job. Examples: {}, { module.hooks.postMapFunction = function (options, callback) { return callback(err, responseData) } |
Post map hook example
This hook sums up the discounts on each line and sets it into a field on the mainline of the JSON object.
function postMap (options) { for(var i=0; i<options.postMapData.length; i++){ var totalDiscount = 0.0; for(var j=0; j<options.postMapData[i].items.length; j++){ if(j<options.postMapData[i].items[j].discount){ totalDiscount += options.postMapData[i].items[j].discount; } } options.postMapData[i].totalDiscount = totalDiscount; } return options.postMapData.map((d) => { return { data: d } }) }
Post submit hook
This hook gets invoked after the records are processed by the import. You can use this hook to further process imported objects and modify the response data received from import for success and error cases. This can also be used to invoke some other process which need to be done at the end of the import.
These hooks run after the record has been successfully synced to the target system, and prior to the flow moving on to the next step of the integration.
Common use cases:
- Format the response to be used in your response mappings
- Validate the response from the target system
- Provide details around error messages
postSubmitFunction: The name of the function can be changed to anything you like.
The function will be passed an 'options' argument and a callback argument.
Post submit hook options argument fields
The first argument 'options' has the following structure:
{ "bearerToken":"", "_importId":"", "_connectionId":"", "_integrationId":"", "_flowId":"", "preMapData":[], "postMapData":[], "responseData":[], "settings":{}, "configuration":{} }
Field Name | Description |
bearerToken | a one-time bearer token which can be used to invoke selected integrator.io API routes. |
_importId | the _importId of the import for which the hook is defined. |
_connectionId | the _id of the connection linked to the import for which the hook is defined. |
_integrationId | the _id of the integration linked to the import for which the hook is defined. |
_flowId | the _id of the flow linked to the import for which the hook is defined. |
preMapData | an array of records representing the page of data before it was mapped. An individual record can be an object {}, or an array [] depending on the data source. |
postMapData | an array of records representing the page of data after it was mapped. An individual record can be an object {}, or an array [] depending on the data source. |
responseData | an array of responses for the page of data that was submitted to the import application. An individual response will have the following structure: { statusCode: 200/422/401, errors: [], ignored: true/false, id: '', _json: {}, dataURI: '' } |
statusCode | 200 is a success. 422 is a data error. 401 means the connection went offline (typically due to an authentication or incorrect password issue). |
errors |
[{code: '', message: '', source: ''}] |
ignored | true if the record was filtered/skipped, false otherwise. |
id | the id from the import application response. |
_json | the complete response data from the import application. |
dataURI | if possible, a URI for the data in the import application (populated only for errored records). |
settings | a container object for all the SmartConnector settings associated with the integration (applicable to SmartConnectors only). |
configuration | an optional configuration object that can be set directly on the import resource (to further customize the hooks behavior). |
Post submit hook callback arguments
The function needs to call back with the following arguments:
Argument Name | Description |
err | an error object to signal a fatal exception and will fail the entire page of records. |
responseData |
the responseData array provided by options.responseData. The length of the responseData array MUST remain unchanged. Elements within the responseData array can be modified to enhance error messages, modify the complete _json response data, etc... module.hooks.postSubmitFunction = function (options, callback) { return callback(err, returnResponseData) } |
Post submit hook examples
Example 1: By default integrator.io will write the error returned by the target system, but in this case the error message isn’t very useful for individuals maintaining the integration. In this hook we will provide more details around the error message to make it clear of what the error actually means.
function postSubmitFunction(options) { for(var rec = 0; rec < options.responseData.length; rec++){ if(!options.responseData[rec].errors) continue; for( var i = 0; i < options.responseData[rec].errors.length; i++){ if(options.responseData[rec].errors[i].source != 'netsuite'){ continue; } if(options.responseData[rec].errors[i].message.endsWith('the order is already closed.')){ //Use this option to extend the error message options.responseData[rec].errors[i].message = "Original Error: " + options.responseData[rec].errors[i].message + " This error usually means that the sales order has already been fulfilled or cancelled in NetSuite."; } } } return options.responseData; }
Example 2: You can use the Post submit hook to ignore known errors in custom flows. In some circumstances, records may generate errors that do not impact the the business needs met by the flow's performance. Known errors on import have to be manually handled on a case-by-case basis, but you can use the Post submit hook to handle such errors automatically.
For example purposes the hook function is named createIOUsersPostSubmitHook. You can use any name you like, but set up the hook function in the destination application's UI (or in the import application that you want to ignore or process the error message). You can do this with the following code:
function createIOUsersPostSubmitHook(options) {
let responseData = processPostSubmitErrors(options, ignoreIOUserExistsErrorFn)
return responseData
}
The createIOUsersPostSubmitHook function will be passed one options argument that has the following structure:
{ code: '', message: '', source: '', ignored: true/false, preMapDataRecord: {}/[]}
- code - the error code.
- message - the error message.
- source - the error source.
- ignored - has the error already been tagged as ignored?
- preMapDataRecord - the full preMapData record.
The function returns an object with the following structure:
{ignored: true/false, betterMessage: ''}
- ignored - should the error be ignored?
- betterMessage - enhanced error message to replace the original message.
function ignoreIOUserExistsErrorFn(options) { let toReturn = {ignored: options.ignored, betterMessage: options.message} if (options.source === 'api.integrator.io' && options.code === 'UserExistsError') { toReturn.ignored = true toReturn.betterMessage = 'The email ' + options.preMapDataRecord.Email + ' already has an integrator.io account.' } return toReturn } function processPostSubmitErrors(postSubmitOptions, myProcessErrorFunction) { postSubmitOptions.responseData.forEach(function(rd, i) { if (rd.statusCode !== 422) return let processedErrors = [] rd.errors.forEach(function(re) { let errsHelper = [] let parsedMessage = re.message try { parsedMessage = JSON.parse(re.message) } catch (ex) {} // Modify the following for your needs. if (typeof parsedMessage === 'object') { if (Array.isArray(parsedMessage)) { errsHelper = parsedMessage } else if (Array.isArray(parsedMessage.errors)) { errsHelper = parsedMessage.errors } else { errsHelper.push({code: re.code, message: re.message}) } } else { errsHelper.push({code: re.code, message: re.message}) } errsHelper.forEach(function(e) { let processedErrorMessage = myProcessErrorFunction({ code: e.code || re.code, message: e.message || re.message, source: re.source, ignored: re.ignored, preMapDataRecord: postSubmitOptions.preMapData[i] }) processedErrors.push({ code: e.code || re.code, message: processedErrorMessage.betterMessage || e.message || re.message, source: re.source, ignored: processedErrorMessage.ignored || re.ignored }) }) }) rd.errors = processedErrors let allErrorsIgnored = true for (let j = 0; j < processedErrors.length; j++) { if (!processedErrors[j].ignored) { allErrorsIgnored = false break } } if (allErrorsIgnored) { rd.ignored = true rd.errors = [] if (!rd._json) rd._json = {} rd._json.ignoredErrors = processedErrors } }) return postSubmitOptions.responseData }
Note: For a more detailed example use case for the postSubmit hook, see Use postSubmit hook to ignore specific errors.
postAggregate hook
This hook gets invoked after the final aggregated file is uploaded to the destination service. Note that this hook only works when the 'skipAggregation' property is 'false'. This hook is passed a read only object.
postAggregrateFunction: The name of the function can be changed to anything you like.
The function will be passed an 'options' argument and a callback argument.
postAggregate hook options fields
The first argument 'options' has the following structure:
{ "bearerToken":"", "_importId":"", "_connectionId":"", "_integrationId":"", "_flowId":"", "postAggregateData":{}, "settings":{}, "configuration":{} }
Field Name | Description |
bearerToken | a one-time bearer token which can be used to invoke selected integrator.io API routes. |
_importId | The _importId of the import for which the hook is defined. |
_connectionId | The _id of the connection linked to the import for which the hook is defined. |
_integrationId | The _id of the integration linked to the import for which the hook is defined. |
_flowId | The _id of the flow linked to the import for which the hook is defined. |
postAggregateData | A container object with the following structure:
{ success: true/false, _json: {} } |
success | True if data aggregation was successful, false otherwise. |
_json | information about the aggregated data transfer. For example, the name of the aggregated file on the FTP site. |
code | error code if data aggregation failed. |
message | error message if data aggregation failed. |
source | error source if data aggregation failed. |
settings | a container object for all the SmartConnector settings associated with the integration (applicable to SmartConnectors only). |
configuration | an optional configuration object that can be set directly on the export resource (to further customize the hooks behavior). |
postAggregate hook callback arguments
The function calls back with the following argument:
Argument Name | Description |
err | an error object to signal a fatal exception and will fail the entire page of records. |
module.hooks.postAggregateFunction = function (options, callback) { return callback(err) }
Comments
5 comments
Excellent article as I get into more javascript hooks. Question: in testing and using the preview functionality of the j-hook window, is there a way to preview with more than a single object in the data[] array? Even with "All" selected in the initial export flow step, I only get data[0].
Thank you!
Jim Kelleher
(As an aside, love that you use i, j, and k as iteration variables. Takes me back to my Fortran days.)
Hi PSA Admin!
Thanks so much for the great feedback! Speaking of old languages, I still try to forget my COBOL days! ;-D I'm checking into the answer on this for you and will get back to you asap.
Hi PSA Admin,
For now, you need to copy and paste or type in additional records into the data[] arg. We're looking at having the ability to preview more than one record in the export panel for next year, which will make this process easier. You letting us know about this need is helping us consider adding the ability to pull more than one record into the Advanced Field Editors (AFEs), so thank you!
I would also like the same functionality described by PSA Admin. Whenever I need to test against more than one object I must copy it from the debugger and then paste it into the template. It would save a lot of time to have this functionality.
Hi Shane Brown,
Thanks so much for letting us know. Could you post this enhancement idea to our community in the enhancement requests area? We are working on a feature to store and view sample data throughout the flow to show how it's being impacted at each step, as a result of PSA Admin's insights in improving this experience. Would you be interested in previewing what we're designing now?
PSA Admin, we're already planning to show you what we're working on :). Thank you so much for your invaluable help!
Please sign in to leave a comment.