integrator-extension

Celigo's integrator-extension provides an easy-to-use framework for extending integrator.io through various functions that you can host on multiple environments (both server-based or server-less). The integrator-extension supports execution of extension functions through two extension types:

• express-integrator-extension: Express-based app running on a server
• lambda-integrator-extension: Amazon Web Services (AWS) Lambda hosting

Contents

Installation

Using npm:

$ npm i --save integrator-extension

Configuration

The integrator-extension provides a loadConfiguration function that accepts a configuration object. Use the configuration object to set up extension functions for both do-it-yourself (DIY) integrations and connectors.

Common configuration properties

diy: This property allows you to create an integration from scratch in integrator.io. Use this field to assign a value containing all hook and wrapper functions for the integration.

Note: The configuration object can have only one diy field.

connectors: A connector represents a fully functional standalone integration that any user can install directly from the integrator.io Marketplace. Set the value for this field to an object containing a list of key-value pairs, where each key is a _connectorId of a given connector and the value is the node module (or object) containing the hooks, wrappers, installer, uninstaller and settings functions for that connector. The configuration properties available vary according to extension type. See the repositories for more details.

Note: The configuration object can have only one connectors field.

Extension functions

Extension functions allow you to write custom code to further customize and enhance integrator.io. Integrator.io supports the following functions:

Note: In this topic, the term value should be interpreted as defined at www.json.org, unless mentioned otherwise.

preSavePage (options, callback)

This function is invoked before saving a page of data that the export retrieved for further processing. You can use this function to add or delete records or modify the existing records in the data passed to the hook preSavePage.

The options argument has the following fields:

{
    code: '',
    message: '',
    source: ''
}

The function returns a callback object with the following fields:

Note: Throwing an exception signals a fatal error and stops the flow.

exports.preSavePageFunction = function (options, callback) {
  // sample code that passes on exported data
  return callback(error, {
    data: options.data,
    errors: options.errors,
    abort: false
  })
} 

preMap (options, callback)

This function is invoked before the fields are mapped to their respective fields in the objects to be imported. This function can be used to reformat the record’s fields before they get mapped.

The name of the function can be changed to anything you like.

The options argument has the following fields:

The function returns an array, and the length must match the options.data array length. Each element in the array represents the actions that should be taken on the record at that index.

Each element in the array should have the following fields:

{
    code: '',
    message: '',
    source: ''
}

Returning an empty object {} for a specific record indicates to integrator.io that the record should be ignored.

Throwing an exception fails the entire page of records.

exports.preMapFunction = function (options, callback) {
  return callback(error, options.data.map((d) => { return { data: d }})
}

postMap (options, callback)

This function is invoked after the fields in the source objects have been mapped to their respective fields in import object. This function can be used to further modify the mapped data.

The name of the function can be changed to anything you like.

The function is passed an options argument with the following fields:

The function returns an array, and the length must match the options.data array length. Each element in the array represents the actions that should be taken on the record at that index.

Each element in the array should have the following fields:

{
    code: '',
    message: '',
    source: ''
}

Returning an empty object {} for a specific record indicates to integrator.io that the record should be ignored.

Returning both data and errors for a specific record indicates that integrator.io should process the record and also log errors on the job.

Throwing an exception fails the entire page of records.

exports.postMapFunction = function (options, callback) {
  return callback(error, options.postMapData.map((d) => { return { data: d }}))
}

postSubmit (options, callback)

This function is invoked after the import processes the records. You can use it to further process imported objects and modify the response data received from an import’s success and error cases. You can also use it to invoke other processes to run after the import.

The name of the function can be changed to anything you like.

The function is passed an options argument with the following fields:

{ statusCode: 200/422/401, errors: [], ignored: true/false,
        id: '', _json: {}, dataURI: '' }

[{code: '', message: '', source: ''}]

The function is expected to return the responseData array provided by options.responseData. The length of the responseData array must remain unchanged. You can modify elements within the responseData array to enhance error messages, modify the complete _json response data, and so forth.

Throwing an exception fails the entire page of records.

exports.postSubmitFunction = function (options, callback) {
  return callback(error, options.responseData)
}

postResponseMap

The name of the function can be changed to anything you like.

This hook runs immediately after response mappings and regular mappings. Use this hook to process records after response mapping before they are passed on to downstream applications in your flow. Any changes that you made to records with the postResponseMap hook will affect ALL downstream applications. You do not need to define a response/results mapping to use this hook. You could also use this hook instead to perform the same mapping type logic. For more detailed information about how postResponseMap works, please read the help text provided in the default function stub.

The function's options argument has the following fields:

{ statusCode: 200/422/401, errors: [], ignored: true/false,
        id: '', _json: {}, dataURI: '' }

[{code: '', message: '', source: ''}]

The function needs to return the postResponseMapData array provided by options.postResponseMapData.

The length of postResponseMapData MUST remain unchanged. Elements within postResponseMapData can be changed however needed.

Throwing an exception will signal a fatal error and fail the entire page of records.

exports.postResponseMapFunction = function (options, callback) {
  return callback(error, options.postResponseMapData)
}

postAggregate

postAggregrateFunction: The name of the function can be changed to anything you like.

The function is passed an options argument with the following fields:

The function doesn't need a return value.

Throwing an exception signals a fatal error.

exports.postAggregateFunction = function (options, callback) {
  return callback(error)
}

formInit

The name of the function can be changed to anything you like.

The function will be passed one options argument that has the following fields:

The function returns a form object for the UI to render.

Throwing an exception signals an error.

exports.formInitFunction = function (options, callback) {
  return callback(error, options.resource.settingsForm.form)
}

preSave

The name of the function can be changed to anything you like.

The function is passed an options argument with the following fields:

The function returns the newResource object to save.

Throwing an exception signals an error.

exports.preSaveFunction = function (options, callback) {
  return callback(error, options.newResource)
}

step

This function represents one step in the installation or uninstallation process. The step functions can be used to create resources, validate bundles, etc.

The function does not need to return anything.

Throwing an exception signals an error.

exports.stepFunction = function (options, callback) {
  return callback(error)
}

initChild

Use this function to create a child integration.

The function does not need to return anything.

Throwing an exception signals an error.

exports.initChild = function (options) {
  return { name: "child integration", installSteps: [...] }
}

update

Use this function to update the integrations belonging to a connector. Whenever, new versions of the connector need to be released or there are any new features in a connector, use the update function to make the required changes for integrations installed in user accounts.

The name of the function can be changed to anything you like.

The function is passed an options argument that has the following fields:

The function does not need to return anything.

Throwing an exception signals an error.

exports.updateFunction = function (options, callback) {
  return callback(error)
}

changeEdition

The name of the function can be changed to anything you like.

The function will be passed one options argument that has the following fields:

The function does not need to return anything.

Throwing an exception signals an error.

exports.changeEditionFunction = function (options, callback) {
  return callback(error)
}

ping

Call this function to verify whether the connection defined for the wrapper adaptor is valid. The options object contains the following properties:

The function calls back with the following arguments:

{ statusCode: 200/401, errors: [{message: 'error message', code: 'error code'}]}

exports.pingConnectionFunction = function (options, callback) {
 return callback(error, response)
}

export

A wrapper export allows you to write a full export and substitute the actual implementation of an integrator.io export. The wrapper export function returns back pages of data and is called by integrator.io again and again until all the records have been returned to integrator.io.

{encrypted: {...}, unencrypted: {...}}

{dateField: ..., lastExecutionTime: ...}

{booleanField: ...}

 {limit: ...}

The function calls back with the following arguments:

{message: 'error message', code: 'error code'} 

exports.exportFunction = function (options, callback) {
  return callback(error, response)
}

Note: No further calls will be made to a wrapper function if the same response is sent more than five times to integrator.io during an export process by the wrapper function. integrator.io will consider this as an infinite loop and stop the export process.

import

A wrapper import allows you to write a full end-to-end import and substitute the actual implementation of an integrator.io import. The options object contains the following properties:

{
encrypted: {...}, 
unencrypted: {...}
}

The function calls back with the following arguments:

{statusCode: 200/401/422, id: string, ignored: boolean, errors: [{code, message}]}

[{message: 'error message', code: 'error code'}]

exports.importFunction = function (options, callback) {
  return callback(error, response)
}

express-integrator-extension

This module extends the integrator-extension and can be used to create an express app which exposes an API that is used by integrator.io to invoke the extension functions and get back the results. This extension type is based on the stack of type server in integrator.io. The baseURI for the API will be same as the hostURI of the stack and the stack's systemToken is used to authenticate the requests made to the API. You can host the express app on a single server or a set of servers behind a load balancer to scale as needed.

express-integrator-extension Installation

Using npm:

$ npm i --save express-integrator-extension

express-integrator-extension usage

var expressExtension = require('express-integrator-extension')
expressExtension.createServer(config, function (err) {
})
expressExtension.stopServer(function (err) {
})

express-integrator-extension createServer(config, callback)

The createServer function loads the configuration and starts an express app. You can set the following configuration fields along with the fields mentioned in integrator-extension configuration:

Getting Started

Use the following steps to create a working Express-based stack:

1. Create a stack in integrator.io.
2. Log in into integrator.io.
3. Click the Options field in the top right corner, and select Stacks.
4. Click the New Stack button.
5. Give an appropriate name for the stack.
6. Select Server as the Type.
7. Set the Host field to the URI hosting the project. If the URI is not available, you can leave it blank and update the stack at a later time.
8. Click Save to create the stack.
9. You will be redirected to the stacks page. To find the token value, click click to display under the System Token column corresponding to the stack you created.
10. Write code
11. Run npm init to create node project in a new folder.
12. Run npm i --save express-integrator-extension.
13. Create a new file functions.js and save the following extension functions in it:

    var obj = { hooks: { preSaveFunction: function (options, callback) { // your code }
    },
    wrappers: { pingFunction: function (options, callback) { // your code } } };
    module.exports = obj;

14. Create a new file index.js and save the following code in it. Either the DIY or connectors configuration should be used.

      var expressExtension = require('express-integrator-extension');
      var functions = require('./functions');
      var systemToken = '************'; // Set this value to the systemToken of the stack created in integrator.io
      var options = {
      diy: functions,
      // connectors: { _connectorId: functions }, // for connectors
      systemToken: systemToken  };
      expressExtension.createServer(options, function (err) {
      });

15. Host the node project on a single server or a set of servers behind a load balancer to scale as needed. Any server based hosting environment like AWS EC2, AWS Opsworks, Rackspace Cloud, Google Cloud, Heroku, etc. can be used to run the code. Update the stack in integrator.io with the endpoint URI of your server or load balancer.

Now, the stack is ready for use and it can be referenced from appropriate exports, imports and connections. To maintain security, systemToken should be stored securely on your server environment. Please refer to best practices of the respective system that you chose to run the express app.

lambda-integrator-extension

This module extends integrator-extension and can be used to host extension functions on AWS Lambda. This allows to run the extension code in a server-less environment that helps in simplifying the deployment and maintenance process. To use this module your stack must be of type lambda in integrator.io. integrator.io uses the AWS IAM User Access Keys provided on the stack to invoke the extension functions and get the results. The associated AWS IAM user must be assigned "lambda:InvokeFunction" permission for the AWS Lambda Function provided on the stack via an attached IAM policy. Detailed steps to create an AWS Lambda stack in integrator.io are given below.

lambda-integrator-extension installation

Using npm:

$ npm i --save lambda-integrator-extension

lambda-integrator-extension usage

var lambdaExtension = require('lambda-integrator-extension')
exports.handler = lambdaExtension.createHandler(config)

lambda-integrator-extension createHandler(callback)

The createHandler function loads the configuration and returns the handler that should be used when creating the AWS Lambda Function. The result of execution of createHandler function needs to be assigned to exports.handler which in turn executes whenever integrator.io sends a request to execute any of the extension functions.

lambda-integrator-extension Getting Started

Use the following steps to create a working AWS Lambda stack.

1. Create an AWS account. As part of the sign-up validation procedure you will receive a phone call.
2. Create an IAM Policy.
   1. Go to Identity And Access Management page by clicking Security Credentials from the drop down under your account name.
   2. On the navigation column on the left, choose Policies.
   3. Click Create Policy at the top of the page.
   4. On the Create Policy page, select Create Your Own Policy.
   5. Give an appropriate name for the policy in name field.
   6. Add the following in the policy document text box:

      {
       "Version": "2012-10-17",
       "Statement": [{
       "Effect": "Allow",
       "Action": [
       "lambda:InvokeFunction"
       ],
       "Resource": [
       "*"
       ]
       }]
       } 

   7. To validate the policy click Validate Policy.
   8. If the validation passes then click Create Policy.
3. Create an IAM User in AWS
   1. Go to the Identity And Access Management page.
   2. Select Users in the left pane.
   3. On the Create Users page enter the user name.
   4. Ensure that Generate an access key for each user is checked.
   5. Click Create.
   6. Click Show User Security Credentials and make a note of the Access Key ID and Secret Access Key.
   7. Click the Permissions tab for the user and attach the policy created above.
4. Write code
   1. Run npm init to create the node project in a new folder.
   2. Run npm i --save lambda-integrator-extension.
   3. Create a new file functions.js and save the following extension functions in it.

      var obj = {
       hooks: {
       preSaveFunction: function (options, callback) {
       // your code
       }
       },
      wrappers: {
       pingFunction: function (options, callback) {
       // your code
       }
       }
       }
      module.exports = obj

   4. Create a new file index.js and save the following code in it. Use either the DIY or connectors configuration.

      var extension = require('lambda-integrator-extension');
       var functions = require('./functions');
      
      var options = { diy: functions };
       // var options = { connectors: { _connectorId: functions } }; // for connectors
       exports.handler = extension.createHandler(options);

   5. Create a zip of the project folder with an archiving utility. It should have package.json, index.js, functions.js, and node_modules in it.
5. Create a Lambda function on AWS
   1. Go to Lambda through Amazon web services dashboard.
   2. Click Create a Lambda Function.
   3. On the Select blueprint page click Skip.
   4. Click Next on the Configure triggers page.
   5. Provide a name for the function.
   6. Select runtime environment as Node.js 8.10.
   7. In the Lambda function code section, select code entry type as Upload a .ZIP file.
   8. Click Upload to upload the zip file that was created earlier.
   9. Set handler to index.handler.
   10. Select Choose an existing role in the Role field.
   11. Select the default lambda_basic_execution role in the Existing role field.
   12. In the Advanced settings section, click Update memory based on your requirement and choose timeout as 300 seconds.
   13. Let the VPC field be set to No VPC.
   14. Click on next button.
   15. Review the configuration and click Create Function.
6. Create a stack in integrator.io
   1. Log in to integrator.io.
   2. Click Options at the top right corner, and select Stacks.
   3. Click New Stack.
   4. Give an appropriate name for the stack.
   5. Select type as AWS Lambda.
   6. Update the Access Key Id, Secret Access Key and AWS Region with the IAM User's properties that you saved earlier.
   7. Set the function name to the function name of the AWS Lambda function that you previously created.
   8. Click Save.

Now, the stack is ready for use and it can be referenced from appropriate exports, imports and connections. The steps above list out the bare minimum setup required to run an AWS Lambda stack. Customize the permissions and other settings according to your own requirements and controls.

Authentication

Bearer tokens

You can generate and access API tokens via Resources > API Tokens, when signed in with Developer mode.

One-time tokens

The integrator.io extension framework supports the use of one-time tokens. The options argument passes one-time tokens to each of your functions and can perform call backs to integrator.io (similarly to bearer tokens). One-time tokens automatically expire after use (or if unused, after 15 minutes of creation). For Integration Apps, one-time tokens can only invoke the resources that belong to the Integration App.

Rate limiting

The integrator.io API is rate limited using a leaky bucket algorithm with a bucket size of 1000 and a fill rate of 300 tokens every 1 second, which approximates to 1,080,000 requests allowed per hour.

Headers

The integrator.io API supports the following custom HTTP headers to facilitate proxying requests directly to a NetSuite RESTlet.

• Integrator-NetSuite-ScriptId
• Integrator-NetSuite-DeployId