Best practices to optimize throughput and performance during high-volume periods (holiday guide 2020)

Contents

• Common leading practices for all eCommerce Integration Apps
  • Concurrency
  • Batch size and page size
  • Export and import
  • Optimize multiple connections in your Integration App
  • Other leading practices
  • Other factors that drive NetSuite performance
• Shopify - NetSuite Integration App
• Amazon - NetSuite Integration App
• eBay - NetSuite Integration App
• Magento 2 - NetSuite Integration App
• Square - NetSuite Integration App

The article provides an overview of the eCommerce Integration App leading practices and recommendations for the holiday season to get maximum throughput. These recommendations can help you to handle high volume data efficiently, especially during the holiday season where the order volume is high.

Common leading practices for all eCommerce Integration Apps

It is recommended to follow the below instructions to configure your Integration Apps during the holiday season for better throughput and performance. There are multiple factors that impact throughput. Let's look at each of the factors below.

• Concurrency
• Batch size and page size
• Export and import
• Other factors that drive the NetSuite performance

Concurrency

Concurrency is defined as two or more events processing at the same time. Each page is processed by a thread or concurrency. If the concurrency is more, then more number of pages are processed parallelly. For more information, see Govern concurrency of NetSuite connections.

Example: In NetSuite, if you have a SuiteCloud Plus license, and if the integration app concurrency per integration tile is 10. It means that the NetSuite account can manage 10 parallel requests at a given time. Therefore, the throughput of the system is directly linked to the concurrency set up.

Note: The concurrency is applicable only for imports (It is not applicable to get data from the source application). Concurrency specific to the other applications is covered in its respective Integration App section. 

Concurrency calculation in NetSuite

SC = SuiteCloud Plus license

A number of concurrent requests = NetSuite Service Tier + Number of SC plus licenses.

Example: If NetSuite service tier "3 or Shared" and has 2 SC+ licenses, it will be: 5 (tier three base concurrency) + 2*10 = 25.

You can view this final number in NetSuite at Setup > Integration > Integration Management > Integration Governance. To increase the concurrency further, contact NetSuite support. 

For additional information, you can refer to these documents. It also provides more details about NetSuite tier information and concurrency cheat sheets.

Important: As of now the integrator.io platform supports a maximum of 25 concurrency per connection. If you require more NetSuite integration concurrency, it is recommended to create a secondary connection record and partition flows to use the two connections (currently supported only for the Amazon-NetSuite Integration App).

Batch size and page size

• Batch size is defined as the number of records fetched in a single request. This parameter is only applicable to the NetSuite export.
• Page size is defined as the number of records to be processed per page. This parameter is applicable to the destination application.

How does the batch size work in accordance with the page size?

If you define the batch size as 500 and page size as 25 when you make a request to NetSuite, it responds back with 500 results that are equally divided into 25 pages, which is 20 records per page. The batch size and page size also depends on the concurrency set on the respective connection. If the concurrency is set to 3, the Integration App submits 3 pages at a time. You have to define batch size, page size, and concurrency logically. 

Example: For the inventory export flow, if you set the batch size as 1000, the page size is 100, and if the concurrency is set as 5, in this scenario, the Integration App processes a minimum of 100 items and a maximum of 500 items per request. If the concurrency is not used by the other Integration App flows, then the Integration App processes a maximum of 500 items per request.

Export and import

Any integration flow has export and import.

• Exports are used to extract data from an application. Exports can run standalone, or in the context of an integration flow. The data is exported from an application in pages. Each page contains a set of data records from the application. For more information on exports, see Export data from NetSuite and Preview export data.
• After you run the integration flow export, it starts to create and assign pages to the import (in an asynchronous manner). Based on the concurrency set on the import connection, the pages are consumed in parallel. 

Important: If the flow fails with SSS_TIME_LIMIT_EXCEEDED error, then reduce the “batchSize” in the export, so that there are less number of records in a single page to be processed by the import.

Optimize multiple connections in your Integration App

You can add additional connections to a single endpoint through which you can exercise greater control over the volume of data that is requested through the connections. This flexibility is key to getting high-volume flows to run more quickly.

When you set up multiple connections for flows based on logical prioritization rules and available “concurrency,” you can optimize the connections for improving data throughput to your API app, file server, or database system.

For more information, see Optimize multiple connections in your Integration App flows for better throughput.

Other leading practices

• Schedule your integration flows in different quarter hours.
  
  Example: If you schedule the inventory flow at 12:10, schedule the fulfillment flow at 12:25, rather than having the same scheduled time.
• In NetSuite, Log Level of all the Celigo scripts should be Error. To verify this, in NetSuite, go to Customization > Scripting > Scripts. Use the Bundle ID or Celigo, which is prefixed to the names of all the scripts, to identify Celigo Scripts. For each script, go to Deployments and set Log Level to Error.
• Uncheck the Always sync inventory levels for entire catalog checkbox to avoid syncing the inventory levels for the entire catalog. Instead, sync only a few items that have changed since the last time the flow ran. To do this, go to Settings > Inventory of your Integration App.
• It is recommended to load all the items before the holiday season. During the holiday season, lower the frequency of item export to as low as once per day. Item export is a data-intensive flow. Therefore, it consumes a lot of processing capacity and time.
• You must refrain from bulk retries, especially for the low priority flows.
• For Inventory or Item or Pricing flow, do not retry for old errors because the data can be outdated.
• Don’t change the key settings or mappings just before the holiday season.

Other factors that drive NetSuite performance

In NetSuite, you need to identify the number of import scripts or workflows deployed on the NetSuite record from Customization > Scripting > Scripting Records. Many times your NetSuite account might have a user event script or workflow that reduces the performance and throughput. It is recommended to reduce or optimize the scripts which are performing heavy processing. You can contact NetSuite support to identify such user events (refer to the image for the typical data provided by NetSuite support).

These scripts can significantly increase the record creation time in NetSuite. It is recommended to use the on-demand functionality to import a single record in NetSuite and check the time required to import the same. 

Based on the integration app settings, records might be resubmitted in NetSuite (for instance overwrite tax), this might slightly increase the record creation time or update time. In this scenario, it is recommended to use the “default customer” setting.

Note: If NetSuite is slow, the jobs pile up in integrator.io and take a lot of time to complete. If you see more concurrent queues (let's say 20), and NetSuite is slower than usual (it might take 15 seconds for record creation), you can leverage the concurrency setting and reduce the export page size. This helps to generate more pages in integrator.io and hence more concurrent queues are consumed in NetSuite to import transactions.

Shopify - NetSuite Integration App

Connections

• Increase NetSuite connection's concurrency level. However, considering Shopify limits, the concurrency level should not be more than 15. To do this, go to Connections. Next to the NetSuite connection, below Actions, click on the three-dotted icon > Edit connection > Advanced > Concurrency Level.
• Shopify concurrency: The recommended concurrency for Shopify (standard) users is 1 and for Shopify Plus is 3. The assigned concurrency is applicable for all the linked integration flows for that particular connection. 
• You can set a maximum of 3 concurrencies to the Shopify Plus connection in the Integration App. If you require more concurrency, it is recommended to create a Private App.
•  Shopify provides concurrency to any private app connection as per business needs.

Sandbox environment integrations

In Shopify, the rate limits work as per the Application-Store combination. It's better if you disable all the sandbox environment flows to see maximum throughput for the production environment flows.

Order sync from Shopify to NetSuite

• Increase the NetSuite connection concurrency to a higher level as required. If you are using the “Shopify Order to NetSuite Order/CashSale Add” batch flow, there won’t be any issue with the higher NetSuite concurrency. The real-time flows utilize only 1 concurrency even if the NetSuite connection has more concurrency. With respect to other batch flow, it utilizes up to the maximum available concurrency limit.
•  “Shopify Order to NetSuite Order/CashSale Add” batch flow:
  1. To run this flow, It’s always recommended to select the Creation time option in the Filter orders by setting. (Flows > Order > next to “Shopify Order to NetSuite Order/CashSale Add” flow > Settings)
  2. It’s recommended to input very minimal lagOffSet in the Lag time to update the order status setting (Flows >  Order > “Shopify order to NetSuite order/cash sale” flow settings) based on the business use case.
     
     
     
     
• If you sync your orders using batch and real-time flows, it’s highly recommended to run the batch flow less frequently (once or twice a day).
• To reduce the item lookup errors, enter a value in the NetSuite SKU field setting and provide default values in the NetSuite item to track order lines when no matching SKU is found in NetSuite and NetSuite item to track Shopify order lines with no SKU value settings ( Settings > Order > Item sub-tab). 
• Undeploy unnecessary scripts on the NetSuite SalesOrder, CashSale (only if you’re sure.)

Tax setup

• Set up tax codes/components properly in NetSuite or use the integratori.io tax configuration in the Settings > General > Tax sub-tab:
  • Add total tax against single line item on the order OR
  • Overwrite taxes with tax with Shopify tax amounts with the Assign the default tax code to all orders checkbox checked.

• Using proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.

Customer sync from Shopify to NetSuite

• Use the Default NetSuite customer if missing on Shopify order setting (Settings > Order section > Customer sub-tab) if customer details are not needed in NetSuite. You can sync the customers later using the “Shopify Customer to NetSuite Customer Add/Update” flow.
  
  Important: In this scenario, the customers are not linked to the orders.
• Try to run the “Shopify Customer to NetSuite Customer Add/Update” and “NetSuite Customer to Shopify Customer” flows before the season and update customer data in Shopify and NetSuite to sync the data ahead.

Transaction sync from Shopify to NetSuite

• Typically, the Shopify transaction data is synced to NetSuite for credit card reconciliation purposes. If the transaction data in NetSuite is not being leveraged, the “Shopify Order Transaction to NetSuite Transaction” flow can be temporarily or permanently disabled (Flows > Order). 

Inventory sync from NetSuite to Shopify

• Run the “NetSuite Inventory to Shopify Inventory Add/Update” flow in delta mode to increase the performance by selecting the appropriate export batch size. To increase performance at Shopify end, consider the Private App option:
  1. Create a Private App in your Shopify account.
  2. Authenticate a new connection in your integrator.io account, using the Private App details.
  3. Assign this new connection to the high volume integration flows.
• Sync your inventory levels in advance to the holiday season and run as per the delta-mode during the sale time.
• Uncheck the Always sync inventory levels for entire catalog checkbox (Settings > Inventory) to avoid setting a large amount of data to Shopify.
• Run the “Shopify Product Id to NetSuite Item Mass Update” flow (Flows > Inventory) in prior as it involves huge data. Try to run it before so that both the systems (Shopify and NetSuite) are in sync.
  Example: Schedule the “Shopify Product Id to NetSuite Item Mass Update” flow once before the holiday season to ensure that the item ID map is synced between Shopify and NetSuite. 

Item sync from Shopify to NetSuite

• Try to load all the items before the holiday season, and during the season, lower the frequency of Item export to as low as once per day. 
• Bulk updates or mass updates to item records should be scheduled during non-peak hours, and preferably should be performed in batches.

Other Shopify integration flows

• Sync refunds and cancellations at a very low frequency.

Shopify payout reconciliation add-on

Recommendation: It is recommended not to run the flows to reconcile during the sale period.

You can sync the previous payouts of all sale days once the load on the below flows reduce:

• Shopify Order to NetSuite Order Add 
• Shopify Order to NetSuite Cash Sale Add 
• Shopify Order to NetSuite Order/CashSale Add 
• NetSuite Fulfillment to Shopify Fulfillment Add 
• Shopify fulfillment to NetSuite item fulfillment
• NetSuite Inventory to Shopify Inventory Add/Update 
• NetSuite Kit and Item Group Inventory to Shopify Inventory Add/Update 
• Shopify Product Id to NetSuite Item Mass Update

Amazon - NetSuite Integration App

Connections

• Use Secondary NetSuite Connection for high volume or low priority flows exclusively. This will help in the faster execution of the high priority flows.
  Examples: 
  • Assign high priority flow such as order sync, so that other integration flows don't block the queue.
  • Settlement flows generate higher volume and lower priority data.
• Set the appropriate concurrency for both the primary and secondary NetSuite connections. The connection serving high priority flow, such as orders and fulfilments should take a higher count of concurrency.

Order sync from Amazon to NetSuite

Note: The legacy SuiteScript (SS) connector is deprecated.

• If you have migrated from the legacy SuiteScript (SS) connector to the integrator.io Integration App and have been live on integrator.io for more than a month, remove the additional lookup criteria to check for old the iSuiteScript (SS) connector orders to optimize the flow. 
• Set the following recommended settings for the date filter so that the flow works in the delta mode.
  • Filter Orders By: Last Modified
  • Duration in Days: empty
  • Go Live Date: empty (in case account is live for months now)
• Be sure to check the Dashboard for any errors, you will have to resolve the errors and retry to sync the orders.
• The throttling limit ~75 feed requests per hour and not more than 15 at a time. So if there are more requests, MWS starts throttling. You can utilize the auto-retry mechanism, however, you can retry the data only up to 5 times, post which an error message is displayed on the Dashboard. You can retry the error to sync the details correctly. For more information, see Throttling: Limits to how often you can submit requests.

Tax setup

• Set up all tax codes/components properly in NetSuite or use the integratori.io tax configuration in the Settings > General > Tax sub-tab:
  • Add total tax against single line item on the order OR
  • Overwrite taxes with tax with Amazon tax amounts with the Assign the default tax code to all orders checkbox checked.
• Using proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.
• The Enable GST/VAT for your marketplace setting is added to allow a dynamic approach for the GST related orders.
• If you have a static GST set up (same tax rate for all orders), in Settings > General > Tax sub-tab, check the Enable GST/VAT for your marketplace checkbox and enter the GST/VAT rate in the Enter GST/VAT rate to apply across all orders setting.
• If you have a dynamic GST set up (tax rate changes for all the orders), check the Enable GST/VAT for your marketplace and Apply GST/VAT rate reported by Amazon checkboxes to deduct the tax from item amount and calculate the tax. 
• Check the Exclude Tax if Marketplace Tax Collection applies checkbox (Settings > General > Tax sub-tab) to avoid the tax collection if Amazon collects and remits tax (varies region to region and is decided by Amazon).

Fulfillment sync from NetSuite to Amazon

• Check the  Allow syncing NetSuite shipments with no tracking number checkbox (Settings > Fulfillment), if you don’t want the Integration App to validate the tracking number.
• Schedule the “NetSuite Fulfillment to Amazon Shipment Add” flow to submit only 15 requests to Amazon per hour.
  Example: Schedule the “NetSuite Fulfillment to Amazon Shipment Add” flow twice in an hour with the batch size as 200 and page size as 50. The Integration App can manage 500 orders with these batch size and page size numbers.
• Check the Automate shipment carrier mapping checkbox to (Settings > Fulfillment), to automatically map the shipping carriers in Amazon. Else, the mapping has to be done manually using the field mappings page.

Inventory sync from NetSuite to Amazon 

• Uncheck the Always sync inventory levels for entire catalog checkbox (Settings > Inventory) to avoid setting a large amount of data to Amazon.
• Schedule the “NetSuite Inventory to Amazon Inventory Add/Update” and “NetSuite Kit Inventory to Amazon Inventory Add/Update” flows to submit only 15 requests to Amazon per hour.
  Example: Schedule the inventory flows once is 2/4 hours in the delta mode.

Item sync from NetSuite to Amazon

• Load all the items before the holiday season, and during the season, lower the frequency of Item sync to as low as once per day. Item export is a heavy flow and takes time and processing on Amazon to perform.
• Bulk updates or mass updates to the item records should be scheduled during non-peak hours, and preferably done in batches.

Pricing sync from NetSuite to Amazon 

• Schedule the “NetSuite Item Price to Amazon Product Price Add/Update” flow to submit only 15 requests to Amazon per hour.
  
  Note: We don’t recommend enabling this flow during the holiday season. However, as the price fluctuates during the holiday season, schedule this flow once in an hour only with the required set of mappings. 

Amazon Transfer Order - InboundShipments add-on

• It is recommended that you must update the inventory levels in Amazon much ahead of the sale period

Amazon FBA Inventory Adjustments add-on

• It is recommended to use this add-on only once or twice a day.
• Configure this add-on using the secondary connection where low priority flows are attached.

eBay - NetSuite Integration App

Order sync (eBay to NetSuite)

• Set the recommended settings for date filter setting so that flow works in the delta mode:
  • Filter Orders By: Last Modified
  • Duration in Days: empty
  • Go Live Date: empty (in case account is live for months now)
• Map the correct fields for SKU so that the duplicate item IDs are not created for items.

Tax setup

• Set up tax properly in NetSuite or use the integratori.io tax configuration in the Settings > General > Tax sub-tab:
  • Add total tax against single line item on the order OR
  • Overwrite taxes with tax with eBay tax amounts with the Assign the default tax code to all orders checkbox checked.
• Using proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.
• Check the Exclude Tax if Marketplace Tax Collection applies checkbox in the Settings > General section > Tax tab in order to avoid the tax collection where eBay collects and remits tax (varies region to region and is decided by eBay).

Inventory sync (NetSuite to eBay)

• Uncheck the Sync All Inventory check box to avoid sending a large amount of data to eBay.
• When you sync the matrix item inventory from NetSuite to eBay, set the eBay Integration App concurrency to 1. After you sync the matrix item inventory, modify the concurrency back to its original level.
  
  Note: It is recommended to sync matrix item inventory during non-holiday season time.
• Disable the “eBay Item Id to NetSuite Mass Update” flow, after you sync all the Item IDs to all eBay items.

Magento 2 - NetSuite Integration App

Order sync

• Keep Magento2 instance optimized, cached, and compiled to improve the API response performance for the order import and item/inventory export.

Tax setup

• Set up tax properly in NetSuite or use the integratori.io tax configuration in the Settings > General > Tax sub-tab:
  • Add total tax against single line item on the order
  • Overwrite taxes with tax with Magento2 tax amounts with the Assign the default tax code to all orders checkbox checked.
• Using proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.

Inventory sync (NetSuite to Magento2) 

• Uncheck the Always sync inventory levels for entire catalog check box (Settings > Inventory) to avoid setting a large amount of data to Magento.

Item sync (NetSuite to Magento2)

• Try to load all the items before the holiday season, and during the season, lower the frequency of Item export to as low as once per day. Item export is a heavy flow and takes time and processing on Magento to perform.

Square - NetSuite Integration App

As the Square-NetSuite Integration App is based on a POS solution, you can sync the data normally following the Integration App functionality. Few suggestions are:

• Schedule the “Square Order to NetSuite Cash Sale” flow with a possible higher frequency.
• If possible, do not sync tenders.
  
  Note: Enable the “Square Tenders to NetSuite Custom Payments” flow if you have to sync refund imports of cash sales that have a gift card applied.
• Set up the Square POS locations and be sure they are synced and selected in the Select active Square locations * setting in the Settings > General section.
• If possible, you can leverage the customer sync and disable the “Square Customer to NetSuite Customer” flow, and add a default customer for all the cash sales.
  
  Note: You cannot sync these customers again.
  
  
• For cash refund sync, it is recommended to run the “Square Refund to NetSuite Cash Refund” flow with a lesser frequency, once or twice in a day for the entire sale season.
• Be sure that the concurrency for the NetSuite connection is 2 or 3, as the remaining concurrency can be utilized for other Integration Apps.