Best practices to optimize eCommerce app throughput & performance during high-volume periods (holiday season)

This article provides an overview of the eCommerce integration best practices and recommendations for the holiday season to achieve maximum throughput. Follow these recommendations to prepare your integrations to handle data efficiently during the holiday season when order volume is high.

Contents

• Common best practices for all eCommerce integrations
  • Configure connections to optimize throughput and governance
  • Configure flow steps to optimize throughput and governance
  • Miscellaneous best practices
• NetSuite performance best practices
• Shopify – NetSuite integration app
• Amazon – NetSuite integration app
• eBay – NetSuite integration app
• Magento 2 – NetSuite integration app
• Payout to Reconciliation Automation for NetSuite
• Square – NetSuite integration app
• BigCommerce – NetSuite integration app
• Shopify - SAP Business ByDesign integration app

Common best practices for all eCommerce integrations

Configure connections to optimize throughput and governance

• Configure connections to optimize throughput and governance
• Leverage multiple connections in your Integration Apps for better throughput
• Optimize multiple connections in your Integration App flows to improve throughput

Configure flow steps to optimize throughput and governance

• Exports and listeners
• Imports

Miscellaneous best practices

• In the integration app > Settings > Inventory section, uncheck the box Always sync inventory levels for the entire catalog to avoid syncing the inventory levels for the entire catalog. Instead, sync only a few items that have changed since the last time the flow ran.
• We recommend loading all items before the holiday season. During the holiday season, lower the frequency of item export to as low as once per day. Item export is a data-intensive flow, consuming a lot of processing capacity and time.
• You must refrain from bulk retries, especially for low-priority flows.
• For Inventory, Item, or Pricing flows, do not process retries for old errors because the data can be out of date.
• Don’t change key settings or mappings just before the holiday season.

NetSuite performance

• If you need to increase the total concurrency level across all your NetSuite connections beyond your current NetSuite imposed account limit, then you will need to contact NetSuite support. You can view your NetSuite concurrency limit here: Setup > Integration > Integration Management > Integration Governance.
• In NetSuite, the Log Level of all Celigo scripts should be Error. To verify in NetSuite, go to Customization > Scripting > Scripts. Look for the bundle ID or Celigo, which is prefixed to the names of all the scripts. For each script, go to Deployments and set Log Level to Error.
• In NetSuite, you can identify the number of import scripts or workflows deployed on the NetSuite record from Customization > Scripting > Scripting Records. Many times your NetSuite account might have a user event script or workflow that reduces the performance and throughput. We recommend reducing or optimizing the scripts that are performing heavy processing. You can contact NetSuite support to identify such user events (refer to the below image for the typical data provided by NetSuite support).These scripts can significantly increase the record creation time in NetSuite. We recommend a) running the on-demand flow to import a single record in NetSuite and b) checking the time required to import it.
• Based on the integration app settings, records might be resubmitted in NetSuite (for instance for overwrite tax), which might slightly increase the record creation time or update time. In this scenario, we recommend using the Default NetSuite customer if missing on the order in the integration app Settings > Order section > Customer subtab.

Shopify – NetSuite integration app

• Shopify concurrency: The recommended concurrency for Shopify (standard) users is 1 and for Shopify Plus is 3. The assigned concurrency governs all the linked integration flows for that particular connection. For any Shopify connection authenticated using the oAuth application, Shopify gives a maximum of 4 concurrencies. If you need more concurrency for the same store, it is recommended to create a custom app.
• If you require more concurrency, we recommend creating a Custom App. Shopify provides additional concurrency to any custom app connection. You can connect to the custom app only using basic authentication.

Important: Do not assign any real-time integration flows to the additional connections created using a custom app.

Sandbox environment integrations

In Shopify, the rate limits work according to your Application-Store combination. It’s better if you disable all of the sandbox environment flows to see maximum throughput for the production environment flows.

Order sync from Shopify to NetSuite

• The Shopify order to NetSuite cash sale (add) and Shopify order to NetSuite sales order (add) real-time flows can sync only one order per page. During the holiday season, it is instead preferable to run the Shopify order to NetSuite order (cash sale or sales order) batch flow, which syncs multiple orders per page.
  

• Shopify order to NetSuite order (cash sale or sales order) batch flow:
  • To run this flow, we strongly recommend selecting the Creation time option in the Filter orders by setting. See Flows > Order > next to the Shopify order to NetSuite order (cash sale or sales order) flow settings.
  • We recommended entering very minimal lagOffSet in the Lag for the next flow run setting – Flows >  Order > Shopify order to NetSuite order (cash sale or sales order) flow settings – based on the business case.

• Set appropriate values in the Lag for the next flow run and Sync Orders created prior to settings so that orders don’t get missed in the batch order flow runs.
  
  Notes:

  • The Lag for the next flow run setting reduces the set amount of time from the “start date” of the flow run.
  • The Sync Orders created prior to setting reduces the set amount of time from the “end date” of the flow run.
• For specific order imports, we recommend running the Sync Shopify order on-demand to NetSuite (add) flow.
• To reduce the item lookup errors, enter a value in the NetSuite SKU field setting and provide default values in the NetSuite item to track order lines when no matching SKU is found in NetSuite and NetSuite item to track Shopify order lines with no SKU value settings (Settings > Order section > Item tab).
• Undeploy unnecessary scripts on the NetSuite sales order and cash sale, if you’re certain that their absence won’t affect your account or flows.
• For integration app premium edition customers: check the Buy online, pick up in-store setting (Settings > Order > POS tab) only if the Shopify POS feature is applicable. It will avoid unnecessary calls to fetch order fulfillment during order import.
• For the integration app premium edition users, check the Buy online, pick up in-store setting (Settings > Order > POS tab) only if the Shopify POS feature is applicable. This will avoid unnecessary calls to fetch the order fulfillments during order import.
• If you’re using the Shopify Markets edition of the integration app, ensure that the Sync records in local currency checkbox is checked for multi-currency transactions to avoid unnecessary variances.
• Map the province to the corresponding non-inventory item for all state taxes to avoid any variance.

Tax setup

• Set up tax codes/components properly in NetSuite or use the integrator.io tax configuration in the Settings > Order > Tax subtab:
  • Add total tax against single line item on the order
      – or –
  • Overwrite taxes with tax with Shopify tax amounts with the Assign the default tax code to all orders box checked.
• Using a proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.
• Ensure that the GST/VAT related settings are in sync in both Shopify and the integration app.

Customer sync

• From Shopify to NetSuite
  • Use the Default NetSuite customer if missing on Shopify order setting (Settings > Order section > Customer subtab) if customer details are not needed in NetSuite. You can sync the customers later by running the Shopify customer to NetSuite customer (add or update) flow.

• • Try to run the Shopify customer to NetSuite customer (add or update) and NetSuite customer to Shopify customer (add or update) flows before the holiday season and update customer data in Shopify and NetSuite to sync the data ahead.

• From NetSuite to Shopify
  • We recommend that you sync customers from NetSuite to Shopify before the holiday season.
  • Set the batch size of the NetSuite export and concurrency of the Shopify import logically to avoid any bottlenecks.
  • In order to sync the line-level address data to Shopify, it’s important to set the “Shopify Store” custom field with the correct store ID on the Address sub record.
  • To avoid conflicts, do not schedule the Shopify customer to NetSuite customer (add or update) and NetSuite customer to Shopify customer (add or update) flows to run at the same time.

Transaction sync from Shopify to NetSuite

• Typically, the Shopify transaction data is synced to NetSuite for credit card reconciliation purposes. If the transaction data in NetSuite is not being leveraged, the Shopify order transaction to NetSuite custom transaction record (add) flow can be temporarily or permanently disabled (Flows > Order).

Inventory sync from NetSuite to Shopify

• Run the NetSuite inventory to Shopify inventory (add or update) and NetSuite kit and item group inventory to Shopify inventory (add or update) flows in delta mode to improve performance by adjusting export batch size. To increase throughput at the Shopify end, consider the Custom App option:
  1. Create a custom app in your Shopify account.
  2. Authenticate a new connection in your integrator.io account, using the custom app details.
  3. Assign this new connection to the high-volume integration flows.
• Sync your inventory levels in advance of the holiday season and run in delta mode during the sale time.
• Uncheck Always sync inventory levels for the entire catalog (Settings > Inventory) to avoid sending a large amount of data to Shopify.
• Run the Shopify product ID to NetSuite item (mass update) flow (Flows > Inventory) prior to the holiday season, since it involves a large volume of data. Try to run it beforehand so that both systems (Shopify and NetSuite) are in sync.
  

Items and matrix items sync (product) from Shopify to NetSuite

• Try to load all the items/matrix items before the holiday season. During the season, lower the frequency of Item exports to as low as once per day.
• Bulk updates or mass updates to item records should be scheduled during off-peak hours, and should preferably be performed in batches.
• If you’re using the “Shopify Markets” edition of the integration app:
  • Ensure that the markets are correctly set up in Shopify.
  • If you’re using the product flow to sync price list updates, ensure that the existing price lists are set up correctly in Shopify.
• The metafield definitions must be correctly set up (data type and validations) in Shopify and if there are any changes to the metafield definitions in Shopify, ensure to save the Sync Shopify metafield definitions setting. If you configured the Specify your Shopify metafield components for products and variants setting, ensure that the metafield mappings for item metafields are removed, and vice versa.

Fulfillment sync 

• From Shopify to NetSuite
  • Schedule the Shopify fulfillment to NetSuite fulfillment (add) flow during off-peak hours.
  • In the Shopify fulfillment to NetSuite fulfillment (add) flow, set the page size of the export from 100 to 500, as appropriate.
• From NetSuite to Shopify
  • Schedule the NetSuite fulfillment to Shopify fulfillment (add) flow during off-peak hours with less frequency.

Refund sync 

• From Shopify to NetSuite
  • Schedule the Shopify refund to NetSuite refund (add) flow during off-peak hours.
  • Set the page size appropriately on the Shopify export.
  • Ensure that the GST/VAT settings in Shopify and the integration app are in sync.
• From NetSuite to Shopify
  • Schedule the NetSuite refund to Shopify refund (add) flow during off-peak hours and with less frequency.
  • For any kind of refund adjustments, ensure that the substitute item is selected for flow to avoid unnecessary errors and miscalculations.

Shopify payout reconciliation add-on

• Avoid setting large offset values in the Lag to bring payout related records setting to ensure better performance. 
• If you uncheck the Different store and payout currency box and try to reconcile payments that have different currency in payout and store, the flow leads to an amount mismatch variance. In order to avoid an unnecessary variance calculation, we recommend checking or unchecking the Different store and payout currency box (Settings > Payout), depending on your integration app and store setup.
• We recommend running the Shopify payout transactions to NetSuite deposit flow on the next day after you sync the orders to NetSuite as sales orders with customer deposits. Otherwise, you risk having open customer deposits accumulate. If you want to first create the billing records in NetSuite and then run the Shopify payout transactions to NetSuite deposit flow, do not sync the sales orders until you import the previous day’s payout report.
  

Example: If there are sales orders from 4 dates and the lag offset is defined as 3, the flow runs on the 5th and it picks the payout report of the 2nd; that is, all of the orders that belong to the 1st day are considered. In this scenario, you shouldn’t bill the orders belonging to days 2 through 4, or at least for the 3rd and 4th on the same day. You must sync the payout report for the 1st and then proceed with billing the next orders to ensure that you do not exceed the 10,000 NetSuite transaction limitation.

Amazon – NetSuite integration app

Connection concurrency

For different domain accounts belonging to the same region, Amazon shares the same request quota and restore limits. If you’re selling in multiple domains within the same marketplace region, it is advisable not to schedule any Amazon integration flows at the same time across multiple domains. This recommendation should ensure that none of the integration flows exceeds the throttling limits.

Example: In the European marketplace region, if you’re selling in Spain and Italy domains, do not schedule the Amazon (FBA/MFN) order to NetSuite order flow at the same time for both domains.

Order sync from Amazon to NetSuite

• Set the following recommended settings for the date filter so that the flow works in the delta mode.
  • Filter orders by: Last modified
  • Duration in bays: empty
  • Go live date: empty (in case the account is live for months now)
• Be sure to check the Dashboard for any errors. Resolve the errors and retry to sync the orders.
• The throttling limit for the ListOrders operation is restored to 1 request per minute with a maximum request quota of 6 at a time. So if there are more requests, MWS starts throttling. You can utilize the auto-retry mechanism; however, you can retry the data only up to 5 times, at which point an error message is displayed on the Dashboard. You can retry the error to sync the details correctly. For more information, see Throttling: Limits to how often you can submit requests.
• We recommend using the Allocate FBA orders to below NetSuite customer(s) setting (Settings > Order FBA section > Customers tab) for the Amazon (FBA) Order to NetSuite Order Add flow. For more information, see Configure Amazon FBA order flow settings.
• It is recommended to specify the value as 0 in the Duration in days setting – Settings > Order (FBA/MFN) section > Order tab – so that the flow doesn’t sync (export) duplicate records.

Tax setup

• Set up all tax codes/components properly in NetSuite or use the integratori.io tax configuration under Settings > General section > Tax tab:
  • Add total tax against single line item on the order
      — or —
  • Overwrite taxes with Amazon tax amounts 
• Using a proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.
• The Enable GST/VAT for your marketplace setting (Settings > General section > Tax tab) allows a dynamic approach for GST-related orders.
• If you have a static GST set up (same tax rate for all orders), in the Settings > General > Tax sub-tab, uncheck the Enable GST/VAT for your marketplace checkbox and enter the GST/VAT rate in the Enter GST/VAT rate to apply across all orders setting.
• If you have a dynamic GST set up (tax rate changes for all the orders), check the Enable GST/VAT for your marketplace and Apply the GST/VAT rate reported by Amazon boxes to deduct the tax from item amount and calculate the tax. 
• Check the Exclude Tax if Marketplace Tax Collection applies box (Settings > General > Tax subtab) to avoid the tax collection if Amazon collects and remits tax (varies region to region as decided by Amazon).

Fulfillment sync from NetSuite to Amazon

• Amazon validates tracking IDs entered while confirming shipments. Invalid tracking IDs provided at the time of shipment confirmation will result in a warning, but the shipment will be confirmed. But these invalid tracking IDs affect the valid tracking rates, which impacts your Amazon account. To avoid this scenario, make sure you are submitting valid tracking IDs.
• Schedule the NetSuite Fulfillment to Amazon Shipment Add flow to submit only 15 requests to Amazon per hour.

Example: Schedule the NetSuite Fulfillment to Amazon Shipment Add flow once in an hour with the batch size at 200, export size at 500, and page size at 100. Those setting would limit the traffic to 5 requests.
Tip: Try to submit a minimal number of requests to Amazon since the limit is 15 requests or a 5 MB integration app process size limit.

• Check the Automate shipment carrier mapping box (Settings > Fulfillment) to automatically map the shipping carriers in Amazon. Otherwise, you will have to configure the mapping in the field mappings page.

Inventory sync from NetSuite to Amazon 

• Uncheck the Always sync inventory levels for the entire catalog box (Settings > Inventory) to avoid sending a large amount of data to Amazon.
• Schedule the NetSuite Inventory to Amazon Inventory Add/Update and NetSuite Kit Inventory to Amazon Inventory Add/Update flows with less frequency, such as once in two hours (based on your requirements) in delta mode.

Product sync from NetSuite to Amazon

• Load all items before the holiday season. Then, during the season, lower the frequency of Item sync to as low as once per day. Item export is a heavy flow and takes considerable time and processing for Amazon to perform.
• Bulk updates or mass updates to the item records should be scheduled during off-peak hours, preferably done in batches.

Pricing sync from NetSuite to Amazon 

• Run the NetSuite Item Price to Amazon Product Price Add/Update flow with less frequency.

Note: We don’t recommend enabling this flow during the holiday season. However, when prices fluctuate during the holiday season, schedule this flow once in an hour with only the required set of mappings.

Important: We recommended staggering the scheduled fulfillment, pricing, and inventory integration flows for the same marketplace region, since they share the same concurrency limit of 15 feed requests per hour.

Auto-billing sync

• We recommend running the NetSuite Order to NetSuite CashSale/Invoice Add flow after the holiday season.

Settlement sync

• Settlement flows require a lot of processing and are a lower priority; avoid running the settlement flows during the holiday season.
• If required during the holiday season, we recommend allotting these flows to the secondary connection.

Amazon Transfer Order – InboundShipments add-on

• We recommend updating the inventory levels in Amazon well ahead of the sale period.

Amazon FBA return management and FBA inventory adjustments add-on

• We recommended running these add-ons to only once or twice a day.
• Configure these add-ons using the secondary connection where low-priority flows are attached.
• The FBA inventory adjustments and FBA return management report operation share the throttling limits in Amazon. So at peak times, a report might get canceled. It’s best to schedule these flows in such a way that the two will never overlap.

eBay – NetSuite integration app

Order sync (eBay to NetSuite)

• Set the recommended settings for the date filter setting so that flow works in delta mode:
  • Filter orders by: Last Modified
  • Duration in eays: empty
  • Go live date: empty (in case account has already gone live)

Tax setup

• Set up tax properly in NetSuite or use the integratori.io tax configuration in the Settings > General > Tax subtab:
  • Add total tax against single line item on the order 
      — or —
  • Overwrite taxes with eBay tax amounts with Assign the default tax code to all orders checked.
• Using a proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to adjust it later.

Check the Exclude Tax if Marketplace Tax Collection applies box in the Settings > General section > Tax tab in order to avoid having eBay collect and remit tax (varies region to region as decided by eBay).

Inventory sync (NetSuite to eBay)

• Uncheck the Sync All Inventory setting to avoid sending a large amount of data to eBay.
• When you sync the matrix item inventory from NetSuite to eBay, set the eBay integration app concurrency to 1. After you sync the matrix item inventory, modify the concurrency back to its original level.

• Disable the eBay Item Id to NetSuite Mass Update flow after you sync all of the Item IDs to all eBay items.

Item and Pricing sync (NetSuite to eBay)

• We recommend syncing the prices and performing heavy item update operations before the holiday season. If required, run these flows during the holiday season with low frequency.

Magento 2 – NetSuite integration app

Store setup:

• In the Settings > General section > General tab, verify the stores that belong to your website in the Select active Magento 2 stores setting.
• If you’re unable to get the record from the newly added store views, then deselect one of the stores in the Select active Magento 2 stores setting and save. Resave the setting with existing and new stores.

Order sync

• Keep the Magento 2 instance optimized, cached, and compiled to improve the API response performance for the order import and item/inventory export.

Tax setup

• Set up tax properly in NetSuite or use the integratori.io tax configuration at Settings > General > Tax.
  • Add total tax against single line item on the order 
      — or —
  • Overwrite taxes with Magento 2 tax amounts with Assign the default tax code to all orders checked.
• Using a proper tax setup will help to reduce the variance and also reduce extra NetSuite API calls to fix the variance.

Inventory sync (NetSuite to Magento2) 

• Uncheck the Always sync inventory levels for the entire catalog box (Settings > Inventory) to avoid sending a large amount of data to Magento.

Item and images sync (NetSuite to Magento2)

• Try to load all of the items and images before the holiday season. During the season, lower the frequency of Item export to as low as once per day. Item export is an intensive flow that takes time and processing for Magento to perform.

Payout to Reconciliation Automation for NetSuite

The Payout to Reconciliation Automation for NetSuite solves the post-holiday season reconciliation so that the open transactions don’t exceed the limit of 10,000.

• NetSuite limitation: Schedule the payout flows at regular time intervals to ensure that the total volume of transactions during any flow run doesn’t increase beyond 10,000.
• Ensure the lag offset is set to a minimal value to improve flow performance.
• For the PayPal payout transactions to NetSuite deposits flow, if the PayPal report has transactions in multiple currencies, ensure that the PayPal primary currency code setting is set correctly. It is intended to avoid any unnecessary variances during the deposit import.
• We recommend clearing all open transactions before the holiday season to avoid exceeding the 10,000 transaction limitation imposed by NetSuite.
• Validate your account settings before the holiday season to avoid integration app setup errors.
• Ensure that all of the order and refund transactions are synced with undeposited fund accounts and with the correct transaction IDs in their respective fields.
• Be sure that the bank deposit account and other GL accounts belong to the same subsidiary and currency.

Square – NetSuite integration app

Since the Square – NetSuite integration app is based on a POS solution, you can sync data as usual. To improve performance, 

• Consider scheduling the Square Order to NetSuite Cash Sale flow at a higher frequency.
• If possible, do not sync tenders.

• Set up the Square POS locations and be sure they are synced and selected in the Select active Square locations setting in the Settings > General section.
• If possible, leverage the customer sync and disable the Square Customer to NetSuite Customer flow. Add a default customer for all cash sales.

• For cash refund sync, we recommend running the “Square Refund to NetSuite Cash Refund” flow with a lower frequency, such as once or twice a day for the entire holiday sale season.
• Set the concurrency for the NetSuite connection to 2 or 3, to leave the remaining concurrency free for other integration apps.
• If your order has any product modifiers, you can select the appropriate NetSuite item in the Sync Square product modifier to NetSuite as settings (Settings > Order) to avoid unwanted variances.
  

Note: On the NetSuite sales order, the description field displays the modifier name only when you add the item description mapping in the Square Order to the NetSuite Order flow.

BigCommerce – NetSuite integration app

• We recommend scheduling the NetSuite Item Image to BigCommerce Product Image Add/Update/Delete flow at a lower frequency.
• To avoid a high-volume data flow run, sync the products and their inventory levels before the holiday season.
• We recommend selecting only the required order statuses during the holiday season.
• Set and schedule the export frequency for the inventory and fulfillment flows as “delta.” 
• Schedule the fulfillment flow at a higher frequency, such as once every 15 minutes.
• It is recommended not to run the following flows during the holiday season:
  • BigCommerce Customer to NetSuite Customer Add/Update
  • NetSuite Item to BigCommerce Product Add/Update
  • NetSuite Item Image to BigCommerce Product Image Add/Update/Delete
  • BigCommerce product ID to NetSuite item (update)
• It is recommended to run the following flows at a lower frequency during non-peak hours:
  • NetSuite Cancellation to BigCommerce Cancellation Add
  • NetSuite Refund to BigCommerce refund Add

Shopify - SAP Business ByDesign integration app

Sandbox environment integrations

In Shopify, the rate limits work as per the Application-Store combination. It's better if you disable all the sandbox environment flows to see maximum throughput for the production environment flows.

• We recommend that you run the ​​Shopify product to SAP Business ByDesign Celigo eTail item map (add or update) flow prior to the holiday season, and turn off the flow if no products are being created in Shopify during the holiday period.
• Ensure that you run the SAP Business ByDesign inventory to Shopify inventory (add or update) flow prior to the holiday season to prevent heavy data transfers during the busy period.
• Only sync orders with the required statuses in Shopify to SAP Business ByDesign, using the  Sync web orders to SAP Business ByDesign based on the Shopify order setting (Settings > Order section). By deselecting statuses that aren’t needed you will reduce the amount of data that is transferred to SAP Business ByDesign.
• Turn off syncing of Shopify order updates by unchecking the Sync Shopify edited orders to SAP Business ByDesign setting (Settings > Order section).