Begin retrieving data from DynamoDB by creating an export that will send your data retrieval request. You can start with a standalone export or add one in Flow Builder, as described below.
Create an export
- Select Flow Builder from the Tools menu.
- Choose DynamoDB for the Source application.
- Select a DynamoDB connection from the Connection list.
- Click Next.
The new export’s Create export pane opens for you to edit the settings:
Name (required): Name the export so that you can easily reference it from other parts of the application.
Description (optional): Enter a description for your export so that other users can quickly understand its purpose without having to examine all the fields and settings. Be sure to highlight any important nuances that others might need to know before using your export in their flows. Update the description whenever you make changes to the export in the future.
Region (required): Select the DynamoDB location where the request is being made, or accept the default us-east-1.
Table name (required): Specify the DynamoDB database table – such as limit-test orders, items, or customers – that you would like to query.
Expression attribute names (required): An expression attribute name is a placeholder in an Amazon DynamoDB expression as an alternative to an actual attribute name. An expression attribute name must begin with a pound sign (#) and be followed by one or more alphanumeric characters, such as { "#n1": "sku", "#n2":"id" }.
Build expression attribute values (required): If you need to compare an attribute with a value, define an expression attribute value as a placeholder. These values substitute for the actual values that you might not know until runtime. An expression attribute value must begin with a colon (:) and be followed by one or more alphanumeric characters, such as { ":p1": "celigo-tshirt",":p2": 99 }.
Key condition expression (required): To specify the search criteria, provide a key condition expression, a string that determines the items to be read from the table or index, such as #n1 = :p1 AND #n2 > :p2.
Filter expression (optional): If you want to export only specific documents from your collection, then enter a filter object. This value must contain a valid JSON string describing a DynamoDB filter object in the correct format and with the correct operators.
Projections (optional): If you want to return only a subset of fields from each DynamoDB document, then enter a projection object. The value of this field must be a valid JSON string describing a DynamoDB projection object in the correct format and with the correct operators.
Export types
Select one of the following export types:
- All – Exports all records matching the Key condition expression each time the export runs
- Delta – Export all records with a value in the Date field that is greater than the last time the flow was run matching the Key condition expression results
- Once – Export any records that have not been exported already and will also automatically update records to mark them as exported
-
Limit – Export a limited number of records, primarily used for testing to avoid syncing lots of data
Note: When you select the export type as Limit - export a set number of records, another setting How many records would you like to export? is enabled for you to enter the maximum number of records to export each time this flow runs. The default value is set to one. You can export a maximum of 100 records.
As demonstrated in the following export type examples, you can validate your expressions by clicking the Preview button. Any errors returned by the source application are displayed. The preview returns the first matching record for all export types, except Delta.
All
Exports all records matching the Key condition expression each time the export runs.
Sample configuration
Region: US East (N. Virginia) [us-east-1]
Table name: limit-test
Expression attribute names: { "#n1": "sku", "#n2": "id" }
Build expression attribute values: { ":p1": "celigo-tshirt",":p2":
99 }
Key condition expression: #n1 = :p1 AND #n2 > :p2
This expression translates to sku = celigo-tshirt AND id > 99 – for example, in plain English, “Export all t-shirts from my Amazon store that were added sequentially starting with the benchmark 100th shirt.”
Delta
Export all records with a date value for the mentioned field in the Date field that is greater than the last time the data flow was run matching the Key condition expression results.
Sample configuration
Region: US East (N. Virginia) [us-east-1]
Table name: delta-test
Expression attribute names: { "#n1": "joining", "#n2": "sku" }
Build expression attribute values: { ":p1": "2019-08-13T19:56:12.008Z", ":p2": "celigo-tshirt" }
Key condition expression: #n1=:p1 AND #n2=:p2
Date field: joining
Select a date field from the export application that integrator.io can use to keep track of records that have changed since the last time the export was run. It is recommended that you pick a system-generated field. For example, many applications maintain a standard Date Last Modified field that always contains the date and time that a record was last changed. You can also select a non-system generated field if you have your own logic in place (in the export application) to set the field accordingly whenever a related change is made to a record.
Once
Export any records that have not been exported already and also automatically update records to mark them as exported.
Sample configuration
Region: US East (N. Virginia) [us-east-1]
Table name: export_once_partition
Expression attribute names: { "#n1": "emp_id" }
Build expression attribute values: { ":p1": 81 }
Key condition expression: #n1 = :p1
Once boolean field: export_once
Provide a Boolean (checkbox) field from the export application that integrator.io can use to keep track of records that have been exported. integrator.io will export only those records where this value is false (unchecked), and integrator.io will also automatically make a subsequent request back into the export application to set this field to true for all the records that were exported (so that those records are not exported again).
Once export partition key: emp_id
Enter the partition key from the table.
Limit
Export a set number of records; used for testing to avoid syncing lots of data.
Note: When you select the export type as Limit - export a set number of records, another setting How many records would you like to export? is enabled for you to enter the maximum number of records to export each time this flow runs while developing and testing an integration. The default value is set to 1. You can export a maximum of 100 records.
Sample configuration
Region: US East (N. Virginia) [us-east-1]
Table name: limit-test
Expression attribute names: { "#n1": "sku", "#n2": "id" }
Build expression attribute values: { ":p1": "celigo-tshirt",":p2":
99 }
Key condition expression: #n1 = :p1 AND #n2 > :p2
Group export records
The Grouping option allows you to manage your flow’s files by grouping records by field. This feature allows you to aggregate your data based on your chosen field(s).
Comments
Please sign in to leave a comment.