Handlebar trim messes up import mapping output (REST API)

Answered

Stefan Lindner Referral Partner

• August 09, 2022 08:46

Hi,

I have the following issue with a import mapping in a custom data flow:

Base is an export from NetSuite which delivers order lines, which are grouped. So we have following input data (simplified):

[
  {
    "id": "261209",
    "recordType": "salesorder",
    "internalid": "261209",
    "orderNumber": "S-003802",
    "amount": "2",
    "item description": "Baselayer Shorts M",
    "sku": "4711-AAA"
  },
  {
    "id": "261209",
    "recordType": "salesorder",
    "internalid": "261209",
    "orderNumber": "S-003802",
    "amount": "3",
    "item description": "Baselayer Shorts L",
    "sku": "4711-BBB"
  }
]

An the following mapping:

Output is looking like expected - new sub-object "shipmentItems" with correct values for SKU, amount and desc.

Cause of data qualitiy issues we have from time to time whitespaces at beginning or end of SKU - so we need a trim-Handlebar to remove these.

But as soon as we use {{trim sku}} for SKU the whole output is messed up and SKU of the first record is used for all other sub-records.

Did not expect this behaviour - is this a bug or works as designed?

Thanks
Stefan

 

 

 

 

 

0

• 

  Steve Klett Strategic Partner

  • August 11, 2022 05:14

  

  

  

  

  

  

  My $0.02 is that when the editor tries to simplify or hide details, it ends up being more of a liability than a help. I would prefer to see the full path/syntax of the data.

  2
• 

  David Gollom Strategic Partner

  • August 09, 2022 16:39

  

  

  

  

  

  When you do your own expression, you need to tell the mapper it's looking at an array-to-array mapping. In your mapping for instance, IIO is really using sometime like [*].amount.  The fact that there's an array on the left side of the expression is being hidden from you.   However, I don't think that literal syntax will work. 

  I always use an export transformation to make the lines into an array nested within the json object. Then you can use something like {{trim lines[*].sku }} as your handlebar expression.  

  Would the $[*] syntax work in this example??  

  1
• 

  Amanjot Singh Group Product Manager

  • August 15, 2022 17:40

  

  

  

  

  Stefan Lindner You could solve this problem easily using mapper 2.0. It's available using the toggle at the top for HTTP/REST imports right now. This way you wouldn't have to create an additional transform in export just to handle arrays. have added the mappings example below for your use case. 
  
  Let me know for any further questions.  
  
  
  
  

  1
• 

  Scott Henderson CTO

  • August 15, 2022 19:11
  • Edited

  

  

  

  To add a little color to the post from Aman, we put a lot of thought into how mapper 2.0 should work with all the different types of data. Mapper 2.0 is a very different mapping engine than 1.0, and there will be a learning curve, but once you get the basics, it will be much easier to map objects, arrays, or any other complex data structures you can imagine.

  Here are some key changes:

  1. Mapper 2.0 supports referencing fields via formal JSONPath syntax (i.e. $.)
  2. Mapper 2.0 supports using handlebars templates (i.e. {{ record. }} or {{ rows. }} ), and we are explicitly wrapping the data context variable with 'record' or 'rows' to make the type of data you are working with explicit.
  3. You cannot use JSONPath within handlebars templates.  These two are mutually exclusive.
  4. We finally made it possible to iterate arrays within arrays within arrays, etc... or to build arrays within arrays within arrays, etc...
  5. We are doing magic behind the scenes when it comes to iterating arrays and referencing individual fields.  In mapper 2.0, you will no longer use any type of '[*]' syntax.  For example, if you are iterating an array within an array within an object like '$.items[*].shipments[*]' you will always reference individual fields using the syntax '$.items.shipments.fieldXYZ' or '{{ record.items.shipments.fieldXYZ }}' and behind the scenes we know where you are in each array, and do the necessary swaps.

  1
• 

  Stefan Lindner Referral Partner

  • August 09, 2022 19:54

  Hi David,

  thanks for your feedback - I have tested a couple of things within the mapper - following expressions are NOT working:
  {{trim [*].sku}}
  {{trim $[*].sku}}

  Interestingly this is working correct (btw. never seen this kind of expression before):
  {{trim *.sku}}

  But you are right, creating a transformation on export side would be the best solution here. :)

  Just wondering, why it's not mentioned in the documentation, that "some magic" is happening in the background for basic array-to-array mappings (without handlebars)...

  Thanks
  Stefan

  0
• 

  Stefan Lindner Referral Partner

  • August 18, 2022 10:46

  Hi all,

  thanks for your very helpful feedback - appreciate that!

  It was an existing flow from pre-"Mapper 2.0" area, so I don't want to re-do the whole mapping only for a "simple" add of trim handlebar. But it was an expensive lesson learned, that it's not so easy as it looks like for array-to-array mappings with Mapper 1.0 and handlebars... (maybe it's worth to document this in any way for all other users?)

  To sum it up, recommendation from Celigo is to use for every new flow Mapper 2.0, even it is in "Beta" status at the moment?

  Thanks
  Stefan

  PS: Is there any way to mark this thread as resolved? ;)

   

  0
• 

  Anitha Abraham Community moderator Principal Technical Writer

  • August 18, 2022 12:08

  

  Thanks, Stefan. I've marked the post as "Answered". 

  0

Please sign in to leave a comment.