Create Amazon S3 Bucket

Comments

13 comments

  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian what is the use case here? There are some templates in the marketplace you can use that show how to create files in S3. You can either create files from records and place in S3 as CSV, JSON, XML, etc. or you can transfer files from one source to S3 without Celigo parsing anything. 

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Hey Tyler-

    I'll outline the scenario as best I can (this is ideal scenario and may be shooting for the stars!). We will be writing data from our salesforce environment on a nightly basis to a folder structure in S3 that resembles this: patient/year=2023/month=11

    The ideal scenario would be that when the export runs for Dec 1st, the flow could check to see if there was a: month=12 object and if not, then create it and write that file to that object (patient/year=2023/month=12). Since the job will be running daily, we would like to check (based off the file being written (see below)) and then potentially create a new object if that file is the first one of the month.

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian gotcha so you want to create folders (aka objects) in S3 if there isn't already one and then place files in those folders. For this, you couldn't use the native S3 connector because it doesn't currently have options to work on records as opposed to files, but you could use a generic HTTP connector. Here is some documentation on how to setup a connection to S3 using the universal HTTP connector. From there, I think you would use the PutObject endpoint.

     

     

    0
  • Youssef Zouhairi Solutions Consultant
    Engaged
    Celigo University Level 4: Legendary

    Hi Dave Guderian,
    In S3 there is nothing such as folders. There are just 2 concepts, buckets and keys. What you see as folders in S3 console is just the UI (an article here detail this
    In your structure above, patient/year=2023/month=11is just the prefix of the filename. I would suggest that you build your filename based on the current date. It will then always create the file with the right structure in the UI of S3. 

    As a result in the UI of S3, you will see folders like this : 

    December 1st , when the flow will have to create the file , it will then create the "structure" based on the current date (which means year=2023 and month=12) , this filename will be then : "patient/year=2023/month=12/patient-01-12-2023_10:10:46.json" (it depends on the format your select for the timestamp part between "patient-" and ".json")

    Hope that answers your question. 

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    You learn something new everyday! Thanks Youssef Zouhairi!

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Youssef and Tyler.... you guys rock! This is an awesome approach and seems pretty straightforward.

    That said, something with the handlebar is off where I keep getting the same error around the dateFormat for the month (have tried multiple times this morning and cant figure out). As always, I super appreciate your help!

    Here is the error I am receiving:

    Here is the handlebar based on Youssef's example above:

    raw/salesforce-ips/patient/year={{dateFormat "YYYY” date}}/month={{dateFormat “MM” date}}/patient_{{timestamp "YYYY-MM-DD_hh-mm-ss" "America/Chicago"}}.json
    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian you can try Celigo AI here to help you build the handlebar expression.

     

    Get current year:

    {{timestamp "YYYY" "America/Chicago"}}

     

    To get current month:

    {{timestamp "MM" "America/Chicago"}}

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian in your case, the issue seems to be with your quotations. It looks like some quotations you have are different than others. This worked for me.

    raw/salesforce-ips/patient/year={{dateFormat "YYYY" date}}/month={{dateFormat "MM" date}}/patient_{{timestamp "YYYY-MM-DD_hh-mm-ss" "America/Chicago"}}.json

     

    See the quotes look off when I copy your statement:

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian are you building this expression in another text editor? Also, we do support curly quotes, but you can't mix and match them. When I fixed the YYYY and left the MM with curly quotes, it worked fine.

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Thanks Tyler. That did the trick, and love the new AI feature (lets just say I have used it a lot!).

    Next question, is I tried to change the timestamp for the file being generated and it still landed in the month=11 key instead of creating a new month=12 key. Is hardcoding the date (like I did) going to be a way that this logic could be tested, or is there somewhere else I need to change it?

    Screenshot showing the file (top) in the month=11 key despite the date (2023-12-01):

    Code block I used showing hard-coding the date:

    raw/salesforce-ips/patient/year={{dateFormat "YYYY" date}}/month={{dateFormat "MM" date}}/patient_{{timestamp "2023-12-01_hh-mm-ss" "America/Chicago"}}.json
    0
  • Youssef Zouhairi Solutions Consultant
    Engaged
    Celigo University Level 4: Legendary

    Hi Dave, 
    The logic we build with the structure in S3 is based on your current date not the hardcoded value (date) you have in the file name. 
    Since we are still in November, any file generated right now will end up in Month=11.
    If you test (run the flow) in December 1st , it will end up in Month=12.


    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Awesome. Thanks so much Youssef! I really appreciate you guys!

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Just to close the loop on this, as Youssef noted a couple posts above this did indeed create a new month=12 key on import today (12/1). Thanks Youssef and Tyler for walking me through this!

    0

Please sign in to leave a comment.