Create Amazon S3 Bucket
Is there a simple way using Celigo to create a new object in Amazon S3? I'm looking through the documentation, but I dont see much in relation to put/post API requests to create a new object within an existing S3 bucket and how to do that within Celigo. It may be a custom script thats needed, but wanted to check with the community to be sure there isn't something I'm missing.
0
Comments
Dave Guderian what is the use case here? There are some templates in the marketplace you can use that show how to create files in S3. You can either create files from records and place in S3 as CSV, JSON, XML, etc. or you can transfer files from one source to S3 without Celigo parsing anything.
Hey Tyler-
I'll outline the scenario as best I can (this is ideal scenario and may be shooting for the stars!). We will be writing data from our salesforce environment on a nightly basis to a folder structure in S3 that resembles this: patient/year=2023/month=11
The ideal scenario would be that when the export runs for Dec 1st, the flow could check to see if there was a: month=12 object and if not, then create it and write that file to that object (patient/year=2023/month=12). Since the job will be running daily, we would like to check (based off the file being written (see below)) and then potentially create a new object if that file is the first one of the month.
Dave Guderian gotcha so you want to create folders (aka objects) in S3 if there isn't already one and then place files in those folders. For this, you couldn't use the native S3 connector because it doesn't currently have options to work on records as opposed to files, but you could use a generic HTTP connector. Here is some documentation on how to setup a connection to S3 using the universal HTTP connector. From there, I think you would use the PutObject endpoint.
Hi Dave Guderian,
In S3 there is nothing such as folders. There are just 2 concepts, buckets and keys. What you see as folders in S3 console is just the UI (an article here detail this )
In your structure above, patient/year=2023/month=11is just the prefix of the filename. I would suggest that you build your filename based on the current date. It will then always create the file with the right structure in the UI of S3.
As a result in the UI of S3, you will see folders like this :
December 1st , when the flow will have to create the file , it will then create the "structure" based on the current date (which means year=2023 and month=12) , this filename will be then : "patient/year=2023/month=12/patient-01-12-2023_10:10:46.json" (it depends on the format your select for the timestamp part between "patient-" and ".json")
Hope that answers your question.
You learn something new everyday! Thanks Youssef Zouhairi!
Youssef and Tyler.... you guys rock! This is an awesome approach and seems pretty straightforward.
That said, something with the handlebar is off where I keep getting the same error around the dateFormat for the month (have tried multiple times this morning and cant figure out). As always, I super appreciate your help!
Here is the error I am receiving:
Here is the handlebar based on Youssef's example above:
raw/salesforce-ips/patient/year={{dateFormat "YYYY” date}}/month={{dateFormat “MM” date}}/patient_{{timestamp "YYYY-MM-DD_hh-mm-ss" "America/Chicago"}}.jsonDave Guderian you can try Celigo AI here to help you build the handlebar expression.
Get current year:
{{timestamp "YYYY" "America/Chicago"}}To get current month:
{{timestamp "MM" "America/Chicago"}}Dave Guderian in your case, the issue seems to be with your quotations. It looks like some quotations you have are different than others. This worked for me.
raw/salesforce-ips/patient/year={{dateFormat "YYYY" date}}/month={{dateFormat "MM" date}}/patient_{{timestamp "YYYY-MM-DD_hh-mm-ss" "America/Chicago"}}.jsonSee the quotes look off when I copy your statement:
Dave Guderian are you building this expression in another text editor? Also, we do support curly quotes, but you can't mix and match them. When I fixed the YYYY and left the MM with curly quotes, it worked fine.
Thanks Tyler. That did the trick, and love the new AI feature (lets just say I have used it a lot!).
Next question, is I tried to change the timestamp for the file being generated and it still landed in the month=11 key instead of creating a new month=12 key. Is hardcoding the date (like I did) going to be a way that this logic could be tested, or is there somewhere else I need to change it?
Screenshot showing the file (top) in the month=11 key despite the date (2023-12-01):
Code block I used showing hard-coding the date:
raw/salesforce-ips/patient/year={{dateFormat "YYYY" date}}/month={{dateFormat "MM" date}}/patient_{{timestamp "2023-12-01_hh-mm-ss" "America/Chicago"}}.jsonHi Dave,
The logic we build with the structure in S3 is based on your current date not the hardcoded value (date) you have in the file name.
Since we are still in November, any file generated right now will end up in Month=11.
If you test (run the flow) in December 1st , it will end up in Month=12.
Awesome. Thanks so much Youssef! I really appreciate you guys!
Just to close the loop on this, as Youssef noted a couple posts above this did indeed create a new month=12 key on import today (12/1). Thanks Youssef and Tyler for walking me through this!
Please sign in to leave a comment.