How to understand the new BigQuery connector "once per page of records"

Comments

3 comments

  • Scott Henderson CTO
    Celigo University Level 1: Skilled
    Answer Pro
    Top Contributor

    Hey Jeremy,

    The docs are not good here, and I passed a note to the doc team to fix this asap, and someone from doc/product team will post an update here once the docs are improved.  Sorry about this.

    At a very high level, the "once per page of records" option is pretty uncommon.  The "bulk insert" is much easier to use.  Is "bulk insert" an option for your use case?  If not, then the key to get the "once per page of records" working is to use the AFE 2.0 UI to help you build the exact SQL query that you would build with any other SQL tool.  Please see the following pics for a very rough example.  To incorporate data from your page, then you would need to iterate through the "batch_of_records" variable using the handlebars #each function.  Hopefully you dont need to go down this path, and I will skip trying to build a proper example here until i hear back that you need this.

     

    0
  • Jeremy Nelson
    Engaged

    Thanks Scott!   My intention with using the once per page of records was to add an "ingestion_date" field that captures the current timestamp using Handlebars.  Would you suggest using a hook for this instead of my original approach?

    0
  • Scott Henderson CTO
    Celigo University Level 1: Skilled
    Answer Pro
    Top Contributor

    You could do this via a hook, which would be easier than using the "once per page of records" option.

    Or (even easier), you can use a field mapping to do this.  Hopefully the following screenshot explains it all.  i.e. click the settings icon for any of your field mappings to see this settings drawer

    0

Please sign in to leave a comment.