Comments

12 comments

  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Vreddhi Bhat we just rolled out functionality for automatically recovering from rate limit errors, but as you see it currently doesn't work on exports, only imports. In some upcoming release that should be enhanced so you won't have to mess with this issue.

     

    That being said, to currently do this, I think your best option would be to create a custom connection to HubSpot, using the universal HTTP connector. Under the connection settings, there is an option to specify a time between api calls. Here you could add some delay between requests to avoid the rate limit.

     

    Hopefully early next year, all connectors will be migrated over to the newer HTTP framework so you wouldn't need to make a custom connection. Significant progress has already been made in that migration, but HubSpot hasn't yet made it.

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Tyler- A followup question on your response. I am also receiving the rate limit error on an export from HubSpot, but it appears that all errors are being autoresolved and the steps are completing successfully. Can you help make sense of if indeed the data "flow" is successful, or if any remediation is needed due to the errors.

    Screenshot showing the rate limit error (note these are falling into the "Resolved errors" bucket and are auto-resolved.

    Screenshot showing the flows successful on each step

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian in your case, the rate limiting is happening on the lookup itself which does support auto resolve of rate limiting errors when one-to-many is not configured. So we would have received the initial rate error, then backed off our concurrent requests, then pinged the endpoint again to get the needed data. It got auto resolved because we were able to successfully handle the initial rate error and get the needed data in a subsequent call.

     

    That being said, it looks like you don't really even need this lookup step because you can get all properties for a company in your initial HubSpot call. You just have to specify the fields you want by putting them in a comma separated list in a parameter field.

    Default without specifying properties:

     

    If specifying properties:

     

    If you want to get a list of all properties for an object, you can use the following endpoint (or look in your HubSpot account):

    https://developers.hubspot.com/docs/api/crm/properties

    crm/v3/properties/{objectType}

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Hey Tyler-

    Thats a great idea, but unfortunately I think we have too many properties (we have 500 for our contacts object, and almost 200 for companies) and may be hitting the limit. Below is the error I receive when I try to put them into the "Configure Search Parameters" field.

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian looks like an internal text field size limit. However, you can work around it by using settings:

    /crm/v3/objects/companies?properties={{{settings.export.properties}}}
    {
      "properties": "hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore,hubspotscore"
    }

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian if you want to get really crazy, you could create a custom form that would allow you to multi select the fields you want (I know in your case you just want to select all). 

    import {
        connections,
        integrations,
        exports,
        imports,
        flows,
        request
    } from 'integrator-api'

    class Form {
        // { fieldMap: {}, layout: { fields: [] } }
        constructor() {
            this.fieldMap = {};
            this.layout = {};
            this.layout.containers = [];
        }

        fields(...fields) {
            this.layout.fields = fields;
        }

        container(type, label, ...fields) {
            this.layout.containers.push({
                type: type,
                containers: [{
                    label: label,
                    fields: fields
                }]
            });
        }

        column(label, ...fields) {
            this.layout.type = 'column';
            this.layout.containers.push({
                label: label,
                fields: fields
            });
        }
    }
      
    class Field {
        constructor(id, name, label, type) {
            this.id = id;
            this.name = name;
            this.label = label;
            this.type = type;
        }

        isRequired() {
            this.required = true;
        }
        
        removeInvalidValues() {
            this.removeInvalidValues = true;
        }

        canBeDeleted() {
            this.showDelete = true;
        }

        isMultiline() {
            this.multiline = true;
        }

        description(description) {
            this.description = description;
        }

        help(helpText) {
            this.helpText = helpText;
        }

        typeOfInput(type) {
            this.inputType = type;
        }

        default(defaultValue) {
            this.defaultValue = defaultValue;
        }

        delimiter(delimiter) {
            this.delimiter = delimiter;
        }

        keyNamePlaceholder(keyName) {
            this.keyName = keyName;
        }

        valueNamePlaceholder(valueName) {
            this.valueName = valueName;
        }

        mode(mode) {
            this.mode = mode;
        }

        addOption(arg1, arg2) {
            if (!this.hasOwnProperty('options')) {
                this.options = [{
                    items: []
                }];
            }

            if (arg2) {
                this.options[0].items.push({
                    label: arg1,
                    value: arg2
                });
            } else {
                this.options[0].items.push(arg1);
            }
        }

        addStaticMapOptions(keyName, keyLabel, keyOptions, valueName, valueLabel, valueOptions) {
            this.keyName = keyName;
            this.keyLabel = keyLabel;
            this.keyOptions = keyOptions;
            this.valueName = valueName;
            this.valueLabel = valueLabel;
            this.valueOptions = valueOptions;
        }    
    }

    function formInit(options) {
        let form = new Form();
        // reference: https://docs.celigo.com/hc/en-us/articles/360059205112-Common-form-fields
        // deconstruct form to make access to fieldMap for prop assignment simpler
        let { fieldMap } = form;
        
        let selectedObject = '';
      let regex = /(?<=objects\/)[^\/?]+/;
        let match = options.resource.http.relativeURI.match(regex);
        if (match) {
          selectedObject = match[0];
        }

        //create field for staticMap
        fieldMap.fieldList = new Field('fieldList','fieldList','Fields(s)', 'multiselect');

        if (selectedObject) {
          let selectOptions = getOptions(options.resource._connectionId,selectedObject);
          for (let s of selectOptions) {
            fieldMap.fieldList.addOption(s.label,s.value);
          }
        }

        fieldMap.fieldList.description('Select the fields you want to select.');
        fieldMap.fieldList.isRequired();
        fieldMap.fieldList.removeInvalidValues();

        //set the layout for the form
        form.fields(...['fieldList']);

        // wrap up by assigning form to options.resource.settingsForm prop; return the form as well
        options.resource.settingsForm.form = form;
        return form;
    }
      
    function getOptions(hubspotConnectionId,selectedObject) {
        return exports.runVirtual({
            "export": {
              "name": "Get all fields",
              "_connectionId": hubspotConnectionId,
              "asynchronous": true,
              "oneToMany": false,
              "sandbox": false,
              "http": {
                "relativeURI": `/crm/v3/properties/${selectedObject}`,
                "method": "GET",
                "successMediaType": "json",
                "errorMediaType": "json",
                "formType": "http",
                "paging": {
                  "method": "token",
                  "path": "paging.next.after",
                  "relativeURI": `/crm/v3/properties/${selectedObject}?after={{{export.http.paging.token}}}`,
                  "lastPageStatusCode": 404
                },
                "response": {
                  "resourcePath": "results"
                }
              },
              "transform": {
                "type": "expression",
                "expression": {
                  "rules": [
                    []
                  ],
                  "rulesTwoDotZero": {
                    "mappings": [
                      {
                        "generate": "label",
                        "dataType": "string",
                        "extract": "$.label",
                        "sourceDataType": "string",
                        "status": "Active"
                      },
                      {
                        "generate": "value",
                        "dataType": "string",
                        "extract": "$.name",
                        "sourceDataType": "string",
                        "status": "Active"
                      }
                    ],
                    "mode": "create"
                  },
                  "version": "2"
                },
                "rules": [
                  []
                ],
                "version": "2"
              },
              "adaptorType": "HTTPExport"
            }
        }).data;
    }
    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    This is awesome Tyler. Like you said I think right now I am just interested in pulling all properties, but I do love this idea for the future and have a couple ideas on how I could use this for some different flows I will be making.

    One thing I noticed this morning, was that my "Delta" loads seem to not be working, in the sense that they are pulling all of the data (so I don't think my {{lastExportDateTime}} handlebar is correct. Here is what I have:

    /crm/v3/objects/companies?properties={{settings.export.properties}}?{{lastExportDateTime}}
    0
  • Kevin Gonzalez

    Hey Dave Guderian, another way of getting Delta data through the Hubspot v3 APIs is by selecting the Search for Companies endpoint and opening the Search Parameters. Once in there, you can add this code and paste it inside the filterGroups section:

    [
    {
    "filters": [
    {
    "propertyName": "hs_lastmodifieddate",
    "operator": "GT",
    "value": "{{lastExportDateTime}}"
    }
    ]
    }
    ]

    Ex:

    You can also add your properties here as well. Here is the article from our Help Center that walks through this. Hope that helps!

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Hey Kevin - Thanks for the feedback. I am using the list V3 endpoint, and it appears that "filters" is not an option.

    0
  • Kevin Gonzalez

    Dave Guderian If you switch the API endpoint to Search for Companies instead of 'List' you should see the other options

    0
  • Dave Guderian
    Engaged
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Thanks Kevin. Is there a particular syntax to use if I wanted to use the "List" endpoint?

    0
  • Tyler Lamparter Principal Product Manager
    Awesome Follow-up
    Engaged
    Top Contributor
    Answer Pro
    Celigo University Level 4: Legendary

    Dave Guderian there is not so you have to use the search option. Use the following setup:

     

    Switch to REST form view. If you don't use the form settings like I showed, just put a string array of property fields you want to use instead of the {{#each}} handlebar block I'm using.

    {
        "properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
        "filterGroups": [
            {
                "filters": [
                    {
                        "propertyName": "hs_lastmodifieddate",
                        "operator": "GTE",
                        "value": "{{lastExportDateTime}}"
                    }
                ]
            }
        ]
    }

    {{#if data._PARENT.paging.next.after}}
     {
        "properties": [{{#each settings.export.fieldList}}"{{this}}"{{^if @last}},{{/if}}{{/each}}],
        "filterGroups": [
            {
                "filters": [
                    {
                        "propertyName": "hs_lastmodifieddate",
                        "operator": "GTE",
                        "value": "{{lastExportDateTime}}"
                    }
                ]
            }
        ],
     "after":"{{data._PARENT.paging.next.after}}"
     }
     {{/if}}

    0

Please sign in to leave a comment.