Skip to main content

Set up an OAuth 2.0 HTTP connection to Azure Data Lake Storage Gen2

Lucian Hymer
  • Updated

ADLS2 documentation: API guide, Authentication

Contents

A. Register an Active Directory app

You must first create an endpoint in Active Directory to sign in via OAuth 2.0:

  1. Navigate to Azure App registrations, and click + New registration.
  1. Register the new application:
    1. Give it whatever name you’d like.
    2. Choose the Supported account types for your setup.
    3. Provide the Redirect URI for the integrator.io OAuth 2.0 callback URL: https://integrator.io/connection/oauth2callback or https://eu.integrator.io/connection/oauth2callback.

  1. Click Register. On the resulting Overview page, record your Application (client) ID and Directory (tenant) ID for use below in the integrator.io connection. 

  1. On the Overview page, click Manage > Certificates & secrets. On the resulting Certificates & secrets page, click + New client secret; give the secret a name and choose an expiration period. Then, click Save.

  1. Record the Value (you can ignore the Secret ID). The token will not be revealed again when you sign back in.

  1. Finally, navigate to Manage > API permissions, and click + Add a permission:
    1. Choose the Azure Data Lake API.
    2. Check user_impersonation.
    3. Click Add permissions.

B. Configure access to the ADLS2 Server

  1. Navigate to your ADLS2 storage account in the Azure portal, and proceed to the Access control (IAM) page.
  2. Click + Add and then Add role assignment.

  1. Set the Role to Storage blob data owner, and select the account that you will be signing in as.
  1. Click Save.

C. Set up an HTTP connection

Start establishing an HTTP connection to Azure Data Lake Storage Gen2 in either of the following ways:

  • From the Resources menu, select Connections. Then, click + Create connection at the top right.
        – or –
  • While working in a new or existing integration, you can add an application to a flow simply by clicking Add source or Add destination/lookup.

In the resulting Application list, select HTTP.

The Create connection pane opens with required and advanced settings.

D. Edit HTTP application details

Note: This article is concerned only with the settings needed for ADLS2 connections. For complete field documentation, see Set up an OAuth 2.0 HTTP connection.

Configure HTTP headers: Add a header for x-ms-version, and provide any version value after 2017, such as 2020-08-04.

Base URI: Paste the Primary endpoint – Blob service value from Azure.

  1. In the Azure portal, navigate to the Endpoints section of your storage account.
  2. Copy the URL in Primary endpoint - Blob service. 

Media type: Accept the default value, JSON.

Success media type: Choose the media type that applies to this connection.

Error media type: Select XML.

E. Configure auth settings

Continuing in the Create connection pane, select OAuth 2.0 for the Authentication type. The settings then become specific to OAuth 2.0 connections:

Grant type: Select Authorization code.

iClient: Click the + button to create an iClient (token ID-secret pair). Enter an identifiable Name along with the following values you copied earlier:

  • Client ID: Paste in the Application (client) ID from the Azure App registration page.
  • Client secret: Paste in the token from the Certificates & secrets page. 

Click Save & close to add the iClient to your integrator.io account and select it for this connection.

Continuing in the Configure auth section, scroll to the Authentication URL setting:

Authentication URL: Enter the following URL, filling in your Directory (tenant) ID from the App Registration page:
https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize

Scopes: Copy the Base URI from above and append /user_impersonation,offline_access.

Access token URL: Enter this URL in the same format as the Authentication URL, replacing /authorize with /token.

Client authentication: Select Send client credentials in body.

Access token headers: Add a header for Content-Type, and give it a value of application/x-www-form-urlencoded.

Continuing in the Configure auth section, scroll to the Location setting:

Location: Select Header.

Header name: Accept the default value, Authorization.

Scheme: Select Bearer.

Authentication fail status code: Enter 403.

F. Save, test, and authorize

Once you have configured the HTTP connection, click Save & authorize to follow the Microsoft prompts and authorize this connection:

G. Integrate with ADLS2

Then, when creating a flow step (such as an export, import, or lookup) to Azure Data Lake Storage Gen2, start by selecting an HTTP application, and then choose your new connection. 

Here’s an example export that simply fetches a JSON file named test.json from a container named container1:

Read more about this connector:

Related articles

Comments

4 comments

Date Votes
  • Noppawat Eakpornpith

    Hello Lucien, 

    Thank you for your guideline, it's very helpful.
    I stuck at the authorization part and hope you could give me some guidance. When I finish creating connection and save & authorize, this error occurred. 

    Sorry, but we were not able to successfully authenticate your connection.
    {"code":"invalid_domain","message":"The domain of request url should be present in iClient.oauth2.validDomainNames","source":"resource","resolved":false,"occurredAt":1689331902762}

    I believe the reason is from the Valid domain names input. Unfortunately, your guideline above doesn't say anything about this. Could you please tell me which part I should take a look at so that I can solve this.

    Sincerely,

    Nate

    0
  • Stephen Brandt Documentation Manager Community moderator
    Engaged
    Great Answer
    Top Contributor
    Celigo University Level 3: Master

    Hi, Noppawat Eakpornpith. I looked for a similar problem and resolution and found the following advice:

    Update the Auth url and access token url as per the API documentation i.e, ("https://login.microsoftonline.com/common/oauth2/authorize" & "https://login.microsoftonline.com/common/oauth2/token") and provide the valid domain as "microsoftonline.com".

    If that doesn't help, please reply and we'll get you to some individual support. 

    1
  • Noppawat Eakpornpith

    Hello, Stephen Brandt

    Thank you, it did work for me. It was stupid of me because I put the valid domain like this "iClient.oauth2.microsoftonline.com" while it's supposed to be just "microsoftonline.com".

    By the way, I tried to import JSON data to Azure database and refer to this API DOC Filesystem - Create - REST API (Azure Storage Services) | Microsoft Learn

    Somehow it kept erroring code 400 with message An HTTP header that&apos;s mandatory for this request is not specified. Do you happen to have any advise about this case? I tried put request header in the doc to configure HTTP headers in celigo one by one but it still keeps erroring the same code.

    Nate

    0
  • Stephen Brandt Documentation Manager Community moderator
    Engaged
    Great Answer
    Top Contributor
    Celigo University Level 3: Master

    Good to hear, Noppawat Eakpornpith, and thanks for circling back. 

    For your new question, I'm a bit out of my depth, and I wouldn't be able to troubleshoot it without observing or gaining access to your flow. That's best done via a support ticket, if you'll forgive me for the handoff. 

    1

Please sign in to leave a comment.