Azure Datalake Gen2

To connect your Azure Datalake Gen2 to Dataloop, follow these setup instructions.

Create App Registration

1. Navigate to “App registrations” on the search bar
2. Create a new registration and choose a meaningful name (e.g. dataloop-app )
3. Locate the app's clientID (Application ID) and tenantID (Directory ID) in the Application overview, you’ll need them for the integration phase

Create a new client secret

1. At the newly created registration, on the left-side panel, navigate to "certificates & secrets"
2. Click on “New client secret" to create a new secret to for the application
3. Fill in a meaningful name for the secret in the description field
4. Define when will the integration expire (We recommend setting it to the longest period possible)
5. NOTICE: Copy and store this secret's value right away, because you won’t have access to it later, and you’ll be needing it for the integration phase.
6. Click add to add the secret to the Application

Create a Storage account

1. Navigate to “Storage accounts” on the search bar
2. Choose the desired Storage account (Or create a storage account if you don't already have one) - Must enable hierarchical namespace
3. When creating Storage Account under advanced tab, go to Data Lake Storage Gen2 and mark Enable hierarchical namespace

For a step-by-step guide on creating a storage account in Azure, read Azure docs.

Create a container 

1. Navigate to “Storage accounts” on the search bar
2. Choose the desired Storage account
3. On the left-side panel Navigate to "Containers”
4. Choose the desired Container (Or create a container if you don't already have one)
   For a step-by-step guide on creating a Container in Azure, read Azure docs.

Add an IAM role assignments to container

1.  Select the chosen container (the one you will integrate with DataLoop)
2. On the left-side panel navigate to "Access Control (IAM)"
3. Click on the Add button
4. Choose “Add role assignment”
5. Under Role section search & choose “Storage Blob Data Contributor”, click on it and click next
6. Under Members section, make sure “Assign access to” is set to “User, group, or service principals”
7. Under Members section, click on “Select members”
8. On the search bar enter your registration app’s name you created earlier, choose it and click select
9. Click “review + assign”
10. It can take 5 minutes for the permissions to be updated and available to use by Dataloop

For a step-by-step guide including screenshots on setting up a policy in Azure, read here.

Create an Azure Integration on DataLoop platform

1.  On the Dataloop platform navigate to the left-side panel and choose integrations
2. Click on the “CREATE NEW INTEGRATION” button
3. Enter a meaningful name for the integration
4. Under Type, choose “Azure Blob”
5. Under Account Name enter your storage account name (Where the container is located)
6. Under Client Id, enter the registration app clientID from earlier 
7. Under Tenant Id, enter the registration app tenantID from earlier 
8. Under Client secret, enter client secret’s value from earlier 
9. Click on create 

Create an Azure Storage Driver on DataLoop platform

1.  On the Dataloop platform navigate to the left-side panel, choose “Data Management” and then click on “Cloud Storage” 
2. Click on the “CREATE DRIVER” button
3. Enter a meaningful name for the storage driver
4. Under Integration choose your relevant Azure Integration
5. Under Type choose “AzureDataLakeGen2” type 
6. Enter your Container name (The one that is relevant to the integration)
7. (Optional) Allow delete items
8. Click “TEST” to test if the connection is successful 
9. Click “Create”