GCP Cloud Storage

To connect your data from GCS storage as a Dataloop dataset, follow these setup instructions.

Create an IAM Role

1. Go to the "IAM & admin" section of the Cloud Console
2. Click on “Roles” in the left navigation panel
3. Click on the "Create role" button
4. Give the role a meaningful title (name), and optional description and ID.
5. Click on “Add Permissions”, then search and add the following permissions:
   1. storage.objects.create
   2. storage.objects.delete
   3. storage.objects.get
   4. storage.objects.list
   5. storage.buckets.get
6. Click on the "Create" button to create the IAM role.

NOTE: “storage.objects.delete” permission allows DataLoop platform to delete dataset items - please see downstream

For a step-by-step guide on creating an IAM role in GCP, read the related GCP documentation.

Create a Service Account

1. Go to the "IAM & admin" section of the Cloud Console
2. Click on “Service Accounts” in the left navigation panel
3. Click on the "Create service account" button.
4. Enter a name and description for the service account
5. Click on the “Create and continue”
6. Click on the "Done" button to create the service account.

For a step-by-step guide on creating a service account in GCP, read the related GCP documentation.

Create a Private Key for the Service Account

1. Go to the "IAM & admin" section of the Cloud Console
2. Click on “Service Accounts” in the left navigation panel
3. Find the desired service account for which you want to add a private key, and click on its name to open its details
   NOTE: The Service Account you recently created
4. Click on the “Keys” tab
5. Click on the "Add key" button.
6. Click on “Create new key”
7. Choose “JSON” as the key type format, and click “Create”

The private key will be generated and downloaded to your computer, and you will need to use it to authenticate as the service account on the DataLoop platform

NOTE: You’ll need the JSON file for the integration phase please save it in a convenient location

Create a Cloud Storage Bucket

1. In the Google Cloud console, go to the Cloud Storage Buckets page
2. Click on the "Create bucket" button
3. Provide a unique bucket name - related to DataLoop
4. Click “Continue”
5. Under “Choose where to store your data” - Choose a location for the bucket and click “Continue”.
6. For all other optional settings use default values
7. Click on the "Create bucket" button

For a step-by-step guide on creating an cloud storage bucket in GCP, read the related GCP documentation

Add permissions to the Cloud Storage Bucket to grant access to the IAM role

1. In the Google Cloud console, go to the Cloud Storage Buckets page
2. Find the desired bucket for which you want to grant access, and click on its name to open its details
   NOTE: The Bucket you recently created
3. Click on the “Permissions” tab
4. Click on the “Grant Access” button
5. Under “Add principals” add your Service account principal
6. Under “Assign roles” choose “custom” and choose the role you recently created 
7. Click on “Save”
   
   For a step-by-step guide including screenshots on setting up a policy in GCP, read here.