AWS Access Key Integration

Create an S3 Bucket

1. Log in to the AWS Management Console
2. Go to the S3 service
3. Click on the "Create bucket" button
4. Provide a unique bucket name - related to DataLoop
5. Make sure Block Public Access settings for this bucket - Block all public access
6. For all other optional settings use default values
7. Click on the "Create bucket" button

For a step-by-step guide on creating an S3 bucket in AWS, read the AWS documentation.

Create an IAM Policy

1. Go to the IAM service
2. Click on “Policies” in the left navigation panel
3. Click on the "Create policy" button
4. Select the JSON tab
5. Define policy document in JSON format, like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowBucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
	            "arn:aws:s3:::<DataLoopBucketName>",
                "arn:aws:s3:::<DataLoopBucketName>/*"
            ]
        }
    ]
}

• Note: Replace <DataLoopBucketName> with the name of the desired S3 bucket.

6. Click on the next button to Review the policy
7. Provide a meaningful name for the policy and an optional description
8. Click the “Create policy” button to finish

Notes: 

a. “s3:DeleteObject” action allows DataLoop platform to delete dataset items - please see downstream
b. The resource arn:aws:s3:::<DataLoopBucketName> is required for the ListBucket action

For a step-by-step guide on creating an IAM policy in AWS, read AWS documentation.

Create an IAM User:

1. Go to the IAM service
2. Click on “Users” in the left navigation panel
3. Click on the "Add users" button
4. Enter a user name for the new user
5. Select "Programmatic access" as the type of access you want to grant the user - Already selected as default
   Note: Use the default access type and don't choose console access
6. Click on the next button 
7. Choose "Attach policies directly" 
8. Select the related policy you created for accessing the S3 bucket
9. Click on the "next" button 
10. Click on the "Create user" button to create the user

For a step-by-step guide on creating an IAM user in AWS, the AWS documentation .

Create an Access Key for the IAM User

1. Go to the IAM service
2. Click on “Users” in the left navigation panel
3. Find and choose the desired user whom you want to create an access key and click on the user's name
   NOTE: The DataLoop user you created
4. Click on the “Security credentials” tab
5. In the “Access keys” section, click on “Create access key”
6. Under “Access key best practices & alternatives” choose “Application running outside AWS”
7. Click “Next”
8. Add an informative and meaningful description that include DataLoop in it
9. Click on “Create access key”
10. Copy and save the access key ID and secret access key in a secure location OR download the CSV file containing the keys
    NOTE: You’ll need the access key ID and secret access key for the integration phase
11. Click on “Done” button

For a step-by-step guide including screenshots on setting up a policy in AWS, read here.

Create an AWS S3 Integration on DataLoop platform

1. On the Dataloop platform navigate to the left-side panel and choose integrations
2. Click on the “CREATE NEW INTEGRATION” button
3. Enter a meaningful name for the integration
4. Under Type, choose “AWS S3”
5. Under Key, enter the IAM user’s access key ID from earlier 
6. Under Secret, enter the IAM user’s secret access key from earlier 
7. Click on Create 

Create an AWS S3 Storage Driver on DataLoop platform

1. On the Dataloop platform navigate to the left-side panel, choose “Data Management” and then click on “Cloud Storage” 
2. Click on the “CREATE DRIVER” button
3. Enter a meaningful name for the storage driver
4. Under Integration choose your relevant AWS S3 Integration
5. Enter your AWS Bucket name (The one that is relevant to the integration)
6. Select the region your S3 bucket is located in 
7. (Optional) Allow delete items
8. Click “TEST” to test if the connection is successful 
9. Click “Create”