Extract and Load: setting up the data lake

The PanoramaDatalake class is set up to connect to an AWS data lake. Its methods can be overridden for other data lake technologies, but this section describes the standard AWS setup, which is the only one supported out of the box.

AWS resources to create

To set up your AWS data lake you will need to:

• Create or use an IAM user or role with permissions to access the S3 buckets, KMS (if the buckets are encrypted), Glue, and Athena.

• Create one S3 bucket to store the data, and another to hold the Athena query results.

  • Encrypted buckets with strict access policies are recommended.

• Create the Panorama database in Athena:

  CREATE DATABASE panorama
  

• Create an Athena workgroup to keep the queries isolated from other projects, and set its Query result location to the results bucket you created above.

These names (bucket, database, workgroup, region) are what you put into the settings file. See Extract and Load: configuration.

IAM permissions

To work with an AWS data lake, create a user (for example panorama-elt) and attach a policy (named, for example, PanoramaELT) with at least the following permissions. Replace <region> and <account id> with the proper values.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "glue:BatchCreatePartition",
                "glue:GetDatabase",
                "athena:StartQueryExecution",
                "glue:CreateTable",
                "athena:GetQueryExecution",
                "athena:GetQueryResults",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:DeleteTable",
                "glue:GetPartitions",
                "glue:UpdateTable"
            ],
            "Resource": [
                "arn:aws:athena:<region>:<account id>:workgroup/panorama",
                "arn:aws:glue:<region>:<account id>:database/panorama",
                "arn:aws:glue:<region>:<account id>:catalog",
                "arn:aws:glue:<region>:<account id>:table/panorama/*"
            ]
        },
        {
            "Sid": "PanoramaS3",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<data bucket>",
                "arn:aws:s3:::<data bucket>/*",
                "arn:aws:s3:::<results bucket>",
                "arn:aws:s3:::<results bucket>/*"
            ]
        }
    ]
}

Replace <data bucket> with the panorama_raw_data_bucket you created and <results bucket> with the Athena query-results bucket.

Important

If you use KMS-encrypted S3 buckets, you must also grant the KMS permissions needed to use the key (typically kms:Decrypt, kms:GenerateDataKey, and kms:DescribeKey) on the key’s ARN.

Warning

The project README shows the S3 statement as "Action": "s3:*" on "Resource": "*", which grants full access to every bucket in the account. Do not use that form in production. The scoped statement above restricts the ELT principal to the two Panorama buckets; add specific actions only if you find they are required. Grant the least privilege that makes extraction work.

Providing the credentials

The agent authenticates with the aws_access_key and aws_secret_access_key values from the datalake section of the settings file, in the aws_region of the Athena engine. See Extract and Load: configuration for where these go.

Once the AWS resources exist and the settings file points at them, continue with the first run to create the tables and views and perform the initial data upload.

Back to Home