.. _elt_datalake:

Extract and Load: setting up the data lake
==========================================

.. contents:: On this page
   :local:
   :depth: 2

The ``PanoramaDatalake`` class is set up to connect to an **AWS** data lake. Its methods can
be overridden for other data lake technologies, but this section describes the standard AWS
setup, which is the only one supported out of the box.

AWS resources to create
-----------------------

To set up your AWS data lake you will need to:

* Create or use an **IAM user or role** with permissions to access the S3 buckets, KMS (if
  the buckets are encrypted), Glue, and Athena.
* Create **one S3 bucket** to store the data, and **another** to hold the Athena query
  results.

  * Encrypted buckets with strict access policies are recommended.

* Create the Panorama **database** in Athena:

  .. code-block:: sql

     CREATE DATABASE panorama

* Create an Athena **workgroup** to keep the queries isolated from other projects, and set
  its *Query result location* to the results bucket you created above.

These names (bucket, database, workgroup, region) are what you put into the settings file.
See :ref:`elt_configuration`.

IAM permissions
---------------

To work with an AWS data lake, create a user (for example ``panorama-elt``) and attach a
policy (named, for example, ``PanoramaELT``) with at least the following permissions. Replace
``<region>`` and ``<account id>`` with the proper values.

.. code-block:: json

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "glue:BatchCreatePartition",
                   "glue:GetDatabase",
                   "athena:StartQueryExecution",
                   "glue:CreateTable",
                   "athena:GetQueryExecution",
                   "athena:GetQueryResults",
                   "glue:GetDatabases",
                   "glue:GetTable",
                   "glue:DeleteTable",
                   "glue:GetPartitions",
                   "glue:UpdateTable"
               ],
               "Resource": [
                   "arn:aws:athena:<region>:<account id>:workgroup/panorama",
                   "arn:aws:glue:<region>:<account id>:database/panorama",
                   "arn:aws:glue:<region>:<account id>:catalog",
                   "arn:aws:glue:<region>:<account id>:table/panorama/*"
               ]
           },
           {
               "Sid": "PanoramaS3",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:DeleteObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": [
                   "arn:aws:s3:::<data bucket>",
                   "arn:aws:s3:::<data bucket>/*",
                   "arn:aws:s3:::<results bucket>",
                   "arn:aws:s3:::<results bucket>/*"
               ]
           }
       ]
   }

Replace ``<data bucket>`` with the ``panorama_raw_data_bucket`` you created and
``<results bucket>`` with the Athena query-results bucket.

.. important::

   If you use KMS-encrypted S3 buckets, you must also grant the KMS permissions needed to use
   the key (typically ``kms:Decrypt``, ``kms:GenerateDataKey``, and ``kms:DescribeKey``) on
   the key's ARN.

.. warning::

   The project README shows the S3 statement as ``"Action": "s3:*"`` on ``"Resource": "*"``,
   which grants full access to **every** bucket in the account. Do not use that form in
   production. The scoped statement above restricts the ELT principal to the two Panorama
   buckets; add specific actions only if you find they are required. Grant the least privilege
   that makes extraction work.

Providing the credentials
-------------------------

The agent authenticates with the ``aws_access_key`` and ``aws_secret_access_key`` values from
the ``datalake`` section of the settings file, in the ``aws_region`` of the Athena engine.
See :ref:`elt_configuration` for where these go.

Once the AWS resources exist and the settings file points at them, continue with the
:ref:`first run <elt_commands>` to create the tables and views and perform the initial data
upload.

`Back to Home <index.html>`_