.. _elt_installation:

Extract and Load: installation
==============================

.. contents:: On this page
   :local:
   :depth: 2

Requirements
------------

* A Linux system.
* Python 3.12.
* ``make`` (``sudo apt install make``).
* ``virtualenv`` (``pip install virtualenv``).

The agent must reach the data sources (MySQL and, for Open edX, MongoDB) and the AWS data
lake (S3, Glue, Athena). It can be installed on the application host, on the database host,
or on a bastion host, as long as it has that network connectivity.

.. tip::

   You can split the installation across multiple hosts, for example one host to reach the
   MySQL tables and another for the MongoDB course structures.

Install from source
--------------------

#. Clone the repository:

   .. code-block:: shell

      git clone https://github.com/aulasneo/panorama-elt

#. Install:

   .. code-block:: shell

      make install

   .. important::

      Do **not** run ``make install`` from inside a virtual environment.

   This command will:

   * Create a ``virtualenv`` in the ``venv`` directory using the ``python3`` interpreter.
   * Install the Python dependencies listed in ``requirements.txt``.
   * Copy the settings file from the template, if it does not already exist.

Running the agent
-----------------

The scripts must run from the virtual environment. By default, running ``panorama.py`` from
the command line uses the Python interpreter in the installed virtual environment
(``venv/bin/python``), because the script starts with ``#! venv/bin/python``.

.. code-block:: shell

   ./panorama.py --help

To use a settings file other than the default ``panorama_settings.yaml``, pass ``--settings``:

.. code-block:: shell

   ./panorama.py --settings panorama_openedx_settings.yaml test-connections

.. note::

   ``--settings`` is a top-level option and must appear **before** the command, as shown
   above. The same applies to ``--debug`` and ``--version``.

Running in Docker
-----------------

The agent can also run inside a Docker container, which avoids installing Python and the
dependencies on the host. See :ref:`elt_docker` for building the image and running one-off
and scheduled jobs.

`Back to Home <index.html>`_