Extract and Load: installation

Requirements

  • A Linux system.

  • Python 3.12.

  • make (sudo apt install make).

  • virtualenv (pip install virtualenv).

The agent must reach the data sources (MySQL and, for Open edX, MongoDB) and the AWS data lake (S3, Glue, Athena). It can be installed on the application host, on the database host, or on a bastion host, as long as it has that network connectivity.

Tip

You can split the installation across multiple hosts, for example one host to reach the MySQL tables and another for the MongoDB course structures.

Install from source

  1. Clone the repository:

    git clone https://github.com/aulasneo/panorama-elt

  2. Install:

    make install

    Important

    Do not run make install from inside a virtual environment.

    This command will:

    • Create a virtualenv in the venv directory using the python3 interpreter.

    • Install the Python dependencies listed in requirements.txt.

    • Copy the settings file from the template, if it does not already exist.

Running the agent

The scripts must run from the virtual environment. By default, running panorama.py from the command line uses the Python interpreter in the installed virtual environment (venv/bin/python), because the script starts with #! venv/bin/python.

./panorama.py --help

To use a settings file other than the default panorama_settings.yaml, pass --settings:

./panorama.py --settings panorama_openedx_settings.yaml test-connections

Note

--settings is a top-level option and must appear before the command, as shown above. The same applies to --debug and --version.

Running in Docker

The agent can also run inside a Docker container, which avoids installing Python and the dependencies on the host. See Extract and Load: running in Docker for building the image and running one-off and scheduled jobs.

Back to Home