.. _elt_docker:

Extract and Load: running in Docker
===================================

.. contents:: On this page
   :local:
   :depth: 2

The agent can run inside a Docker container, which keeps the host free of Python and
dependencies. A ``Dockerfile`` is included. The image contains only the Panorama ELT code and
its Python dependencies; the settings file is deliberately kept **outside** the image and
bind-mounted at runtime, so configuration changes never require a rebuild.

Building the image
------------------

.. code-block:: shell

   docker build -t aulasneo/panorama-elt:$(python3 -c "from panorama_elt.__about__ import __version__; print(__version__)") -t aulasneo/panorama-elt:latest .

Runtime directories
-------------------

The container generally uses two bind mounts:

* ``/config`` — settings files, for example ``/config/panorama_openedx_settings.yaml``.
* ``/work`` — temporary CSV work files generated during extraction.

The application writes temporary CSV files to the current working directory and deletes them
after upload. Set the container working directory to ``/work`` to control where those files
are created.

.. important::

   Mount ``/config`` **read-only** for regular extraction jobs. Mount it read-write only for
   commands that update the settings file, such as ``set-tables`` and ``set-tables-fields``.

   The container must have network access to the data sources and the data lake.

Example: one-off command
------------------------

.. code-block:: shell

   docker run --rm \
     -v /etc/panorama:/config:ro \
     -v /var/lib/panorama/work:/work \
     -w /work \
     aulasneo/panorama-elt:latest \
     --settings /config/panorama_openedx_settings.yaml test-connections

One-time setup and maintenance
------------------------------

Create the tables and views the same way, with the mounted settings file:

.. code-block:: shell

   docker run --rm -it \
     -v /etc/panorama:/config:ro \
     -v /var/lib/panorama/work:/work \
     -w /work \
     aulasneo/panorama-elt:latest \
     --settings /config/panorama_openedx_settings.yaml create-datalake-tables --all

.. code-block:: shell

   docker run --rm -it \
     -v /etc/panorama:/config:ro \
     -v /var/lib/panorama/work:/work \
     -w /work \
     aulasneo/panorama-elt:latest \
     --settings /config/panorama_openedx_settings.yaml create-table-views --all

For commands that update the settings file, mount ``/config`` read-write:

.. code-block:: shell

   docker run --rm -it \
     -v /etc/panorama:/config:rw \
     -v /var/lib/panorama/work:/work \
     -w /work \
     aulasneo/panorama-elt:latest \
     --settings /config/panorama_openedx_settings.yaml set-tables-fields --all

Initial full upload
-------------------

Run the first, forced upload of partitioned tables once:

.. code-block:: shell

   docker run --rm --name panorama-elt \
     -v /etc/panorama:/config:ro \
     -v /var/lib/panorama/work:/work \
     -w /work \
     aulasneo/panorama-elt:latest \
     --settings /config/panorama_openedx_settings.yaml extract-and-load --all --force

Periodic extraction from cron
-----------------------------

After the tables and views exist, schedule ``extract-and-load`` from the host crontab. This
example runs every hour, keeps the settings outside the image, and writes logs on the host:

.. code-block:: text

   0 * * * * docker run --rm -v /etc/panorama:/config:ro -v /var/lib/panorama/work:/work -w /work aulasneo/panorama-elt:latest --settings /config/panorama_openedx_settings.yaml extract-and-load --all >> /var/log/panorama/panorama.log 2>&1

`Back to Home <index.html>`_