Extract and Load: running in Docker

The agent can run inside a Docker container, which keeps the host free of Python and dependencies. A Dockerfile is included. The image contains only the Panorama ELT code and its Python dependencies; the settings file is deliberately kept outside the image and bind-mounted at runtime, so configuration changes never require a rebuild.

Building the image

docker build -t aulasneo/panorama-elt:$(python3 -c "from panorama_elt.__about__ import __version__; print(__version__)") -t aulasneo/panorama-elt:latest .

Runtime directories

The container generally uses two bind mounts:

• /config — settings files, for example /config/panorama_openedx_settings.yaml.

• /work — temporary CSV work files generated during extraction.

The application writes temporary CSV files to the current working directory and deletes them after upload. Set the container working directory to /work to control where those files are created.

Important

Mount /config read-only for regular extraction jobs. Mount it read-write only for commands that update the settings file, such as set-tables and set-tables-fields.

The container must have network access to the data sources and the data lake.

Example: one-off command

docker run --rm \
  -v /etc/panorama:/config:ro \
  -v /var/lib/panorama/work:/work \
  -w /work \
  aulasneo/panorama-elt:latest \
  --settings /config/panorama_openedx_settings.yaml test-connections

One-time setup and maintenance

Create the tables and views the same way, with the mounted settings file:

docker run --rm -it \
  -v /etc/panorama:/config:ro \
  -v /var/lib/panorama/work:/work \
  -w /work \
  aulasneo/panorama-elt:latest \
  --settings /config/panorama_openedx_settings.yaml create-datalake-tables --all

docker run --rm -it \
  -v /etc/panorama:/config:ro \
  -v /var/lib/panorama/work:/work \
  -w /work \
  aulasneo/panorama-elt:latest \
  --settings /config/panorama_openedx_settings.yaml create-table-views --all

For commands that update the settings file, mount /config read-write:

docker run --rm -it \
  -v /etc/panorama:/config:rw \
  -v /var/lib/panorama/work:/work \
  -w /work \
  aulasneo/panorama-elt:latest \
  --settings /config/panorama_openedx_settings.yaml set-tables-fields --all

Initial full upload

Run the first, forced upload of partitioned tables once:

docker run --rm --name panorama-elt \
  -v /etc/panorama:/config:ro \
  -v /var/lib/panorama/work:/work \
  -w /work \
  aulasneo/panorama-elt:latest \
  --settings /config/panorama_openedx_settings.yaml extract-and-load --all --force

Periodic extraction from cron

After the tables and views exist, schedule extract-and-load from the host crontab. This example runs every hour, keeps the settings outside the image, and writes logs on the host:

0 * * * * docker run --rm -v /etc/panorama:/config:ro -v /var/lib/panorama/work:/work -w /work aulasneo/panorama-elt:latest --settings /config/panorama_openedx_settings.yaml extract-and-load --all >> /var/log/panorama/panorama.log 2>&1

Back to Home