Tutor plugin: what it deploys and operations

This page describes what the plugin actually builds, injects, and schedules, so you can operate it and troubleshoot. Most behavior is gated by PANORAMA_MODE.

Images it builds

The plugin registers two buildable Docker images:

  • panorama-elt — the extraction agent (PANORAMA_DOCKER_IMAGE).

  • panorama-elt-logs — a Fluent Bit image for shipping tracking logs (PANORAMA_LOGS_DOCKER_IMAGE).

They are built, pulled, and pushed with the usual Tutor commands, for example tutor images build panorama-elt. It also installs panorama-openedx-backend into the openedx image (only when PANORAMA_MFE_ENABLED is true), which is why you rebuild openedx after enabling the plugin.

Settings it injects into the LMS

The plugin patches the LMS common settings with the values the backend reads:

  • PANORAMA_MODE, PANORAMA_ENABLE_STUDENT_VIEW, PANORAMA_DEFAULT_USER_ARN;

  • PANORAMA_AWS_ACCESS_KEY, PANORAMA_AWS_SECRET_ACCESS_KEY, PANORAMA_REGION, PANORAMA_AWS_ACCOUNT_ID.

It also sets MFE_CONFIG["PANORAMA_URL"] so the platform knows where the MFE lives, and registers the Panorama MFE (repository, version, port 2100) served at /panorama/.

Initialization tasks

On tutor {local|k8s} do init (run during launch / quickstart), the plugin runs:

  • ELT init (panorama-elt service), skipped in DEMO mode:

    • test-connections — verify it can reach the data sources and data lake.

    • In CUSTOM mode only: create-datalake-tables --all then create-table-views --all (so you do not run the first run by hand).

  • Backend migrations (lms service), only in SAAS and CUSTOM modes: it runs the Django migrations for panorama_openedx_backend.

Periodic extraction

Extraction is scheduled from the PANORAMA_CRONTAB value (default 55 * * * *, i.e. hourly). It is skipped entirely in DEMO mode.

  • Local / Docker Compose. A panorama service runs a cron container that periodically executes panorama.py ... extract-and-load --all.

  • Kubernetes. A CronJob named panorama-elt runs the same extraction on the schedule.

Note

The always-present panorama-job Job (Kubernetes) is what backs the manual extract-and-load command described below; the scheduled CronJob is what runs extraction automatically.

Tracking-log shipping

Tracking logs are shipped to the data lake by Fluent Bit (the panorama-elt-logs image), which enables event-based analytics. This is not run in DEMO or FREE mode (FREE extracts only relational and courseware data):

  • Local. A panorama_logs Fluent Bit service tails the LMS logs and uploads them, using PANORAMA_FLB_LOG_LEVEL, PANORAMA_LOGS_TOTAL_FILE_SIZE, and PANORAMA_LOGS_UPLOAD_TIMEOUT.

  • Kubernetes. A Fluent Bit DaemonSet ships node/pod logs, gated by PANORAMA_RUN_K8S_FLUENTBIT and only in SAAS / CUSTOM modes.

Important

On Kubernetes, run the Fluent Bit DaemonSet in only one namespace. If you have multiple Open edX namespaces, set PANORAMA_RUN_K8S_FLUENTBIT: false in all but one to avoid duplicate log collection.

Running extraction on demand

The plugin adds an extract-and-load do-command so you can trigger extraction manually (useful for a first full load, or after config changes):

tutor local do extract-and-load --all
tutor local do extract-and-load --tables student_courseenrollment,auth_user
tutor local do extract-and-load --all --force

The options mirror the ELT agent: --all / -a, --tables / -t, --force, and --debug. See Extract and Load: command reference and first run for what they do. On Kubernetes use tutor k8s do extract-and-load ....

Tip

For a first full load of partitioned tables in CUSTOM mode, run the do-command once with --force after init has created the tables and views.

Troubleshooting pointers

  • Nothing is extracted. Confirm PANORAMA_MODE is not DEMO and that the crontab / CronJob exists. Check test-connections output from the init task.

  • Dashboards do not load. The backend requires HTTPS and rejects local/dev hosts in DEMO mode; see Open edX Backend: configuration. Verify the MFE is served at /panorama/ and MFE_CONFIG["PANORAMA_URL"] is set.

  • Access denied on AWS. Recheck the IAM policy and Lake Formation grants (Tutor plugin: configuration).

  • Pods OOM-killed (Kubernetes). Set PANORAMA_K8S_JOB_MEMORY_REQUEST / PANORAMA_K8S_JOB_MEMORY_LIMIT.

Back to Home