Architecture overview

Panorama is a stack, not a single application. This page shows how the components fit together so that the rest of the manual makes sense in context.

The data pipeline

Panorama moves data through a series of layers, from raw source data to finished dashboards:

Data sources            Extraction            Data lake (AWS)              BI / dashboards
------------            ----------            --------------              ---------------
MySQL          ┐                        ┌─ raw tables (CSV on S3)   ┐
MongoDB        ├──►  panorama-elt  ──►  ├─ table views (typed)      ├──►  Amazon QuickSight
Tracking logs  │      (Extract &        ├─ curated views (joins,    │      dashboards
CSV / Excel    ┘       Load agent)      │   filters, calc fields)   ┘        │
Other systems                           └─ Athena / Glue catalog             │
                                                                             ▼
                                                          panorama-openedx-backend (LMS)
                                                                             │
                                                                             ▼
                                                                frontend-app-panorama (MFE)
                                                                             │
                                                                             ▼
                                                                        End users

  1. Extract and Load. panorama-elt connects to each data source, extracts the data in tabular form, and uploads it unmodified to the data lake as CSV files. Keeping the raw data intact means transformations happen later and can be re-run without re-extracting.

  2. Raw tables. In the data lake catalog (AWS Glue / Athena), each uploaded table is registered as an external table over the CSV files. All columns are strings at this layer.

  3. Table views. The first transformation layer. For each raw table Panorama creates a view with the same columns but with numeric and date fields cast to proper types, and the literal string 'NULL' replaced with real NULL values.

  4. Curated views. Further pipeline steps, implemented as SQL views, add joins, filters, and calculated fields. Panorama ships example views for Open edX (enrollments, course access roles, manual enrollment history).

  5. Dashboards. A BI tool (Amazon QuickSight in the standard Panorama setup) reads the curated views and renders dashboards.

  6. In-LMS delivery. For Open edX, panorama-openedx-backend generates signed embed URLs for QuickSight dashboards, and frontend-app-panorama renders them to users inside the LMS.

The components

Component

Repository

Role

Panorama Extract and Load

panorama-elt

CLI agent that extracts data from the sources and loads it into the data lake, and provides helpers to create the raw tables and the first pipeline layer.

Panorama Open edX Backend

panorama-openedx-backend

Django app for Open edX that provides the API the MFE calls to obtain dashboard embed URLs, user access, and the running mode. It does not extract or load any data.

Panorama MFE

frontend-app-panorama

Micro-frontend that shows the dashboards to users, served by default at <mfe base url>/panorama.

Tutor plugin for Panorama

tutor-contrib-panorama

Convenience plugin that wires everything together for Open edX: extraction, tracking logs, the backend, and the MFE.

Note

The extraction side (panorama-elt) and the presentation side (panorama-openedx-backend + frontend-app-panorama) are independent. The backend and MFE only need dashboards to exist in your BI tool; they do not care how the data got there. Likewise the ELT agent has no knowledge of the LMS UI.

Where the AWS pieces live

The standard Panorama data lake uses:

  • Amazon S3 — stores the raw CSV files (and a separate bucket for Athena query results).

  • AWS Glue Data Catalog — holds the table and partition metadata.

  • Amazon Athena — the SQL query engine used to create tables and views and to serve the BI tool.

  • Amazon QuickSight — the dashboards and embedding layer.

The ELT agent’s data lake setup section covers exactly which resources and permissions are required.

Back to Home