.. _architecture:

Architecture overview
=====================

.. contents:: On this page
   :local:
   :depth: 2

Panorama is a stack, not a single application. This page shows how the components fit
together so that the rest of the manual makes sense in context.

The data pipeline
-----------------

Panorama moves data through a series of layers, from raw source data to finished
dashboards::

    Data sources            Extraction            Data lake (AWS)              BI / dashboards
    ------------            ----------            --------------              ---------------
    MySQL          ┐                        ┌─ raw tables (CSV on S3)   ┐
    MongoDB        ├──►  panorama-elt  ──►  ├─ table views (typed)      ├──►  Amazon QuickSight
    Tracking logs  │      (Extract &        ├─ curated views (joins,    │      dashboards
    CSV / Excel    ┘       Load agent)      │   filters, calc fields)   ┘        │
    Other systems                           └─ Athena / Glue catalog             │
                                                                                 ▼
                                                              panorama-openedx-backend (LMS)
                                                                                 │
                                                                                 ▼
                                                                    frontend-app-panorama (MFE)
                                                                                 │
                                                                                 ▼
                                                                            End users

1. **Extract and Load.** ``panorama-elt`` connects to each data source, extracts the data in
   tabular form, and uploads it **unmodified** to the data lake as CSV files. Keeping the raw
   data intact means transformations happen later and can be re-run without re-extracting.

2. **Raw tables.** In the data lake catalog (AWS Glue / Athena), each uploaded table is
   registered as an *external* table over the CSV files. All columns are strings at this
   layer.

3. **Table views.** The first transformation layer. For each raw table Panorama creates a
   view with the same columns but with numeric and date fields cast to proper types, and the
   literal string ``'NULL'`` replaced with real ``NULL`` values.

4. **Curated views.** Further pipeline steps, implemented as SQL views, add joins, filters,
   and calculated fields. Panorama ships example views for Open edX (enrollments, course
   access roles, manual enrollment history).

5. **Dashboards.** A BI tool (Amazon QuickSight in the standard Panorama setup) reads the
   curated views and renders dashboards.

6. **In-LMS delivery.** For Open edX, ``panorama-openedx-backend`` generates signed embed
   URLs for QuickSight dashboards, and ``frontend-app-panorama`` renders them to users inside
   the LMS.

The components
--------------

.. list-table::
   :header-rows: 1
   :widths: 22 20 58

   * - Component
     - Repository
     - Role
   * - :ref:`Panorama Extract and Load <elt_overview>`
     - `panorama-elt <https://github.com/aulasneo/panorama-elt>`_
     - CLI agent that extracts data from the sources and loads it into the data lake, and
       provides helpers to create the raw tables and the first pipeline layer.
   * - :ref:`Panorama Open edX Backend <backend_overview>`
     - `panorama-openedx-backend <https://github.com/aulasneo/panorama-openedx-backend>`_
     - Django app for Open edX that provides the API the MFE calls to obtain dashboard embed
       URLs, user access, and the running mode. It does **not** extract or load any data.
   * - :ref:`Panorama MFE <mfe_overview>`
     - `frontend-app-panorama <https://github.com/aulasneo/frontend-app-panorama>`_
     - Micro-frontend that shows the dashboards to users, served by default at
       ``<mfe base url>/panorama``.
   * - :ref:`Tutor plugin for Panorama <tutor_overview>`
     - `tutor-contrib-panorama <https://github.com/aulasneo/tutor-contrib-panorama>`_
     - Convenience plugin that wires everything together for Open edX: extraction, tracking
       logs, the backend, and the MFE.

.. note::

   The extraction side (``panorama-elt``) and the presentation side
   (``panorama-openedx-backend`` + ``frontend-app-panorama``) are independent. The backend and
   MFE only need dashboards to exist in your BI tool; they do not care how the data got there.
   Likewise the ELT agent has no knowledge of the LMS UI.

Where the AWS pieces live
-------------------------

The standard Panorama data lake uses:

* **Amazon S3** — stores the raw CSV files (and a separate bucket for Athena query results).
* **AWS Glue Data Catalog** — holds the table and partition metadata.
* **Amazon Athena** — the SQL query engine used to create tables and views and to serve the
  BI tool.
* **Amazon QuickSight** — the dashboards and embedding layer.

The ELT agent's :ref:`data lake setup <elt_datalake>` section covers exactly which resources
and permissions are required.

`Back to Home <index.html>`_