Extract and Load: non-Open edX sources

The default settings file is prepared for Open edX: it contains the most common tables with all the field information for the recommended setup, and it includes the MongoDB scripts to obtain course structures, which are specific to Open edX.

The MySQL data source, however, works for any MySQL installation. This page describes how to point Panorama at a non-Open edX system.

Configuring a generic MySQL source

  1. Configure the MySQL connection settings (mysql_host, mysql_username, mysql_password, mysql_database) for the data source.

  2. Set the base_prefix in the datalake section to a word that identifies your system, for example wordpress, erp, or forms. This groups the system’s tables together in the data lake and in the default Athena table names.

  3. Record the tables to extract. Pass a comma-separated list with no spaces between the table names:

    ./panorama.py set-tables -t table_one,table_two,table_three

  4. Retrieve each table’s fields from the database:

    ./panorama.py set-tables-fields --all

    You can also target a single data source or table:

    ./panorama.py set-tables-fields --datasource=my_mysql
./panorama.py set-tables-fields --table=table_one

  5. Optionally, set each table’s data lake table name and/or field partitions (see Extract and Load: configuration and Extract and Load: the data pipeline).

From here, the first run workflow is the same as for Open edX: create the raw tables, create the views, and do the initial (forced) upload.

CSV and Excel sources

The agent also supports generic CSV (type: csv) and Excel (type: xls) data sources. See the panorama_csv_settings_example.yaml and the Excel examples in the repository for the exact structure. As with MySQL, the data is uploaded unmodified and the typed views handle casting.

Note

Additional data processing can be performed on the data lake CSV files by external tools. Take care to write the results back in the correct format and to register the data lake tables accordingly, so the raw table and its view stay consistent.

Moodle and other systems

Panorama can connect to Moodle directly or through a Moodle plugin, and to many other systems out of the box or with modest customization. The repository includes a Moodle example settings file. For help wiring up Moodle or another source, contact Aulasneo.

Back to Home