Cases & Voices
Improving the preservation of data generated in different geographic points
Through the Case Study - LiMoNet (Lightning Monitoring Network), learn how MiLab, the platform as a service deployed to support data management, the preservation of digital history and the collaborative work of research groups associated with RedCLARA supports the LiMoNet data preservation challenge.
What do electrical storms and particles from space have to do with each other?
Cosmic rays are high-energy particles from outer space that constantly bombard the Earth. Because they are uncommon, studying them requires particle detectors that, due to their high sensitivity, can generate anomalous results in the presence of atmospheric phenomena such as electrical discharges.
The LiMoNet Lightning Monitoring Network is a prototype for detecting and recording those events (lightning) that can cause detectors to malfunction.
This network was developed in the Halley group of the Universidad Industrial de Santander (UIS, Colombia) within the scope of its link as an associate member of the Pierre Auger Observatory, which detects, among others, large-scale optical events that occur in the upper atmosphere and are used by the LiMoNet to correlate with thunderstorm episodes.
The challenge of data preservation
As it is a project to collect data from different sources and which must then be correlated in a centralized way, LiMoNet required an adequate mechanism to automate and guarantee the preservation of data files generated from the network stations, which are distributed at different geographic points. These files must be kept available, secure and complete for further analysis by the work team.
The solution
To face the challenge of preserving more than 5,800 aggregated files in 187 datasets, the MiLab platform is being used. MiLab allowed not only the preservation of information, but also the execution of the calibration analysis of the stations, an objective achieved through the use of the different services of the platform: dataLab for the preservation of data in a safe and accessible environment for all researchers, G -Lab for collaborative management of codes or scripts that were developed or evolved within the workgroup, chatLab to facilitate communication between the team and compLab to perform computational analysis efficiently.
Impact and results
- A methodology was established for the preservation of datasets in an automated way from the stations to the dataLab service, guaranteeing their preservation and facilitating their use by the research group.
- The group now has a software creation environment to manage the computer codes used in the analyses. This environment, in addition to preserving the content, facilitates the collaborative work of people in different geographic locations.
- A user-friendly computing environment was created to perform calibration analysis of the LiMoNet station network.
About MiLab
MiLab is the platform as a service developed to support data management, the preservation of digital history and the collaborative work of research groups associated to RedCLARA. Currently, MiLab is being used by more than 10 multinational groups in the region thanks to the collaborative management of the National Research and Education Networks and makes use of the CEDIA (Ecuadorian academic network) cloud infrastructure service, which allows the use of the speed that connects research and teaching institutions in Latin America. More information at https://milab.redclara.net/docs.
About the Pierre Auger Observatory
The Pierre Auger Observatory is located on the vast plain known as Pampa Amarilla, in western Argentina. It studies the highest energy particles in the Universe, which hit the Earth from all directions, the so-called cosmic rays. Cosmic rays with low and moderate energies are well known, while those with extremely high energies remain highly mysterious. By detecting and studying these rare particles, the Pierre Auger Observatory is solving the puzzles of their origin and existence. More information at https://www.auger.org/.
About Halley
The Halley Group, from the Universidad Industrial de Santander - part of RENATA, the Colombian academic network - is a Science Center for the dissemination of scientific knowledge that has Astronomy as its central axis. Its work has reach in different settings, from educational institutions at all levels to the community at large. The Halley Group leads the promotion of Astronomy through outreach, dissemination and research projects in astronomy and aerospace sciences in northeastern Colombia. More information at https://halley.uis.edu.co/.
In-depth technical solutions
- A channel for online discussions was created in the chatLab space of the Halley group.
- A group was created in the G-Lab service for the preservation of computer code and network documentation. User roles for the workgroup have also been assigned.
- Added projects in G-Lab to perform calibration analysis, data ingestion into the dataLab service and network documentation. These can be consulted at https://gitmilab.redclara.net/halleyUIS/limonet.
- Datasets were added to the dataLab service using the dataverse API; these can be found at https://dataverse.redclara.net/dataverse/limonet. The script used for the ingestion process can be found at https://gitmilab.redclara.net/halleyUIS/limonet/gestion-de-datos. In addition to ingestion, it is responsible for cataloging the datasets with the metadata generated from the stations.
- The compLab computing service was used for the calibration analysis of the network stations; the notebook used can be found at https://gitmilab.redclara.net/halleyUIS/limonet/analisis-de-datos. In this service, the integration features between MiLab services were used, specifically:
- The dataverse API for accessing datasets from the computing environment.
- The jupyterlab-git plugin was used to access and preserve new versions of analysis code hosted on the G-Lab service.
