Data Management
The following is a description of the ORMGP’s data management practice. All data can be cited using the link below.
Contents
Data Warehousing
The ORMGP Database (MSSQL)
Our main MSSQL Server database holds all the information to support our decision-making tools. The ORMGP data systems are active, in that they are automatically updated at near real-time. From hourly and daily intervals, a series of web-scraping routines have been written to access open data made available from many sources including international, federal, provincial, municipal and conservation authority agencies. The objective is to centralize hydrogeological and hydrological data for the south-central Ontario (Canada) jurisdiction that currently makes-up the ORMGP.
Delft-FEWS
The Delft-FEWS platform offers a streamlined user interface that allows practitioners to visualize time-varying spatially-distributed data from a variety of sources and formats. For instance, spatial/vector/raster time-series datasets common to climatology differ from the point/location datasets contained in the SQL database. We maintain a Delft-FEWS platform to manage these data formats. Our ORMGP-FEWS system periodically ($\leq$ 1 day) scrapes data from federal, provincial, municipal and partner agencies to support a variety of APIs serving near-real-time historical climatologies to our partner users.
More information:
- About the Delft-FEWS product by Deltares.
- More information on ORMGP web data scraping and the sources scraped on a nightly basis.
ORMGP Servers
Databases and REST-APIs
- Our main MSSQL Server database
- VertiGIS Studio/Geocortex — Our principle web mapping server used by partners to access our database and products.
- golang server — multi-functional REST API serving a variety of data products and interpolation tools.
- fews server — home for the ORMGP-FEWS data management system.
- image server — hosts many of the data layers produced by the ORMGP (and discussed above).
- data server — REST-API hosting GET-only requests to the ORMGP central database.
Web Scraping
- ORMGP-cron: a linux server with scheduled web-scraping tasks mostly written in Python.
- ORMGP-FEWS: our operational Delft-FEWS system that scrapes, stores, interpolates and regenerates data served on ORMGP-cron.
Data Sources
Geology
Hydrogeology
- ORMGP monitoring sites
- spotflow/baseflow surveys
- Database groundwater monitoring locations:
Monitoring locations with >35 monitoring dates
Elevation, Drainage and Topology
For groundwater elevations, digital elevation data are used to correct depths of measure to metres above sea level (masl). Therefore, vertical accuracy and datum are tied to the digital elevation (terrain) model (DEM) we used as reference. Currently, correction are taken to the 10m 2006 OMNR Digital Elevation Model - Version 2.0.0 - Provincial Tiled Dataset.
Overland flow drainage and their pathways (topology) are defined using the 30m Provincial Digital Elevation Model - South (CGVD2013).
Sub-watershed characterization and flow topology
Hydro-meteorological station data
- Data source References: climate and streamflow
- Baseflow/Hydrograph Separation
- Spatial Interpolation of point data
- Near-Real-Time Data Acquisition, updated nightly, including:
- Daily data
- Min/max daily temperature
- Atmospheric/Barometric pressure
- Precipitation (rainfall and snowfall)
- Snowmelt (sourced and computed)
- Potential evaporation (computed)
- Streamflow
- Hourly data, updated every 6-hours, including:
- Precipitation (distributed, re-analysed, 6hr accumulations)
- Air temperature
- Atmospheric/Barometric pressure
- Relative humidity
- Windspeed and direction
- Potential evaporation flux
- Daily data
Technologies
A number of open-source and proprietary data analysis tools have been employed in our overall data management system. In contribution to the open-source community, much of our work has also left open on our Github.
Tools (external links)
- Github pages — the hosting of technical information, like what you see now.
- Python
- Beautiful Soup — web scraping
- NumPy, Pandas — data manipulation
- (and many more)
- Go
- FORTRAN/C++
- VB.NET/C#
- R and RStudio — a free, open-source “…software environment for statistical computing and graphics.”
- R-Shiny — online, real-time data analytics
- Leaflet for R — web mapping
- ggplot2 — data visualization
- dygraphs for R — dynamic time-series visualization
- Surfer for geo-statistical analysis such as surface kriging.
Databases (external links)
- MSSQL
- Delft-FEWS