Quality Control of Oceanographic Data
Project description
CoTeDe is an Open Source Python package to quality control (QC) oceanographic data such as temperature and salinity. It was designed to attend individual scientists as well as real-time operations on large data centers. To achieve that, CoTeDe is highly customizable, giving the user full control to compose the desired set of tests including the specific parameters of each test, or choose from a list of preset QC procedures.
I believe that we can do better than we have been doing with more flexible classification techniques, which includes machine learning. My goal is to minimize the burden on manual expert QC improving the consistency, performance, and reliability of the QC procedure for oceanographic data, especially for real-time operations.
CoTeDe is the result from several generations of quality control systems that started in 2006 with real-time QC of TSGs and were later expanded for other platforms including CTDs, XBTs, gliders, and others.
Why CoTeDe
CoTeDe contains several QC procedures that can be easily combined in different ways:
Pre-set standard tests according to the recommendations by GTSPP, EGOOS, XBT, Argo or QARTOD;
Custom set of tests, including user defined thresholds;
Two different fuzzy logic approaches: as proposed by Timms et. al 2011 & Morello et. al. 2014, and using usual defuzification by the bisector;
A novel approach based on Anomaly Detection, described by Castelao 2021 (available since 2014 http://arxiv.org/abs/1503.02714).
Each measuring platform is a different realm with its own procedures, metadata, and meaningful visualization. So CoTeDe focuses on providing a robust framework with the procedures and lets each application, and the user, to decide how to drive the QC. For instance, the pySeabird package is another package that understands CTD and uses CoTeDe as a plugin to QC.
Documentation
A detailed documentation is available at http://cotede.readthedocs.org, while a collection of notebooks with examples is available at http://nbviewer.ipython.org/github/castelao/CoTeDe/tree/master/docs/notebooks/
Citation
If you use CoTeDe, or replicate part of it, in your work/package, please consider including the reference:
Castelão, G. P., (2020). A Framework to Quality Control Oceanographic Data. Journal of Open Source Software, 5(48), 2063, https://doi.org/10.21105/joss.02063
@article{Castelao2020,
doi = {10.21105/joss.02063},
url = {https://doi.org/10.21105/joss.02063},
year = {2020},
publisher = {The Open Journal},
volume = {5},
number = {48},
pages = {2063},
author = {Guilherme P. Castelao},
title = {A Framework to Quality Control Oceanographic Data},
journal = {Journal of Open Source Software}
}
For the Anomaly Detection techinique specifically, which was implemented in CoTeDe, please include the reference:
Castelão, G. P. (2021). A Machine Learning Approach to Quality Control Oceanographic Data. Computers & Geosciences, https://doi.org/10.1016/j.cageo.2021.104803
@article{Castelao2021,
doi = {10.1016/j.cageo.2021.104803},
url = {https://doi.org/10.1016/j.cageo.2021.104803},
year = {2021},
publisher = {Elsevier},
author = {Guilherme P. Castelao},
title = {A Machine Learning Approach to Quality Control Oceanographic Data},
journal = {Computers and Geosciences}
}
If you are concerned about reproducibility, please include the DOI provided by Zenodo on the top of this page, which is associated with a specific release (version).
History
0.20 - Jul, 2018
Removing dependency on pySeabird and pyArgo. Inversion of roles to generalize CoTeDe for other uses. Before CoTeDe would depend on pySeabird, but now CoTeDe is an optional requirement for pySeabird to QC CTD and TSG.
0.19
CARS features and flags
0.17 - Mar, 2016
Implementing fuzzy procedures inside CoTeDe, thus removing dependency on scikit-fuzzy. scikit-fuzzy is broken, hence compromising tests and development of CoTeDe.
0.16 - Mar, 2016
Using external package OceansDB to handle climatologies and bathymetry.
0.15 - Dec, 2015
Moved procedures to handle climatology to external standalone packages.
0.14 - Aug, 2015
Interface for human calibration of anomaly detection
Implemented fuzzy logic criteria
0.13 - July, 2015
Major improvements in the anomaly detection submodule
Partial support to thermosalinographs (TSG)
Working on WOA test to generalize for profiles and tracks
Adding .json to default QC configuration filenames
Moved load_cfg from qc to utils
0.12
Since 0.9 some of the most important changes.
Following OceanSites vocabulary for variable names (PRES, TEMP, PSAL…)
Partial support to Argo profiles
Added density invertion test
Included haversine to avoid dependency on MAUD.
tox and travis support.
0.9 - Dec, 2013
A few people already had access but at this point it went open publicly.
0.7.3
Creating fProfileQC()
0.5.4 - Nov, 2013
Including Tukey53H test
0.5.0
Implemented ProfileQCCollection (later moved to PySeabird).
0.4 - Sep, 2013
Gradient and spike tests with depth conditional thresholds.
CruiseQC (later replaced by ProfileQCCollection).
Use default threshold values for the QC tests.
0.1 - May 24, 2013
Renamed to CoTeDe. Another major refactoring.
QC_ML - 2011
Renamed to QC_ML, a machine learning approach to quality control hydrographic data, the initial prototype of Anomaly Detection approach. I refactored the system I developed to quality control TSG, to evaluate the PIRATA’s CTD stations for INPE. At that point I migrated from my personal Subversion server to Bitbucket, and I lost the detailed history and logs before that.
2008
Modified to parse Seabird CTDs so that the .cnv files could be directly QCed.
2006
A system to automaticaly quality control TSG data on realtime for AOML-NOAA. The data was handled in a PostgreSQL database, and only the traditional tests were applied, i.e. a sequence of binary tests (spike, gradient, valid position …).
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file cotede-0.23.8.tar.gz.
File metadata
- Download URL: cotede-0.23.8.tar.gz
- Upload date:
- Size: 52.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/51.1.2.post20210112 requests-toolbelt/0.9.1 tqdm/4.55.1 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1e1a42068393929b62b9bea3772fcd0395879780ec9fed7cfcb3fbf857b94d2c |
|
| MD5 | ba6deb63819e9cd71566efc5cb1eaccd |
|
| BLAKE2b-256 | 63e189ea03e5ee0755cbd4cb844d43c76b65699476c18d73a18fd32f22c21999 |
File details
Details for the file cotede-0.23.8-py2.py3-none-any.whl.
File metadata
- Download URL: cotede-0.23.8-py2.py3-none-any.whl
- Upload date:
- Size: 70.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/51.1.2.post20210112 requests-toolbelt/0.9.1 tqdm/4.55.1 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e418b6a9423cb788a74877f85ffc0c0cd982505146d4c5ea1c6ca39cb2ad11a0 |
|
| MD5 | 07993f2319529b76fad88c605eec8b64 |
|
| BLAKE2b-256 | 70b8959d90928385ddd2ff5d34b97aa4c50a5920641f8a92e705e5c03a69afd5 |
