Bienvenue sur le netCDF Operator (NCO) site
Current stable NCO version is 5.2.9 released Friday, 18-Oct-2024 19:52:17 UTC
The NCO toolkit manipulates and analyzes data stored in
netCDF-accessible
formats, including
DAP,
HDF4,
HDF5, and, most recently,
Zarr.
NCO exploits the geophysical expressivity and logic of many
CF
(Climate & Forecast) metadata conventions,
the flexible description of physical dimensions translated by
UDUnits,
the network transparency of
OPeNDAP,
the storage capabilities (e.g., compression, chunking, groups) of
HDF (the Hierarchical Data Format),
many powerful mathematical and statistical algorithms of
GSL (the GNU Scientific Library),
and the lossy and lossless compression codecs provided by the
CCR (Community Codec Repository).
NCO is
fast,
powerful,
and
free.
Recent Releases & Milestones
- 2024 Nov ??: 5.3.0 In Progress...
- 2024 Oct 18: 5.2.9 S1D features
- 2024 Sep 02: 5.2.8 ncclimo --rgn_stt
- 2024 Jul 22: 5.2.7 ncclimo/ncremap fixes
- 2024 Jun 20: 5.2.6 Fix 5.2.5 ncremap
- 2024 Jun 19: 5.2.5 MPAS-A, --qnt
- 2024 Apr 15: 5.2.4 CF lossy metadata
- 2024 Apr 03: 5.2.3 ncclimo --rgn_avg
- 2024 Mar 22: 5.2.2 ncchecker
- 2024 Feb 18: 5.2.1 Quick fix to 5.2.0
- 2024 Feb 16: 5.2.0 S1D ELM restart unpacking
- 2023 Nov 07: 5.1.9 ncremap updates
- 2023 Sep 17: 5.1.8 NC_STRING support
- 2023 Jul 26: 5.1.7 ncremap/ncclimo compression
- 2023 May 12: 5.1.6 Coupler regridding
- 2023 Mar 14: 5.1.5 Native EAMXX vertical support
- 2023 Jan 10: 5.1.4 Depth Interpolation
- 2022 Nov 28: 5.1.3 EAMxx fix
- 2022 Nov 18: 5.1.2 EAMxx
- 2022 Oct 28: 5.1.1 NCZarr, stdin
- 2022 Jul 12: 5.1.0 Compression filters
- 2022 May 02: 5.0.7 SGS + Sub-SGS fixes
- 2022 Feb 02: 5.0.6 Small fixes
- 2022 Jan 24: 5.0.5 BitRound
- 2021 Dec 04: 5.0.4 Splitter API revamp
- 2021 Oct 06: 5.0.3 Granular BitRound
- 2021 Sep 24: 5.0.2 MOAB/mbtempest
- 2021 Jul 13: 5.0.1 ncremap MPI support
- 2021 Jun 18: 5.0.0 ncremap masks, fixes
- 2021 May 14: 4.9.9 ncremap masks, fixes
- 2021 Mar 21: 4.9.8 Small features/fixes
- Milestones from 200301–202012 (versions 2.8.4–4.9.7)
- News and Announcements from 199801–200212 (versions 1.1.0–2.8.3) and earlier
What is NCO?
The netCDF Operators (NCO) comprise about a dozen standalone,
command-line programs that take
netCDF,
HDF, and/or
DAP
files as input, then operate (e.g., derive new fields, compute
statistics, print, hyperslab, manipulate metadata, regrid) and output
the results to screen or files in text, binary, or netCDF formats.
NCO aids analysis of gridded and unstructured scientific data.
The shell-command style of NCO allows users to manipulate and analyze
files interactively, or with expressive scripts that avoid some
overhead of higher-level programming environments.
Traditional geoscience data analysis requires users to work with
numerous flat (data in one level or namespace) files.
In that paradigm instruments or models produce, and then repositories
archive and distribute, and then researchers request and analyze,
collections of flat files.
NCO works well with that paradigm, yet it also embodies the necessary
algorithms to transition geoscience data analysis from relying solely
on traditional (or “flat”) datasets to allowing newer
hierarchical (or “nested”) datasets.
The next logical step is to support and enable combining all
datastreams that meet user-specified criteria into a
single or small number of files that hold all the
science-relevant data organized in hierarchical structures.
NCO (and no other software to our knowledge) can do this now.
We call the resulting data storage, distribution, and analysis
paradigm Group-Oriented Data Analysis and Distribution
(GODAD).
GODAD lets the scientific question organize the data, not the
ad hoc granularity of all relevant datasets.
The User Guide illustrates
GODAD
techniques for climate data analysis:
Note that the “averagers”
(ncra and ncwa) are misnamed because they perform
many non-linear statistics as well, e.g., total, minimum, RMS.
Moreover, ncap2 implements a powerful domain language which
handles arbitrarily complex algebra, calculus, and statistics (using GSL).
The operators are as general as
netCDF
itself: there are no restrictions on the contents of input file(s).
NCO's internal routines are completely dynamic and impose no limit on
the number or sizes of dimensions, variables, and files.
NCO is designed to be used both interactively and with large batch jobs.
The default operator behavior is often sufficient for everyday needs,
and there are numerous command line (i.e., run-time) options, for
special cases.
How to Contribute: Volunteer, Endorse, or Donate
NCO has always (since 1995) been Open Source.
SourceForge.net started hosting NCO
in March, 2000.
This facilitated collaboration, code contributions, and support.
In March, 2015, NCO development moved to GitHub.com.
We continue to use the SourceForge discussion fora for historical
continuity (seventeen years of searchable Q&A).
No matter what your programming background there is a task you can help with.
From re-organizing the TODO list itself, to improving this cheesy
webpage, to documentation, to designing and implementing new features
and interfaces, we need your help!
The project homepage
contains mailing lists, discussion forums, and instructions to make
contributing to NCO easy.
Many users feel unable to volunteer their time.
An equally effective contribution in the long-run would be your
endorsement, which may make the difference
between a declined and an accepted proposal.
An endorsement can be a few sentences that describes how NCO benefits your work or research.
E-mail your endorsement to
Charlie “my surname is zender” Zender
with Subject: “NCO Proposal Endorsement”.
This information is useful advocating for more NCO support.
“What future proposals?” you ask, “Aren't you already funded?”
Yes, in 2012 NASA funded us to implement netCDF4 groups and HDF support,
and in 2014 NASA funded us to improve regridding.
These funds are/were primarily for development and maintainance of
specific features.
To realize our grander ambition, i.e., to shift geoscience data
analysis from a flat- to a hierarchical-paradigm
(GODAD),
will require a sustained effort and software ecosystem that
understands and implement hierarchical dataset concepts.
And it's hard to sell a federal agency on the wisdom of investing in a paradigm shift!
Other more prosaic tasks that need work are, for example,
I/O parallelization (!!!), user-defined and non-atomic types, more CF conventions,
cloud-services, JSON back-end, and user-defined ncap2 functions.
If you send an endorsement, please include (at least) your Name, Title, and Institutional affiliation.
Lastly, as of June, 2003, if you're more strapped for time than money
and want to contribute something back, consider a monetary donation.
This may incentivize us to tackle your favorite TODO items.
Inspired by President Obama's plan to bring more transparency to
government investment, these homepage donation counters track the
influence of your monetary donations on NCO development:
Donations received between 20030624 and 20240105: US$319.77. Thank you, donors!
NCO features “incentivized” by these donations: More emoticons in the documentation :)
NSF Compression Project
The National Science Foundation Grant
NSF OAC-2004993
funded our
CANDEE Project,
“Elements: Advanced Lossless and Lossy Compression Algorithms for netCDF Datasets in Earth and Engineering Sciences (CANDEE)”
from 20200701–20230630 as part of the
Cyberinfrastructure for Sustained Scientific Innovation
(CSSI)
program.
UCI's goal is to enable modern lossy and lossless compression in
geoscientific and engineering research that employs the netCDF/HDF
data format.
We first developed, with our partner E. Hartnett (NOAA), the
Community Codec Repository
(CCR)
a robust, extensible software infrastructure for making compression algorithms
transparently available for netCDF users.
We then developed the Granular BitGroom quantization algorithm, and
implemented it, and the BitRound algorithm, and the Zstandard codec,
into the CCR and into the netCDF-C library for wider adoption.
We are quantifying the efficacy of quantization/compression at
reducing dataset size, and thereby storage, power, and associate GHG
emissions.
We are proposing adoption of these methods in CMIP7.
DOE E3SM Project
The Energy Exascale Earth System Model (E3SM)
project,
a part of DOE's Earth System Modeling (ESM)
program,
has since 2015 supported our development of parallel climatology, timeseries
splitting, and regridding workflows to help analyze E3SM output.
Most NCO development during these years is a direct result of this DOE support.
NASA OSTFL 2020 Project
The National Aeronautics and Space Administration (NASA)
Open Source Tools, Frameworks, and Libraries
(OSTFL 2020)
program funded our project “NASA-Oriented Improvements to and Maintenance and Support of the netCDF Operators (NCO) Toolkit”
from 20220901–202400831.
This OSTFL project funds NCO feature development, fostering
community through user support, and improving software maintenance and
infrastructure.
Our pre-proposal survey of NASA users highlighted three priority
areas for the first track, feature development:
1) Optimized access to/from/within cloud-storage environments, and
2) Improved interoperability with Python, and
3) Transparent access to more efficient compression algorithms.
These features will support the ongoing transition of many NASA
scientists to fast, modern, local and cloud-based data analysis.
The user-support aspect of this project takes place through the
existing support channels (SourceForge, GitHub) as well as our new
“office hours”.
Users are invited to ask questions and discuss issues with the NCO
team on Monday at noon (Pacific Time) on Zoom at
https://uci.zoom.us/my/zender.
Monday telecons will occasionally not occur due to holidays, travel,
and/or schedule conflicts.
Ideas for a way to verify meeting status and cancellations are
welcome.
Feedback is welcome about this exciting new outreach effort.
NCO-Related Publications and Presentations
-
Zender, C. S. (2022),
What Geoscientists Want: Short and Sweet Commands with Eco-friendly Data,
Presented to the Computational and Information Systems Lab (CISL) Seminar Series,
National Center for Atmospheric Research,
Boulder, CO, February–16, 2022 (virtual).
PDF
-
Zender, C. S. (2021),
What Geoscientists Want: Short and Sweet Commands with Eco-friendly Data,
Presented to the 104th American Geophysical Union Fall Meeting,
New Orleans, LA, December 13 17, 2021.
Abstract IN32C-01.
MP4,
PDF
-
Hartnett, E. J., C. S. Zender, W. Fisher, D. Heimbigner, H. Lei, B. Curtis, and K. Gerheiser (2021),
Quantization and Next-Generation Zlib Compression for Fully Backward-Compatible, Faster, and More Effective Data Compression in NetCDF Files,
Presented by E. Hartnett to the 104th American Geophysical Union (AGU) Fall Meeting,
New Orleans, LA, December 13 17, 2021.
Abstract IN15A-05.
PDF
-
Zender, C. S. (2021),
Using the self-describing netCDF format to accelerate scientific workflows,
Presented to the US Climate Variability and Predictability Program (CLIVAR) Working Group on Emerging Data Science Tools for Climate Variability and Predictability,
May 3, 2021 (virtual).
MP4,
PDF
-
Hartnett, E. J. and C. S. Zender (2021),
Additional netCDF Compression Options with the Community Codec Repository (CCR),
Presented by E. Hartnett to the 37th AMS Conference on Environmental Information Processing Technologies (EIPT) at the American Meteorological Society Annual Meeting,
Paper 573, January 10–15 (virtual).
PDF
-
Jelenak, A., P. Leonard, and C. S. Zender (2019):
NASA Dataset Interoperability Recommendations for Earth Science,
Presented by A. Jelenak to the Earth Science Information Partners (ESIP) 2019 Winter Meeting,
Bethesda, MD, January 15–17, 2019.
PDF
-
Zender, C. S. (2018):
NCO-JSON: A Flexible, Complete JavaScript Object Notation for netCDF,
Presented to the American Geophysical Union Fall Meeting,
Washington, DC, December 10–14, 2018.
Eos Trans. AGU, 98(54), Fall Meet. Suppl., Abstract IN31B-30.
iPoster PDF
-
Zender, C. S. (2018),
NCO-JSON: JavaScript Object Notation (JSON) for netCDF.
Presented to the Earth Science Information Partners (ESIP) 2018 Summer Meeting,
Tucson, AZ, July 17–20, 2018.
PDF
-
Zender, C. S. and D. Lee (2018),
CF-2 Group Proposal.
Presented to the Earth Science Information Partners (ESIP) 2018 Summer Meeting,
Tucson, AZ, July 17–20, 2018.
PDF
-
Lee, D. and Zender, C. S. (2018),
CF-2 Group Proposal.
Presented by D. Lee to the 2018 netCDF-CF Workshop,
National Centre for Atmospheric Science (NCAS), University of Reading,
Reading, UK, June 19–20, 2018.
PDF
-
Zender, C. S. and J. D. Silver (2017),
Bit-Grooming: Shave Your Bits with Razor-sharp Precision.
Presented to the American Geophysical Union Fall Meeting,
New Orleans, LA, December 11–15, 2017.
Eos Trans. AGU, 97(54), Fall Meet. Suppl., Abstract IN11B-0037.
PDF
-
Zender, C. S. (2017):
CF2-Group: Draft Climate/Forecast Conventions for Hierarchical Data+Metadata.
Presented to the Earth Science Data Systems Working Group (ESDSWG)
Dataset Interoperability Working Group (DIWG).
Online meeting, September 20, 2017.
PDF
-
Zender, C. S. (2017),
New Methods to Generate, Regrid, and Split Climate Data.
Presented to the DOE Accelerated Climate Modeling for Energy (ACME) All-Hands Meeting,
Potomac, MD, June 5–7, 2017.
PDF
-
Silver, J. D. and C. S. Zender (2017), The compression-error trade-off for large gridded data sets, Geosci. Model Dev., 10, 413–423, doi:10.5194/gmd-10-413-2017.
PDF
-
Zender, C. S. and J. D. Silver (2016),
Bit Grooming: Put Climate Data on a Diet with Precision-Preserving Lossy Compression.
Presented to the DOE Accelerated Climate Modeling for Energy (ACME) PI Meeting,
Denver, CO, November 9–11, 2016.
PDF
-
Zender, C. S. (2016), Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+), Geosci. Model Dev., 9, 3199–3211, doi:10.5194/gmd-9-3199-2016.
PDF
-
Zender, C. S. (2016): Regrid Curvilinear, Rectangular, and Unstructured Data (CRUD) with ncremap, a new netCDF Operator. Presented to the Earth Science Data Systems Working Group (ESDSWG) Meeting, Greenbelt, MD, April 6--8, 2016.
PDF
-
Zender, C. S. (2016): Regridding Swath, Curvilinear, Rectangular, and Unstructured Data (SCRUD). Presented to the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC), Goddard Space Flight Center, Greenbelt, MD, April 5, 2016.
PDF
-
Zender, C. S. (2015): Regrid Curvilinear, Rectangular, and Unstructured Data (CRUD) with ncremap, a new netCDF Operator. Presented to the American Geophysical Union Fall Meeting, San Francisco, CA, December 14--18, 2015. Eos Trans. AGU, 95(54), Fall Meet. Suppl., Abstract IN31A-1744.
PDF
-
Zender, C. S. (2015): Optimizing Intrinsic Parallelism to generate climatologies with netCDF Operators (NCO). Presented to the DOE Accelerated Climate Modeling for Energy (ACME) PI Meeting, Albuquerque, NM, November 2--4, 2015.
PDF
-
Zender, C. S., P. Vicente, and W. Wang (2015): Use netCDF Operators (NCO) to Improve Data Interoperability and Usability. Presented to the Earth Science Data Systems Working Group (ESDSWG) Meeting, Greenbelt, MD, March 24--26, 2015.
PDF
-
Zender, C. S., P. Vicente, and W. Wang (2014): Simplifying and accelerating model evaluation by NASA satellite data. Presented to the Earth Science Data Systems Working Group (ESDSWG) Meeting, Greenbelt, MD, March 24--26, 2014.
PDF
-
Zender, C. S. (2014): Use Hierarchical Storage and Analysis to Exploit Intrinsic Parallelism. Presented to the NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC), Goddard Space Flight Center, Greenbelt, MD, March 27, 2014.
PDF
-
Zender, C. S., P. Vicente and W. Wang (2013): Use Hierarchical Storage and Analysis to Exploit Intrinsic Parallelism. Presented at the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 9–13, 2013. Eos Trans. AGU, 93(53), Fall Meet. Suppl., Abstract IN52A-06.
PDF
-
Zender, C. S., P. Vicente and W. Wang (2013): The Future of Model Evaluation. Presented to the Chapman University Symposium on Big Data and Analytics: 44th Symposium on the Interface of Computing Science and Statistics, Chapman University, Orange, CA, April 4–6, 2013. PDF
-
Zender, C. S., P. Vicente and W. Wang (2012): NCO: Simpler and faster model evaluation by NASA satellite data via unified file-level netCDF and HDF-EOS data post-processing tools.. Presented at the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 3–7, 2012. Eos Trans. AGU, 93(53), Fall Meet. Suppl., Abstract IN34A-07.
PDF
-
Zender, C. S., P. Vicente and W. Wang (2012): Simplifying and accelerating model evaluation by NASA satellite data.. Presented to the Earth Science Data Systems Working Group (ESDSWG) Meeting, Annapolis MD, November 13–15, 2012.
PDF
- Presentation and Publications from 2006–2011
Release Highlights
Stable releases receive unique tags and their tarballs are created and stored
here at GitHub.
Identical copies of those tarballs are also stored
here on SourceForge for historical
continuity.
You may retrieve the source of tagged versions directly with, e.g.,
git clone -b 5.2.9 http://github.com/nco/nco.git nco-5.2.9.
- NCO 5.3.1: (Future)
netCDF4 user-defined types support;
ncks --xtn better extensive variable treatment?;
ncks --bsa Byte-swap algoritm;
ncremap small-circle weights;
- NCO 5.3.0: (In Progress, features in-progress or complete include)
ncremap CF Convention for longitude vertices in map-files
- NCO 5.2.9: (Current Stable Release)
ncclimo fix Bash syntax;
ncks --s1d pft order, topounits;
ncremap ECMWF vrt, EAMxx pdq
- NCO 5.2.8:
ncclimo --rgn_stt, --sum_scl;
ncks --chk_map area_b fix;
ncremap fix sub-SGS issue
- NCO 5.2.7:
all Finalize CF quantization metadata;
ncks --chk_map area diagnostics;
ncremap fix SGS issue from 5.2.6
- NCO 5.2.6:
ncremap one-line bugfix to 5.2.5
- NCO 5.2.5:
all output maximum_relative_error for NCO BitRound algorithm;
all --qnt_alg support for NCO algorithms;
ncclimo support --xcl_var in timeseries mode;
ncremap support level coordinate (ERA5) for vertical interpolation;
ncclimo/ncremap support MPAS-A
- NCO 5.2.4:
all CF lossy metadata support for NCO algorithms;
ncclimo/ncremap add debug info
- NCO 5.2.3: (Current Stable Release)
ncclimo --rgn_avg fixes;
ncremap documentation updates
- NCO 5.2.2:
ncchecker debuts;
ncclimo --rgn_avg includes time_bounds in ELM/CLM;
ncremap MBTR --monotonicity
- NCO 5.2.1: (Current Stable Release)
ncclimo/ncremap MPI fix
- NCO 5.2.0:
all CF Convention for lossy compression metadata;
ncclimo automatically generate monthly filenames for timeseries;
ncclimo fix globbing, fix Dec. climatology_bounds in DJF-mode;
ncks --chk_bnd CF/DIWG bounds-variable checking;
ncremap TR --mono*, ncoidw, MWF alg_lst, trfv2
- NCO 5.1.9:
all Windows path separator fix;
ncks --chk_xtn DIWG filename extension check;
ncra/ncrcat interleave fix;
ncremap --alg_typ supports TR bilin, intbilin;
ncremap --mpt_mss option replaces default
- NCO 5.1.8:
all NC_STRING attributes equivalent to NC_CHAR;
ncks --chk_chr CF/DIWG identifier compliance check;
ncks --chk_mss DIWG missing_value check;
ncremap infer MPAS radian grids without units
- NCO 5.1.7:
ncclimo/ncremap --cmp_sng accept human-legible floating-point, unsigned codec parameters;
ncremap CF Convention for longitude vertices in regridded files
- NCO 5.1.6:
all overwrite NCZarr stores yet more safely;
all CF handling on by default;
ncap2 --cmp_sng fix;
ncremap vertical extrapolation boundaries more robust;
ncremap support Coupler history files;
ncremap --col_dmn column-dimension name;
ncremap vertical pressure->pressure timeseries support;
ncremap non-spatio-temporal dimension support in vertical interpolation;
- NCO 5.1.5:
all overwrite NCZarr stores more safely;
ncremap plev_nm, --prs_stt, mss_val fixes;
ncremap EAMxx w/plev;
ncremap --ps_rtn
- NCO 5.1.4:
ncremap SIMD fix on Compy;
ncremap depth-based vertical interpolation
- NCO 5.1.3:
ncremap column-MRV vertical interpolation
- NCO 5.1.2:
ncremap --vrt_in, --vrt_out, --ps_nm;
ncclimo/ncremap -P eamxx
- NCO 5.1.1:
all support NCZarr I/O;
all support stdin for input filenames;
all support Blosc codecs (with netCDF 4.9.1+);
ncz2psx debuts for NCZarr;
ncremap --add_fll uses sgs_frc < rnr_thr
ncremap --vrt_xtr=lnr,zero methods
- NCO 5.1.0:
all --cmp_sng to access modern codecs in netCDF, CCR, or HDF5;
ncremap fix two sub-SGS issues
- NCO 5.0.7:
all BitGroom 52 not 53 explicit bits;
ncclimo/ncremap fix -P bug, warn if SGS not invoked;
ncremap --add_fll use mss_val_cmp_dbl
- NCO 5.0.6:
ncclimo/ncremap small bug-fixes
- NCO 5.0.5:
all quantize add BitRound;
ncatted Prepend mode;
ncclimo climos filenames from stdin/positional arguments
- NCO 5.0.4:
ncclimo --stdin API, --permute API, --wnt_md;
ncremap improve ERWG, MOAB invocation and support;
ncremap --area_dgn, --alg=idw;
ncwa fix averaging with compression
- NCO 5.0.3:
all quantize add DigitRound, default to Granular BitGroom;
ncremap fix MOAB support
- NCO 5.0.2:
all regridder, bitmasks respect NC_FILL_FLOAT/DOUBLE;
ncclimo splitter support for regular expressions;
ncremap support MOAB/mbtempest
- NCO 5.0.1:
ncremap support MPI-enabled weight generators
- NCO 5.0.0:
ncks -X support wrapped longitude;
ncremap fixes for mask_a/b, MPI, msk_nm, stdin;
ncremap --add_fll, --msk_apl, --no_permute
- NCO 4.9.9:
Windows fix DAP https support;
ncks --chk_map account for masks;
ncclimo --mth_[srt/end] in high-frequency splitter, fix tpd, dpf inferral;
ncremap curvilinear inferral, mask fixes, integer rounding, TR allow_no_overlap
- NCO 4.9.8:
ncclimo/ncremap MALI climos;
ncclimo --mth_[srt/end] in monthly splitter;
ncremap fix MPAS grid inferral;
ncremap deconvolve S1D sparse matrix fields (for CLM/ELM)
- Release Highlights from 200001–202012 (versions 1.1.47–4.9.7)
Pre-built Executables
Pre-built binary executables are available for many platforms.
Our source tarballs are always up-to-date, and work on our
development systems (Fedora, Ubuntu, and MacOS).
We also attempt to provide (theoretically) platform-independent sources
in the most common Linux package formats (Debian and RPM).
Below are one-line installation instructions and links to these and to
packages for other platforms created by volunteers.
Anyone willing to perform regular regression testing and porting
of NCO to other platforms is welcome.
We highly recommend the Anaconda method to first-time users, as it
works on all platforms and (along with Homebrew) is perhaps the most
up-to-date pre-packaged version.
Previous versions of these executables are still available by searching
the directory index here.
AIX on IBM mainframes
- nco-4.2.5.aix53.tar.gz (9.5M): Executables AIX 5.3-compatible (last updated Tuesday, 29-Jan-2013 19:02:41 UTC). Maintained by NCO Project.
Newer (beta- or pre-release) packages are sometimes available for AIX users as described here.
Thanks to NSF for supporting AIX machines at NCAR over the years.
Anaconda is a coordinated,
cross-platform Python environment that utilizes the conda package manager.
Anaconda can be easily installed into a user-owned directory.
This bypasses the normal headache of relying on system-administrators to install the latest NCO on shared systems like supercomputers.
Miniconda (rather than the full Anaconda install) suffices for most purposes.
Up-to-date Anaconda-compatible versions of NCO for Linux, MacOS, and Windows are maintained at conda-forge.
Install NCO in your Anaconda framework with one command ‘conda install -c conda-forge nco’.
Or, alternatively, permanently add conda-forge (which teems with goodies besides NCO) to your automatically-searched channels with
‘conda config --add channels conda-forge’, then
install NCO with ‘conda install nco’.
The default NCO installed by conda is generally within a month of the latest release.
- nco-5.2.9 Executables Anaconda-compatible. Maintained by Filipe Fernandes.
Thanks to Rich Signell, Filipe Fernandes, Pedro Vicente, Xylar Asay-Davis and others for developing and maintaining the NCO package for conda.
Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux
- The Fedora NCO RPMs are usually up-to-date so that ‘sudo dnf install nco’ installs a recent version.
RHEL NCO RPMs are documented at the Fedora site.
OpenSUSE keeps NCO RPMs here.
A comprehensive list of pre-built RPMs for many OS’s is here.
Volunteers have updated and maintained fairly up-to-date NCO packages in Fedora since it was added by Ed Hill in about 2004.
Thanks to Patrice Dumas, Ed Hill, Orion Poplawski, and Manfred Schwarb for packaging NCO RPMs over the years.
Thanks to Gavin Burris and Kyle Wilcox for documenting build procedures for RHEL and CentOS.
The most up-to-date executables are probably those in the tarball below. Those unfamiliar with installing executables from tarballs may try the (older) DMG files (you may need to add /opt/local/bin to your executable path to access those operators).
- nco-5.2.9.macos.15.0.tar.gz ([an error occurred while processing this directive]): Executables MacOS 15.0-compatible (last updated [an error occurred while processing this directive]). Maintained by NCO Project.
- Fink packages for NCO: Currently NCO 4.9.5. Maintained by Alexander Hansen.
- Homebrew packages for NCO: Currently NCO 5.1.5. Install with ‘brew install nco’. Maintained by Carlo Cabrera, Ian Lancaster, Alejandro Soto, and others over the years.
- MacPorts infrastructure for NCO: Portfile: Currently NCO 5.1.1. Install with ‘sudo port install nco’. Maintained by Takeshi Enomoto.
Installing a full linux subsystem in Windows 10 is straightforward:
Search (e.g., in Google) for "install linux in windows 10".
The first link, for example, is
here.
Follow those instructions.
We suggest installing Ubuntu.
Once you have, open a shell window and enter sudo apt-get install nco, just as you
would do on any Debian or Ubuntu Linux platform.
(These instructions are copied from Adrian Tompkins.)
These native Windows executables (64bit) are stand-alone, i.e., do not require users to have any additional software.
This is a new feature as of 20120615, please send us feedback.
To build NCO from source yourself using MSVC with CMake, please see example in nco/cmake/build.bat.
- nco-4.9.5.windows.mvs.exe ( 27M): Windows Self-Extracting Installer (last updated Thursday, 24-Sep-2020 15:29:26 UTC). Maintained by Pedro Vicente.
Microsoft Windows (running Cygwin environment, compiled with GNU-toolchain)
In June, 2021 NCO became an officially supported Cygwin package, and can now be installed via the Cygwin setup installer.
Thanks to Marco Atzeri, Mark Hadfield, and Pedro Vicente for creating Cygwin ports and tarballs over the years.
Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin.
The NCO Users Guide is available for reading in these formats:
- DVI Device Independent (kdvi, xdvi)
- HTML Hypertext (any browser)
- Info GNU Info (M-x Info, emacs)
- PDF Portable Document Format (acroread, evince, kpdf, okular, xpdf)
- Postscript Printing (ghostview, kghostview)
- TeXInfo Documentation Source code (emacs)
- Text Plain text (more)
- XML Extensible Markup Language (firefox)
nco.texi is the most up-to-date.
Files nco.dvi, nco.ps, and nco.pdf
contain all the mathematical formulae (typeset with TeX) missing from
the screen-oriented formats.
The screen-oriented formats—nco.html,
nco.info, nco.txt, and nco.xml—contain
all the documentation except the highly mathematical sections.
Wenshan Wang of UCI contributed a Quick Reference Card
(last updated Tuesday, 30-Nov-2021 22:02:49 UTC) suitable for
printing, framing, and/or carrying on your person at all times.
Other documentation:
- This abbreviation key unlocks the mysteries of the source code abbreviations and acronyms.
FAQ: Frequently Asked Questions
These questions show up almost as frequently as my mother.
But they are more predictable:
- I still have questions, how do I contact the NCO project?
The NCO project has various Q&A and discussion forums described
below.
- Where can I find prebuilt NCO executables?
Pre-built executables of some versions of NCO for the operating
systems described above (Debian-compatible
GNU/Linux, Fedora/RedHat GNU/Linux, Gentoo GNU/Linux, and
MacOS).
Otherwise, you may be on your own.
- Does NCAR support NCO?
The NCAR CISL Consulting Service Group (CSG) supports NCO like other
software packages.
The NCAR CISL-suported executables are made available through
“modules” so try module load nco.
If you notice problems with the NCO installation on CISL machines, or
if you would benefit from a more recent release or patch, then ask
cislhelp.
If you have a comment, suggestion, or bug report, then contact the
developers as described below.
- Is there an easy way to keep up with new NCO releases?
Subscribe to the
nco-announce
mailing list.
This list is for NCO-related announcements, not for questions.
nco-announce is very low volume—one message every few months.
Help/Support/Contacts:
If you have support questions or comments please contact us via the
Project Forums (rather than personal e-mail) so other users can
benefit from and contribute to our exchange.
Let us know how NCO is working for you—we'd like to hear.
Have you read the documentation and browsed the
forums to see if your question/comment has been reported before?
Please read the Guide's suggestions for productive
Help Requests and Bug Reports.
- Where should I ask my questions on how to use NCO?
On the Help site.
- Where should I post suggestions/comments on NCO features and usage?
On the Discussion site.
- Where are NCO development and new features discussed?
On the Developer site.
- Where do I file an NCO bug-report?
On the Bug Tickets site.
ANNOUNCE/ChangeLog/README/TODO
Files containing useful information about the current distribution:
- ANNOUNCE Notes on current release
- ChangeLog Change History since 1997 (version 0.9)
- README Platforms and software required
- TODO An unordered list of features and fixes we plan
Source Code
The simplest way to acquire the source is to download the compressed tarball:
-
nco-5.2.9.tar.gz
(6.5M compressed tar-file)
MD5(src/nco-5.2.9.tar.gz)= f4b1f97735e5713c036cd4a3fb8f9217
SHA1(src/nco-5.2.9.tar.gz)= 51089da4e2b7cd2e34b8465f1a118403e800d36e
SHA256(src/nco-5.2.9.tar.gz)= 6245886e2a18a4821b0fb768cf9906de09aeb47c303462c8e85f0d1a4f34956d
The best way to acquire the source and occasionally update to the
latest features is with Git.
The browsable Repository
contains up-to-the-minute sources and is the easiest way to stay
synchronized with NCO features.
Updating NCO source requires some familiarity with development tools,
especially Git and
Make.
You may retrieve any NCO source code distribution you wish from
GitHub.
To avoid the churn that comes with using the latest snapshot,
potential contributors may wish to retrieve a recent tagged (i.e.,
stable and released) version,
This command retrieves the entire NCO repository (< 20 MB) and
then checks out NCO version 5.2.9:
git clone https://github.com/nco/nco.git;cd nco;git checkout 5.2.9
However, the most efficient way to actively track and develop NCO is to
retrieve the current (“bleeding edge”) NCO snapshot.
This command retrieves the latest NCO master branch into a local
directory named nco:
git clone git@github.com:/nco/nco.git
, or
git clone https://github.com/nco/nco.git ~/nco
Track changes to the development version using
cd nco;git pull
One difference between running a "tagged" release
(e.g., 5.2.9) and the development version is that the
tagged release operators will print a valid version number (e.g.,
5.2.9) when asked to do so with the -r flag
(e.g., ncks -r).
Snapshots will include the sub-version and patch-level in the version
information (e.g., 5.2.9-alpha06).
Compilation Requirements
Still having trouble building NCO from source? Read these (much older)
Build Hints
Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)
HPCCs unfortunately do not utilize modern package systems like RPMs or
.debs, or do so on old OSs with no access to newer RPMs and .debs.
Institution-supported executables are usually available with module load nco.
These stable releases are often many versions (up to two years) old.
Thanks to funding from external grants, DOE, NCAR, and UCI HPCC
users may find more recent pre-built NCO executableses the
personal directories shown below.
These are usually built from a recent tagged-version of NCO (e.g.,
4.9.X-alphaY) not from the “bleeding-edge” of master which
is usually untagged.
One way to use these pre-built executables is to prepend them to
your executable and library search paths, e.g.,
export PATH="~zender/bin:${PATH}",
export LD_LIBRARY_PATH="~zender/lib:${LD_LIBRARY_PATH}"
- ANL OLCF Chrysalis chrysalis.lcrc.anl.gov: ~ac.zender/bin_chrysalis
- ANL LCRC Blues blues.lcrc.anl.gov: ~zender/bin_blues
- LLNL acme1.llnl.gov: ~zender1/bin
- NCAR CISL derecho.hpc.ucar.edu: ~zender/bin
- NERSC Perlmutter perlmutter-p1.nersc.gov: ~zender/bin_perlmutter
- ORNL OLCF Andes andes.olcf.ornl.gov: ~zender/bin_andes
- ORNL OLCF Frontier frontier.olcf.ornl.gov: ~zender/bin_frontier
- PNNL Compy compy.pnl.gov: ~zender/bin
Known Problems from 2021 (version 4.9.9) Onwards
Generic NCO Run-time Problems:
No known generic NCO run-time problems with recent releases.
Operator-specific Run-time Problems:
- netCDF4 Renaming bugs:
Unfortunately from 2007–present (April, 2021) the netCDF
library (versions 4.0.0–4.8.0) has bugs and/or limitations
that prevent ncrename (and other netCDF4-based software)
from correctly renaming coordinate variables, dimensions, groups,
and attributes in netCDF4 files.
(To our knowledge the netCDF library calls for renaming always work
well on netCDF3 files so one workaround to netCDF4 bugs is convert to
netCDF3, rename, then convert back).
A summary of renaming limitations associated with particular versions
of netCDF4 is maintained in the online manual
here.
Important updates will also be posted here on the homepage.
There are still known bugs with renaming features as of netCDF library
version 4.8.0 (March, 2021).
Recent Platform-specific Run-time Problems:
No known platform-specific problems with recent releases.
Older Problems 2013 (version 4.2.4) through 2021 (version 4.9.9)
Oldest Problems 2003 (version 2.7.3) through 2012 (version 4.2.3)
People:
Current Developers (please contact us via the project forums not via email):
-
Charlie Zender,
Professor of Earth System Science
(ESS) and of
Computer Science
(CS).
Role: Project PI.
Contributions: Core library, porting, release manager
Related Research:
1. Group-Oriented Data Analysis and Distribution
(GODAD).
2. Extend empirically verified analytic model
(described here)
for terascale data reduction of gridded datasets to account for
cluster- and network-effects.
3. Enable and optimize NCO for intra-file-level parallelism using
netCDF4/HDF5 parallel filesystem features.
Other Interests: Atmospheric Physics, Climate Change
-
Henry Butowsky,
software engineer
Roles: Scientific programmer
Current Research:
1. Efficient complex data analysis with storage-layer constraints.
2. Develop and thread the ncap2 interpreter.
Other Interests: Compilers and interpreters.
-
Pedro Vicente,
software engineer, 201206—201405, then NOAA, continues as volunteer contributor
Alumni Developers: