/ Announce / Developer / Discussion / Help / Manual / Project / Source / [ACME] [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]

Bienvenue sur le netCDF Operator (NCO) site

Current stable NCO version is 5.2.9 released Friday, 18-Oct-2024 19:52:17 UTC

The NCO toolkit manipulates and analyzes data stored in netCDF-accessible formats, including DAP, HDF4, HDF5, and, most recently, Zarr. NCO exploits the geophysical expressivity and logic of many CF (Climate & Forecast) metadata conventions, the flexible description of physical dimensions translated by UDUnits, the network transparency of OPeNDAP, the storage capabilities (e.g., compression, chunking, groups) of HDF (the Hierarchical Data Format), many powerful mathematical and statistical algorithms of GSL (the GNU Scientific Library), and the lossy and lossless compression codecs provided by the CCR (Community Codec Repository). NCO is fast, powerful, and free.

Recent Releases & Milestones

What is NCO?

The netCDF Operators (NCO) comprise about a dozen standalone, command-line programs that take netCDF, HDF, and/or DAP files as input, then operate (e.g., derive new fields, compute statistics, print, hyperslab, manipulate metadata, regrid) and output the results to screen or files in text, binary, or netCDF formats. NCO aids analysis of gridded and unstructured scientific data. The shell-command style of NCO allows users to manipulate and analyze files interactively, or with expressive scripts that avoid some overhead of higher-level programming environments.

Traditional geoscience data analysis requires users to work with numerous flat (data in one level or namespace) files. In that paradigm instruments or models produce, and then repositories archive and distribute, and then researchers request and analyze, collections of flat files. NCO works well with that paradigm, yet it also embodies the necessary algorithms to transition geoscience data analysis from relying solely on traditional (or “flat”) datasets to allowing newer hierarchical (or “nested”) datasets.

The next logical step is to support and enable combining all datastreams that meet user-specified criteria into a single or small number of files that hold all the science-relevant data organized in hierarchical structures. NCO (and no other software to our knowledge) can do this now. We call the resulting data storage, distribution, and analysis paradigm Group-Oriented Data Analysis and Distribution (GODAD). GODAD lets the scientific question organize the data, not the ad hoc granularity of all relevant datasets. The User Guide illustrates GODAD techniques for climate data analysis:

Note that the “averagers” (ncra and ncwa) are misnamed because they perform many non-linear statistics as well, e.g., total, minimum, RMS. Moreover, ncap2 implements a powerful domain language which handles arbitrarily complex algebra, calculus, and statistics (using GSL). The operators are as general as netCDF itself: there are no restrictions on the contents of input file(s). NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases.

How to Contribute: Volunteer, Endorse, or Donate

NCO has always (since 1995) been Open Source. SourceForge.net started hosting NCO in March, 2000. This facilitated collaboration, code contributions, and support.

In March, 2015, NCO development moved to GitHub.com. We continue to use the SourceForge discussion fora for historical continuity (seventeen years of searchable Q&A). No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

Many users feel unable to volunteer their time. An equally effective contribution in the long-run would be your endorsement, which may make the difference between a declined and an accepted proposal. An endorsement can be a few sentences that describes how NCO benefits your work or research. E-mail your endorsement to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. This information is useful advocating for more NCO support. “What future proposals?” you ask, “Aren't you already funded?” Yes, in 2012 NASA funded us to implement netCDF4 groups and HDF support, and in 2014 NASA funded us to improve regridding. These funds are/were primarily for development and maintainance of specific features. To realize our grander ambition, i.e., to shift geoscience data analysis from a flat- to a hierarchical-paradigm (GODAD), will require a sustained effort and software ecosystem that understands and implement hierarchical dataset concepts. And it's hard to sell a federal agency on the wisdom of investing in a paradigm shift! Other more prosaic tasks that need work are, for example, I/O parallelization (!!!), user-defined and non-atomic types, more CF conventions, cloud-services, JSON back-end, and user-defined ncap2 functions. If you send an endorsement, please include (at least) your Name, Title, and Institutional affiliation.

Lastly, as of June, 2003, if you're more strapped for time than money and want to contribute something back, consider a monetary donation. This may incentivize us to tackle your favorite TODO items.
Inspired by President Obama's plan to bring more transparency to government investment, these homepage donation counters track the influence of your monetary donations on NCO development:
Donations received between 20030624 and 20240105: US$319.77. Thank you, donors!
NCO features “incentivized” by these donations: More emoticons in the documentation :)

NSF Compression Project

The National Science Foundation Grant NSF OAC-2004993 funded our CANDEE Project, “Elements: Advanced Lossless and Lossy Compression Algorithms for netCDF Datasets in Earth and Engineering Sciences (CANDEE)” from 20200701–20230630 as part of the Cyberinfrastructure for Sustained Scientific Innovation (CSSI) program.

UCI's goal is to enable modern lossy and lossless compression in geoscientific and engineering research that employs the netCDF/HDF data format. We first developed, with our partner E. Hartnett (NOAA), the Community Codec Repository (CCR) a robust, extensible software infrastructure for making compression algorithms transparently available for netCDF users. We then developed the Granular BitGroom quantization algorithm, and implemented it, and the BitRound algorithm, and the Zstandard codec, into the CCR and into the netCDF-C library for wider adoption. We are quantifying the efficacy of quantization/compression at reducing dataset size, and thereby storage, power, and associate GHG emissions. We are proposing adoption of these methods in CMIP7.

DOE E3SM Project

The Energy Exascale Earth System Model (E3SM) project, a part of DOE's Earth System Modeling (ESM) program, has since 2015 supported our development of parallel climatology, timeseries splitting, and regridding workflows to help analyze E3SM output. Most NCO development during these years is a direct result of this DOE support.

NASA OSTFL 2020 Project

The National Aeronautics and Space Administration (NASA) Open Source Tools, Frameworks, and Libraries (OSTFL 2020) program funded our project “NASA-Oriented Improvements to and Maintenance and Support of the netCDF Operators (NCO) Toolkit” from 20220901–202400831.

This OSTFL project funds NCO feature development, fostering community through user support, and improving software maintenance and infrastructure. Our pre-proposal survey of NASA users highlighted three priority areas for the first track, feature development: 1) Optimized access to/from/within cloud-storage environments, and 2) Improved interoperability with Python, and 3) Transparent access to more efficient compression algorithms. These features will support the ongoing transition of many NASA scientists to fast, modern, local and cloud-based data analysis. The user-support aspect of this project takes place through the existing support channels (SourceForge, GitHub) as well as our new “office hours”. Users are invited to ask questions and discuss issues with the NCO team on Monday at noon (Pacific Time) on Zoom at https://uci.zoom.us/my/zender. Monday telecons will occasionally not occur due to holidays, travel, and/or schedule conflicts. Ideas for a way to verify meeting status and cancellations are welcome. Feedback is welcome about this exciting new outreach effort.

NCO-Related Publications and Presentations

Release Highlights

Stable releases receive unique tags and their tarballs are created and stored here at GitHub. Identical copies of those tarballs are also stored here on SourceForge for historical continuity. You may retrieve the source of tagged versions directly with, e.g., git clone -b 5.2.9 http://github.com/nco/nco.git nco-5.2.9.

Pre-built Executables

Pre-built binary executables are available for many platforms. Our source tarballs are always up-to-date, and work on our development systems (Fedora, Ubuntu, and MacOS). We also attempt to provide (theoretically) platform-independent sources in the most common Linux package formats (Debian and RPM). Below are one-line installation instructions and links to these and to packages for other platforms created by volunteers. Anyone willing to perform regular regression testing and porting of NCO to other platforms is welcome. We highly recommend the Anaconda method to first-time users, as it works on all platforms and (along with Homebrew) is perhaps the most up-to-date pre-packaged version. Previous versions of these executables are still available by searching the directory index here.

AIX on IBM mainframes

Anaconda (for *NIX, MacOS, and Windows)

Debian and Ubuntu GNU/Linux

Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux

Gentoo GNU/Linux

MacOS Darwin

Microsoft Windows (version 10 and later) via Linux install

Installing a full linux subsystem in Windows 10 is straightforward: Search (e.g., in Google) for "install linux in windows 10". The first link, for example, is here. Follow those instructions. We suggest installing Ubuntu. Once you have, open a shell window and enter sudo apt-get install nco, just as you would do on any Debian or Ubuntu Linux platform. (These instructions are copied from Adrian Tompkins.)

Microsoft Windows (version 9 and earlier) (native build, compiled with Visual Studio 2019, use this if unsure)

These native Windows executables (64bit) are stand-alone, i.e., do not require users to have any additional software. This is a new feature as of 20120615, please send us feedback. To build NCO from source yourself using MSVC with CMake, please see example in nco/cmake/build.bat.

Microsoft Windows (running Cygwin environment, compiled with GNU-toolchain)

In June, 2021 NCO became an officially supported Cygwin package, and can now be installed via the Cygwin setup installer. Thanks to Marco Atzeri, Mark Hadfield, and Pedro Vicente for creating Cygwin ports and tarballs over the years. Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin.

Python Bindings Source and Documentation

Documentation and Users Guide

The NCO Users Guide is available for reading in these formats:

nco.texi is the most up-to-date. Files nco.dvi, nco.ps, and nco.pdf contain all the mathematical formulae (typeset with TeX) missing from the screen-oriented formats. The screen-oriented formats—nco.html, nco.info, nco.txt, and nco.xml—contain all the documentation except the highly mathematical sections. Wenshan Wang of UCI contributed a Quick Reference Card (last updated Tuesday, 30-Nov-2021 22:02:49 UTC) suitable for printing, framing, and/or carrying on your person at all times.

Other documentation:

FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:

Help/Support/Contacts:

If you have support questions or comments please contact us via the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.

ANNOUNCE/ChangeLog/README/TODO

Files containing useful information about the current distribution:

Source Code

The simplest way to acquire the source is to download the compressed tarball:

The best way to acquire the source and occasionally update to the latest features is with Git. The browsable Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Updating NCO source requires some familiarity with development tools, especially Git and Make. You may retrieve any NCO source code distribution you wish from GitHub. To avoid the churn that comes with using the latest snapshot, potential contributors may wish to retrieve a recent tagged (i.e., stable and released) version, This command retrieves the entire NCO repository (< 20 MB) and then checks out NCO version 5.2.9:

git clone https://github.com/nco/nco.git;cd nco;git checkout 5.2.9

However, the most efficient way to actively track and develop NCO is to retrieve the current (“bleeding edge”) NCO snapshot. This command retrieves the latest NCO master branch into a local directory named nco:

git clone git@github.com:/nco/nco.git

, or

git clone https://github.com/nco/nco.git ~/nco

Track changes to the development version using

cd nco;git pull

One difference between running a "tagged" release (e.g., 5.2.9) and the development version is that the tagged release operators will print a valid version number (e.g., 5.2.9) when asked to do so with the -r flag (e.g., ncks -r). Snapshots will include the sub-version and patch-level in the version information (e.g., 5.2.9-alpha06).

Compilation Requirements

Still having trouble building NCO from source? Read these (much older) Build Hints

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

HPCCs unfortunately do not utilize modern package systems like RPMs or .debs, or do so on old OSs with no access to newer RPMs and .debs. Institution-supported executables are usually available with module load nco. These stable releases are often many versions (up to two years) old. Thanks to funding from external grants,

DOE, NCAR, and UCI HPCC users may find more recent pre-built NCO executableses the personal directories shown below. These are usually built from a recent tagged-version of NCO (e.g., 4.9.X-alphaY) not from the “bleeding-edge” of master which is usually untagged. One way to use these pre-built executables is to prepend them to your executable and library search paths, e.g., export PATH="~zender/bin:${PATH}", export LD_LIBRARY_PATH="~zender/lib:${LD_LIBRARY_PATH}"

Known Problems from 2021 (version 4.9.9) Onwards

  • Older Problems 2013 (version 4.2.4) through 2021 (version 4.9.9)
  • Oldest Problems 2003 (version 2.7.3) through 2012 (version 4.2.3)

    • People:

    Current Developers (please contact us via the project forums not via email):

    Alumni Developers:

    Get NCO netCDF Operators at SourceForge.net. Fast, secure and Free Open Source software downloads
    / Announce / Developer / Discussion / Help / Manual / Project / Source / [ANTLR] [CF] [DIWG] [E3SM] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]