/ Announce / Developer / Discussion / Help / Manual / Project / Source / [ACME] [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]


Welcome to the netCDF Operator (NCO) Homepage

Current NCO version is 4.4.7 last updated on Wednesday, 26-Nov-2014 21:13:01 UTC

The NCO toolkit manipulates and analyzes data stored in netCDF-accessible formats, including DAP, HDF4, and HDF5. It exploits the geophysical expressivity of many CF (Climate & Forecast) metadata conventions, the flexible description of physical dimensions translated by UDUnits, the network transparency of OPeNDAP, the storage features (e.g., compression, chunking, groups) of HDF (the Hierarchical Data Format), and many powerful mathematical and statistical algorithms of GSL (the GNU Scientific Library). NCO is fast, powerful, and free.


Recent Releases & Milestones


What is NCO?

The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF, HDF, and/or DAP files as input, then operate (e.g., derive new data, compute statistics, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids analysis of gridded scientific data. The shell-command style of NCO allows users to manipulate and analyze files interactively, or with expressive scripts that avoid some overhead of higher-level programming environments.

Traditional geoscience data analysis requires users to work with numerous flat (data in one level or namespace) files. In that paradigm instruments or models produce, and then repositories archive and distribute, and then researchers request and analyze, collections of flat files. NCO works well with that paradigm, yet it also embodies the necessary algorithms to transition geoscience data analysis from relying solely on traditional (or “flat”) datasets to allowing newer hierarchical (or “nested”) datasets.

The next logical step is to support and enable combining all datastreams that meet user-specified criteria into a single or small number of files that hold all the science-relevant data organized in hierarchical structures. NCO (and no other software to our knowledge) can do this now. We call the resulting data storage, distribution, and analysis paradigm Group-Oriented Data Analysis and Distribution (GODAD). GODAD lets the scientific question organize the data, not the ad hoc granularity of all relevant datasets. The User Guide illustrates GODAD techniques for climate data analysis:

Note that the “averagers” (ncra and ncwa) are misnamed because they perform many non-linear statistics as well, e.g., total, minimum, RMS. Moreover, ncap2 implements a powerful domain language which handles arbitrarily complex algebra, calculus, and statistics (using GSL). The operators are as general as netCDF itself: there are no restrictions on the contents of input file(s). NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases.


How to Contribute: Volunteer, Endorse, or Donate

In March, 2000, NCO became an Open Source project hosted by SourceForge.net. This facilitates collaboration, code contributions, and support. There is a TODO list hundreds of items long! No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

Many users feel unable to volunteer their time. An equally effective contribution in the long-run would be your endorsement, which may make the difference between a declined and an accepted proposal. An endorsement can be a few sentences that describes how NCO benefits your work or research. E-mail your endorsement to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. This information is useful advocating for more NCO support. “What future proposals?” you ask, “Aren't you already funded?” Yes, in 2012 NASA funded us to implement netCDF4 groups and HDF support, and in 2014 NASA funded us to improve SLD handling. These funds are/were primarily for development and maintainance of specific features. To realize our grander ambition, i.e., to shift geoscience data analysis from a flat- to a hierarchical-paradigm (GODAD), will require a sustained effort and software ecosystem that understands and implement hierarchical dataset concepts. And it's hard to sell a federal agency on the wisdom of investing in a paradigm shift! Other more prosaic tasks that need work are, for example, I/O parallelization (!!!), user-defined and non-atomic types, more CF conventions, cloud-services, JSON back-end, and user-defined ncap2 functions. If you send an endorsement, please include (at least) your Name, Title, and Institutional affiliation.

Lastly, as of June, 2003, if you're more strapped for time than money and want to contribute something back, consider a monetary donation. This may incentivize us to tackle your favorite TODO items.
Inspired by President Obama's plan to bring more transparency to government investment, these homepage donation counters track the influence of your monetary donations on NCO development:
Donations received between 20030624 and 20140615: US$144.55. Thank you, donors!
NCO features “incentivized” by these donations: More emoticons in the documentation :)

NASA ACCESS 2013 Project

The National Aeronautics and Space Administration (NASA) Cooperative Agreement NNX14AH55A funds our project, “Easy Access to and Analysis of NASA and Model Swath-like Data” from 20140701–20160630 as part of the Advancing Collaborative Connections for Earth System Science (ACCESS) program. This URL, http://nco.sf.net#prp_axs, points to the most up-to-date information on the ACCESS 2013 project.

This ACCESS project provides resources to implement support in NCO for Swath-like data (SLD), i.e., dataset with non-rectangular and/or time-varying spatial grids in which one or more coordinates are multi-dimensional. It is often challenging and time-consuming to work with SLD, including all NASA Level 2 satellite-retrieved data, non-rectangular subsets of Level 3 data, and model data (e.g., CMIP5) that are increasingly stored on non-rectangular grids. Spatially intelligent software tools is a key component of the Group-Oriented Data Analysis and Distribution (GODAD) paradigm we are developing for geoscience data analysis.

We are currently recruiting a programmer (aka software engineer) or postdoc based at UCI for at least two years, to accomplish our ACCESS objectives. As described in the proposal, this person will be responsible for incorporating geospatial features and parallelism into NCO. See the ads for more details. (PDF, TXT).

NASA ACCESS 2011 Project

The National Aeronautics and Space Administration (NASA) Cooperative Agreement NNX12AF48A funded our project, “Simplifying and accelerating model evaluation by NASA satellite data” from 20120208–20140207 as part of the Advancing Collaborative Connections for Earth System Science (ACCESS) program. We appreciate the proposal reviewers for and the staff of the Earth Science Division (ESD) in the Science Mission Directorate (SMD) Earth Science Data Systems (ESDS) Office. NCO development was completely voluntary and without institutional support from August, 2008–February, 2012. This NASA support dramatically changed the scale and pace of NCO development. This URL, http://nco.sf.net#prp_access, points to the most up-to-date information on the ACCESS 2011 project.

This ACCESS project provided human resources to implement support in NCO for hierarchical group features in netCDF4 files and for NCO to process HDF files. Groups are a powerful HDF feature that many NASA satellite datasets exploit, and that sources of netCDF data (e.g., the CMIP5 archive of climate simulations for IPCC AR5) could better exploit. By supporting groups in a generic fashion, NCO helps make possible more widespread adoption of hierarchical data analysis, aka Group-Oriented Data Analysis and Distribution (GODAD).

We recruited two personnel, a programmer (aka software engineer) and a graduate student (scientific specialist) both based at UCI for at least two years, to accomplish our ACCESS objectives. As described in the proposal, the responsibilities of positions are roughly segregated as follows: The programmer helped re-factor the code-base to support groups and improved the build system so that now we have native Windows builds. The graduate student researcher is analyzing and intercomparing snow cover and snow albedo datasets from NASA MODIS and MISR datasets (in HDF format) and CMIP5 simulations (in netCDF format). Her analysis informs the development of GODAD so that NCO can solve the most important problems real-world researchers encounter in evaluating GCMs against satellite data.


NSF SEI Project

The National Science Foundation Grant NSF IIS-0431203 funded our SEI Project, “SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data” from 20040901–20080831 as part of the Science and Engineering Informatics (SEI) program. We appreciate the proposal reviewers for and staff of the Divisions of Information and Intelligent Systems (IIS) and Shared Cyberinfrastructure (SC) in the Directorate for Computer and Information Science and Engineering (CISE). Until September 2004, NCO development was completely voluntary and without institutional support. This NSF support dramatically changed the scale and pace of NCO development. This URL, http://nco.sf.net#prp_sei, points to the most up-to-date information on the NCO SEI proposal.

The NSF project provided the human and technical resources to analyze, design, and implement novel advanced methods for distributed data analysis and reduction (DDRA) of geophysical data. The efficacy of these powerful methods were demonstrated by analyses of distributed climate prediction and observation datasets. On the human side, we recruited full-time help for specific areas of NCO, alongside our current and future volunteers. We hired a programmer to help design, implement, and release major code changes. We hired a graduate student researcher to pursue innovative Ph.D. research using “grid-aware” distributed memory, shared memory, and client-server software technologies to improve DDRA of climate datasets. Another graduate student helped us design, implement, and release major code changes.

On the institutional and hardware side, this project connected the Earth System Modeling Facility (ESMF) to the California Institute for Telecommunications and Information Technology (Cal-IT2) OptIPuter at the San Diego Supercomputer Center (SDSC). These supercomputers dedicated a TB of storage each to OPeNDAP-served climate simulation datasets for DDRA. After the proof-of-concept NCO DDRA was complete, we attempted a DDRA intercomparison of the internationally distributed CMIP3 global climate change simulations that reside on the Earth System Grid (ESG). Our techniques tremendously accelerated the analysis of different climate prediction scenarios for the same model, and among different climate models (see publications).


Publications and Presentations


Release Highlights

Releases receive unique CVS tags and their tarballs are stored here on SourceForge and here on our server at UCI. You can also retrieve the source of tagged versions directly with CVS.

Get NCO Binary Executables

NCO developers are too short-handed to provide pre-built binary executables for all platforms. Our source tarballs are always up-to-date, and work on our development systems (Fedora, Ubuntu, and Mac OS X). We also attempt to provide (theoretically) platform-independent sources in the most common Linux package formats (Debian and RPM). Below are links to these and to packages for other platforms created by volunteers. Anyone willing to perform regular regression testing and porting of NCO to other platforms is welcome. Previous versions of these binaries are still available by searching the directory index here.

AIX on IBM mainframes

Debian and Ubuntu GNU/Linux

Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux

Gentoo GNU/Linux

Mac OS X/Darwin

Microsoft Windows (native build, compiled with Visual Studio 2010, use this if unsure)

These native Windows executables should be stand-alone, i.e., not require users to have any additional software. This is a new feature as of 20120615, please send us feedback. To build NCO from source yourself using MSVC or Qt, please see the NCO Qt/MSVC build page.

Microsoft Windows (running Cygwin environment, compiled with GNU-toolchain)

Before using, first install curl (in the "Web" category of Cygwin Setup), and set export UDUNITS2_XML_PATH='/usr/local/share/udunits/udunits2.xml' (or wherever udunits2.xml is installed). Thanks to Mark Hadfield and Pedro Vicente for creating Cygwin tarballs. Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin over the years.

Python Bindings Source and Documentation

Documentation and User's Guide

The NCO User's Guide is available for reading in these formats:

nco.texi is the most up-to-date. Files nco.dvi, nco.ps, and nco.pdf are contain all the mathematical formulae (typeset with TeX) missing from the screen-oriented formats. The screen-oriented formats—nco.html, nco.info, nco.txt, and nco.xml—contain all the documentation except the highly mathematical sections.

Other documentation:


FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:


Help/Support/Contacts:

If you have support questions or comments please contact us via the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.


README/ChangeLog/TODO

Files containing useful information about the current distribution:


Get NCO Source Code

The best way to acquire the source is with CVS. The second best way is to download the source as a compressed tarfile:

NCO's browsable CVS Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Retrieving NCO requires some familiarity with GNU development tools, especially CVS and Make. Sourceforge provides generic instructions for accessing their CVS servers. First log into the NCO CVS server:

cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login

Hit return when asked for the CVS password. This login procedure is only required the first time you access the CVS server. Then you may retrieve any NCO distribution you wish. Usually you wish to retrieve a recent tagged (i.e., released) version. This command retrieves and places NCO version 4.4.7 (which is tagged as nco-4_4_7 due to CVS rules) into local directory nco-4.4.7:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-4_4_7 -d nco-4.4.7 nco

This command retrieves the current (“bleeding edge”) development version of NCO into a local directory named nco:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco

Track changes to the development version using

cd nco;cvs update -kk

One difference between running a "tagged" release (e.g., nco-4_4_7) and the development version is that the tagged release operators will print a valid version number (e.g., 4.4.7) when asked to do so with the -r flag (e.g., ncks -r). The development version simply places today's date in place of the version. Once the autotools builds are working more robustly, the confusion over versions should largely disappear.

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

The Doxygen documentation is infrequently (i.e., never since Daniel left) updated.


Compilation Requirements

Still having trouble building NCO from source? Read these Build Hints

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

UC Irvine and NCAR users may find pre-built, almost up-to-date NCO executables in the following locations. Users at NCAR/NWSC should first try the CISL-supported executables with module load nco. The unsatisfied or adventurous may try my personal executables which are built from the “main trunk” of NCO, not a tagged version, and therefore may behave slightly differently.


Known Problems from 2013 (version 4.2.4) Onwards

  • Known Problems through 2012 (version 4.2.3)

  • People:

    Current Developers (please contact us via the project forums not via email):

    Alumni Developers:


    Get NCO netCDF Operators at SourceForge.net. Fast, secure and Free Open Source software downloads
    / Announce / Developer / Discussion / Help / Manual / Project / Source / [ACME] [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]