/ Announce / Developer / Discussion / Help / Manual / Homepage / Project / Source / [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]


Welcome to the netCDF Operator (NCO) Homepage

Current NCO version is 4.4.3 last updated on Thursday, 24-Apr-2014 03:52:49 UTC

NCO manipulates data stored in netCDF-accessible formats, including DAP, HDF4, and HDF5. It also exploits the geophysical expressivity of many CF (Climate & Forecast) metadata conventions, the flexible description of physical dimensions translated by UDUnits, the network transparency of OPeNDAP, the storage features (e.g., compression, chunking, groups) of HDF (the Hierarchical Data Format), and many powerful mathematical and statistical algorithms of GSL (the GNU Scientific Library). NCO is fast, powerful, and free.


NCO Milestones


What is NCO?

The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF or HDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids manipulation and analysis of gridded scientific data. The shell-command style of NCO allows users to manipulate and analyze files interactively, or with simple scripts that avoid some overhead (and power) of higher level programming environments. The NCO User's Guide illustrates their use with examples of climate data analysis:

Note that the “averagers” (ncra and ncwa) are misnamed because they perform many non-linear operations as well, e.g., total, minimum, maximum, RMS. Moreover, ncap2 implements a powerful domain language which handles arbitrarily complex operations. The operators are as general as netCDF itself: there are no restrictions on the contents of input file(s). NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases. NCO works well and is used on most modern operating systems.


How to Contribute

In March, 2000, NCO became an Open Source project hosted by SourceForge.net. This facilitates collaboration, code contributions, and support. There is a TODO list about 200 items long! No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The NCO project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

As of June, 2003, you may easily donate your hard-earned money to NCO. Send your donation to NCO developers as a reward for good behavior, or as an incentive to tackle more TODO items.

Inspired by President Obama's plan to bring more transparency to government investment, these homepage donation counters track the influence of your monetary donations on NCO development:
Donations received between 20030624 and 20120301: US$144.55. Thank you, donors!
NCO features “incentivized” by these donations: More emoticons in the documentation :)

NASA ACCESS Project

The National Aeronautics and Space Administration Cooperative Agreement (CA) NASA 11-ACCESS11-32 funded our ACCESS Project, “Simplifying and accelerating model evaluation by NASA satellite data” from 20120208–20140207 as part of the Advancing Collaborative Connections for Earth System Science (ACCESS) program. We appreciate the proposal reviewers for and the staff of the Earth Science Division (ESD) Research and Analysis (R\&A) in the Science Mission Directorate (SMD). NCO development was completely voluntary and without institutional support from August, 2008–February, 2012. This NASA support dramatically changed the scale and pace of NCO development. This URL, http://nco.sf.net#prp_access, points to the most up-to-date information on the ACCESS proposal.

The ACCESS project provides the human resources to implement support in NCO for hierarchical group features in netCDF4 files and for NCO to process (using wrappers) HDF-EOS files. Groups are a powerful HDF feature that some NASA satellite datasets (e.g., HIRDLS) exploit, and that sources of netCDF data (e.g., the CMIP5 archive of climate simulations for IPCC AR5) could better exploit. By supporting groups in a generic fashion, NCO hopes to remove one of the barriers to more widespread adoption of netCDF4 features.

We recruited (i.e., these positions have been filled) two personnel, a programmer (aka software engineer) and a graduate student (scientific specialist) both based at UCI for at least two years, to accomplish our ACCESS objectives. As described in the proposal, the responsibilities of positions are roughly segregated as follows: The programmer is re-factoring the code-base to support groups, writing wrappers for NCO, an improving the build system (e.g., now we have native Windows builds!). The graduate student researcher is analyzing and intercomparing snow cover and snow albedo datasets from NASA MODIS and MISR datasets (in HDF format) and CMIP5 simulations (in netCDF format). Her analysis informs the development of NCO commands and wrappers for HDF atasets so that we solve the most important problems real-world researchers encounter in evaluating GCMs against NASA data.

We continue to encourage you to e-mail NCO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to implement netCDF4 groups and HDF-EOS wrappers in NCO. Our ambition is get one more grant funded to fully support netCDF4 capabilities in NCO. For example, user-defined types, multiple record coordinates, and some more CF conventions. Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.

Pre-ACCESS CMIP5/MODIS analysis scripts

NSF SEI Project

The National Science Foundation Grant NSF IIS-0431203 funded our SEI Project, “SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data” from 20040901–20080831 as part of the Science and Engineering Informatics (SEI) program. We appreciate the proposal reviewers for and staff of the Divisions of Information and Intelligent Systems (IIS) and Shared Cyberinfrastructure (SC) in the Directorate for Computer and Information Science and Engineering (CISE). Until September 2004, NCO development was completely voluntary and without institutional support. This NSF support dramatically changed the scale and pace of NCO development. This URL, http://nco.sf.net#prp_sei, points to the most up-to-date information on the NCO SEI proposal.

The NSF project provided the human and technical resources to analyze, design, and implement novel advanced methods for distributed data analysis and reduction (DDRA) of geophysical data. The efficacy of these powerful methods were demonstrated by analyses of distributed climate prediction and observation datasets. On the human side, we recruited full-time help for specific areas of NCO, alongside our current and future volunteers. We hired a programmer to help design, implement, and release major code changes. We hired a graduate student researcher to pursue innovative Ph.D. research using “grid-aware” distributed memory, shared memory, and client-server software technologies to improve DDRA of climate datasets. Another graduate student helped us design, implement, and release major code changes.

On the institutional and hardware side, this project connected the Earth System Modeling Facility (ESMF) to the California Institute for Telecommunications and Information Technology (Cal-IT2) OptIPuter at the San Diego Supercomputer Center. These supercomputers dedicated a TB of storage each to OPeNDAP-served climate simulation datasets for DDRA. After the proof-of-concept NCO DDRA was complete, we attempted a DDRA intercomparison of the internationally distributed global climate change simulations that reside on the Earth System Grid. Our techniques tremendously accelerated the analysis of different climate prediction scenarios for the same model, and among different climate models (see publications).

We encourage you to e-mail NCO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to optimize and “grid-enable” NCO. There are still many desirable and unexplored methods for distributed data analysis and reduction (DDRA) that would benefit geosciences. For example, have you tried analyzing multiple NASA satellite datasets in HDF5 format? Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.


Publications and Presentations


Version Summary

Releases receive unique CVS tags and their tarballs are stored here on SourceForge and here on our server at UCI. You can also retrieve the source of tagged versions directly with CVS.

Get NCO Binary Executables

NCO developers are too short-handed to provide pre-built binary executables for all platforms. Our source tarballs are always up-to-date, and work on our development systems (Ubuntu-flavored Debian GNU/Linux for x86_64) and in our high-end scientific computing environments (AIX 5 for Power4/5/6/7). We also attempt to provide (theoretically) platform-independent sources in the most common UNIX packaging formats (Debian and RPM). Below are links to these and to packages created by volunteers who port NCO to other platforms. Volunteers willing to perform regular regression testing and porting of NCO to other platforms are welcome. Previous versions of these binaries are still available by searching the directory index here.

AIX on IBM mainframes

Debian and Ubuntu GNU/Linux

Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux

Gentoo GNU/Linux

Mac OS X/Darwin

Python Bindings Source and Documentation

Microsoft Windows (native build, compiled with Visual Studio 2010, use this if unsure)

These native Windows executables should be stand-alone, i.e., not require users to have any additional software. This is a new feature as of 20120615, please send us feedback. To build NCO from source yourself using MSVC or Qt, please see the NCO Qt/MSVC build page.

Microsoft Windows (running Cygwin environment, compiled with GNU-toolchain)


Documentation and User's Guide

The NCO User's Guide is available for reading in these formats:

nco.texi is the most up-to-date. Files nco.dvi, nco.ps, and nco.pdf are contain all the mathematical formulae (typeset with TeX) missing from the screen-oriented formats. The screen-oriented formats—nco.html, nco.info, nco.txt, and nco.xml—contain all the documentation except the highly mathematical sections.

Other documentation:


FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:


README/ChangeLog/TODO

Files containing useful information about the current NCO distribution:


Get NCO Source Code

The best way to acquire NCO sources is with CVS. The second best way is to download the source as a compressed tarfile:

NCO's browsable CVS Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Retrieving NCO requires some familiarity with GNU development tools, especially CVS and Make. Sourceforge provides generic instructions for accessing their CVS servers. First log into the NCO CVS server:

cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login

Hit return when asked for the CVS password. This login procedure is only required the first time you access the CVS server. Then you may retrieve any NCO distribution you wish. Usually you wish to retrieve a recent tagged (i.e., released) version. This command retrieves and places NCO version 4.4.3 (which is tagged as nco-4_4_3 due to CVS rules) into local directory nco-4.4.3:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-4_4_3 -d nco-4.4.3 nco

This command retrieves the current (“bleeding edge”) development version of NCO into a local directory named nco:

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco

Track changes to the development version using

cd nco;cvs update -kk

One difference between running a "tagged" release (e.g., nco-4_4_3) and the development version is that the tagged release operators will print a valid version number (e.g., 4.4.3) when asked to do so with the -r flag (e.g., ncks -r). The development version simply places today's date in place of the version. Once the autotools builds are working more robustly, the confusion over versions should largely disappear.

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

Please be aware that the documentation may be slightly inaccurate and infrequently updated. Comments and suggestions are certainly welcomed on the forums.


Compilation Requirements

Hints to Build-from-Source:


Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

UC Irvine and NCAR users may find pre-built, almost up-to-date NCO executables in the following locations. Users at NCAR/NWSC should first try the CISL-supported executables with module load nco. The unsatisfied or adventurous may try my personal executables which are built from the “main trunk” of NCO, not a tagged version, and therefore may behave slightly differently.


Benchmarks

We have benchmarked NCO in SMP (e.g., AIX) and MPI (e.g., Linux) environments. The full NCO Benchmarks (in PDF form) are available here. Please send us your suggestions for improved configurations. The NCO distribution includes all the custom benchmarking software, and is designed to allow you to contribute your own benchmarks. Contact us if you're interested.

Current tests include data stored locally and across OPeNDAP servers. We will benchmark NCO with new features (e.g., netCDF4) and hardware (e.g., parallel filesystems) as time permits.

There are two parallel versions of NCO—one uses OpenMP on SMP (including hyperthreaded) machines and the other uses the Message Passing Interface (MPI) on distributed architectures. OpenMP requires a compiler that supports it. Commercial compilers which we have used include those from IBM and the Portland Group. Intel's Linux compiler is free (beer). The MPI operators will run with most MPI implementations, including LAM, MPICH, and MPICH2. LAM installs easily with our own testing environment (latest Ubuntu).


Known Problems

Problems with recent releases:
  • ncra MRO missing value bug: Versions 4.3.6—4.3.9 of ncra could treat missing values incorrectly during double-precision arithmetic. A symptom was that missing values could be replaced by strange numbers like, well, infinity or zero. This mainly affects ncra in MRO (multi-record output) mode, and the symptoms should be noticeable. The workaround is to run the affected versions of ncra using the --flt switch, so that single-precision floating point numbers are not promoted prior to arithmetic. The solution is to upgrade to NCO 4.4.0.
  • ncwa hyperslabbing while averaging bug: Versions 4.3.3—4.3.5 of ncwa could return incorrect answers when user-specified hyperslabs were simultaneously extracted. In such cases, hyperslab limits were not consistently applied. This could produce incorrect answers that look correct. This bug only affected hyperslabbed statistics (those produced by simultaneously invoking -a and -d switches); “global averages” were unaffected. We urge all ncwa users to upgrade to NCO 4.3.6.
  • ncpdq unpacking bug with auxiliary coordinates: Versions 4.3.2–4.3.3 of ncpdq did not correctly store unpacked variables. These versions unpacked (when specified with -U or -P upk) the values, but inadvertently stored their original packing attributes with the unpacked values. This would lead further operators to assume that the values were still packed. Hence consecutive operations could lead to incorrect values. Fixed in version 4.3.4. All ncpdq users are encouraged to upgrade. NB: this bug did not affect the other arithmetic operators which unpack data prior to arithmetic.
  • Problems with older releases:


    People:

    Current Developers (please communicate with us using the project forums rather than contacting us individually):

    Alumni Developers:


    Contacting the Project:

    Contact us through the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.


    Get NCO netCDF Operators at SourceForge.net. Fast, secure and Free Open Source software downloads
    / Announce / Developer / Discussion / Help / Homepage / Manual / Project / Source / [ANTLR] [CF] [DIWG] [GSL] [netCDF] [OPeNDAP] [SWAMP] [UDUnits]