Welcome to the netCDF Operator (NCO) Homepage

Current NCO version is 4.0.8 last updated on Tuesday, 06-Dec-2011 22:25:17 UTC

News & Milestones What is NCO? Contributing NCO→SDO Project Publications Version Summary Binaries Documentation FAQ README/ChangeLog/TODO Source Code Compiling UCI and NCAR Benchmarks Known Problems People Contact us

The original NCO logo…and (in order of date received) contenders from Tony Freeman (patch), Rich Signell (cogs), Rob Hetland (numeric cog), and Andrea Cimatoribus (spiral). Which do you prefer? The Mandelblob, the patch, a cog, the spiral? Can you improve these or create a better one? Send it in! Keep the size <250×250 pixels.

NCO Milestones

• 2012 Jan 01?: Funding for NASA ACCESS Project begins
• 2012 Jan 2?: NCO 4.0.9
• 2011 Jun 03: Submit NASA ACCESS proposal
• 2011 Apr 28: NCO 4.0.8 (HPSS support, NOFILL workaround, upgrade recommended)
• 2011 Feb 21: NCO 4.0.7 (upgrade recommended)
• 2011 Jan 15: NCO 4.0.6
• 2010 Oct 13: NCO 4.0.5
• 2010 Sep 24: NCO 4.0.4 (chunking everywhere)
• 2010 Sep 02: NCO 4.0.3
• 2010 Jun 27: NCO 4.0.2
• 2010 Apr 05: NCO 4.0.1
• 2010 Jan 05: NCO 4.0.0 (bilinear interpolation, calendar attributes)
• 2009 Jul 15: NCO 3.9.9 (chunking, re-basing, GSL stats/interp)
• 2009 Apr 29: NCO 3.9.8 (GSL RNG/PDF functions)
• 2009 Mar 18: NCO 3.9.7
• 2009 Jan 22: NCO 3.9.6 (GSL special functions) (upgrade highly recommended)
• 2008 Sep 24: ncap2 documentation
• 2008 Aug 31: NSF funding ends
• 2008 Aug 08: NCO Wiki!
• 2008 May 22: SWAMP Talk at CCGRID08 in Lyon, France
• 2008 May 11: NCO 3.9.5
• 2008 Apr 28: NCO overview paper in EMS
• 2008 Mar 04: NCO 3.9.4
• 2007 Dec 10: SWAMP Poster at Fall AGU, San Francisco, California
• 2007 Dec 07: NCO 3.9.3 (SWAMP!)
• 2007 Dec 04: NCO scaling paper in IJHPCA
• 2007 Aug 29: NCO 3.9.2 (_FillValues)
• 2007 Aug 27: SWAMP site spins-off from NCO
• 2007 Jun 29: NCO 3.9.1 (netCDF4 deflation)
• 2007 May 26: NCO 3.9.0 (netCDF4 atomic types)
• 2007 May 02: SWAMP Talk at GPC, Paris, France
• 2007 Apr 20: NCO 3.2.0
• 2007 Mar 29: NCO 3.1.9 x86_64 RPMs
• 2007 Mar 10: NCO 3.1.9 (i386/x86_64 debs and i386 RPMs)
• 2007 Feb 22: SWAMP Talk at OPeNDAP Developer's Workshop, Boulder, Colorado
• 2007 Jan 15: Talk and Extended Abstract on NCO Scaling at AMS IIPS, San Antonio, Texas
• 2007 Jan 09: Poster on NCO Scaling at Fall AGU, San Francisco, California
• 2006 Dec 09: Poster on SWAMP at Fall AGU, San Francisco, California
• 2006 Dec 06: NCO 3.1.8
• 2006 Nov 11: NCO 3.1.7
• 2006 Sep 18: NCO 3.1.6
• 2006 Aug 29: NCO 3.1.5
• 2006 Jun 07: NCO 3.1.4
• 2006 May 30: NCO 3.1.3
• 2006 Apr 21: NCO 3.1.2
• 2006 Jan 30: NCO 3.1.1
• 2005 Dec 02: NCO 3.1.0 (MPI!)
• 2005 Oct 28: NCO 3.0.3
• 2005 Oct 17: NCO parallel benchmarks
• 2005 Aug 22: NCO 3.0.2
• 2005 Jun 10: NCO 3.0.1
• 2005 Mar 24: NCO 3.0.0
• 2004 Sep 07: NCO 2.9.9 (ncpdq packs data quietly)
• 2004 Sep 01: Funding for NSF SEI Project begins
• 2004 Aug 07: NCO 2.9.8 (ncpdq re-orders dimensions)
• 2004 Mar 04: Submit NSF SEI proposal
• Ancient NCO News & Announcements

What is NCO?

The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids manipulation and analysis of gridded scientific data. The shell-command style of NCO allows users to manipulate and analyze files interactively, or with simple scripts that avoid some overhead (and power) of higher level programming environments. The NCO User's Guide illustrates their use with examples of climate data analysis:

• ncap2 netCDF Arithmetic Processor (examples)
• ncatted netCDF ATTribute EDitor (examples)
• ncbo netCDF Binary Operator (includes ncadd, ncsubtract, ncmultiply, ncdivide) (examples)
• ncea netCDF Ensemble Averager (examples)
• ncecat netCDF Ensemble conCATenator (examples)
• ncflint netCDF FiLe INTerpolator (examples)
• ncks netCDF Kitchen Sink (examples)
• ncpdq netCDF Permute Dimensions Quickly, Pack Data Quietly (examples)
• ncra netCDF Record Averager (examples)
• ncrcat netCDF Record conCATenator (examples)
• ncrename netCDF RENAMEer (examples)
• ncwa netCDF Weighted Averager (examples)

Note that the “averagers” (ncea and ncra) are misnamed because they perform many non-linear operations as well, e.g., total, minimum, maximum, RMS. Moreover, ncap2 implements a powerful domain language which handles arbitrarily complex operations. The operators are as general as netCDF itself: there are no restrictions on the contents of the netCDF file(s) used as input. NCO's internal routines are completely dynamic and impose no limit on the number or sizes of dimensions, variables, and files. NCO is designed to be used both interactively and with large batch jobs. The default operator behavior is often sufficient for everyday needs, and there are numerous command line (i.e., run-time) options, for special cases. NCO works well and is used on most modern operating systems.

How to Contribute

In March, 2000, NCO became an Open Source project hosted by SourceForge.net. This facilitates collaboration, code contributions, and support. There is a TODO list about 200 items long! No matter what your programming background there is a task you can help with. From re-organizing the TODO list itself, to improving this cheesy webpage, to documentation, to designing and implementing new features and interfaces, we need your help! The NCO project homepage contains mailing lists, discussion forums, and instructions to make contributing to NCO easy.

As of June, 2003, you may easily donate your hard-earned money to NCO. Send your donation to NCO developers as a reward for good behavior, or as an incentive to tackle more TODO items.

NASA ACCESS Project

The National Aeronautics and Space Administration Cooperative Agreement (CA) NASA 11-ACCESS11-32 funded our ACCESS Project, “Simplifying and accelerating model evaluation by NASA satellite data” from 20120101–20131231 as part of the Advancing Collaborative Connections for Earth System Science (ACCESS) program. We appreciate the proposal reviewers for and staff of the Earth Science Division (ESD) Research and Analysis (R\&A) in the Science Mission Directorate (SMD). NCO development has been completely voluntary and without institutional support since August, 2008. This NASA support dramatically changes the scale and pace of NCO development. This URL, http://nco.sf.net#prp_access, points to the most up-to-date information on the ACCESS proposal.

The ACCESS project provides the human resources to implement support in NCO for hierarchical group features in netCDF4 files and for NCO to process (using wrappers) HDF-EOS files. Groups are a powerful HDF feature that some NASA satellite datasets (e.g., HIRDLS) exploit, and that sources of netCDF data (e.g., the CMIP5 archive of climate simulations for IPCC AR5) could better exploit. By supporting groups in a generic fashion, NCO hopes to remove one of the barriers to more widespread adoption of netCDF4 features.

We plan to recruit two personnel, a programmer (aka software engineer) and a graduate student (scientific specialist) each for at least two years, to accomplish our ACCESS objectives. As described in the proposal, the responsibilities of positions are roughly segregated as follows: The programmer will re-factor the code-base to support groups (50%), improve the NCO build system and user support (25%), and support NASA ESDSWG activities (25%). The graduate student researcher analyze and intercompare snow cover and snow albedo datasets from NASA MODIS and MISR datasets (in HDF-EOS format) and NCAR CESM simulations (in netCDF format). The analysis will inform the development of wrappers for NCO commands to analyze HDF-EOS datasets. Contact us if you are seriously interested in these positions on a full-time, part-time, or pay-per-feature basis.

We continue to encourage you to e-mail NCO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to implement netCDF4 groups and HDF-EOS wrappers in NCO. Our ambition is get one more grant funded to fully support netCDF4 capabilities in NCO. For example, user-defined types, multiple record coordinates, and some more CF conventions. Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.

NSF SEI Project

The National Science Foundation Grant NSF IIS-0431203 funded our SEI Project, “SEI(GEO): Scientific Data Operators Optimized for Efficient Distributed Interactive and Batch Analysis of Tera-Scale Geophysical Data” from 20040901–20080831 as part of the Science and Engineering Informatics (SEI) program. We appreciate the proposal reviewers for and staff of the Divisions of Information and Intelligent Systems (IIS) and Shared Cyberinfrastructure (SC) in the Directorate for Computer and Information Science and Engineering (CISE). Until September 2004, NCO development was completely voluntary and without institutional support. This NSF support dramatically changed the scale and pace of NCO development. This URL, http://nco.sf.net#prp_sei, points to the most up-to-date information on the NCO SEI proposal.

The NSF project provided the human and technical resources to analyze, design, and implement novel advanced methods for distributed data analysis and reduction (DDRA) of geophysical data. The efficacy of these powerful methods were demonstrated by analyses of distributed climate prediction and observation datasets. On the human side, we recruited full-time help for specific areas of NCO, alongside our current and future volunteers. We have a full-time programmer to help design, implement, and release major code changes. We have a graduate student researcher pursuing innovative Ph.D. research in using “grid-aware” distributed memory, shared memory, and client-server software technologies to improve DDRA of climate datasets. Another graduate student helps us design, implement, and release major code changes. Contact us if you are seriously interested in tackling these problems on a full-time, part-time, or pay-per-feature basis.

On the institutional and hardware side, this project connected the Earth System Modeling Facility (ESMF) to the California Institute for Telecommunications and Information Technology (Cal-IT2) OptIPuter at the San Diego Supercomputer Center. These supercomputers dedicated a TB of storage each to OPeNDAP-served climate simulation datasets for DDRA. After the proof-of-concept NCO DDRA was complete, we attempted a DDRA intercomparison of the internationally distributed global climate change simulations that reside on the Earth System Grid. Our techniques tremendously accelerated the analysis of different climate prediction scenarios for the same model, and among different climate models (see publications).

We encourage you to e-mail NCO endorsements to Charlie “my surname is zender” Zender with Subject: “NCO Proposal Endorsement”. An endorsement can be a sentence or a page, describing how NCO benefits your work or research. The information in your endorsement will be used to create supplementary documents to NSF and NASA for future NCO proposal advocacy. “What future proposals?” you ask, “Didn't you just get funded?” Yes, we are funded to optimize and “grid-enable” NCO. There are still many desirable and unexplored methods for distributed data analysis and reduction (DDRA) that would benefit geosciences. For example, have you tried analyzing multiple NASA satellite datasets in HDF5 format? Need I say more? If you send an endorsement, remember to include (at least) your Name, Title, and Institutional affiliation in the body.

Publications and Presentations

• Wang, D. L., C. S. Zender, and S. F. Jenks (2008), Compiling the uncompilable: A case for shell script compilation, Submitted to ACM Trans. Softw. Engin. Method.. BibTeX PDF (© 2008 by ACM)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2009), Efficient Clustered Server-side Data Analysis Workflows using SWAMP, Earth Sci. Inform., 2(3), 141–155, doi:10.1007/s12145-009-0021-z. BibTeX PDF (© 2009 by ESI)
• Wang, D. L., C. S. Zender, and S. F. Jenks: Cluster Workflow Execution of Retargeted Data Analysis Scripts, in Cluster Computing and the Grid, 2008. CCGRID '08. 8th IEEE International Symposium on. Lyons, France, 19 22 May, 2008. IEEE Computer Society, 449–458, doi:10.1109/CCGRID.2008.69. BibTeX PDF (© 2008 by IEEE)
• Zender, C. S. (2008), Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO), Environ. Modell. Softw., 23(10), 1338–1342, doi:10.1016/j.envsoft.2008.03.004. BibTeX PDF (CSZ) PDF (EMS) (© 2008 by Elsevier Ltd.)
• Wang, D. L., C. S. Zender and S. F. Jenks (2007): A System for Scripted Data Analysis at Remote Data Centers. Presented at the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 10–14, 2007. Eos Trans. AGU, 88(52), Fall Meet. Suppl., Abstract IN11B-0469. BibTeX PDF (DLW)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side parallel data reduction and analysis, in Advances in Grid and Pervasive Computing, Second International Conference, GPC 2007, Paris, France, May 2–4, 2007, Proceedings. IEEE Lecture Notes in Computer Science, vol. 4459, edited by C. Cérin and K.-C. Li, pp. 744–750, Springer-Verlag, Berlin/Heidelberg. BibTeX PDF (DLW)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Parallel Data Reduction and Analysis. Presented by D. Wang to the International Conference on Grid and Pervasive Computing (GPC), Paris, France, May 2–4, 2007. PDF
• Zender, C. S., and H. J. Mangalam (2007), Scaling Properties of Common Statistical Operators for Gridded Datasets, Int. J. High Perform. Comput. Appl, 21(4), 485–498, doi:10.1177/1094342007083802. BibTeX PDF (CSZ) PDF (IJHPCA) (© 2007 by SAGE Publications)
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing. Presented by D. Wang to the OPeNDAP Developer's Workshop, Boulder, CO, February 21–23, 2007. PDF
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), Server-side Data Reduction and Analysis with Script Workflow Analysis for Multi-Processing. Presented by D. Wang to the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, San Antonio, TX, January 14–18, 2007. PDF
• Wang, D. L., C. S. Zender, and S. F. Jenks (2007), DAP-enabled Server-side Data Reduction and Analysis, Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.2., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA. BibTeX PDF
• Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO). Presented to the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, January 14–18, San Antonio, TX, January 14–18, 2007. PDF
• Zender, C. S., and D. L. Wang (2007), High performance distributed data reduction and analysis with the netCDF Operators (NCO), Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Paper 3B.4., January 14–18, San Antonio, TX. American Meteorological Society, AMS Press, Boston, MA. BibTeX PDF
• Wang, D. , Zender, C. S., and S. F. Jenks (2006): Server-side netCDF Data Reduction and Analysis. Presented by D. Wang to the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 5–9, 2006. Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0826. PDF
• Zender, C. S., H. J. Mangalam, and D. L. Wang (2006): Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets. Presented to the Fall Meeting of the American Geophysical Union, San Francisco, CA, December 5–9, 2006. Eos Trans. AGU, 87(52), Fall Meet. Suppl., Abstract IN53B-0827. PDF

Version Summary

• NCO 4.1.0: (Future) netCDF4 compound types?; World peace? Stride works with DAP?;
• NCO 4.0.9: (In Progress, features already in beta include the following) Fix over-zealous ncap2 packing; Fix ncea -y ttl; Warn when appending variables of suspicious record length; Warn when NC_MAX_ATTRS exceeded;
• NCO 4.0.8: (Current Stable Release) Support HPSS; Workaround netCDF NOFILL bug (upgrade recommended); Support CF bounds; CF noleap not no_leap; Fix ncatted overwrite-mode; Fix NC_BYTE printing;
• NCO 4.0.7: Fix degenerate hyperslab bug (upgrade recommended);
• NCO 4.0.6: clang-compliance; Fix ncra YYYYMMDD-date bug; Extract all associated coordinates; Tighten ncbo warnings; ncpdq warns when _FillValue outside packing range;
• NCO 4.0.5: Fix ncks hyperslabbing bug; Fix ncra CF "coordinate" bug;
• NCO 4.0.4: Chunking in all operators; preserve deflation; ncks print formatting; Fix ncrename bug; Do not pack coordinate variables;
• NCO 4.0.3: Fix configure for ncap2, netCDF4, UDUnits2, netCDF-DAP; Fix ncbo bug under AIX xlC; Fix ncra bug for some CF-compliant dates.
• NCO 4.0.2: Consistent deflation, chunking with netCDF4_classic; Fix ncap2 RAM vars, 1-D arrays; Fix ncatted appending NULs.
• NCO 4.0.1: Easily create/remove record dimensions with ncks --mk_rec_dmn, --fix_rec_dmn;
• NCO 4.0.0: Bilinear interpolation, GSL fitting; Chunking improvements; Better support CF calendar and coordinate attributes;
• NCO 3.9.9: netCDF4 chunking and NC_STRING support; GSL statistics, interpolation functions; UDUnits2 time-since support; ncrcat re-basing;
• NCO 3.9.8: ncap2 supports sort(), array(); better DAP/libnetcdf and UDUnits2 builds; GSL distribution/RNG functions;
• NCO 3.9.7: Improve GSL-build support; ncap2 threading improvements; ncks --no_dmn_var_nm printing;
• NCO 3.9.6: Fix OpenMP MSA bug (upgrade highly recommended); Memory management to improve threading; ncatted workaround for netCDF4 _FillValue; disable OpenMP with netCDF4/HDF; no math on type NC_CHAR; ncap2 documentation, better missing value propagation, #include files, GSL math functions;
• NCO 3.9.5: Multi-slabbing (MSA) and -X everywhere; ncecat -M; netCDF4-enabled builds on Cygwin; ncap2 builds on Solaris;
• NCO 3.9.4: File consanguinity; ncks: CF auxiliary coords, LZ stats; ncap2: masks, index arrays, print attributes, OpenMP, atan2(), pow();
• NCO 3.9.3: SWAMP release; ncecat -u; wget URLs; GCC 4.2; ncap2 OpenMP;
• NCO 3.9.2: _FillValue supplants missing_value; UDUnits2 support; ncap2 type-conversion support and printing; netCDF4 configure fixes; Change licenses to GPL3 and FDL1.2;
• NCO 3.9.1: Support lossless netCDF4/HDF deflation; Improve Intel compiler support; ncap2 for/while loops; various and sundry build fixes;
• NCO 3.9.0: Support netCDF4 types (ubyte, ushort, uint, int64, and uint64); Control ncks metadata copying; Fix ncbo packed valued bug (upgrade recommended); Improve IRIX 6.5 support;
• NCO 3.2.0: More forgiving exclusion list (-x -v var_nm) handling; Fix rmssdn normalization; Support Mac OS X on Intel; Documentation bugfixes;
• NCO 3.1.9: ncap2 in RPM packages; Quieter output; AIX configure supports ncap2; Update to DAP for netCDF 3.6.2; Fix nc[erw]a for coordinate min/max/ttl; ncecat allows files to differ in record dimension size;
• NCO 3.1.8: Support _FillValue with compile-time switch; Debian package synchronized, includes ncap2
• NCO 3.1.7: ncap2 “double-parsing”, array initialization, supercedes ncap
• NCO 3.1.6: Support stride in all hyperslabbing operators; change more WARNINGs to INFOs
• NCO 3.1.5: New ncap2 array and hyperslab features; change some WARNINGs to INFOs, add Pathscale and update PGI and Intel compiler support
• NCO 3.1.4: Fix ncbo memory problem; report timer results
• NCO 3.1.3: ncap2 automatically propagates attributes to new/derived variables; NCO handles record variables with zero records; support CF coordinates convention; ncwa -b retains degenerate dimensions; build fixes for Cray T3E, Solaris, Cygwin.
• NCO 3.1.2: ncap2 control structures (conditionals, loops, etc.); keep non-processed data packed in output file; binary operators ncbo and ncflint work when only one variable has missing_value attribute; change ncra and ncea to always average coordinate variables
• NCO 3.1.1: NCO builds (again) with C++ compilers; ncap2 development code; server-side extensions
• NCO 3.1.0: MPI operators are bug-free; fix ./configure for MPI and netCDF4; MRV algorithm speeds up ncwa 5—10 times; ncap random number generator rnd_nbr()
• NCO 3.0.3: Eliminate all critical read regions (accelerates all SMP operators); builds with netCDF4 alpha; read NETCDF4_CLASSIC (i.e., HDF) files automatically; write NETCDF4 files on request; benchmark results quantify parallelism.
• NCO 3.0.2: Fewer critical read regions; all known memory leaks plugged; better Intel compiler support; SFTP protocol support; FTP protocol supports .netrc-based passwords; better model grid-property-awareness (e.g., lat_bnds); ncatted ncks, ncrename support metadata header padding with --hdr; ncbo supports heterogeneous variable ordering; ncatted bugfix
• NCO 3.0.1: benchmarks; ncks printing tweaks; memory cleanup; ncap double-parse speedup and exponentiation bugfix; thread operators by default
• NCO 3.0.0: New ncks -P print switch; ncap optimizations; -Z option supports 64-bit offsets (requires netCDF 3.6); NEC SX build improvements; support GCC 3.4.X; nco_put_vara() overloads in libnco_c++
• NCO 2.9.9: ncpdq packs/unpacks entire files; ncbo is threaded
• NCO 2.9.8: ncpdq released; ncwa --msk_sng consolidated mask switch; C99 compiler required
• NCO 2.9.7: Read input file lists from stdin; write nco_input_file_list attributes; add nco_put_att<int,short,long> functions to libnco_c++; large file support (LFS); write nco_openmp_thread_number attribute
• NCO 2.9.6: Support -o fl_out syntax; 64-bit ABI for x86_64; Debian builds libnco_c++
• NCO 2.9.5: ncflint handles missing_value data better, is commutative when weights are equal
• NCO 2.9.4: ncecat works on pure scalar files; change to cvs.sf.net; AIX GCC builds
• NCO 2.9.3: ncra and ncea now work correctly when missing_value equals zero or any intermediate sums
• NCO 2.9.2: ncap S/V, S%V, and S^V operations fixed (S = scalar, V = variable)
• NCO 2.9.1: ncwa works again on packed NC_FLOAT data (broke when?)
• NCO 2.9.0: Complete ncap modulo operator (V%S) and generic exponentiation (V^S), scalar divided by variable (S/V); workaround stack memory problem with GCC extensions; fix make check
• NCO 2.8.8: --enable-[optimize/debug]-custom options for GCC work again (broke in 2.8.6)
• NCO 2.8.7: ncap packing works again (broke in 2.8.4); Cygwin builds work again
• NCO 2.8.6: AIX and IRIX builds work again; custom GCC builds with -Werror
• NCO 2.8.5: Solaris make install works; ncap changes attribute values correctly; Support DODS 3.4+, deprecate 3.3-; add --enable-[optimize/debug]-custom options to ./configure
• NCO 2.8.4: Index-based hyperslabs work again (broke in 2.7.3, upgrade highly recommended); ncap peak memory usage reduced still more; support for AMD Opteron x86_64 architecture
• NCO 2.8.3: Fix big ncap memory leak; fix regressions; remove scary packing warnings
• NCO 2.8.2: ncbo handles distinct missing_values correctly; improve error diagnostics
• NCO 2.8.1: “Variable wildcarding”—extended regular expressions in -v var,... lists—support in all operators
• NCO 2.8.0: New “binary” operator ncbo, obsoletes ncdiff; Debian integration
• NCO 2.7.4: Autotools build support for AIX; fix UDUnits confusion with floating point hyperslab specifications
• NCO 2.7.3: Support UDUnits (and some CF) conventions, e.g., ncks -d wvl "1 picometer","1 furlong" -d time,"2001-03-19 06:00:0.0","2001-03-19 18:00:0.0"
• NCO 2.7.2: Printing (ncks -H) plays well with multi-slabbing
• NCO 2.7.1: (Packaged) Fallback to B. Sittler's getopt() for long option support; store fixed, packed variables correctly with ncap.
• NCO 2.7.0: Support for GNU/POSIX long options; ncks introduces multi-slabbing (= multiple hyperslabs) capability
• NCO 2.6.6: Fix and document ncrename attribute renaming; add type conversion functions to ncap
• NCO 2.6.5: Cray, Sun build improvements
• NCO 2.6.4: Mac OS X, NEC, SGI build improvements
• NCO 2.6.3: Mac OS X builds work seamlessly; documentation fixed
• NCO 2.6.2: Major improvements in autotools support; ncap supports user-defined dimensions
• NCO 2.6.1: Minor fix to unpacking support
• NCO 2.6.0: All arithmetic operators (ncap, ncdiff, ncea, ncflint, ncra) support unpacking
• NCO 2.5.6: Handle missing_value correctly for integer variables in ncra, merge autoconf build
• NCO 2.5.5: Handle files without existing global “History” attribute correctly
• NCO 2.5.4: Add support NEC-SX support; update Cray support; small build changes
• NCO 2.5.3: Fix native builds under MS Windows with Cygwin ncap; hint to ncwa
• NCO 2.5.2: Add math functions to ncap; hint to ncwa
• NCO 2.5.1: ncatted deletes all attributes of att_var when att_nm not specified
• NCO 2.5.0: man(1) pages from Debian distribution
• NCO 2.4.6: Free (speech) sort routines
• NCO 2.4.5: Arithmetic speedups
• NCO 2.4.4: 64-bit AIX builds
• NCO 2.4.3: Handle non-standard “History” attributes
• NCO 2.4.2: ncap supports nesting files with #include
• NCO 2.4.0: ncks supports writing native binary output with -b switch
• NCO 2.3.1: ncap builds on all Unices
• NCO 2.3.0: Library redesign
• NCO 2.2.4: ncap fully supports packing and unpacking variables
• NCO 2.2.3: ncap supports manually unpacking variables stored with scale_factor and add_offset
• NCO 2.2.2: fix rare ncwa crash with weights
• NCO 2.2.1: fix ncap divide and fmodf bugs
• NCO 2.2.0: (RPM available) 20020203: ncap public debut (has division error, please upgrade)
• NCO 2.1.3: ncap implements LHS casting
• NCO 2.1.2: C++ API to netcdf; libnco_c++ open for business
• NCO 2.1.0: ncap full lexer/parser operational
• NCO 2.0.3: small fixes to 2.0.0 (e.g., ncks printing)
• NCO 2.0.2: Add packing/unpacking capabilities based on scale_factor and add_offset convention
• NCO 2.0.1: Add user-configurable multi-threaded capability to arithmetic-intensive operators ncwa, ncea, and ncra
• NCO 2.0.0: (RPM available) 20010507: Switched all internal calls to netCDF 3.x API
• NCO 1.2.2: 20010211: Allow weights with ncwa in min/max/ttl operations; easier building with HDF4. ncrename can now work on specific variables rather than whole file. ncks has -q option for cleaner printing
• NCO 1.2.1: 20000828: Added type-conversion of integers to floating point before arithmetic; fixed broken -v option
• NCO 1.2: 20000730: Added non-linear operations to ncra/ncea and ncwa: min,max,total,rms,rmssdn,sqrt. Added DODS compliance.
• NCO 1.1.48: 20000515: Potentially serious bugfix to ncdiff, recommend upgrade
• NCO 1.1.47: 20000510: Improved ncra and ncrcat to allow lazier specifications of input files

Get NCO Binary Executables

NCO developers are too short-handed to provide pre-built binary executables for all platforms. Our source tarballs are always up-to-date, and work on our development systems (Ubuntu-flavored Debian GNU/Linux for x86_64) and in our high-end scientific computing environments (AIX 5 for Power4/5/6/7). We also attempt to provide (theoretically) platform-independent sources in the most common UNIX packaging formats (Debian and RPM). Below are links to these and to packages created by volunteers who port NCO to other platforms. Volunteers willing to perform regular regression testing and porting of NCO to other platforms are welcome. Previous versions of these binaries are still available by searching the directory index here.

AIX on IBM mainframes

• nco-4.0.8.aix53.tar.gz ( 25M): Executables AIX 5.3-compatible (last updated Tuesday, 26-Apr-2011 23:05:32 UTC)
Newer (beta- or pre-release) packages are sometimes available for AIX users as described here. Thanks to NSF for supporting AIX machines at NCAR over the years.

Debian and Ubuntu GNU/Linux

Debian NCO and Ubuntu NCO homepages. ‘aptitude install nco’ installs the standard NCO for your Debian-compatible OS. NCO packages in the Debian/Ubuntu repositories (i.e., Sid and Natty) generally lag the packages distributed here by 6–12 months. Newer (beta- or pre-release) packages are often available for intrepid Debian/Ubuntu users as described here.
Debian package for most recent NCO release (install with, e.g., ‘dpkg --install nco_4.0.8-1_i386.deb’):
• nco_4.0.8-1_amd64.deb (1.2M): Executables AMD64-compatible (last updated Thursday, 28-Apr-2011 21:38:13 UTC)
• nco_4.0.8-1.dsc (1.2K): Description (last updated Thursday, 28-Apr-2011 21:38:13 UTC)
• nco_4.0.8-1_amd64.changes (1.7K): Changes since last deb package (last updated Thursday, 28-Apr-2011 21:38:12 UTC)
• nco_4.0.8.orig.tar.gz (2.3M): Upstream Source (last updated Thursday, 28-Apr-2011 21:38:14 UTC)
• nco_4.0.8-1.diff.gz (1.7K): Debian patch to upstream source (last updated Thursday, 28-Apr-2011 21:38:12 UTC)
Thanks to Daniel Baumann, Barry deFreese, Francesco Lovergine, Brian Mays, Rorik Peterson, and Matej Vela for their help packaging NCO for Debian over the years.

Fedora, RedHat Enterprise Linux (RHEL), and Community ENTerprise Operating System (CentOS) GNU/Linux

The Fedora NCO RPMs are usually up-to-date so that ‘yum install nco’ might install, e.g.,
nco-3.9.8-1.fc11.x86_64.rpm: Executables for x86_64/Fedora Core 11-compatible environments
If not, try our own most recent self-built NCO RPMs (install with, e.g., ‘yum install nco-3.9.5-1.fc7.i386.rpm’):
nco-3.9.5-1.fc7.i386.rpm (790K): Executables for i386/Fedora Core 7-compatible environments (last updated Monday, 12-May-2008 12:56:53 UTC)
nco-3.9.5-1.fc7.src.rpm (1.5M): Source (last updated Monday, 12-May-2008 12:56:52 UTC)
nco-3.9.5-1.x86_64.rpm (952K): Executables for x86_64/CentOS 5-compatible environments (last updated Monday, 04-Aug-2008 19:31:27 UTC)
Scripts for generating executables for recent (4.0.x, ~2011) versions of NCO for RedHat Enterprise Linux 5 (RHEL5) and CentOS5 platforms are described here.
Older information on building x86_64 RPMs is available here.
Volunteers have updated and maintained fairly up-to-date NCO packages in Fedora since it was added by Ed Hill in about 2004. Thanks to Patrice Dumas, Ed Hill, and Orion Poplawski for packaging NCO RPMs over the years. Thanks to Gavin Burris and Kyle Wilcox for documenting build procedures for RHEL and CentOS.

Gentoo GNU/Linux

Gentoo GNU/Linux homepage for NCO. Portage packages by George Shapavalov and Patrick Kursawe:
• nco-3.9.4: Latest(?) Gentoo package

Mac OS X/Darwin

Use the DMG files if you do not have MacPorts installed. You may need to add /opt/local/bin to your executable path to access the operators.
• nco-4.0.3_x86_10.6.dmg ( 34M): For Mac OS 10.6 (last updated Friday, 17-Sep-2010 22:15:57 UTC). Maintained by Chad Cantwell.
• nco-4.0.7_x86_10.6.dmg ( 35M): For Mac OS 10.6 (last updated Wednesday, 02-Mar-2011 20:17:21 UTC). Maintained by Chad Cantwell.
• Fink packages for NCO: Currently NCO 3.9.5. Maintained by Alexander Hansen.
• MacPorts infrastructure for NCO: Portfile for NCO 3.9.9. Maintained by Takeshi Enomoto.

Microsoft Windows running Cygwin

• nco-4.0.6.win32.cygwin.tar.gz ( 14M): Executables Win32-compatible (last updated Thursday, 10-Feb-2011 03:06:38 UTC).
First install curl (in the "Web" category of Cygwin setup), and point the environment variable UDUNITS2_XML_PATH to the udunits2.xml file, e.g., with export UDUNITS2_XML_PATH='/usr/local/share/udunits/udunits2.xml'. Thanks to Cygnus Solutions and RedHat Inc. for developing and supporting Cygwin over the years.

Documentation and User's Guide

The NCO User's Guide is available for reading in these formats:

• DVI Device Independent (kdvi, xdvi)
• HTML Hypertext (any browser)
• Info GNU Info (M-x Info, emacs)
• PDF Portable Document Format (acroread, evince, kpdf, okular, xpdf)
• Postscript Printing (ghostview, kghostview)
• TeXInfo Documentation Source code (emacs)
• Text Plain text (more)
• XML Extensible Markup Language (firefox)

Wiki:

• As of August, 2008, NCO has a Wiki. The Wiki contains a Cookbook of NCO examples not found in the manual. Volunteers are encourage to add their own “NCO Hacks” to this cookbook!

Other documentation:

• This abbreviation key unlocks the mysteries of the source code abbreviations and acronyms.
• Unidata netCDF for Developers Workshop NCO examples

FAQ: Frequently Asked Questions

These questions show up almost as frequently as my mother. But they are more predictable:

• I still have questions, how do I contact the NCO project? The NCO project has various Q&A and discussion forums described below.
• Where can I find prebuilt NCO executables? Pre-built executables of some versions of NCO for the operating systems described above (Debian-compatible GNU/Linux, Fedora/RedHat GNU/Linux, Gentoo GNU/Linux, and Mac OS X). If you have an account at UCI or SDSC, try the locations in my personal home directories listed below (AIX, IRIX). At NCAR, first try /usr/local/bin (AIX, IRIX). Otherwise, you are on your own.
• Does NCAR support NCO? The NCAR CISL Technical Consulting Group (TCG) supports NCO like other community software packages such as lapack. No other NCAR division has coordinated NCO support. The NCAR CISL-suported executables are in /usr/local/bin. If you notice problems with the NCO installation on SCD machines, or if you would benefit from a more recent release or patch, then inform the relevant system administrators, e.g., NCAR User Support. If you have a comment, suggestion, or bug report, then contact the developers as described below.
• Is there an easy way to keep up with new NCO releases? Subscribe to the nco-announce mailing list. This list is for NCO-related announcements, not for questions. nco-announce is very low volume—one message every few months.

README/ChangeLog/TODO

Files containing useful information about the current NCO distribution:

• README Platforms and software required by NCO
• ChangeLog History of NCO changes
• TODO An unordered list of features and fixes we plan for NCO

Get NCO Source Code

• nco-4.0.8.tar.gz (4.6M compressed tar-file)
  MD5(src/nco-4.0.8.tar.gz)= edd1e5dab719b4bfc2cd07ec840f4f1d
  SHA1(src/nco-4.0.8.tar.gz)= 98f97fc21bab1190053c08bfae77b61b149bfd50

NCO's browsable CVS Repository contains up-to-the-minute sources and is the easiest way to stay synchronized with NCO features. Retrieving NCO requires some familiarity with GNU development tools, especially CVS and Make. Sourceforge provides generic instructions for accessing their CVS servers. First log into the NCO CVS server:

cvs -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco login

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -r nco-4_0_8 -d nco-4.0.8 nco

cvs -z3 -d:pserver:anonymous@nco.cvs.sf.net:/cvsroot/nco co -kk nco

cd nco;cvs update -kk

Developer NCO Source Documentation

Automated source documentation, created by the Doxygen tool is available. Some developers find this documentation helpful, as it can clarify code and data relationships in the code.

• Source documentation for NCO and NetCDF4(alpha13)

Please be aware that the documentation may be slightly inaccurate and infrequently updated. Comments and suggestions are certainly welcomed on the forums.

Compilation Requirements

Best Practices: Building NCO yourself can be painful, so we recommend you first try the pre-built binaries for your system, e.g.,
sudo aptitude install nco # Debian-based systems like Ubuntu
sudo yum-install nco # RPM-based systems like Fedora
If the pre-built executables do not satisfy you, then the recommended first steps to build (i.e., compile, for the most part) NCO from source code are to install ANTLR 2.7.x (not version 3.x!) (required for ncap2), GSL (desirable for ncap2), netCDF (absolutely required), OPeNDAP (enables network transparency), and UDUnits (allows dimensional unit transformations) before building NCO. If possible build these all with the same compiler (and switches). NCO is mostly written in C99, and although you may mix and match compilers, this is often difficult in practice and is not recommended. The exception is ncap2 which is written in C++. ANTLR, OPeNDAP, and NCO must be built with the same C++ compiler to properly resolve the C++ name-mangling. NCO does not yet support newer ANTLR versions because the ANTLR 3.x C++ interface is incomplete. It is possible and often preferable to install most of the pre-requisite and optional software from pre-compiled packages on modern Debian-based systems with
sudo aptitude install antlr libantlr-dev # ANTLR
sudo aptitude install libcurl4-gnutls-dev libexpat1-dev libxml2-dev # DAP-prereqs
sudo aptitude install bison flex gcc g++ # GNU toolchain
sudo aptitude install gsl-bin libgsl0-dev # GSL
sudo aptitude install libnetcdf6 libnetcdf-dev netcdf-bin # netCDF and DAP
sudo aptitude install udunits-bin libudunits2-0 libudunits2-dev # UDUnits
or, on RPM-based systems, with
sudo yum install antlr antlr-c++-devel # ANTLR
sudo yum install curl-devel libxml2-devel # DAP-prereqs
sudo yum install libdap libdap-devel libnc-dap libnc-dap-devel # DAP
sudo yum install bison flex gcc g++ # GNU toolchain
sudo yum install gsl gsl-devel # GSL
sudo yum install netcdf netcdf-devel # netCDF
sudo yum install librx librx-devel # RX
sudo yum install udunits udunits-devel # UDUnits

Hints to Build-from-Source:

Generic Build Hints:
• The two NCO build mechanisms: NCO has two build mechanisms: (1) GNU configure and (2)  a custom Makefile (nco/bld/Makefile) designed for hand-editing. Try the traditional GNU configure method first, i.e., ./configure;make;make check;make install. This method involves no hand-editing of makefiles, but trouble-free builds with it are less common than we would like. The file nco/configure.eg contains a table which describes the success you can expect building NCO with the configure mechanism. If the simple configure recipe above does not work, savvier users may find it straightforward to modify and use the following, more advanced configure recipe:

  # gcc/g++ Sysadmins use/modify this to install/update NCO in /usr/local, based on netCDF4 also in /usr/local (works on givre 20100727):
  export GNU_TRP=`~/nco/autobld/config.guess`
  cd ~/nco;/bin/rm -f *.${GNU_TRP}.foo;make distclean
  CPPFLAGS='-DHAVE_NETCDF4_H -I/usr/local/include' LDFLAGS='-L/usr/local/lib' ./configure --prefix='/usr/local' --enable-ncap2 --enable-netcdf4 > nco.configure.${GNU_TRP}.foo 2>&1
  /bin/cp -f config.log nco.config.log.${GNU_TRP}.foo
  /bin/cp -f libtool nco.libtool.${GNU_TRP}.foo
  make clean;make > nco.make.${GNU_TRP}.foo 2>&1
  make check >> nco.make.${GNU_TRP}.foo 2>&1
  sudo make install >> nco.make.${GNU_TRP}.foo 2>&1
  

  This is the recipe that the developers use to test configure on the development machines. Please contribute patches required to get the configure mechanism working properly for NCO on your platform. The regressions archive contains gory details of successful (and failed) builds for many platforms. Scripts for generating executables for recent (4.0.x, ~2011) versions of NCO for RedHat Enterprise Linux 5 (RHEL5) and CentOS5 platforms are described here.
  All platforms continue to work fine with the nco/bld/Makefile method:

  export MY_BIN_DIR='/home/username/bin' # Executables installed here
  export MY_OBJ_DIR='/home/username/obj' # Object files placed here
  export MY_LIB_DIR='/home/username/lib' # Libraries built here
  mkdir -p ${MY_BIN_DIR} ${MY_OBJ_DIR} ${MY_LIB_DIR}
  cd ~/nco/bld
  make dir
  make all
  make ncap2
  make test
  

  Disadvantages are that nco/bld/Makefile (1) Usually requires editing to fix your platforms paths; (2) Does not support shared libraries; and (3) Requires a degree from Hogwarts to understand.
• Duplicate symbols errors: When building NCO with the netCDF4 library and headers, some compilers emit errors due to duplicate symbol definitions. This arises because of the complexity of the build system. Essentially, NCO has to define some netCDF API prototypes itself to handle the corner case where the netCDF4 distribution is used to build netCDF3 libraries. We have tried to address this issue using autoconf/configure but it is beyond our abilities. The fix is to manually tell NCO that the netCDF4 header file netcdf.h is in fact being used so that NCO should not attempt to define these prototypes. Do this, e.g., with CPPFLAGS='-DHAVE_NETCDF4_H' or USR_TKN='-DHAVE_NETCDF4_H' if you use configure.
• Build Chain Version Incompatibilities: NCO requires GNU make version ≥ 3.78.1 to support target-specific variable values in bld/Makefile. NCO most often fails to build because ncap fails to build. Two of the most common reasons that ncap fails to build are (1) GNU bison is out-of-date (upgrade bison) and (2) --enable-optimize-* is specified to configure (do not specify it).
  
• C99-compliance & old compilers: NCO versions ≥ 2.9.8 require the compiler to support the 1999 ISO C-Standard called C99. NCO takes advantage of two C99 features, restricted pointers and designated initializers. To build NCO versions ≥ 2.9.8 without a C99-compliant compiler, simply undefine the restrict type qualifier before compiling NCO. Do this, e.g., with CPPFLAGS='-Drestrict='. More details on C99 compatibility are in the header file nco/src/nco/nco.h.
  
• Old netCDF libraries: NCO versions ≥ 3.0.0 require netCDF versions ≥ 3.6 to take advantage of the new 64-bit offset file type. The library and include directories (e.g., /usr/lib and /usr/include, respectively) for netCDF version ≥ 3.6 must be on the path searched by the compiler, and must precede the locations of any older netCDF installations. To build NCO versions ≥ 3.0.0 with older netCDF versions < 3.6, simply define the pre-processor token NC_64BIT_OFFSET to 0 before compiling NCO. Do this, e.g., with CPPFLAGS='-DNC_64BIT_OFFSET=0' or USR_TKN='-DNC_64BIT_OFFSET=0' if you use configure or bld/Makefile, respectively. Users attempting to build NCO with OPeNDAP versions ≤ 3.6.x may need this workaround because OPeNDAP did not fully support netCDF version 3.6.x functionality until OPeNDAP versions ≥ 3.7.0. (The indicator for this problem is a build error like "nco_fl_utl.c", line 30.20: 1506-045 (S) Undeclared identifier NC_64BIT_OFFSET.) All modern NCO versions require netCDF versions ≥ 3.5.0 to support nc_type NC_NAT in NCO source code. NCO versions 3.9.0 and greater (May, 2007) require netCDF versions ≥ 3.6.0 to resolve the nc_inq_format() function.
• ncap build hints: ncap is a sophisticated and somewhat fragile program. ncap requires flex to build correctly. AT&T lex will not work because it does not understand EOF rules used in ncap_lex.l. bison version 1.29 (the default version on early Mac OS X) is buggy and does not parse ncap_yacc.y correctly; previous and later bison's work correctly. bison version 1.25 will not build ncap on Solaris. Please upgrade bison to the latest version (e.g., 1.875+) before reporting ncap build problems. If ncap keeps you from building NCO, and you do not need ncap, configure with --disable-ncap (and know that ncwa will fail to build as a result).
• Build Hints with GSL:
  NCO versions 3.9.6— support automatic building and linking for GSL versions 1.4—. Earlier GSL versions are not supported.
  
• Build Hints with DAP:
  NCO versions 3.9.9— support automatic building and linking for DAP supplied from libnetcdf versions 4.1-beta1 and beyond
  NCO versions 3.1.3— support automatic building and linking for OPeNDAP versions 3.6.x and GCC versions 4.0+.
  NCO versions 3.0.1—3.1.2 support automatic building and linking for OPeNDAP versions 3.5.x and GCC versions 3.4+.
  NCO versions 2.8.5—3.0.0 support automatic building and linking for OPeNDAP versions 3.4.x and GCC versions 3.3+.
  NCO versions ?.?.?—2.8.4 support automatic building and linking for DODS versions 3.2.x and GCC versions 3.2+.
  No other version combinations are supported.
• libnco_c++ build hints: libnco_c++ requires a good C++ compiler to build correctly. Few people need libnco_c++, so the workaround is not to build it. This is done by configuring with --disable-nco_cplusplus.

Platform-Specific Build Hints:
• AIX build hints: NCO versions 2.8.7+ do not build with autotools under AIX when --enable-optimize-* is selected. The workaround is to not use --enable-optimize-*. NCO versions 2.9.7+ do not build shared libraries with AIX xlc compilers because of silly AIX library dependencies activated by OpenMP directives. The workaround is to configure with --disable-shared.
• IRIX build hints: The most common problem with building on SGI IRIX is and out-of-date toolchain.
• Mac OS X build hints: Mac OS X users report success building NCO without patches so long as they (1) Install the latest version of bison, (2) Disable shared libraries and (3) Disable extended regular expressions. This is done with ./configure --disable-regex --disable-shared Building NCO with shared libraries on Mac OS X on may be possible. To try, first re-build and re-install the netCDF library using CFLAGS=-fno-common. Apparently netCDF, unlike most codes, does not set this automatically. Then build NCO normally. Mac OS X ships with a broken version of the POSIX cut utility which causes many NCO regression tests to appear to fail. In fact the NCO commands succeed, but the cut output fails. The workaround is not to heed the regression tests, or to install a working cut, e.g., from the GNU coreutils package.
• Solaris build hints: All recent NCO versions build with autotools on Solaris 2.7—2.9. The most common problem with building on Solaris with cc/CC is that bison may be out-of-date. Install an up-to-date bison (e.g., 1.875+). Solaris 2.10, i.e., Solaris10, had the audacity to change the prototype of the ctime_r() function and so does not build NCO 3.0.1. The fix is to upgrade to the current NCO version.
• SX build hints: NEC SX does not understand the S_ISLNK macro, preferring instead the (non-POSIX) S_IFLNK macro. This macro appears in the file nco_fl_utl.c. The patch I wrote for this may be broken. Will someone with NEC SX please send us a working patch?
• Windows build hints: Building on Microsoft Windows is possible with Cygwin. The resulting executables will behave normally on local files, but will not have some (most?) of NCO's network functionality (e.g., scp, OPeNDAP). See the User's Guide for more details.

Using NCO at UCI, NCAR, and other High Performance Computing Centers (HPCCs)

UC Irvine and NCAR users may find pre-built, almost up-to-date NCO executables in the following locations. These executables are unsupported and are almost guaranteed to hail from different NCO versions and thus to behave slightly differently. (They are built from the main trunk, not a tagged version). Users at NCAR should try the supported executables in /usr/local/bin first.

• CGD network (Solaris 2.x): /home/zender/bin/SUNMP
• ESS greenplanet.ps.uci.edu (Linux 2.6.x): /home/zender/bin/LINUX
• ESS dust.ess.uci.edu (Linux 2.6.x): /home/zender/bin/LINUXAMD64
• CISL bluefire.ucar.edu (AIX 5.3): ~zender/bin/AIX
• CISL mirage0.ucar.edu (Linux 2.6.x): ~zender/bin/LINUXAMD64

We have benchmarked NCO in SMP (e.g., AIX) and MPI (e.g., Linux) environments. The full NCO Benchmarks (in PDF form) are available here. Please send us your suggestions for improved configurations. The NCO distribution includes all the custom benchmarking software, and is designed to allow you to contribute your own benchmarks. Contact us if you're interested.

There are two parallel versions of NCO—one uses OpenMP on SMP (including hyperthreaded) machines and the other uses the Message Passing Interface (MPI) on distributed architectures. OpenMP requires a compiler that supports it. Commercial compilers which we have used include those from IBM and the Portland Group. Intel's Linux compiler is free (beer). The MPI operators will run with most MPI implementations, including LAM, MPICH, and MPICH2. LAM installs easily with our own testing environment (latest Ubuntu).

Known Problems

Recent Generic Run-time Problems:
• NOFILL bug: All NCO versions except 4.0.8 may create corrupt netCDF3 files when linked to any netCDF version prior to 4.1.3. The corruption occurs silently (without warning or error messages). The problem has been seen "in the wild" only on filesystems with large block sizes (e.g., Lustre), although it may be more widespread. It is caused by a netCDF bug that NCO triggers by invoking NOFILL mode for faster writes. Hence it is called the NOFILL bug. The bug is hard to trigger, it depends on a rare interaction of filesystem block-size, hyperslab size, and order-of-variable writing. The bug exists in all versions of netCDF through 4.1.2. The recommended temporary workaround is to use NCO version 4.0.8, which disables NOFILL mode (and thus writes files more slowly). NCO 4.0.8 and will workaround the NOFILL bug on all versions of netCDF (i.e., 4.1.2 and earlier). Hence NCO 4.0.8 will always correctly write netCDF3 files. The NOFILL patch included in NCO 4.0.8 will probably be removed in NCO 4.0.9, assuming netCDF 4.1.3, which fixes the underlying bug, has been released and is stable. At that time the netCDF3 write speed of NCO will increase to what it once was. Other temporary workarounds include creating only netCDF4 files (e.g., ncks -4 ...) instead of netCDF3 files. The real, long-term solution to the NOFILL bug is to re-build (any version of) NCO against netCDF version 4.1.3+, which, when released, will contain the necessary netCDF library fix.

Recent Operator-specific Run-time Problems: No known operator-specific problems with recent releases.

Recent Platform-specific Run-time Problems: No known platform-specific problems with recent releases.

Older Generic Run-time Problems:
• Degenerate hyperslabbing bug: Versions ???—4.0.6 could return incorrect hyperslabs when user-specified hyperslabs did not include at least one point. In such cases, instead of returning no data, hyperslabs could return all data. To determine whether your NCO is affected by this bug, run these commands: ncks -O -v lat -d lat,20.,20.001 ~/nco/data/in.nc ~/foo.nc;ncks -H ~/foo.nc If the returned hyperslab contains any data, then your NCO is buggy (because that hyperslab should be empty). This can lead to incorrect answers for hyperslabs that should be empty. Analogous problems would occur with empty auxiliary coordinate bounding boxes. Although most users do not specify empty hyperslabs, we urge all users to upgrade to NCO 4.0.7+ just to be safe.
• Threading problems with MSA: NCO version 3.9.5 has a nasty bug that causes threaded arithmetic operators, e.g., ncea to produce incorrect results under some conditions. The problem may occur whenever OpenMP is enabled and the operators run on a multi-core CPU with more than one thread. These incorrect answers, if generated, are relatively easy to notice. The number of threads used to generate a file is, by default, recorded in the global attribute nco_openmp_thread_number which may be examined with ncks -M foo.nc | grep nco_openmp_thread_number. The only action that will correct a file that you think (or know) contains corrupted data because of this NCO bug is to re-process the file with a non-buggy NCO version. Version 3.9.5 is buggy and should be upgraded ASAP. Be careful with data processed using this NCO version on multi-core CPUs. The (one-line!) patch to fix this bug in 3.9.5 is here.
• Index-based hyperslab problems: NCO versions 2.7.3—2.8.3 have a nasty bug that causes index-based hyperslabs, e.g., -d lat,1, to behave like value-based hyperslabs, e.g., -d lat,1.0 under some conditions. Unfortunately, the incorrect answers generated may be hard to notice! This problem was most often enountered by users trying to assemble monthly averages using the stride feature of ncrcat. One common symptom is that the time-offset of the output file is incorrect. Versions 2.7.3—2.8.3 are buggy and should be upgraded ASAP. Re-do any data-processing that used index-based hyperslabbing with these versions of NCO.

Older Operator-specific Run-time Problems:
• ncra bug handling CF coordinates attributes that contain the name of the record coordinate: Versions 4.0.3—4.0.4 of ncra incorrectly treat the record variable (usually time) as a fixed variable if it is specified in the coordinates attribute of any variable in a file processed with CCM/CCSM/CF metadata conventions. This bug caused core dumps, and even weirder behavior like creating imaginary time slices in the ouput. Upgrade recommended if you work with NCAR CCSM/CESM model output. One workaround that does not require NCO upgrades is to remove the record coordinate name (usually time) from the coordinates attribute of all variables in CF-compliant files before processing the file with ncra.
• ncra bug averaging YYYYMMDD-format date variables in CCSM/CF-compliant files: Versions ???—4.0.5 of ncra contain a bug which produces an incorrect average (usually zero) of the date variable which many CCSM/CF-compliant files use to track model dates in the human-readable YYYYMMDD-format. Averaging YYYYMMDD-format integers is intrinsically difficult, since such dates have calendar assumptions built-in. NCO attempts this in CCSM/CF-compliant files by using the nbdate (beginning date) and time (days since nbdate) variables to find the average date, converting that to YYYYMMDD, and writing that as the average value of date.
• ncks bug hyperslabbinging fixed netCDF4 dimensions: Versions 4.0.3—4.0.4 of ncks contain a bug which triggers a core-dump when hyperslabbing (along a non-record dimension) a netCDF4-format input file into a netCDF4-format output file, e.g., ncks -d 0,1,lat in4.nc out4.nc. Three workarounds that do not require NCO upgrades (or downgrades) are to explicitly specify chunking with, e.g., ncks --cnk_plc=all -d 0,1,lat in4.nc out4.nc, or, to use ncea instead of ncks for hyperslabbing, e.g., ncea -d 0,1,lat in4.nc out4.nc (ncea does a no-op when there is only one input file), or to write to a netCDF3 file, ncks -3 -d 0,1,lat in4.nc out3.nc.
• Core dump with ncks: Printing variables to screen with ncks can trigger a segfault in NCO 3.9.9—4.0.3. Users may upgrade, downgrade, or apply this one-line patch to 3.9.9 sources: Remove this line “*cnk_sz=(size_t)NULL;” —near line 751 of nco/src/nco/nco_netcdf.c— should fix the problem. The problem in later NCO versions is due to a different bug and this patch will not work.
• ncrename erroneous error exit: Versions 4.0.1—4.0.3 of ncrename contain a bug where commands like ncrename -a .old_nm,new_nm in.nc out.nc would, if old_nm did not exist, write the correct file and then exit with an error message although no error had occurred. The files written were fine, and the error message can be safely ignored. This was due to not clearing an extraneous return code.
• ncbo segmentation fault: ncbo versions 4.0.0—4.0.2 incorrectly refreshed internal metadata, leading to segmentation faults and core dumps with some exacting compilers, notably xlC on AIX.
• ncra segmentation fault: ncra versions 4.0.0—4.0.1 mishandled some CF-compliant dates, leading to segmentation faults and core dumps.
• Arithmetic problems with ncap division, modulo, and exponentiation: ncap versions < 3.0.1 incorrectly exponentiate variables to variable powers (V^V). We recommend that all ncap users upgrade.
  ncap versions up to 2.9.1 incorrectly handle division, modulo, and exponentiation operations of the form S/V, S%V, and S^V where first operand (S) is scalar (i.e., either typed directly in the ncap script or converted from an attribute) and the second operand (V) is a full variable (i.e., stored in a file or computed by ncap). Instead of the requested quantity, ncap returned V/S, V%S, and V^S. In other words ncap treated some non-commutative operations as commutative. This is now fixed. The V/V, V%V, V^V, V/S, V%S, V^S, S/S, S%S, and S^S operations were never affected. We recommend that all ncap users upgrade.
• Incorrect ncbo output for packed input: ncbo versions ???—3.2.0 incorrectly write differences of packed input. This only affects packed variables.
• Problems with ncflint and missing_values: The algorithm ncflint used to perform interpolation in versions up to 2.9.4 was not commutative. It returned the weighted valid datum when the other datum was missing_value, or it returned missing_value, depending on the order the input files were specified. As of version 2.9.5, ncflint always returns missing_value when either input datum is missing_value. Possible future implementations are discussed here.
• Problems with ncra and ncea when missing_value = 0.0: The algorithm ncra and ncea used to perform arithmetic in versions up to 2.9.2 breaks if missing_value is 0.0. Why, you ask? Running average (or total, etc.) algorithms must initialize the answer to 0.0. This is done since the sum accumulates in place as ncra and ncea proceeds across records and files. (Normalizing this accumulation by the total number of records is the last step). The old algorithm compared both the current running average and the new record to the missing_value. If either comparison matched, then nothing accumulated for that record. This zero-initialization led to a state where it was impossible to ever recognize valid data. As a result nothing accumulated and the answer was always zero. The record and ensemble averages would also fail (in a non-obvious) way whenever an intermediate sum equalled missing_value. The chances of the latter event ever happening are exceedingly remote. The new algorithm compares only the new record to the missing_value. This fixes both problems and is faster, too.
• Packing problems with ncwa: NCO versions ???—2.9.0 have a bug that causes ncwa to fail (produce garbage answers) when processing packed NC_FLOAT data. Version 2.9.1 fixes this problem. This problem may have been noticed most by OPeNDAP users since many netCDF climate datasets served by OPeNDAP are packed NC_FLOATs. Upgrade to 2.9.1 if you use ncwa on packed data.
• Packing problems with ncap: NCO versions 2.8.4—2.8.6 have a bug that causes the ncap intrinsic packing function pack() to fail. Version 2.8.7 fixes this problem.

Older Platform-specific Run-time Problems:
• Float-valued intrinsic arithmetic functions in ncap on AIX: ncap versions through 4.0.4 have a bug that causes all float-valued intrinsic math functions to fail under AIX. Float-valued math functions are the ISO C99 functions, e.g., cosf(), fabsf(), logf(). The user does not invoke these functions directly— the user always specifies the generic function name, e.g., cos(), abs(), log(). NCO automatically calls the native single precision (i.e., float-valued) math functions when the generic function argument is a native float (e.g., naked constants like 1.0f or variables stored as NC_FLOAT). Double precision arguments cause NCO to invoke the standard (double-valued) form of the generic function, e.g., cos(), fabs(), log().

People:

(Please communicate with us using the project forums rather than contacting us individually)

• Charlie Zender, Professor of Earth System Science (ESS) and former Director, UCI Earth System Modeling Facility (ESMF). Role: Project PI. Contributions: NCO core library, porting, release manager Related Research: 1. Extend empirically verified analytic model for terascale data reduction of gridded datasets to account for cluster- and network- effects. 2. Enable and optimize NCO for intra-file-level parallelism using netCDF4/HDF5 parallel filesystem features. Other Interests: Atmospheric Physics, Climate Change.
• Stephen Jenks, Former Assistant Professor of Electrical Engineering and Computer Science (EECS) and Director, Scalable Parallel and Distributed Systems (SPDS) lab. Roles: Collaborator. Thesis advisor. Related Research: 1. Consulting on SWAMP design and implementation. Other Interests: Cache memory usage
• Dr. Daniel Wang: Earned EECS Ph.D. (2008) with Jenks and Zender at UCI, now at SLAC. Role: Architect distributed data reduction and analysis solutions for geoscience workflows Related Research: 1. Use of compiler techniques and whole program analysis to discover and exploit parallelism in scripted workflows. 2. Resource management in combined computation-data service environments. 3. Design and implement SWAMP to provide automated server-side data analysis capabilities to the geoscience community. Other Interests: Large data, data management, programming languages and practice.
• Henry Butowsky, Henry Butowsky, software engineer. Roles: Scientific programmer Current Research: 1. Efficient complex data analysis with storage-layer constraints. 2. Develop and thread the ncap2 interpreter. Other Interests: Compilers and interpreters.
• Dr. Scott Capps, Earned Earth System Science Ph.D. (2009) with Zender at UCI, now at UCLA. Related Research: 1. Test NCO on real tera-scale geoscience analysis problems and identify necessary improvements in ncap2 and in SWAMP. 2. Role of satellite-observed sub-gridscale wind variability on air-surface exchange. Other Interests: Weather, climate

Thanks also to past NCO contributors.

Contacting the Project:

Contact us through the Project Forums (rather than personal e-mail) so other users can benefit from and contribute to our exchange. Let us know how NCO is working for you—we'd like to hear. Have you read the documentation and browsed the forums to see if your question/comment has been reported before? Please read the Guide's suggestions for productive Help Requests and Bug Reports.

• Where should I ask my questions on how to use NCO? On the Help site.
• Where should I post suggestions/comments on NCO features and usage? On the Discussion site.
• Where are NCO development and bug-squashing discussed? At the Developer site.