NCO 4.7.5-alpha01 User Guide

This file documents NCO, a collection of utilities to manipulate and analyze netCDF files.

Copyright © 1995–2018 Charlie Zender

This is the first edition of the NCO User Guide,
and is consistent with version 2 of texinfo.tex.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. The license is available online at

The original author of this software, Charlie Zender, wants to improve it with the help of your suggestions, improvements, bug-reports, and patches.
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)
3200 Croul Hall
Department of Earth System Science
University of California, Irvine
Irvine, CA 92697-3100

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

NCO User Guide

Note to readers of the NCO User Guide in HTML format: The NCO User Guide in PDF format (also on SourceForge) contains the complete NCO documentation.
This HTML documentation is equivalent except it refers you to the printed (i.e., DVI, PostScript, and PDF) documentation for description of complex mathematical expressions. Also, images appear only in the PDF document due to SourceForge limitations.

The netCDF Operators, or NCO, are a suite of programs known as operators. The operators facilitate manipulation and analysis of data stored in the self-describing netCDF format, available from ( Each NCO operator (e.g., ncks) takes netCDF input file(s), performs an operation (e.g., averaging, hyperslabbing, or renaming), and outputs a processed netCDF file. Although most users of netCDF data are involved in scientific research, these data formats, and thus NCO, are generic and are equally useful in fields from agriculture to zoology. The NCO User Guide illustrates NCO use with examples from the field of climate modeling and analysis. The NCO homepage is, and the source code is maintained at

This documentation is for NCO version 4.7.5-alpha01. It was last updated 14 April 2018. Corrections, additions, and rewrites of this documentation are gratefully welcome.

Charlie Zender

Next: , Previous: , Up: Top   [Contents][Index]


NCO is the result of software needs that arose while I worked on projects funded by NCAR, NASA, and ARM. Thinking they might prove useful as tools or templates to others, it is my pleasure to provide them freely to the scientific community. Many users (most of whom I have never met) have encouraged the development of NCO. Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo da Silva, John Sheldon, and William Weibel for stimulating suggestions and correspondence. Your encouragment motivated me to complete the NCO User Guide. So if you like NCO, send me a note! I should mention that NCO is not connected to or officially endorsed by Unidata, ACD, ASP, CGD, or Nike.

Charlie Zender
May 1997
Boulder, Colorado

Major feature improvements entitle me to write another Foreword. In the last five years a lot of work has been done to refine NCO. NCO is now an open source project and appears to be much healthier for it. The list of illustrious institutions that do not endorse NCO continues to grow, and now includes UCI.

Charlie Zender
October 2000
Irvine, California

The most remarkable advances in NCO capabilities in the last few years are due to contributions from the Open Source community. Especially noteworthy are the contributions of Henry Butowsky and Rorik Peterson.

Charlie Zender
January 2003
Irvine, California

NCO was generously supported from 2004–2008 by US National Science Foundation (NSF) grant IIS-0431203. This support allowed me to maintain and extend core NCO code, and others to advance NCO in new directions: Gayathri Venkitachalam helped implement MPI; Harry Mangalam improved regression testing and benchmarking; Daniel Wang developed the server-side capability, SWAMP; and Henry Butowsky, a long-time contributor, developed ncap2. This support also led NCO to debut in professional journals and meetings. The personal and professional contacts made during this evolution have been immensely rewarding.

Charlie Zender
March 2008
Grenoble, France

The end of the NSF SEI grant in August, 2008 curtailed NCO development. Fortunately we could justify supporting Henry Butowsky on other research grants until May, 2010 while he developed the key ncap2 features used in our climate research. And recently the NASA ACCESS program commenced funding us to support netCDF4 group functionality. Thus NCO will grow and evade bit-rot for the foreseeable future.

I continue to receive with gratitude the thanks of NCO users at nearly every scientific meeting I attend. People introduce themselves, shake my hand and extol NCO, often effusively, while I grin in stupid embarassment. These exchanges lighten me like anti-gravity. Sometimes I daydream how many hours NCO has turned from grunt work to productive research for researchers world-wide, or from research into early happy-hours. It’s a cool feeling.

Charlie Zender
April, 2012
Irvine, California

The NASA ACCESS 2011 program generously supported (Cooperative Agreement NNX12AF48A) NCO from 2012–2014. This allowed us to produce the first iteration of a Group-oriented Data Analysis and Distribution (GODAD) software ecosystem. Shifting more geoscience data analysis to GODAD is a long-term plan. Then the NASA ACCESS 2013 program agreed to support (Cooperative Agreement NNX14AH55A) NCO from 2014–2016. This support permits us to implement support for Swath-like Data (SLD). Most recently, the DOE has funded me to implement NCO re-gridding and parallelization in support of their ACME program. After many years of crafting NCO as an after-hours hobby, I finally have the cushion necessary to give it some real attention. And I’m looking forward to this next, and most intense yet, phase of NCO development.

Charlie Zender
June, 2015
Irvine, California

Next: , Previous: , Up: Top   [Contents][Index]


This manual describes NCO, which stands for netCDF Operators. NCO is a suite of programs known as operators. Each operator is a standalone, command line program executed at the shell-level like, e.g., ls or mkdir. The operators take netCDF files (including HDF5 files constructed using the netCDF API) as input, perform an operation (e.g., averaging or hyperslabbing), and produce a netCDF file as output. The operators are primarily designed to aid manipulation and analysis of data. The examples in this documentation are typical applications of the operators for processing climate model output. This stems from their origin, though the operators are as general as netCDF itself.

Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

Next: , Previous: , Up: Introduction   [Contents][Index]

1.1 Availability

The complete NCO source distribution is currently distributed as a compressed tarfile from and from The compressed tarfile must be uncompressed and untarred before building NCO. Uncompress the file with ‘gunzip nco.tar.gz’. Extract the source files from the resulting tarfile with ‘tar -xvf nco.tar’. GNU tar lets you perform both operations in one step with ‘tar -xvzf nco.tar.gz’.

The documentation for NCO is called the NCO User Guide. The User Guide is available in PDF, Postscript, HTML, DVI, TeXinfo, and Info formats. These formats are included in the source distribution in the files nco.pdf,, nco.html, nco.dvi, nco.texi, and*, respectively. All the documentation descends from a single source file, nco.texi 1. Hence the documentation in every format is very similar. However, some of the complex mathematical expressions needed to describe ncwa can only be displayed in DVI, Postscript, and PDF formats.

A complete list of papers and publications on/about NCO is available on the NCO homepage. Most of these are freely available. The primary refereed publications are ZeM06 and Zen08. These contain copyright restrictions which limit their redistribution, but they are freely available in preprint form from the NCO.

If you want to quickly see what the latest improvements in NCO are (without downloading the entire source distribution), visit the NCO homepage at The HTML version of the User Guide is also available online through the World Wide Web at URL To build and use NCO, you must have netCDF installed. The netCDF homepage is

New NCO releases are announced on the netCDF list and on the nco-announce mailing list

Next: , Previous: , Up: Introduction   [Contents][Index]

1.2 How to Use This Guide

Detailed instructions about how to download the newest version, and how to complie source code, as well as a FAQ and descriptions of Known Problems etc. are on our homepage (

There are twelve operators in the current version (4.7.5-alpha01). The function of each is explained in Reference Manual. Many of the tasks that NCO can accomplish are described during the explanation of common NCO Features (see Shared features). More specific use examples for each operator can be seen by visiting the operator-specific examples in the Reference Manual. These can be found directly by prepending the operator name with the xmp_ tag, e.g., Also, users can type the operator name on the shell command line to see all the available options, or type, e.g., ‘man ncks’ to see a help man-page.

NCO is a command-line language. You may either use an operator after the prompt (e.g., ‘$’ here), like,

$ operator [options] input [output]

or write all commands lines into a shell script, as in the CMIP5 Example (see CMIP5 Example).

If you are new to NCO, the Quick Start (see Quick Start) shows simple examples about how to use NCO on different kinds of data files. More detailed “real-world” examples are in the CMIP5 Example. The Index is presents multiple keyword entries for the same subject. If these resources do not help enough, please see Help Requests and Bug Reports.

Next: , Previous: , Up: Introduction   [Contents][Index]

1.3 Operating systems compatible with NCO

In its time on Earth, NCO has been successfully ported and tested on so many 32- and 64-bit platforms that if we did not write them down here we would forget their names: IBM AIX 4.x, 5.x, FreeBSD 4.x, GNU/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64, LinuxAMD64, SGI IRIX 5.x and 6.x, MacOS X 10.x, DEC OSF, NEC Super-UX 10.x, Sun SunOS 4.1.x, Solaris 2.x, Cray UNICOS 8.x–10.x, and Microsoft Windows (95, 98, NT, 2000, XP, Vista, 7, 8, 10). If you port the code to a new operating system, please send me a note and any patches you required.

The major prerequisite for installing NCO on a particular platform is the successful, prior installation of the netCDF library (and, as of 2003, the UDUnits library). Unidata has shown a commitment to maintaining netCDF and UDUnits on all popular UNIX platforms, and is moving towards full support for the Microsoft Windows operating system (OS). Given this, the only difficulty in implementing NCO on a particular platform is standardization of various C-language API system calls. NCO code is tested for ANSI compliance by compiling with C99 compilers including those from GNU (‘gcc -std=c99 -pedantic -D_BSD_SOURCE -D_POSIX_SOURCE’ -Wall) 2, Comeau Computing (‘como --c99’), Cray (‘cc’), HP/Compaq/DEC (‘cc’), IBM (‘xlc -c -qlanglvl=extc99’), Intel (‘icc -std=c99’), LLVM (‘clang’), NEC (‘cc’), PathScale (QLogic) (‘pathcc -std=c99’), PGI (‘pgcc -c9x’), SGI (‘cc -c99’), and Sun (‘cc’). NCO (all commands and the libnco library) and the C++ interface to netCDF (called libnco_c++) comply with the ISO C++ standards as implemented by Comeau Computing (‘como’), Cray (‘CC’), GNU (‘g++ -Wall’), HP/Compaq/DEC (‘cxx’), IBM (‘xlC’), Intel (‘icc’), Microsoft (‘MVS’), NEC (‘c++’), PathScale (Qlogic) (‘pathCC’), PGI (‘pgCC’), SGI (‘CC -LANG:std’), and Sun (‘CC -LANG:std’). See nco/bld/Makefile and nco/src/nco_c++/Makefile.old for more details and exact settings.

Until recently (and not even yet), ANSI-compliant has meant compliance with the 1989 ISO C-standard, usually called C89 (with minor revisions made in 1994 and 1995). C89 lacks variable-size arrays, restricted pointers, some useful printf formats, and many mathematical special functions. These are valuable features of C99, the 1999 ISO C-standard. NCO is C99-compliant where possible and C89-compliant where necessary. Certain branches in the code are required to satisfy the native SGI and SunOS C compilers, which are strictly ANSI C89 compliant, and cannot benefit from C99 features. However, C99 features are fully supported by modern AIX, GNU, Intel, NEC, Solaris, and UNICOS compilers. NCO requires a C99-compliant compiler as of NCO version 2.9.8, released in August, 2004.

The most time-intensive portion of NCO execution is spent in arithmetic operations, e.g., multiplication, averaging, subtraction. These operations were performed in Fortran by default until August, 1999. This was a design decision based on the relative speed of Fortran-based object code vs. C-based object code in late 1994. C compiler vectorization capabilities have dramatically improved since 1994. We have accordingly replaced all Fortran subroutines with C functions. This greatly simplifies the task of building NCO on nominally unsupported platforms. As of August 1999, NCO built entirely in C by default. This allowed NCO to compile on any machine with an ANSI C compiler. In August 2004, the first C99 feature, the restrict type qualifier, entered NCO in version 2.9.8. C compilers can obtain better performance with C99 restricted pointers since they inform the compiler when it may make Fortran-like assumptions regarding pointer contents alteration. Subsequently, NCO requires a C99 compiler to build correctly 3.

In January 2009, NCO version 3.9.6 was the first to link to the GNU Scientific Library (GSL). GSL must be version 1.4 or later. NCO, in particular ncap2, uses the GSL special function library to evaluate geoscience-relevant mathematics such as Bessel functions, Legendre polynomials, and incomplete gamma functions (see GSL special functions).

In June 2005, NCO version 3.0.1 began to take advantage of C99 mathematical special functions. These include the standarized gamma function (called tgamma() for “true gamma”). NCO automagically takes advantage of some GNU Compiler Collection (GCC) extensions to ANSI C.

As of July 2000 and NCO version 1.2, NCO no longer performs arithmetic operations in Fortran. We decided to sacrifice executable speed for code maintainability. Since no objective statistics were ever performed to quantify the difference in speed between the Fortran and C code, the performance penalty incurred by this decision is unknown. Supporting Fortran involves maintaining two sets of routines for every arithmetic operation. The USE_FORTRAN_ARITHMETIC flag is still retained in the Makefile. The file containing the Fortran code, nco_fortran.F, has been deprecated but a volunteer (Dr. Frankenstein?) could resurrect it. If you would like to volunteer to maintain nco_fortran.F please contact me.

Previous: , Up: Compatability   [Contents][Index]

1.3.1 Compiling NCO for Microsoft Windows OS

NCO has been successfully ported and tested on most Microsoft Windows operating systems including: XP SP2/Vista/7/10. Support is provided for compiling either native Windows executables, using the Microsoft Visual Studio Compiler (MVSC), or with Cygwin, the UNIX-emulating compatibility layer with the GNU toolchain. The switches necessary to accomplish both are included in the standard distribution of NCO.

With Microsoft Visual Studio compiler, one must build NCO with C++ since MVSC does not support C99. Support for Qt, a convenient integrated development environment, was deprecated in 2017. As of NCO version 4.6.9 (September, 2017) please build native Windows executables with CMake:

cd ~/nco/cmake
make install

The file nco/cmake/build.bat shows how deal with various path issues.

As of NCO version 4.7.1 (December, 2017) the Conda package for NCO is available from the conda-forge channel on all three smithies: Linux, MacOS, and Windows.

# Recommended install with Conda
conda config --add channels conda-forge # Permananently add conda-forge
conda install nco
# Or, specify conda-forge explicitly as a one-off:
conda install -c conda-forge nco

Using the freely available Cygwin (formerly gnu-win32) development environment 4, the compilation process is very similar to installing NCO on a UNIX system. Set the PVM_ARCH preprocessor token to WIN32. Note that defining WIN32 has the side effect of disabling Internet features of NCO (see below). NCO should now build like it does on UNIX.

The least portable section of the code is the use of standard UNIX and Internet protocols (e.g., ftp, rcp, scp, sftp, getuid, gethostname, and header files <arpa/nameser.h> and <resolv.h>). Fortunately, these UNIX-y calls are only invoked by the single NCO subroutine which is responsible for retrieving files stored on remote systems (see Remote storage). In order to support NCO on the Microsoft Windows platforms, this single feature was disabled (on Windows OS only). This was required by Cygwin 18.x—newer versions of Cygwin may support these protocols (let me know if this is the case). The NCO operators should behave identically on Windows and UNIX platforms in all other respects.

Next: , Previous: , Up: Introduction   [Contents][Index]

1.4 Symbolic Links

NCO relies on a common set of underlying algorithms. To minimize duplication of source code, multiple operators sometimes share the same underlying source. This is accomplished by symbolic links from a single underlying executable program to one or more invoked executable names. For example, nces and ncrcat are symbolically linked to the ncra executable. The ncra executable behaves slightly differently based on its invocation name (i.e., ‘argv[0]’), which can be nces, ncra, or ncrcat. Logically, these are three different operators that happen to share the same executable.

For historical reasons, and to be more user friendly, multiple synonyms (or pseudonyms) may refer to the same operator invoked with different switches. For example, ncdiff is the same as ncbo and ncpack is the same as ncpdq. We implement the symbolic links and synonyms by the executing the following UNIX commands in the directory where the NCO executables are installed.

ln -s -f ncbo ncdiff    # ncbo --op_typ='-'
ln -s -f ncra nces      # ncra --pseudonym='nces'
ln -s -f ncra ncrcat    # ncra --pseudonym='ncrcat'
ln -s -f ncbo ncadd     # ncbo --op_typ='+'
ln -s -f ncbo ncsubtract # ncbo --op_typ='-'
ln -s -f ncbo ncmultiply # ncbo --op_typ='*'
ln -s -f ncbo ncdivide   # ncbo --op_typ='/'
ln -s -f ncpdq ncpack    # ncpdq
ln -s -f ncpdq ncunpack  # ncpdq --unpack
# NB: Windows/Cygwin executable/link names have '.exe' suffix, e.g.,
ln -s -f ncbo.exe ncdiff.exe

The imputed command called by the link is given after the comment. As can be seen, some these links impute the passing of a command line argument to further modify the behavior of the underlying executable. For example, ncdivide is a pseudonym for ncbo --op_typ='/'.

Next: , Previous: , Up: Introduction   [Contents][Index]

1.5 Libraries

Like all executables, the NCO operators can be built using dynamic linking. This reduces the size of the executable and can result in significant performance enhancements on multiuser systems. Unfortunately, if your library search path (usually the LD_LIBRARY_PATH environment variable) is not set correctly, or if the system libraries have been moved, renamed, or deleted since NCO was installed, it is possible NCO operators will fail with a message that they cannot find a dynamically loaded (aka shared object or ‘.so’) library. This will produce a distinctive error message, such as ‘ /usr/local/bin/nces: fatal: can't open file: errno=2’. If you received an error message like this, ask your system administrator to diagnose whether the library is truly missing 5, or whether you simply need to alter your library search path. As a final remedy, you may re-compile and install NCO with all operators statically linked.

Next: , Previous: , Up: Introduction   [Contents][Index]

1.6 netCDF2/3/4 and HDF4/5 Support

netCDF version 2 was released in 1993. NCO (specifically ncks) began soon after this in 1994. netCDF 3.0 was released in 1996, and we were not exactly eager to convert all code to the newer, less tested netCDF implementation. One netCDF3 interface call (nc_inq_libvers) was added to NCO in January, 1998, to aid in maintainance and debugging. In March, 2001, the final NCO conversion to netCDF3 was completed (coincidentally on the same day netCDF 3.5 was released). NCO versions 2.0 and higher are built with the -DNO_NETCDF_2 flag to ensure no netCDF2 interface calls are used.

However, the ability to compile NCO with only netCDF2 calls is worth maintaining because HDF version 4, aka HDF4 or simply HDF, 6 (available from HDF) supports only the netCDF2 library calls (see There are two versions of HDF. Currently HDF version 4.x supports the full netCDF2 API and thus NCO version 1.2.x. If NCO version 1.2.x (or earlier) is built with only netCDF2 calls then all NCO operators should work with HDF4 files as well as netCDF files 7. The preprocessor token NETCDF2_ONLY exists in NCO version 1.2.x to eliminate all netCDF3 calls. Only versions of NCO numbered 1.2.x and earlier have this capability.

HDF version 5 became available in 1999, but did not support netCDF (or, for that matter, Fortran) as of December 1999. By early 2001, HDF5 did support Fortran90. Thanks to an NSF-funded “harmonization” partnership, HDF began to fully support the netCDF3 read interface (which is employed by NCO 2.x and later). In 2004, Unidata and THG began a project to implement the HDF5 features necessary to support the netCDF API. NCO version 3.0.3 added support for reading/writing netCDF4-formatted HDF5 files in October, 2005. See File Formats and Conversion for more details.

HDF support for netCDF was completed with HDF5 version version 1.8 in 2007. The netCDF front-end that uses this HDF5 back-end was completed and released soon after as netCDF version 4. Download it from the netCDF4 website.

NCO version 3.9.0, released in May, 2007, added support for all netCDF4 atomic data types except NC_STRING. Support for NC_STRING, including ragged arrays of strings, was finally added in version 3.9.9, released in June, 2009. Support for additional netCDF4 features has been incremental. We add one netCDF4 feature at a time. You must build NCO with netCDF4 to obtain this support.

NCO supports many netCDF4 features including atomic data types, Lempel-Ziv compression (deflation), chunking, and groups. The new atomic data types are NC_UBYTE, NC_USHORT, NC_UINT, NC_INT64, and NC_UINT64. Eight-byte integer support is an especially useful improvement from netCDF3. All NCO operators support these types, e.g., ncks copies and prints them, ncra averages them, and ncap2 processes algebraic scripts with them. ncks prints compression information, if any, to screen.

NCO version 3.9.1 (June, 2007) added support for netCDF4 Lempel-Ziv deflation. Lempel-Ziv deflation is a lossless compression technique. See Deflation for more details.

NCO version 3.9.9 (June, 2009) added support for netCDF4 chunking in ncks and ncecat. NCO version 4.0.4 (September, 2010) completed support for netCDF4 chunking in the remaining operators. See Chunking for more details.

NCO version 4.2.2 (October, 2012) added support for netCDF4 groups in ncks and ncecat. Group support for these operators was complete (e.g., regular expressions to select groups and Group Path Editing) as of NCO version 4.2.6 (March, 2013). See Group Path Editing for more details. Group support for all other operators was finished in the NCO version 4.3.x series completed in December, 2013.

Support for netCDF4 in the first arithmetic operator, ncbo, was introduced in NCO version 4.3.0 (March, 2013). NCO version 4.3.1 (May, 2013) completed this support and introduced the first example of automatic group broadcasting. See ncbo netCDF Binary Operator for more details.

netCDF4-enabled NCO handles netCDF3 files without change. In addition, it automagically handles netCDF4 (HDF5) files: If you feed NCO netCDF3 files, it produces netCDF3 output. If you feed NCO netCDF4 files, it produces netCDF4 output. Use the handy-dandy ‘-4’ switch to request netCDF4 output from netCDF3 input, i.e., to convert netCDF3 to netCDF4. See File Formats and Conversion for more details.

When linked to a netCDF library that was built with HDF4 support 8, NCO automatically supports reading HDF4 files and writing them as netCDF3/netCDF4/HDF5 files. NCO can only write through the netCDF API, which can only write netCDF3/netCDF4/HDF5 files. So NCO can read HDF4 files, perform manipulations and calculations, and then it must write the results in netCDF format.

NCO support for HDF4 has been quite functional since December, 2013. For best results install NCO versions 4.4.0 or later on top of netCDF versions 4.3.1 or later. Getting to this point has been an iterative effort where Unidata improved netCDF library capabilities in response to our requests. NCO versions 4.3.6 and earlier do not explicitly support HDF4, yet should work with HDF4 if compiled with a version of netCDF (4.3.2 or later?) that does not unexpectedly die when probing HDF4 files with standard netCDF calls. NCO versions 4.3.7–4.3.9 (October–December, 2013) use a special flag to circumvent netCDF HDF4 issues. The user must tell these versions of NCO that an input file is HDF4 format by using the ‘--hdf4’ switch.

When compiled with netCDF version 4.3.1 (20140116) or later, NCO versions 4.4.0 (January, 2014) and later more gracefully handle HDF4 files. In particular, the ‘--hdf4’ switch is obsolete. Current versions of NCO use netCDF to determine automatically whether the underlying file is HDF4, and then take appropriate precautions to avoid netCDF4 API calls that fail when applied to HDF4 files (e.g., nc_inq_var_chunking(), nc_inq_var_deflate()). When compiled with netCDF version 4.3.2 (20140423) or earlier, NCO will report that chunking and deflation properties of HDF4 files as HDF4_UNKNOWN, because determining those properties was impossible. When compiled with netCDF version 4.3.3-rc2 (20140925) or later, NCO versions 4.4.6 (October, 2014) and later fully support chunking and deflation features of HDF4 files. The ‘--hdf4’ switch is supported (for backwards compatibility) yet redundant (i.e., does no harm) with current versions of NCO and netCDF.

Converting HDF4 files to netCDF: Since NCO reads HDF4 files natively, it is now easy to convert HDF4 files to netCDF files directly, e.g.,

ncks        fl.hdf # Convert HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks --hdf4 fl.hdf # Convert HDF4->netCDF4 (NCO 4.3.7-4.3.9)

The most efficient and accurate way to convert HDF4 data to netCDF format is to convert to netCDF4 using NCO as above. Many HDF4 producers (NASA!) love to use netCDF4 types, e.g., unsigned bytes, so this procedure is the most typical. Conversion of HDF4 to netCDF4 as above suffices when the data will only be processed by NCO and other netCDF4-aware tools.

However, many tools are not fully netCDF4-aware, and so conversion to netCDF3 may be desirable. Obtaining any netCDF file from an HDF4 is easy:

ncks -3 fl.hdf      # HDF4->netCDF3 (NCO 4.4.0+, netCDF 4.3.1+)
ncks -4 fl.hdf      # HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
ncks -6 fl.hdf      # HDF4->netCDF3 64-bit  (NCO 4.4.0+, ...)
ncks -7 -L 1 fl.hdf # HDF4->netCDF4 classic (NCO 4.4.0+, ...)
ncks --hdf4 -3 fl.hdf # HDF4->netCDF3 (netCDF 4.3.0-)
ncks --hdf4 -4 fl.hdf # HDF4->netCDF4 (netCDF 4.3.0-)
ncks --hdf4 -6 fl.hdf # HDF4->netCDF3 64-bit  (netCDF 4.3.0-)
ncks --hdf4 -7 fl.hdf # HDF4->netCDF4 classic (netCDF 4.3.0-)

As of NCO version 4.4.0 (January, 2014), these commands work even when the HDF4 file contains netCDF4 atomic types (e.g., unsigned bytes, 64-bit integers) because NCO can autoconvert everything to atomic types supported by netCDF3 9.

As of NCO version 4.4.4 (May, 2014) both ncl_convert2nc and NCO have built-in, automatic workarounds to handle element names that contain characters that are legal in HDF though are illegal in netCDF. For example, slashes and leading special characters are are legal in HDF and illegal in netCDF element (i.e., group, variable, dimension, and attribute) names. NCO converts these forbidden characters to underscores, and retains the original names of variables in automatically produced attributes named hdf_name 10.

Finally, in February 2014, we learned that the HDF group has a project called H4CF (described here) whose goal is to make HDF4 files accessible to CF tools and conventions. Their project includes a tool named h4tonccf that converts HDF4 files to netCDF3 or netCDF4 files. We are not yet sure what advantages or features h4tonccf has that are not in NCO, though we suspect both methods have their own advantages. Corrections welcome.

As of 2012, netCDF4 is relatively stable software. Problems with netCDF4 and HDF libraries have mainly been fixed. Binary NCO distributions shipped as RPMs and as debs have used the netCDF4 library since 2010 and 2011, respectively.

One must often build NCO from source to obtain netCDF4 support. Typically, one specifies the root of the netCDF4 installation directory. Do this with the NETCDF4_ROOT variable. Then use your preferred NCO build mechanism, e.g.,

export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism

We carefully track the netCDF4 releases, and keep the netCDF4 atomic type support and other features working. Our long term goal is to utilize more of the extensive new netCDF4 feature set. The next major netCDF4 feature we are likely to utilize is parallel I/O. We will enable this in the MPI netCDF operators.

Previous: , Up: Introduction   [Contents][Index]

1.7 Help Requests and Bug Reports

We generally receive three categories of mail from users: help requests, bug reports, and feature requests. Notes saying the equivalent of “Hey, NCO continues to work great and it saves me more time everyday than it took to write this note” are a distant fourth.

There is a different protocol for each type of request. The preferred etiquette for all communications is via NCO Project Forums. Do not contact project members via personal e-mail unless your request comes with money or you have damaging information about our personal lives. Please use the Forums—they preserve a record of the questions and answers so that others can learn from our exchange. Also, since NCO is government-funded, this record helps us provide program officers with information they need to evaluate our project.

Before posting to the NCO forums described below, you might first register your name and email address with or else all of your postings will be attributed to nobody. Once registered you may choose to monitor any forum and to receive (or not) email when there are any postings including responses to your questions. We usually reply to the forum message, not to the original poster.

If you want us to include a new feature in NCO, check first to see if that feature is already on the TODO list. If it is, why not implement that feature yourself and send us the patch? If the feature is not yet on the list, then send a note to the NCO Discussion forum.

Read the manual before reporting a bug or posting a help request. Sending questions whose answers are not in the manual is the best way to motivate us to write more documentation. We would also like to accentuate the contrapositive of this statement. If you think you have found a real bug the most helpful thing you can do is simplify the problem to a manageable size and then report it. The first thing to do is to make sure you are running the latest publicly released version of NCO.

Once you have read the manual, if you are still unable to get NCO to perform a documented function, submit a help request. Follow the same procedure as described below for reporting bugs (after all, it might be a bug). That is, describe what you are trying to do, and include the complete commands (run with ‘-D 5’), error messages, and version of NCO (with ‘-r’). Post your help request to the NCO Help forum.

If you think you used the right command when NCO misbehaves, then you might have found a bug. Incorrect numerical answers are the highest priority. We usually fix those within one or two days. Core dumps and sementation violations receive lower priority. They are always fixed, eventually.

How do you simplify a problem that reveal a bug? Cut out extraneous variables, dimensions, and metadata from the offending files and re-run the command until it no longer breaks. Then back up one step and report the problem. Usually the file(s) will be very small, i.e., one variable with one or two small dimensions ought to suffice. Run the operator with ‘-r’ and then run the command with ‘-D 5’ to increase the verbosity of the debugging output. It is very important that your report contain the exact error messages and compile-time environment. Include a copy of your sample input file, or place one on a publicly accessible location, of the file(s). If you are sure it is a bug, post the full report to the NCO Project buglist. Otherwise post all the information to NCO Help forum.

Build failures count as bugs. Our limited machine access means we cannot fix all build failures. The information we need to diagnose, and often fix, build failures are the three files output by GNU build tools, nco.config.log.${GNU_TRP}.foo, nco.configure.${GNU_TRP}.foo, and nco.make.${GNU_TRP}.foo. The file shows how to produce these files. Here ${GNU_TRP} is the “GNU architecture triplet”, the chip-vendor-OS string returned by config.guess. Please send us your improvements to the examples supplied in The regressions archive at contains the build output from our standard test systems. You may find you can solve the build problem yourself by examining the differences between these files and your own.

Next: , Previous: , Up: Top   [Contents][Index]

2 Operator Strategies

Next: , Previous: , Up: Strategies   [Contents][Index]

2.1 Philosophy

The main design goal is command line operators which perform useful, scriptable operations on netCDF files. Many scientists work with models and observations which produce too much data to analyze in tabular format. Thus, it is often natural to reduce and massage this raw or primary level data into summary, or second level data, e.g., temporal or spatial averages. These second level data may become the inputs to graphical and statistical packages, and are often more suitable for archival and dissemination to the scientific community. NCO performs a suite of operations useful in manipulating data from the primary to the second level state. Higher level interpretive languages (e.g., IDL, Yorick, Matlab, NCL, Perl, Python), and lower level compiled languages (e.g., C, Fortran) can always perform any task performed by NCO, but often with more overhead. NCO, on the other hand, is limited to a much smaller set of arithmetic and metadata operations than these full blown languages.

Another goal has been to implement enough command line switches so that frequently used sequences of these operators can be executed from a shell script or batch file. Finally, NCO was written to consume the absolute minimum amount of system memory required to perform a given job. The arithmetic operators are extremely efficient; their exact memory usage is detailed in Memory Requirements.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.2 Climate Model Paradigm

NCO was developed at NCAR to aid analysis and manipulation of datasets produced by General Circulation Models (GCMs). GCM datasets share many features with other gridded scientific datasets and so provide a useful paradigm for the explication of the NCO operator set. Examples in this manual use a GCM paradigm because latitude, longitude, time, temperature and other fields related to our natural environment are as easy to visualize for the layman as the expert.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.3 Temporary Output Files

NCO operators are designed to be reasonably fault tolerant, so that a system failure or user-abort of the operation (e.g., with C-c) does not cause loss of data. The user-specified output-file is only created upon successful completion of the operation 11. This is accomplished by performing all operations in a temporary copy of output-file. The name of the temporary output file is constructed by appending .pid<process ID>.<operator name>.tmp to the user-specified output-file name. When the operator completes its task with no fatal errors, the temporary output file is moved to the user-specified output-file. This imbues the process with fault-tolerance since fatal error (e.g., disk space fills up) affect only the temporary output file, leaving the final output file not created if it did not already exist. Note the construction of a temporary output file uses more disk space than just overwriting existing files “in place” (because there may be two copies of the same file on disk until the NCO operation successfully concludes and the temporary output file overwrites the existing output-file). Also, note this feature increases the execution time of the operator by approximately the time it takes to copy the output-file 12. Finally, note this fault-tolerant feature allows the output-file to be the same as the input-file without any danger of “overlap”.

Over time many “power users” have requested a way to turn-off the fault-tolerance safety feature of automatically creating a temporary file. Often these users build and execute production data analysis scripts that are repeated frequently on large datasets. Obviating an extra file write can then conserve significant disk space and time. For this purpose NCO has, since version 4.2.1 in August, 2012, made configurable the controls over temporary file creation. The ‘--wrt_tmp_fl’ and equivalent ‘--write_tmp_fl’ switches ensure NCO writes output to an intermediate temporary file. This is and has always been the default behavior so there is currently no need to specify these switches. However, the default may change some day, especially since writing to RAM disks (see RAM disks) may some day become the default. The ‘--no_tmp_fl’ switch causes NCO to write directly to the final output file instead of to an intermediate temporary file. “Power users” may wish to invoke this switch to increase performance (i.e., reduce wallclock time) when manipulating large files. When eschewing temporary files, users may forsake the ability to have the same name for both output-file and input-file since, as described above, the temporary file prevented overlap issues. However, if the user creates the output file in RAM (see RAM disks) then it is still possible to have the same name for both output-file and input-file.

ncks # Default: create then move to
ncks --wrt_tmp_fl # Same as default
ncks --no_tmp_fl # Create directly on disk
ncks --no_tmp_fl # ERROR-prone! Overwrite with itself
ncks --create_ram --no_tmp_fl # Create in RAM, write to disk
ncks --open_ram --no_tmp_fl # Read into RAM, write to disk

There is no reason to expect the fourth example to work. The behavior of overwriting a file while reading from the same file is undefined, much as is the shell command ‘cat foo > foo’. Although it may “work” in some cases, it is unreliable. One way around this is to use ‘--create_ram’ so that the output file is not written to disk until the input file is closed, See RAM disks. However, as of 20130328, the behavior of the ‘--create_ram’ and ‘--open_ram’ examples has not been thoroughly tested.

The NCO authors have seen compelling use cases for utilizing the RAM switches, though not (yet) for combining them with ‘--no_tmp_fl’. NCO implements both options because they are largely independent of eachother. It is up to “power users” to discover which best fit their needs. We welcome accounts of your experiences posted to the forums.

Other safeguards exist to protect the user from inadvertently overwriting data. If the output-file specified for a command is a pre-existing file, then the operator will prompt the user whether to overwrite (erase) the existing output-file, attempt to append to it, or abort the operation. However, in processing large amounts of data, too many interactive questions slows productivity. Therefore NCO also implements two ways to override its own safety features, the ‘-O’ and ‘-A’ switches. Specifying ‘-O’ tells the operator to overwrite any existing output-file without prompting the user interactively. Specifying ‘-A’ tells the operator to attempt to append to any existing output-file without prompting the user interactively. These switches are useful in batch environments because they suppress interactive keyboard input.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.4 Appending Variables

Adding variables from one file to another is often desirable. This is referred to as appending, although some prefer the terminology merging 13 or pasting. Appending is often confused with what NCO calls concatenation. In NCO, concatenation refers to splicing a variable along the record dimension. The length along the record dimension of the output is the sum of the lengths of the input files. Appending, on the other hand, refers to copying a variable from one file to another file which may or may not already contain the variable 14. NCO can append or concatenate just one variable, or all the variables in a file at the same time.

In this sense, ncks can append variables from one file to another file. This capability is invoked by naming two files on the command line, input-file and output-file. When output-file already exists, the user is prompted whether to overwrite, append/replace, or exit from the command. Selecting overwrite tells the operator to erase the existing output-file and replace it with the results of the operation. Selecting exit causes the operator to exit—the output-file will not be touched in this case. Selecting append/replace causes the operator to attempt to place the results of the operation in the existing output-file, See ncks netCDF Kitchen Sink.

The simplest way to create the union of two files is

ncks -A

This puts the contents of into The ‘-A’ is optional. On output, is the union of the input files, regardless of whether they share dimensions and variables, or are completely disjoint. The append fails if the input files have differently named record dimensions (since netCDF supports only one), or have dimensions of the same name but different sizes.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.5 Simple Arithmetic and Interpolation

Users comfortable with NCO semantics may find it easier to perform some simple mathematical operations in NCO rather than higher level languages. ncbo (see ncbo netCDF Binary Operator) does file addition, subtraction, multiplication, division, and broadcasting. It even does group broadcasting. ncflint (see ncflint netCDF File Interpolator) does file addition, subtraction, multiplication and interpolation. Sequences of these commands can accomplish simple yet powerful operations from the command line.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.6 Statistics vs. Concatenation

The most frequently used operators of NCO are probably the statisticians (i.e., tools that do statistics) and concatenators. Because there are so many types of statistics like averaging (e.g., across files, within a file, over the record dimension, over other dimensions, with or without weights and masks) and of concatenating (across files, along the record dimension, along other dimensions), there are currently no fewer than five operators which tackle these two purposes: ncra, nces, ncwa, ncrcat, and ncecat. These operators do share many capabilities 15, though each has its unique specialty. Two of these operators, ncrcat and ncecat, concatenate hyperslabs across files. The other two operators, ncra and nces, compute statistics across (and/or within) files 16. First, let’s describe the concatenators, then the statistics tools.

Next: , Previous: , Up: Statistics vs. Concatenation   [Contents][Index]

2.6.1 Concatenators ncrcat and ncecat

Joining together independent files along a common record dimension is called concatenation. ncrcat is designed for concatenating record variables, while ncecat is designed for concatenating fixed length variables. Consider five files,,, … each containing a year’s worth of data. Say you wish to create from them a single file, containing all the data, i.e., spanning all five years. If the annual files make use of the same record variable, then ncrcat will do the job nicely with, e.g., ncrcat 8?.nc The number of records in the input files is arbitrary and can vary from file to file. See ncrcat netCDF Record Concatenator, for a complete description of ncrcat.

However, suppose the annual files have no record variable, and thus their data are all fixed length. For example, the files may not be conceptually sequential, but rather members of the same group, or ensemble. Members of an ensemble may have no reason to contain a record dimension. ncecat will create a new record dimension (named record by default) with which to glue together the individual files into the single ensemble file. If ncecat is used on files which contain an existing record dimension, that record dimension is converted to a fixed-length dimension of the same name and a new record dimension (named record) is created. Consider five realizations,,, … of 1985 predictions from the same climate model. Then ncecat 85?.nc glues together the individual realizations into the single file, If an input variable was dimensioned [lat,lon], it will have dimensions [record,lat,lon] in the output file. A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file. Normally this means all the input files are the same size, and contain data on different realizations of the same variables. See ncecat netCDF Ensemble Concatenator, for a complete description of ncecat.

ncpdq makes it possible to concatenate files along any dimension, not just the record dimension. First, use ncpdq to convert the dimension to be concatenated (i.e., extended with data from other files) into the record dimension. Second, use ncrcat to concatenate these files. Finally, if desirable, use ncpdq to revert to the original dimensionality. As a concrete example, say that files,, … contain time-evolving datasets from spatially adjacent regions. The time and spatial coordinates are time and x, respectively. Initially the record dimension is time. Our goal is to create a single file that contains joins all the spatially adjacent regions into one single time-evolving dataset.

for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
  ncpdq -a x,time x_${idx}.nc foo_${idx}.nc  # Make x record dimension
ncrcat foo_??.nc       # Concatenate along x
ncpdq -a time,x # Revert to time as record dimension

Note that ncrcat will not concatenate fixed-length variables, whereas ncecat concatenates both fixed-length and record variables along a new record variable. To conserve system memory, use ncrcat where possible.

Next: , Previous: , Up: Statistics vs. Concatenation   [Contents][Index]

2.6.2 Averagers nces, ncra, and ncwa

The differences between the averagers ncra and nces are analogous to the differences between the concatenators. ncra is designed for averaging record variables from at least one file, while nces is designed for averaging fixed length variables from multiple files. ncra performs a simple arithmetic average over the record dimension of all the input files, with each record having an equal weight in the average. nces performs a simple arithmetic average of all the input files, with each file having an equal weight in the average. Note that ncra cannot average fixed-length variables, but nces can average both fixed-length and record variables. To conserve system memory, use ncra rather than nces where possible (e.g., if each input-file is one record long). The file output from nces will have the same dimensions (meaning dimension names as well as sizes) as the input hyperslabs (see nces netCDF Ensemble Statistics, for a complete description of nces). The file output from ncra will have the same dimensions as the input hyperslabs except for the record dimension, which will have a size of 1 (see ncra netCDF Record Averager, for a complete description of ncra).

Previous: , Up: Statistics vs. Concatenation   [Contents][Index]

2.6.3 Interpolator ncflint

ncflint can interpolate data between or two files. Since no other operators have this ability, the description of interpolation is given fully on the ncflint reference page (see ncflint netCDF File Interpolator). Note that this capability also allows ncflint to linearly rescale any data in a netCDF file, e.g., to convert between differing units.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.7 Large Numbers of Files

Occasionally one desires to digest (i.e., concatenate or average) hundreds or thousands of input files. Unfortunately, data archives (e.g., NASA EOSDIS) may not name netCDF files in a format understood by the ‘-n loop’ switch (see Specifying Input Files) that automagically generates arbitrary numbers of input filenames. The ‘-n loop’ switch has the virtue of being concise, and of minimizing the command line. This helps keeps output file small since the command line is stored as metadata in the history attribute (see History Attribute). However, the ‘-n loop’ switch is useless when there is no simple, arithmetic pattern to the input filenames (e.g.,,, … Moreover, filename globbing does not work when the input files are too numerous or their names are too lengthy (when strung together as a single argument) to be passed by the calling shell to the NCO operator 17. When this occurs, the ANSI C-standard argc-argv method of passing arguments from the calling shell to a C-program (i.e., an NCO operator) breaks down. There are (at least) three alternative methods of specifying the input filenames to NCO in environment-limited situations.

The recommended method for sending very large numbers (hundreds or more, typically) of input filenames to the multi-file operators is to pass the filenames with the UNIX standard input feature, aka stdin:

# Pipe large numbers of filenames to stdin
/bin/ls | grep ${CASEID}_'......'.nc | ncecat -o

This method avoids all constraints on command line size imposed by the operating system. A drawback to this method is that the history attribute (see History Attribute) does not record the name of any input files since the names were not passed on the command line. This makes determining the data provenance at a later date difficult. To remedy this situation, multi-file operators store the number of input files in the nco_input_file_number global attribute and the input file list itself in the nco_input_file_list global attribute (see File List Attributes). Although this does not preserve the exact command used to generate the file, it does retains all the information required to reconstruct the command and determine the data provenance.

A second option is to use the UNIX xargs command. This simple example selects as input to xargs all the filenames in the current directory that match a given pattern. For illustration, consider a user trying to average millions of files which each have a six character filename. If the shell buffer cannot hold the results of the corresponding globbing operator, ??????.nc, then the filename globbing technique will fail. Instead we express the filename pattern as an extended regular expression, ......\.nc (see Subsetting Files). We use grep to filter the directory listing for this pattern and to pipe the results to xargs which, in turn, passes the matching filenames to an NCO multi-file operator, e.g., ncecat.

# Use xargs to transfer filenames on the command line
/bin/ls | grep ${CASEID}_'......'.nc | xargs -x ncecat -o

The single quotes protect the only sensitive parts of the extended regular expression (the grep argument), and allow shell interpolation (the ${CASEID} variable substitution) to proceed unhindered on the rest of the command. xargs uses the UNIX pipe feature to append the suitably filtered input file list to the end of the ncecat command options. The -o switch ensures that the input files supplied by xargs are not confused with the output file name. xargs does, unfortunately, have its own limit (usually about 20,000 characters) on the size of command lines it can pass. Give xargs the ‘-x’ switch to ensure it dies if it reaches this internal limit. When this occurs, use either the stdin method above, or the symbolic link presented next.

Even when its internal limits have not been reached, the xargs technique may not be sophisticated enough to handle all situations. A full scripting language like Perl or Python can handle any level of complexity of filtering input filenames, and any number of filenames. The technique of last resort is to write a script that creates symbolic links between the irregular input filenames and a set of regular, arithmetic filenames that the ‘-n loop’ switch understands. For example, the following Perl script creates a monotonically enumerated symbolic link to up to one million .nc files in a directory. If there are 999,999 netCDF files present, the links are named to

# Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(<STDIN>){chop;symlink $_,sprintf("",$idx++);}'
ncecat -n 999999,6,1
# Remove symbolic links when finished
/bin/rm ??????.nc

The ‘-n loop’ option tells the NCO operator to automatically generate the filnames of the symbolic links. This circumvents any OS and shell limits on command-line size. The symbolic links are easily removed once NCO is finished. One drawback to this method is that the history attribute (see History Attribute) retains the filename list of the symbolic links, rather than the data files themselves. This makes it difficult to determine the data provenance at a later date.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.8 Large Datasets

Large datasets are those files that are comparable in size to the amount of random access memory (RAM) in your computer. Many users of NCO work with files larger than 100 MB. Files this large not only push the current edge of storage technology, they present special problems for programs which attempt to access the entire file at once, such as nces and ncecat. If you work with a 300 MB files on a machine with only 32 MB of memory then you will need large amounts of swap space (virtual memory on disk) and NCO will work slowly, or even fail. There is no easy solution for this. The best strategy is to work on a machine with sufficient amounts of memory and swap space. Since about 2004, many users have begun to produce or analyze files exceeding 2 GB in size. These users should familiarize themselves with NCO’s Large File Support (LFS) capabilities (see Large File Support). The next section will increase your familiarity with NCO’s memory requirements. With this knowledge you may re-design your data reduction approach to divide the problem into pieces solvable in memory-limited situations.

If your local machine has problems working with large files, try running NCO from a more powerful machine, such as a network server. If you get a memory-related core dump (e.g., ‘Error exit (core dumped)’) on a GNU/Linux system, or the operation ends before the entire output file is written, try increasing the process-available memory with ulimit:

ulimit -f unlimited

This may solve constraints on clusters where sufficient hardware resources exist yet where system administrators felt it wise to prevent any individual user from consuming too much of resource. Certain machine architectures, e.g., Cray UNICOS, have special commands which allow one to increase the amount of interactive memory. On Cray systems, try to increase the available memory with the ilimit command.

The speed of the NCO operators also depends on file size. When processing large files the operators may appear to hang, or do nothing, for large periods of time. In order to see what the operator is actually doing, it is useful to activate a more verbose output mode. This is accomplished by supplying a number greater than 0 to the ‘-D debug-level’ (or ‘--debug-level’, or ‘--dbg_lvl’) switch. When the debug-level is nonzero, the operators report their current status to the terminal through the stderr facility. Using ‘-D’ does not slow the operators down. Choose a debug-level between 1 and 3 for most situations, e.g., nces -D 2 A full description of how to estimate the actual amount of memory the multi-file NCO operators consume is given in Memory Requirements.

Next: , Previous: , Up: Strategies   [Contents][Index]

2.9 Memory Requirements

Many people use NCO on gargantuan files which dwarf the memory available (free RAM plus swap space) even on today’s powerful machines. These users want NCO to consume the least memory possible so that their scripts do not have to tediously cut files into smaller pieces that fit into memory. We commend these greedy users for pushing NCO to its limits!

This section describes the memory NCO requires during operation. The required memory depends on the underlying algorithms, datatypes, and compression, if any. The description below is the memory usage per thread. Users with shared memory machines may use the threaded NCO operators (see OpenMP Threading). The peak and sustained memory usage will scale accordingly, i.e., by the number of threads. In all cases the memory use refers to the uncompressed size of the data. The netCDF4 library automatically decompresses variables during reads. The filesize can easily belie the true size of the uncompressed data. In other words, the usage below can be taken at face value for netCDF3 datasets only. Chunking will also affect memory usage on netCDF4 operations. Memory consumption patterns of all operators are similar, with the exception of ncap2.

Next: , Previous: , Up: Memory Requirements   [Contents][Index]

2.9.1 Single and Multi-file Operators

The multi-file operators currently comprise the record operators, ncra and ncrcat, and the ensemble operators, nces and ncecat. The record operators require much less memory than the ensemble operators. This is because the record operators operate on one single record (i.e., time-slice) at a time, whereas the ensemble operators retrieve the entire variable into memory. Let MS be the peak sustained memory demand of an operator, FT be the memory required to store the entire contents of all the variables to be processed in an input file, FR be the memory required to store the entire contents of a single record of each of the variables to be processed in an input file, VR be the memory required to store a single record of the largest record variable to be processed in an input file, VT be the memory required to store the largest variable to be processed in an input file, VI be the memory required to store the largest variable which is not processed, but is copied from the initial file to the output file. All operators require MI = VI during the initial copying of variables from the first input file to the output file. This is the initial (and transient) memory demand. The sustained memory demand is that memory required by the operators during the processing (i.e., averaging, concatenation) phase which lasts until all the input files have been processed. The operators have the following memory requirements: ncrcat requires MS <= VR. ncecat requires MS <= VT. ncra requires MS = 2FR + VR. nces requires MS = 2FT + VT. ncbo requires MS <= 3VT (both input variables and the output variable). ncflint requires MS <= 3VT (both input variables and the output variable). ncpdq requires MS <= 2VT (one input variable and the output variable). ncwa requires MS <= 8VT (see below). Note that only variables that are processed, e.g., averaged, concatenated, or differenced, contribute to MS. Variables that do not appear in the output file (see Subsetting Files) are never read and contribute nothing to the memory requirements.

Further note that some operators perform internal type-promotion on some variables prior to arithmetic (see Type Conversion). For example, ncra, nces, and ncwa all promote integer types to double-precision floating-point prior to arithmetic, then perform the arithmetic, then demote back to the original integer type after arithmetic. This preserves the on-disk storage type while obtaining the precision advantages of double-precision floating-point arithmetic. Since version 4.3.6 (released in September, 2013), NCO also by default converts single-precision floating-point to double-precision prior to arithmetic, which incurs the same RAM penalty. Hence, the sustained memory required for integer variables and single-precision floats are two or four-times their on-disk, uncompressed, unpacked sizes if they meet the rules for automatic internal promotion. Put another way, disabling auto-promotion of single-precision variables (with ‘--flt’) considerably reduces the RAM footprint of arithmetic operators.

The ‘--open_ram’ switch (and switches that invoke it like ‘--ram_all’ and ‘--diskless_all’) incurs a RAM penalty. These switches cause each input file to be copied to RAM upon opening. Hence any operator invoking these switches utilizes an additional FT of RAM (i.e., MS += FT). See RAM disks for further details.

ncwa consumes between two and eight times the memory of an NC_DOUBLE variable in order to process it. Peak consumption occurs when storing simultaneously in memory one input variable, one tally array, one input weight, one conformed/working weight, one weight tally, one input mask, one conformed/working mask, and one output variable. NCO’s tally arrays are of type C-type long, whose size is eight-bytes on all modern computers, the same as NC_DOUBLE 18. When invoked, the weighting and masking features contribute up to three-eighths and two-eighths of these requirements apiece. If weights and masks are not specified (i.e., no ‘-w’ or ‘-a’ options) then ncwa requirements drop to MS <= 3VT (one input variable, one tally array, and the output variable). The output variable is the same size as the input variable when averaging only over a degenerate dimension. However, normally the output variable is much smaller than the input, and is often a simple scalar, in which case the memory requirements drop by 1VT since the output array requires essentially no memory.

All of this is subject to the type promotion rules mentioned above. For example, ncwa averaging a variable of type NC_FLOAT requires MS <= 16VT (rather than MS <= 8VT) since all arrays are (at least temporarily) composed of eight-byte elements, twice the size of the values on disk. Without mask or weights, the requirements for NC_FLOAT are MS <= 6VT (rather than MS <= 3VT as for NC_DOUBLE) due to temporary internal promotion of both the input variable and the output variable to type NC_DOUBLE. The ‘--flt’ option that suppresses promotion reduces this to MS <= 4VT (the tally elements do not change size), and to MS <= 3VT when the output array is a scalar.

The above memory requirements must be multiplied by the number of threads thr_nbr (see OpenMP Threading). If this causes problems then reduce (with ‘-t thr_nbr’) the number of threads.

Previous: , Up: Memory Requirements   [Contents][Index]

2.9.2 Memory for ncap2

ncap2 has unique memory requirements due its ability to process arbitrarily long scripts of any complexity. All scripts acceptable to ncap2 are ultimately processed as a sequence of binary or unary operations. ncap2 requires MS <= 2VT under most conditions. An exception to this is when left hand casting (see Left hand casting) is used to stretch the size of derived variables beyond the size of any input variables. Let VC be the memory required to store the largest variable defined by left hand casting. In this case, MS <= 2VC.

ncap2 scripts are complete dynamic and may be of arbitrary length. A script that contains many thousands of operations, may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find. Sometimes a malloc() (or new[]) failure is the only noticeable clue to their existence. If you have good reasons to believe that a memory allocation failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would be very interested in receiving a detailed bug report.

Previous: , Up: Strategies   [Contents][Index]

2.10 Performance

An overview of NCO capabilities as of about 2006 is in Zender, C. S. (2008), “Analysis of Self-describing Gridded Geoscience Data with netCDF Operators (NCO)”, Environ. Modell. Softw., doi:10.1016/j.envsoft.2008.03.004. This paper is also available at

NCO performance and scaling for arithmetic operations is described in Zender, C. S., and H. J. Mangalam (2007), “Scaling Properties of Common Statistical Operators for Gridded Datasets”, Int. J. High Perform. Comput. Appl., 21(4), 485-498, doi:10.1177/1094342007083802. This paper is also available at

It is helpful to be aware of the aspects of NCO design that can limit its performance:

  1. No data buffering is performed during nc_get_var and nc_put_var operations. Hyperslabs too large to hold in core memory will suffer substantial performance penalties because of this.
  2. Since coordinate variables are assumed to be monotonic, the search for bracketing the user-specified limits should employ a quicker algorithm, like bisection, than the two-sided incremental search currently implemented.
  3. C_format, FORTRAN_format, signedness, scale_format and add_offset attributes are ignored by ncks when printing variables to screen.
  4. In the late 1990s it was discovered that some random access operations on large files on certain architectures (e.g., UNICOS) were much slower with NCO than with similar operations performed using languages that bypass the netCDF interface (e.g., Yorick). This may have been a penalty of unnecessary byte-swapping in the netCDF interface. It is unclear whether such problems exist in present day (2007) netCDF/NCO environments, where unnecessary byte-swapping has been reduced or eliminated.

Next: , Previous: , Up: Top   [Contents][Index]

3 Shared Features

Many features have been implemented in more than one operator and are described here for brevity. The description of each feature is preceded by a box listing the operators for which the feature is implemented. Command line switches for a given feature are consistent across all operators wherever possible. If no “key switches” are listed for a feature, then that particular feature is automatic and cannot be controlled by the user.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.1 Internationalization

Availability: All operators

NCO support for internationalization of textual input and output (e.g., Warning messages) is nascent. We introduced the first foreign language string catalogues (French and Spanish) in 2004, yet did not activate these in distributions because the catalogues were nearly empty. We seek volunteers to populate our templates with translations for their favorite languages.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.2 Metadata Optimization

Availability: All operators
Short options: None
Long options: ‘--hdr_pad’, ‘--header_pad

NCO supports padding headers to improve the speed of future metadata operations. Use the ‘--hdr_pad’ and ‘--header_pad’ switches to request that hdr_pad bytes be inserted into the metadata section of the output file. There is little downside to padding a header with kilobyte of space, since subsequent manipulation of the file will annotate the history attribute with all commands, let alone any explicit metadata additions with ncatted.

ncks --hdr_pad=1000 # Pad header with  1 kB space
ncks --hdr_pad=10000 # Pad header with 10 kB space

Future metadata expansions will not incur the netCDF3 performance penalty of copying the entire output file unless the expansion exceeds the amount of header padding. This can be beneficial when it is known that some metadata will be added at a future date. The operators which benefit most from judicious use of header padding are ncatted and ncrename, since they only alter metadata.

This optimization exploits the netCDF library nc__enddef() function, which behaves differently with different versions of netCDF. It will improve speed of future metadata expansion with CLASSIC and 64bit netCDF files, though not necessarily with NETCDF4 files, i.e., those created by the netCDF interface to the HDF5 library (see File Formats and Conversion).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.3 OpenMP Threading

Availability: ncclimo, ncks, ncremap
Short options: ‘-t
Long options: ‘--thr_nbr’, ‘--threads’, ‘--omp_num_threads

NCO supports shared memory parallelism (SMP) when compiled with an OpenMP-enabled compiler. Threads requests and allocations occur in two stages. First, users may request a specific number of threads thr_nbr with the ‘-t’ switch (or its long option equivalents, ‘--thr_nbr’, ‘--threads’, and ‘--omp_num_threads’). If not user-specified, OpenMP obtains thr_nbr from the OMP_NUM_THREADS environment variable, if present, or from the OS, if not.

Caveat: Unfortunately, threading does not improve NCO throughput (i.e., wallclock time) because nearly all NCO operations are I/O-bound. This means that NCO spends negligible time doing anything compared to reading and writing. The only exception is regridding with ncremap which uses ncks under-the-hood. As of 2017, threading works only for regridding, thus this section is relevant only to ncclimo, ncks, and ncremap. We have seen some and can imagine other use cases where ncwa, ncpdq, and ncap2 (with long scripts) will complete faster due to threading. The main benefits of threading so far have been to isolate the serial from parallel portions of code. This parallelism is now exploited by OpenMP but then runs into the I/O bottleneck during output. The bottleneck will be ameliorated for large files by the use of MPI-enabled calls in the netCDF4 library when the underlying filesystem is parallel (e.g., PVFS or JFS). Implementation of the parallel output calls in NCO is not a goal of our current funding and would require new volunteers or funding.

NCO may modify thr_nbr according to its own internal settings before it requests any threads from the system. Certain operators contain hard-code limits to the number of threads they request. We base these limits on our experience and common sense, and to reduce potentially wasteful system usage by inexperienced users. For example, ncrcat is extremely I/O-intensive so we restrict thr_nbr <= 2 for ncrcat. This is based on the notion that the best performance that can be expected from an operator which does no arithmetic is to have one thread reading and one thread writing simultaneously. In the future (perhaps with netCDF4), we hope to demonstrate significant threading improvements with operators like ncrcat by performing multiple simultaneous writes.

Compute-intensive operators (ncremap) benefit most from threading. The greatest increases in throughput due to threading occur on large datasets where each thread performs millions, at least, of floating-point operations. Otherwise, the system overhead of setting up threads probably outweighs the speed enhancements due to SMP parallelism. However, we have not yet demonstrated that the SMP parallelism scales beyond four threads for these operators. Hence we restrict thr_nbr <= 4 for all operators. We encourage users to play with these limits (edit file nco_omp.c) and send us their feedback.

Once the initial thr_nbr has been modified for any operator-specific limits, NCO requests the system to allocate a team of thr_nbr threads for the body of the code. The operating system then decides how many threads to allocate based on this request. Users may keep track of this information by running the operator with dbg_lvl > 0.

By default, threaded operators attach one global attribute, nco_openmp_thread_number, to any file they create or modify. This attribute contains the number of threads the operator used to process the input files. This information helps to verify that the answers with threaded and non-threaded operators are equal to within machine precision. This information is also useful for benchmarking.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.4 Command Line Options

Availability: All operators

NCO achieves flexibility by using command line options. These options are implemented in all traditional UNIX commands as single letter switches, e.g., ‘ls -l’. For many years NCO used only single letter option names. In late 2002, we implemented GNU/POSIX extended or long option names for all options. This was done in a backward compatible way such that the full functionality of NCO is still available through the familiar single letter options. Many features of NCO introduced since 2002 now require the use of long options, simply because we have nearly run out of single letter options. More importantly, mnemonics for single letter options are often non-intuitive so that long options provide a more natural way of expressing intent.

Extended options, also called long options, are implemented using the system-supplied getopt.h header file, if possible. This provides the getopt_long function to NCO 19.

The syntax of short options (single letter options) is -key value (dash-key-space-value). Here, key is the single letter option name, e.g., ‘-D 2’.

The syntax of long options (multi-letter options) is --long_name value (dash-dash-key-space-value), e.g., ‘--dbg_lvl 2’ or --long_name=value (dash-dash-key-equal-value), e.g., ‘--dbg_lvl=2’. Thus the following are all valid for the ‘-D’ (short version) or ‘--dbg_lvl’ (long version) command line option.

ncks -D 3        # Short option, preferred form
ncks -D3         # Short option, alternate form
ncks --dbg_lvl=3 # Long option, preferred form
ncks --dbg_lvl 3 # Long option, alternate form

The third example is preferred for two reasons. First, ‘--dbg_lvl’ is more specific and less ambiguous than ‘-D’. The long option format makes scripts more self documenting and less error-prone. Often long options are named after the source code variable whose value they carry. Second, the equals sign = joins the key (i.e., long_name) to the value in an uninterruptible text block. Experience shows that users are less likely to mis-parse commands when restricted to this form.

Next: , Previous: , Up: Command Line Options   [Contents][Index]

3.4.1 Truncating Long Options

GNU implements a superset of the POSIX standard. Their superset accepts any unambiguous truncation of a valid option:

ncks -D 3        # Short option
ncks --dbg_lvl=3 # Long option, full form
ncks --dbg=3     # Long option, OK unambiguous truncation
ncks --db=3      # Long option, OK unambiguous truncation
ncks --d=3       # Long option, ERROR ambiguous truncation

The first four examples are equivalent and will work as expected. The final example will exit with an error since ncks cannot disambiguate whether ‘--d’ is intended as a truncation of ‘--dbg_lvl’, of ‘--dimension’, or of some other long option.

NCO provides many long options for common switches. For example, the debugging level may be set in all operators with any of the switches ‘-D’, ‘--debug-level’, or ‘--dbg_lvl’. This flexibility allows users to choose their favorite mnemonic. For some, it will be ‘--debug’ (an unambiguous truncation of ‘--debug-level’, and other will prefer ‘--dbg’. Interactive users usually prefer the minimal amount of typing, i.e., ‘-D’. We recommend that re-usable scripts employ long options to facilitate self-documentation and maintainability.

This manual generally uses the short option syntax in examples. This is for historical reasons and to conserve space in printed output. Users are expected to pick the unambiguous truncation of each option name that most suits their taste.

Previous: , Up: Command Line Options   [Contents][Index]

3.4.2 Multi-arguments

As of NCO version 4.6.2 (November, 2016), NCO accepts multiple key-value pair options for a single feature to be joined together into a single extended argument called a multi-argument, sometimes abbreviated MTA. Only four NCO features accept multiple key-value pairs that can be aggregated into multi-arguments. These features are: Global Attribute Addition options indicated via ‘--gaa’ (see Global Attribute Addition); Image Manipulation indicated via ‘--trr20, Precision-Preserving Compression options indicated via ‘--ppc’ (see Precision-Preserving Compression); and Regridding options are indicated via ‘--rgr’ (see Regridding). Arguments to these four indicator options take the form of key-value pairs, e.g., ‘--rgr key=val’. These four features have so many options that making each key its own command line option would pollute the namespace of NCO’s global options. Yet supplying multiple options to each indicator option one-at-a-time can result in command lines overpopulated with indicator switches (e.g., ‘--rgr’):

ncks --rgr grd_ttl='Title' --rgr --rgr latlon=129,256 \
     --rgr lat_typ=fv --rgr lon_typ=grn_ctr ...

Multi-arguments combine all the indicator options into one option that receives a single argument that comprises all the original arguments glued together by a delimiter, which is, by default, ‘#’. Thus the multi-argument version of the above example is

ncks --rgr grd_ttl='Title',256#lat_typ=fv#lon_typ=grn_ctr

Note the aggregation of all key=val pairs into a single argument. NCO simply splits this argument at each delimiter, and processes the sub-arguments as if they had been passed with their own indicator option. Multi-arguments produce the same results, and may be mixed with, traditional indicator options supplied one-by-one.

As mentioned previously, the multi-argument delimiter string is, by default, the hash-sign ‘#’. When any key=val pair contains the default delimiter, the user must specify a custom delimiter string so that options are parsed correctly. The options to change the multi-argument delimiter string are ‘--mta_dlm=delim_string’ or ‘--dlm_mta=delim_string’, where delim_string can be any single or multi-character string that (1) is not contained in any key or val string; and (2) will not confuse the shell. For example, to use multi-arguments to pass a string that includes the hash symbol (the default delimiter is ‘#’), one must also change the delimiter so something besides hash, e.g., a colon ‘:’:

ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="hash # is in value" 
ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="Thu Sep 15 13\:03\:18 PDT 2016"
ncks --dlm="csz" --gaa foo=barcszfoo2=bar2cszfoo3,foo4="Long text"

In the second example, the colons that are escaped with the backslash become literal characters. Many characters have special shell meanings and so must be escaped by a single or double backslash or enclosed in single quotes to prevent interpolation. These special characters include ‘:’, ‘$’, ‘%’, ‘*’, ‘@’, and ‘&’. If val is a long text string that could contain the default delimiter, then delimit with a unique multi-character string such as ‘csz’ in the third example.

As of NCO version 4.6.7 (May, 2017), multi-argument flags no longer need be specified as key-value pairs. By definition a flag sets a boolean value to either True or False. Previously MTA flags had to employ key-value pair syntax, e.g., ‘--rgr infer=Y’ or ‘--rgr no_cll_msr=anything’ in order to parse correctly. Now the MTA parser accepts flags in the more intuitive syntax where they are listed by name, i.e., the flag name alone indicates the flag to set, e.g., ‘--rgr infer’ or ‘--rgr no_cll_msr’ are valid. A consequence of this is that flags in multi-argument strings appear as straightforward flag names, e.g., ‘--rgr infer#no_cll_msr#latlon=129,256’. It is also valid to prefix flags in multi-arument strings with single or double-dashes to make the flags more visible, e.g., ‘--rgr latlon=129,256#--infer#-no_cll_msr’.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.5 Sanitization of Input

Availability: All operators

NCO is often installed in system directories (although not with Conda), and on some production machines it may have escalated privileges. Since NCO manipulates files by using system calls (e.g., to move and copy them) it makes sense to audit it for vulnerabilities and protect it from malicious users trying to exploit security gaps. Securing NCO against malicious attacks is multi-faceted, and involves careful memory management and auditing of user-input. As of version 4.7.3 (March, 2018), NCO implements a whitelist of characters allowed in user-specified filenames. The purpose of the whitelist is to prevent users from injecting malicious strings into filenames that could be used by attackers. The whitelist allows only these characters:

1234567890_-.@ :%/

The backslash character \ is also whitelisted (Windows only). This whitelist allows filenames to be URLs, include username prefixes, and standard non-alphabetic characters. The implied blacklist includes these characters


This blacklist rules-out strings that may contain dangerous commands and injection attacks. If you would like any of these characters whitelisted, please contact us and include a compelling real-world use-case.

The whitelist method is straightforward, and does not seem to interfere with NCO’s globbing feature. While the whitelist currently applies only to filenames (which are directly handled by system() calls, the method is applicable to other user-input such as variable lists, hyperslab arguments, etc. Hence, the whitelist may be applied to other user-input in the future.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.6 Specifying Input Files

Availability (-n): nces, ncecat, ncra, ncrcat
Availability (-p): All operators
Short options: ‘-n’, ‘-p
Long options: ‘--nintap’, ‘--pth’, ‘--path

It is important that users be able to specify multiple input files without typing every filename in full, often a tedious task even by graduate student standards. There are four different ways of specifying input files to NCO: explicitly typing each, using UNIX shell wildcards, and using the NCO-n’ and ‘-p’ switches (or their long option equivalents, ‘--nintap’ or ‘--pth’ and ‘--path’, respectively). Techniques to augment these methods to specify arbitrary numbers (e.g., thousands) and patterns of filenames are discussed separately (see Large Numbers of Files).

To illustrate these methods, consider the simple problem of using ncra to average five input files,,, …, and store the results in Here are the four methods in order. They produce identical answers.

ncra 8[56789].nc
ncra 8?.nc
ncra -p input-path
ncra -n 5,2,1

The first method (explicitly specifying all filenames) works by brute force. The second method relies on the operating system shell to glob (expand) the regular expression 8[56789].nc. The shell then passes the valid filenames (those which match the regular expansion) to ncra. In this case ncra never knows that a regular expression was used, because the shell intercepts and expands and matches the regular expression before ncra is actually invoked. The third method is uses globbing with a different regular expression that is less safe (it will also match unwanted files such as and if present). The fourth method uses the ‘-p input-path’ argument to specify the directory where all the input files reside. NCO prepends input-path (e.g., /data/username/model) to all input-files (though not to output-file). Thus, using ‘-p’, the path to any number of input files need only be specified once. Note input-path need not end with ‘/’; the ‘/’ is automatically generated if necessary.

The last method passes (with ‘-n’) syntax concisely describing the entire set of filenames 21. This option is only available with the multi-file operators: ncra, ncrcat, nces, and ncecat. By definition, multi-file operators are able to process an arbitrary number of input-files. This option is very useful for abbreviating lists of filenames representable as alphanumeric_prefix+numeric_suffix+.+filetype where alphanumeric_prefix is a string of arbitrary length and composition, numeric_suffix is a fixed width field of digits, and filetype is a standard filetype indicator. For example, in the file, we have alphanumeric_prefix = ccm3_h, numeric_suffix = 0001, and filetype = nc.

NCO decodes lists of such filenames encoded using the ‘-n’ syntax. The simpler (three-argument) ‘-n’ usage takes the form -n file_number,digit_number,numeric_increment where file_number is the number of files, digit_number is the fixed number of numeric digits comprising the numeric_suffix, and numeric_increment is the constant, integer-valued difference between the numeric_suffix of any two consecutive files. The value of alphanumeric_prefix is taken from the input file, which serves as a template for decoding the filenames. In the example above, the encoding -n 5,2,1 along with the input file name tells NCO to construct five (5) filenames identical to the template except that the final two (2) digits are a numeric suffix to be incremented by one (1) for each successive file. Currently filetype may be either be empty, nc, h5, cdf, hdf, hd5, or he5. If present, these filetype suffixes (and the preceding .) are ignored by NCO as it uses the ‘-n’ arguments to locate, evaluate, and compute the numeric_suffix component of filenames.

Recently the ‘-n’ option has been extended to allow convenient specification of filenames with “circular” characteristics. This means it is now possible for NCO to automatically generate filenames which increment regularly until a specified maximum value, and then wrap back to begin again at a specified minimum value. The corresponding ‘-n’ usage becomes more complex, taking one or two additional arguments for a total of four or five, respectively: -n file_number,digit_number,numeric_increment[,numeric_max[,numeric_min]] where numeric_max, if present, is the maximum integer-value of numeric_suffix and numeric_min, if present, is the minimum integer-value of numeric_suffix. Consider, for example, the problem of specifying non-consecutive input files where the filename suffixes end with the month index. In climate modeling it is common to create summertime and wintertime averages which contain the averages of the months June–July–August, and December–January–February, respectively:

ncra -n 3,2,1
ncra -n 3,2,1,12
ncra -n 3,2,1,12,1

The first example shows that three arguments to the ‘-n’ option suffice to specify consecutive months (06, 07, 08) which do not “wrap” back to a minimum value. The second example shows how to use the optional fourth and fifth elements of the ‘-n’ option to specify a wrap value. The fourth argument to ‘-n’, when present, specifies the maximum integer value of numeric_suffix. In the example the maximum value is 12, and will be formatted as 12 in the filename string. The fifth argument to ‘-n’, when present, specifies the minimum integer value of numeric_suffix. The default minimum filename suffix is 1, which is formatted as 01 in this case. Thus the second and third examples have the same effect, that is, they automatically generate, in order, the filenames,, and as input to NCO.

As of NCO version 4.5.2 (September, 2015), NCO supports an optional sixth argument to ‘-n’, the month-indicator. The month-indicator affirms to NCO that the right-most digits being manipulated in the generated filenames correspond to month numbers (with January formatted as 01 and December as 12). Moreover, it assumes digits to the left of the month are the year. The full (six-argument) ‘-n’ usage takes the form -n file_number,digit_number,month_increment,max_month,min_month,‘yyyymm. The ‘yyyymm’ string is a clunky way (can you think of a clearer way?) to tell NCO to enumerate files in year-month mode. When present, ‘yyyymm’ string causes NCO to automatically generate series of filenames whose right-most two digits increment from min_month by month_increment up to max_month and then the leftmost digits (i.e., the year) increment by one, and the whole process is reapeated until the file_number filenames are generated.

ncrcat -n 3,6,1,12,1
ncrcat -n 3,6,1,12,1,yyyymm
ncrcat -n 3,6,1,12,12,yyyymm

The first command concatenates three files (,, into the output file. The second command concatenates three files (,, The ‘yyyymm’-indicator causes the left-most digits to increment each time the right-most two digits reach their maximum and then wrap. The first command does not have the indicator so it is always 1985. The third command concatenates three files (,,

Next: , Previous: , Up: Shared features   [Contents][Index]

3.7 Specifying Output Files

Availability: All operators
Short options: ‘-o
Long options: ‘--fl_out’, ‘--output

NCO commands produce no more than one output file, fl_out. Traditionally, users specify fl_out as the final argument to the operator, following all input file names. This is the positional argument method of specifying input and ouput file names. The positional argument method works well in most applications. NCO also supports specifying fl_out using the command line switch argument method, ‘-o fl_out’.

Specifying fl_out with a switch, rather than as a positional argument, allows fl_out to precede input files in the argument list. This is particularly useful with multi-file operators for three reasons. Multi-file operators may be invoked with hundreds (or more) filenames. Visual or automatic location of fl_out in such a list is difficult when the only syntactic distinction between input and output files is their position. Second, specification of a long list of input files may be difficult (see Large Numbers of Files). Making the input file list the final argument to an operator facilitates using xargs for this purpose. Some alternatives to xargs are heinous and undesirable. Finally, many users are more comfortable specifying output files with ‘-o fl_out’ near the beginning of an argument list. Compilers and linkers are usually invoked this way.

Users should specify fl_out using either (not both) method. If fl_out is specified twice (once with the switch and once as the last positional argument), then the positional argument takes precedence.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.8 Accessing Remote Files

Availability: All operators
Short options: ‘-p’, ‘-l
Long options: ‘--pth’, ‘--path’, ‘--lcl’, ‘--local

All NCO operators can retrieve files from remote sites as well as from the local file system. A remote site can be an anonymous FTP server, a machine on which the user has rcp, scp, or sftp privileges, NCAR’s Mass Storage System (MSS), or an OPeNDAP server. Examples of each are given below, following a brief description of the particular access protocol.

To access a file via an anonymous FTP server, simply supply the remote file’s URL. Anonymous FTP usually requires no further credentials, e.g., no .netrc file is necessary. FTP is an intrinsically insecure protocol because it transfers passwords in plain text format. Users should access sites using anonymous FTP, or better yet, secure FTP (SFTP, see below) when possible. Some FTP servers require a login/password combination for a valid user account. NCO allows transactions that require additional credentials so long as the required information is stored in the .netrc file. Usually this information is the remote machine name, login, and password, in plain text, separated by those very keywords, e.g.,

machine login zender password bushlied

Eschew using valuable passwords for FTP transactions, since .netrc passwords are potentially exposed to eavesdropping software 22.

SFTP, i.e., secure FTP, uses SSH-based security protocols that solve the security issues associated with plain FTP. NCO supports SFTP protocol access to files specified with a homebrew syntax of the form


Note the second colon following the top-level-domain, tld. This syntax is a hybrid between an FTP URL and standard remote file syntax.

To access a file using rcp or scp, specify the Internet address of the remote file. Of course in this case you must have rcp or scp privileges which allow transparent (no password entry required) access to the remote machine. This means that ~/.rhosts or ~/ssh/authorized_keys must be set accordingly on both local and remote machines.

To access a file on a High Performance Storage System (HPSS) (such as that at NCAR, ECMWF, LANL, DKRZ, LLNL) specify the full HPSS pathname of the remote file. NCO will attempt to detect whether the local machine has direct (synchronous) HPSS access. In this case, NCO attempts to use the Hierarchical Storage Interface (HSI) command hsi get 23.

The following examples show how one might analyze files stored on remote systems.

ncks -l .
ncks -l . s
ncks -l .
ncks -l . /ZENDER/nco/ # NCAR (broken old MSS path)
ncks -l . /home/zender/nco/ # NCAR
ncks -l . 

The first example works verbatim if your system is connected to the Internet and is not behind a firewall. The second example works if you have sftp access to the machine The third example works if you have rcp or scp access to the machine The fourth and fifth examples work on NCAR computers with local access to the HPSS hsi get command 24. The sixth command works if your local version of NCO is OPeNDAP-enabled (this is fully described in OPeNDAP), or if the remote file is accessible via wget. The above commands can be rewritten using the ‘-p input-path’ option as follows:

ncks -p -l .
ncks -p s -l .
ncks -p -l .
ncks -p /ZENDER/nco -l .
ncks -p /home/zender/nco -l . # HPSS
ncks -p \ 
     -l .

Using ‘-p’ is recommended because it clearly separates the input-path from the filename itself, sometimes called the stub. When input-path is not explicitly specified using ‘-p’, NCO internally generates an input-path from the first input filename. The automatically generated input-path is constructed by stripping the input filename of everything following the final ‘/’ character (i.e., removing the stub). The ‘-l output-path’ option tells NCO where to store the remotely retrieved file. It has no effect on locally-retrieved files, or on the output file. Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file. If ‘-l’ is not specified then NCO internally generates an output-path by simply setting output-path equal to input-path stripped of any machine names. If ‘-l’ is not specified and the remote file resides on the NCAR HPSS system, then the leading character of input-path, ‘/’, is also stripped from output-path. Specifying output-path as ‘-l ./’ tells NCO to store the remotely retrieved file and the output file in the current directory. Note that ‘-l .’ is equivalent to ‘-l ./’ though the latter is syntactically more clear.

Previous: , Up: Remote storage   [Contents][Index]

3.8.1 OPeNDAP

The Distributed Oceanographic Data System (DODS) provides useful replacements for common data interface libraries like netCDF. The DODS versions of these libraries implement network transparent access to data via a client-server data access protocol that uses the HTTP protocol for communication. Although DODS-technology originated with oceanography data, it applyies to virtually all scientific data. In recognition of this, the data access protocol underlying DODS (which is what NCO cares about) has been renamed the Open-source Project for a Network Data Access Protocol, OPeNDAP. We use the terms DODS and OPeNDAP interchangeably, and often write OPeNDAP/DODS for now. In the future we will deprecate DODS in favor of DAP or OPeNDAP, as appropriate 25.

NCO may be DAP-enabled by linking NCO to the OPeNDAP libraries. This is described in the OPeNDAP documentation and automagically implemented in NCO build mechanisms 26. The ./configure mechanism automatically enables NCO as OPeNDAP clients if it can find the required OPeNDAP libraries 27. in the usual locations. The $DODS_ROOT environment variable may be used to override the default OPeNDAP library location at NCO compile-time. Building NCO with bld/Makefile and the command make DODS=Y adds the (non-intuitive) commands to link to the OPeNDAP libraries installed in the $DODS_ROOT directory. The file doc/ contains a generic script intended to help users install OPeNDAP before building NCO. The documentation at the OPeNDAP Homepage is voluminous. Check there and on the DODS mail lists. to learn more about the extensive capabilities of OPeNDAP 28.

Once NCO is DAP-enabled the operators are OPeNDAP clients. All OPeNDAP clients have network transparent access to any files controlled by a OPeNDAP server. Simply specify the input file path(s) in URL notation and all NCO operations may be performed on remote files made accessible by a OPeNDAP server. This command tests the basic functionality of OPeNDAP-enabled NCO clients:

% ncks -O -o ~/ -C -H -v one -l /tmp \
% ncks -H -v one ~/
one = 1

The one = 1 outputs confirm (first) that ncks correctly retrieved data via the OPeNDAP protocol and (second) that ncks created a valid local copy of the subsetted remote file. With minor changes to the above command, netCDF4 can be used as both the input and output file format:

% ncks -4 -O -o ~/ -C -H -v one -l /tmp \
% ncks -H -v one ~/
one = 1

And, of course, OPeNDAP-enabled NCO clients continue to support orthogonal features such as UDUnits (see UDUnits Support):

% ncks -u -C -H -v wvl -d wvl,'0.4 micron','0.7 micron' \
% wvl[0]=5e-07 meter

The next command is a more advanced example which demonstrates the real power of OPeNDAP-enabled NCO clients. The ncwa client requests an equatorial hyperslab from remotely stored NCEP reanalyses data of the year 1969. The NOAA OPeNDAP server (hopefully!) serves these data. The local ncwa client then computes and stores (locally) the regional mean surface pressure (in Pa).

ncwa -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \ \ ~/

All with one command! The data in this particular input file also happen to be packed (see Methods and functions), although this complication is transparent to the user since NCO automatically unpacks data before attempting arithmetic.

NCO obtains remote files from the OPeNDAP server (e.g., rather than the local machine. Input files are first copied to the local machine, then processed. The OPeNDAP server performs data access, hyperslabbing, and transfer to the local machine. This allows the I/O to appear to NCO as if the input files were local. The local machine performs all arithmetic operations. Only the hyperslabbed output data are transferred over the network (to the local machine) for the number-crunching to begin. The advantages of this are obvious if you are examining small parts of large files stored at remote locations.

Natually there are many versions of OPeNDAP servers supplying data and bugs in the server can appear to be bugs in NCO. However, with very few exceptions 29 an NCO command that works on a local file must work across an OPeNDAP connection or else there is a bug in the server. This is because NCO does nothing special to handle files served by OPeNDAP, the whole process is (supposed to be) completely transparent to the client NCO software. Therefore it is often useful to try NCO commands on various OPeNDAP servers in order to isolate whether a problem may be due to a bug in the OPeNDAP server on a particular machine. For this purpose, one might try variations of the following commands that access files on public OPeNDAP servers:

# Strided access to HDF5 file
ncks -v Time -d Time,0,10,2
# Strided access to netCDF3 file
ncks -O -D 1 -d time,1 -d lev,0 -d lat,0,100,10 -d lon,0,100,10 -v u_velocity ~/

These servers were operational at the time of writing, March 2014. Unfortunately, administrators often move or rename path directories. Recommendations for additional public OPeNDAP servers on which to test NCO are welcome.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.9 Retaining Retrieved Files

Availability: All operators
Short options: ‘-R
Long options: ‘--rtn’, ‘--retain

In order to conserve local file system space, files retrieved from remote locations are automatically deleted from the local file system once they have been processed. Many NCO operators were constructed to work with numerous large (e.g., 200 MB) files. Retrieval of multiple files from remote locations is done serially. Each file is retrieved, processed, then deleted before the cycle repeats. In cases where it is useful to keep the remotely-retrieved files on the local file system after processing, the automatic removal feature may be disabled by specifying ‘-R’ on the command line.

Invoking -R disables the default printing behavior of ncks. This allows ncks to retrieve remote files without automatically trying to print them. See ncks netCDF Kitchen Sink, for more details.

Note that the remote retrieval features of NCO can always be used to retrieve any file, including non-netCDF files, via SSH, anonymous FTP, or msrcp. Often this method is quicker than using a browser, or running an FTP session from a shell window yourself. For example, say you want to obtain a JPEG file from a weather server.

ncks -R -p -l . storm.jpg

In this example, ncks automatically performs an anonymous FTP login to the remote machine and retrieves the specified file. When ncks attempts to read the local copy of storm.jpg as a netCDF file, it fails and exits, leaving storm.jpg in the current directory.

If your NCO is DAP-enabled (see OPeNDAP), then you may use NCO to retrieve any files (including netCDF, HDF, etc.) served by an OPeNDAP server to your local machine. For example,

ncks -R -l . -p \ \

It may occasionally be useful to use NCO to transfer files when your other preferred methods are not available locally.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.10 File Formats and Conversion

Availability: ncap2, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-3’, ‘-4’, ‘-5’, ‘-6’, ‘-7
Long options: ‘--3’, ‘--4’, ‘--5’, ‘--6’, ‘--64bit_offset’, ‘--7’, ‘--fl_fmt’, ‘--netcdf4

All NCO operators support (read and write) all three (or four, depending on how one counts) file formats supported by netCDF4. The default output file format for all operators is the input file format. The operators listed under “Availability” above allow the user to specify the output file format independent of the input file format. These operators allow the user to convert between the various file formats. (The operators ncatted and ncrename do not support these switches so they always write the output netCDF file in the same format as the input netCDF file.)

Next: , Previous: , Up: File Formats and Conversion   [Contents][Index]

3.10.1 File Formats

netCDF supports five types of files: CLASSIC, 64BIT_OFFSET, 64BIT_DATA, NETCDF4, and NETCDF4_CLASSIC. The CLASSIC format is the traditional 32-bit offset written by netCDF2 and netCDF3. As of 2005, nearly all netCDF datasets were in CLASSIC format. The 64BIT_OFFSET (originally called plain old 64BIT) format was added in Fall, 2004. As of 2010, many netCDF datasets were in 64BIT_OFFSET format. As of 2013, an increasing number of netCDF datasets were in NETCDF4_CLASSIC format. The 64BIT_DATA (aka CDF5 or PNETCDF) format was added in January, 2016.

The NETCDF4 format uses HDF5 as the file storage layer. The files are (usually) created, accessed, and manipulated using the traditional netCDF3 API (with numerous extensions). The NETCDF4_CLASSIC format refers to netCDF4 files created with the NC_CLASSIC_MODEL mask. Such files use HDF5 as the back-end storage format (unlike netCDF3), though they incorporate only netCDF3 features. Hence NETCDF4_CLASSIC files are entirely readable by applications that use only the netCDF3 API (though the applications must be linked with the netCDF4 library). NCO must be built with netCDF4 to write files in the new NETCDF4 and NETCDF4_CLASSIC formats, and to read files in these formats. Datasets in the default CLASSIC or the newer 64BIT_OFFSET formats have maximum backwards-compatibility with older applications. NCO has deep support for NETCDF4 formats. If backwards compatibility is important, and your datasets are too large for netCDF3, use NETCDF4_CLASSIC instead of CLASSIC format files. NCO support for the NETCDF4 format is complete and many high-performance disk/RAM efficient workflows utilize this format.

As mentioned above, all operators write use the input file format for output files unless told otherwise. Toggling the short option ‘-6’ or the long option ‘--6’ or ‘--64bit_offset’ (or their key-value equivalent ‘--fl_fmt=64bit_offset’) produces the netCDF3 64-bit offset format named 64BIT_OFFSET. NCO must be built with netCDF 3.6 or higher to produce a 64BIT_OFFSET file. As of NCO version 4.6.9 (September, 2017), toggling the short option ‘-5’ or the long options ‘--5’, ‘--64bit_data’, ‘--cdf5’, or ‘--pnetcdf’ (or their key-value equivalent ‘--fl_fmt=64bit_data’) produces the netCDF3 64-bit data format named 64BIT_DATA. This format is widely used by MPI-enabled modeling codes because of its long association with PnetCDF. NCO must be built with netCDF 4.4 or higher to produce a 64BIT_DATA file.

Using the ‘-4’ switch (or its long option equivalents ‘--4’ or ‘--netcdf4’), or setting its key-value equivalent ‘--fl_fmt=netcdf4’ produces a NETCDF4 file (i.e., with all supported HDF5 features). Using the ‘-7’ switch (or its long option equivalent ‘--730, or setting its key-value equivalent ‘--fl_fmt=netcdf4_classic’ produces a NETCDF4_CLASSIC file (i.e., with all supported HDF5 features like compression and chunking but without groups or new atomic types). Operators given the ‘-3’ (or ‘--3’) switch without arguments will (attempt to) produce netCDF3 CLASSIC output, even from netCDF4 input files.

Note that NETCDF4 and NETCDF4_CLASSIC are the same binary format. The latter simply causes a writing application to fail if it attempts to write a NETCDF4 file that cannot be completely read by the netCDF3 library. Conversely, NETCDF4_CLASSIC indicates to a reading application that all of the file contents are readable with the netCDF3 library. NCO has supported reading/writing basic NETCDF4 and NETCDF4_CLASSIC files since October, 2005.

Next: , Previous: , Up: File Formats and Conversion   [Contents][Index]

3.10.2 Determining File Format

Input files often end with the generic .nc suffix that leaves (perhaps by intention) the internal file format ambiguous. There are at least three ways to discover the internal format of a netCDF-supported file. These methods determine whether it is a classic (32-bit offset) or newer 64-bit offset netCDF3 format, or is a netCDF4 format. Each method returns the information using slightly different terminology that becomes easier to understand with practice.

First, examine the first line of global metadata output by ‘ncks -M’:

% ncks -M
Summary of filetype = NC_FORMAT_CLASSIC, 0 groups ...
% ncks -M
Summary of filetype = NC_FORMAT_64BIT_OFFSET, 0 groups ...
% ncks -M
Summary of filetype = NC_FORMAT_CDF5, 0 groups ...
% ncks -M
Summary of filetype = NC_FORMAT_NETCDF4_CLASSIC, 0 groups ...
% ncks -M
Summary of filetype = NC_FORMAT_NETCDF4, 0 groups ...

This method requires a netCDF4-enabled NCO version 3.9.0+ (i.e., from 2007 or later). As of NCO version 4.4.0 (January, 2014), ncks will also print the extended or underlying format of the input file. The extended filetype will be one of the six underlying formats that are accessible through the netCDF API. These formats are NC_FORMATX_NC3 (classic and 64-bit versions of netCDF3 formats), NC_FORMATX_NC_HDF5 (classic and extended versions of netCDF4, and “pure” HDF5 format), NC_FORMATX_NC_HDF4 (HDF4 format), NC_FORMATX_PNETCDF (PnetCDF format), NC_FORMATX_DAP2 (accessed via DAP2 protocol), and NC_FORMATX_DAP4 (accessed via DAP4 protocol). For example,

% ncks -D 2 -M hdf.hdf
Summary of hdf.hdf: filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF4), 0 groups ...
% ncks -D 2 -M
Summary of \
  filetype = NC_FORMAT_CLASSIC (representation of extended/underlying \
  filetype NC_FORMATX_DAP2), 0 groups  
% ncks -D 2 -M
Summary of filetype = NC_FORMAT_NETCDF4 (representation of \
  extended/underlying filetype NC_FORMAT_HDF5), 0 groups  

The extended filetype determines some of the capabilities that netCDF has to alter the file.

Second, query the file with ‘ncdump -k’:

% ncdump -k
% ncdump -k
64-bit offset
% ncdump -k
% ncdump -k
netCDF-4 classic model
% ncdump -k

This method requires a netCDF4-enabled netCDF 3.6.2+ (i.e., from 2007 or later).

The third option uses the POSIX-standard od (octal dump) command:

% od -An -c -N4
   C   D   F 001
% od -An -c -N4
   C   D   F 002
% od -An -c -N4
   C   D   F 005
% od -An -c -N4
 211   H   D   F
% od -An -c -N4
 211   H   D   F

This option works without NCO and ncdump. Values of ‘C D F 001’ and ‘C D F 002’ indicate 32-bit (classic) and 64-bit netCDF3 formats, respectively, while values of ‘211 H D F’ indicate either of the newer netCDF4 file formats.

Next: , Previous: , Up: File Formats and Conversion   [Contents][Index]

3.10.3 File Conversion

Let us demonstrate converting a file from any netCDF-supported input format into any netCDF output format (subject to limits of the output format). Here the input file may be in any of these formats: netCDF3 (classic, 64bit_offset, 64bit_data), netCDF4 (classic and extended), HDF4, HDF5, HDF-EOS (version 2 or 5), and DAP. The switch determines the output format written in the comment: 31

ncks --fl_fmt=classic # netCDF3 classic
ncks --fl_fmt=64bit_offset # netCDF3 64bit-offset
ncks --fl_fmt=64bit_data # netCDF3 64bit-data
ncks --fl_fmt=cdf5 # netCDF3 64bit-data
ncks --fl_fmt=netcdf4_classic # netCDF4 classic
ncks --fl_fmt=netcdf4 # netCDF4 
ncks -3 # netCDF3 classic
ncks --3 # netCDF3 classic
ncks -6 # netCDF3 64bit-offset
ncks --64 # netCDF3 64bit-offset
ncks -5 # netCDF3 64bit-data
ncks --5 # netCDF3 64bit-data
ncks -4 # netCDF4 
ncks --4 # netCDF4 
ncks -7 # netCDF4 classic
ncks --7 # netCDF4 classic

Of course since most operators support these switches, the “conversions” can be done at the output stage of arithmetic or metadata processing rather than requiring a separate step. Producing (netCDF3) CLASSIC or 64BIT_OFFSET or 64BIT_DATA files from NETCDF4_CLASSIC files always works.

Previous: , Up: File Formats and Conversion   [Contents][Index]

3.10.4 Autoconversion

Because of the dearth of support for netCDF4 amongst tools and user communities (including the CF conventions), it is often useful to convert netCDF4 to netCDF3 for certain applications. Until NCO version 4.4.0 (January, 2014), producing netCDF3 files from netCDF4 files only worked if the input files contained no netCDF4-specific features (e.g., atomic types, multiple record dimensions, or groups). As of NCO version 4.4.0, ncks supports autoconversion of many netCDF4 features to their closest netCDF3-compatible representations. Since converting netCDF4 to netCDF3 results in loss of features, “automatic down-conversion” may be a more precise description of what we term autoconversion.

NCO employs three algorithms to downconvert netCDF4 to netCDF3:

  1. Autoconversion of atomic types: Autoconversion automatically promotes NC_UBYTE to NC_SHORT, and NC_USHORT to NC_INT. It automatically demotes the three types NC_UINT, NC_UINT64, and NC_INT64 to NC_INT. And it converts NC_STRING to NC_CHAR. All numeric conversions work for attributes and variables of any rank. Two numeric types (NC_UBYTE and NC_USHORT) are promoted to types with greater range (and greater storage). This extra range is often not used so promotion perhaps conveys the wrong impression. However, promotion never truncates values or loses data (this perhaps justifies the extra storage). Three numeric types (NC_UINT, NC_UINT64 and NC_INT64) are demoted. Since the input range is larger than the output range, demotion can result in numeric truncation and thus loss of data. In such cases, it would possible to convert the data to floating-point values instead. If this feature interests you, please be the squeaky wheel and let us know.

    String conversions (to NC_CHAR) work for all attributes, but not for variables. This is because attributes are at most one-dimensional and may be of any size whereas variables require gridded dimensions that usually do not fit the ragged sizes of text strings. Hence scalar NC_STRING attributes are correctly converted to and stored as NC_CHAR attributes in the netCDF3 output file, but NC_STRING variables are not correctly converted. If this limitation annoys or enrages you, please let us know by being the squeaky wheel.

  2. Convert multiple record dimensions to fixed-size dimensions. Many netCDF4 and HDF5 datasets have multiple unlimited dimensions. Since a netCDF3 file may have at most one unlimited dimension, all but possibly one unlimited dimension from the input file must be converted to fixed-length dimensions prior to storing netCDF4 input as netCDF3 output. By invoking --fix_rec_dmn all the user ensures the output file will adhere to netCDF3 conventions and the user need not know the names of the specific record dimensions to fix. See ncks netCDF Kitchen Sink for a description of the ‘--fix_rec_dmn’ option.
  3. Flattening (removal) of groups. Many netCDF4 and HDF5 datasets have group hierarchies. Since a netCDF3 file may not have any groups, groups in the input file must be removed. This is also called “flattening” the hierarchical file. See Group Path Editing for a description of the GPE option ‘-G :’ to flatten files.

Putting the three algorithms together, one sees that the recipe to convert netCDF4 to netCDF4 becomes increasingly complex as the netCDF4 features in the input file become more elaborate:

# Convert file with netCDF4 atomic types
ncks -3 in.nc4 out.nc3
# Convert file with multiple record dimensions + netCDF4 atomic types
ncks -3 --fix_rec_dmn=all in.nc4 out.nc3
# Convert file with groups, multiple record dimensions + netCDF4 atomic types
ncks -3 -G : --fix_rec_dmn=all in.nc4 out.nc3

Future versions of NCO may automatically invoke the record dimension fixation and group flattening when converting to netCDF3 (rather than requiring it be specified manually). If this feature would interest you, please let us know.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.11 Large File Support

Availability: All operators
Short options: none
Long options: none

NCO has Large File Support (LFS), meaning that NCO can write files larger than 2 GB on some 32-bit operating systems with netCDF libraries earlier than version 3.6. If desired, LFS support must be configured when both netCDF and NCO are installed. netCDF versions 3.6 and higher support 64-bit file addresses as part of the netCDF standard. We recommend that users ignore LFS support which is difficult to configure and is implemented in NCO only to support netCDF versions prior to 3.6. This obviates the need for configuring explicit LFS support in applications (such as NCO) that now support 64-bit files directly through the netCDF interface. See File Formats and Conversion for instructions on accessing the different file formats, including 64-bit files, supported by the modern netCDF interface.

If you are still interested in explicit LFS support for netCDF versions prior to 3.6, know that LFS support depends on a complex, interlocking set of operating system 32 and netCDF support issues. The netCDF LFS FAQ describes the various file size limitations imposed by different versions of the netCDF standard. NCO and netCDF automatically attempt to configure LFS at build time.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.12 Subsetting Files

Options -g grp
Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-g
Long options: ‘--grp’ and ‘--group
Options -v var and -x
Availability: (ncap2), ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-v’, ‘-x
Long options: ‘--variable’, ‘--exclude’ or ‘--xcl
Options --unn
Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options:
Long options: ‘--unn’ and ‘--union
Options --grp_xtr_var_xcl
Availability: ncks
Short options:
Long options: ‘--gxvx’ and ‘--grp_xtr_var_xcl

Subsetting variables refers to explicitly specifying variables and groups to be included or excluded from operator actions. Subsetting is controlled by the ‘-v var[,…]’ and ‘-x’ options for directly specifying variables. Specifying groups, whether in addition to or instead of variables, is quite similar and is controlled by the ‘-g grp[,…]’ and ‘-x’ options. A list of variables or groups to extract is specified following the ‘-v’ and ‘-g’ options, e.g., ‘-v time,lat,lon’ or ‘-g grp1,grp2’. Both options may be specified simultaneously and NCO will extract the intersection of the lists, i.e., only variables of the specified names found in groups of the specified names. The ‘--unn’ option causes NCO to extract the union, rather than the intersection, of the specified groups and variables. Not using the ‘-v’ or ‘-g’ option is equivalent to specifying all variables or groupp, respectively. The ‘-x’ option causes the list of variables specified with ‘-v’ to be excluded rather than extracted. Thus ‘-x’ saves typing when you only want to extract fewer than half of the variables in a file.

Variables or groups explicitly specified for extraction with ‘-v var[,…]’ or ‘-g grp[,…]must be present in the input file or an error will result. Variables explicitly specified for exclusion with ‘-x -v var[,…]’ need not be present in the input file. To accord with the sophistication of the underlying hierarchy, group subsetting is controlled by a few powerful yet subtle syntactical distinctions. When learning this syntax it is helpful to keep in mind the similarity between group hierarchies and directory structures.

As of NCO 4.4.4 (June, 2014), ncks (alone) supports an option to include specified groups yet exclude specified variables. The ‘--grp_xtr_var_xcl’ switch (with long option equivalent ‘--gxvx’) extracts all contents of groups given as arguments to ‘-g grp[,…]’, except for variables given as arguments to ‘-v var[,…]’. Use this when one or a few variables in hierarchical files are not to be extracted, and all other variables are. This is useful when coercing netCDF4 files into netCDF3 files such as with converting, flattening, or dismembering files (see Flattening Groups).

ncks --grp_xtr_var_xcl -g g1 -v v1 # Extract all of group g1 except v1

Two properties of subsetting, recursion and anchoring, are best illustrated by reminding the user of their UNIX equivalents. The UNIX command mv src dst moves src and all its subdirectories (and all their subdirectories etc.) to dst. In other words mv is, by default, recursive. In contrast, the UNIX command cp src dst moves src, and only src, to dst, If src is a directory, not a file, then that command fails. One must explicitly request to copy directories recursively, i.e., with cp -r src dst. In NCO recursive extraction (and copying) of groups is the default (like with mv, not with cp). Recursion is turned off by appending a trailing slash to the path.

These UNIX commands also illustrate a property we call anchoring. The command mv src dst moves (recursively) the source directory src to the destination directory dst. If src begins with the slash character then the specified path is relative to the root directory, otherwise the path is relative to the current working directory. In other words, an initial slash character anchors the subsequent path to the root directory. In NCO an initial slash anchors the path at the root group. Paths that begin and end with slash characters (e.g., //, /g1/, and /g1/g2/) are both anchored and non-recursive.

Consider the following commands, all of which may be assumed to end with ‘’:

ncks -g  g1  # Extract, recursively, all groups with a g1 component
ncks -g  g1/ # Extract, non-recursively, all groups terminating in g1
ncks -g /g1  # Extract, recursively, root group g1
ncks -g /g1/ # Extract, non-recursively root group g1
ncks -g //   # Extract, non-recursively the root group

The first command is probably the most useful and common. It would extract these groups, if present, and all their direct ancestors and children: /g1, /g2/g1, and /g3/g1/g2. In other words, the simplest form of ‘-g grp’ grabs all groups that (and their direct ancestors and children, recursively) that have grp as a complete component of their path. A simple string match is insufficient, grp must be a complete component (i.e., group name) in the path. The option ‘-g g1’ would not extract these groups because g1 is not a complete component of the path: /g12, /fg1, and /g1g1. The second command above shows how a terminating slash character / cancels the recursive copying of groups. An argument to ‘-g’ which terminates with a slash character extracts the group and its direct ancestors, but none of its children. The third command above shows how an initial slash character / anchors the argument to the root group. The third command would not extract the group /g2/g1 because the g1 group is not at the root level, but it would extract, any group /g1 at the root level and all its children, recursively. The fourth command is the non-recursive version of the third command. The fifth command is a special case of the fourth command.

As mentioned above, both ‘-v’ and ‘-g’ options may be specified simultaneously and NCO will, by default, extract the intersection of the lists, i.e., the specified variables found in the specified groups 33. The ‘--unn’ option causes NCO to extract the union, rather than the intersection, of the specified groups and variables. Consider the following commands (which may be assumed to end with ‘’):

# Intersection-mode subsetting (default)
ncks -g  g1  -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
ncks -g /g1  -v v1 # Yes: /g1/v1, /g1/g2/v1. No: /v1, /g2/v1, /g2/g1/v1
ncks -g  g1/ -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -v  g1/v1     # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
ncks -g /g1/ -v v1 # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...
ncks -v /g1/v1     # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...

# Union-mode subsetting (invoke with --unn or --union)
ncks -g  g1  -v v1 --unn # All variables in  g1 or progeny, or named v1
ncks -g /g1  -v v1 --unn # All variables in /g1 or progeny, or named v1
ncks -g  g1/ -v v1 --unn # All variables in  g1 or named v1
ncks -g /g1/ -v v1 --unn # All variables in /g1 or named v1

The first command (‘-g g1 -v v1’) extracts the variable v1 from any group named g1 or descendent g1. The second command extracts v1 from any root group named g1 and any descendent groups as well. The third and fourth commands are equivalent ways of extracting v1 only from the root group named g1 (not its descendents). The fifth and sixth commands are equivalent ways of extracting the variable v1 only from the root group named g1. Subsetting in union-mode (with ‘--unn’) causes all variables to be extracted which meet either one or both of the specifications of the variable and group specifications. Union-mode subsetting is simply the logical “OR” of intersection-mode subsetting. As discussed below, the group and variable specifications may be comma separated lists of regular expressions for added control over subsetting.

Remember, if averaging or concatenating large files stresses your systems memory or disk resources, then the easiest solution is often to subset (with ‘-g’ and/or ‘-v’) to retain only the most important variables (see Memory Requirements).

ncks # Extract all groups and variables
ncks -v scl   # Extract variable scl from all groups
ncks -g g1    # Extract group g1 and descendents
ncks -x -g g1 # Extract all groups except g1 and descendents
ncks -g g2,g3 -v scl # Extract scl from groups g2 and g3

Overwriting and appending work as expected:

# Replace scl in group g2 in with scl from group g2 from
ncks -A -g g2 -v scl

Due to its special capabilities, ncap2 interprets the ‘-v’ switch differently (see ncap2 netCDF Arithmetic Processor). For ncap2, the ‘-v’ switch takes no arguments and indicates that only user-defined variables should be output. ncap2 neither accepts nor understands the -x and -g switches.

Regular expressions the syntax that NCO use pattern-match object names in netCDF file against user requests. The user can select all variables beginning with the string ‘DST’ from an input file by supplying the regular expression ‘^DST’ to the ‘-v’ switch, i.e., ‘-v '^DST'’. The meta-characters used to express pattern matching operations are ‘^$+?.*[]{}|’. If the regular expression pattern matches any part of a variable name then that variable is selected. This capability is also called wildcarding, and is very useful for sub-setting large data files.

Extended regular expressions are defined by the POSIX grep -E (aka egrep) command. As of NCO 2.8.1 (August, 2003), variable name arguments to the ‘-v’ switch may contain extended regular expressions. As of NCO 3.9.6 (January, 2009), variable names arguments to ncatted may contain extended regular expressions. As of NCO 4.2.4 (November, 2012), group name arguments to the ‘-g’ switch may contain extended regular expressions.

Because of its wide availability, NCO uses the POSIX regular expression library regex. Regular expressions of arbitary complexity may be used. Since netCDF variable names are relatively simple constructs, only a few varieties of variable wildcards are likely to be useful. For convenience, we define the most useful pattern matching operators here:


Matches the beginning of a string


Matches the end of a string


Matches any single character

The most useful repetition and combination operators are


The preceding regular expression is optional and matched at most once


The preceding regular expression will be matched zero or more times


The preceding regular expression will be matched one or more times


The preceding regular expression will be joined to the following regular expression. The resulting regular expression matches any string matching either subexpression.

To illustrate the use of these operators in extracting variables and groups, consider file with groups g0g9, and subgroups s0s9, in each of those groups, and file with variables Q, Q01Q99, Q100, QAAQZZ, Q_H2O, X_H2O, Q_CO2, X_CO2.

ncks -v '.+'               # All variables (default)
ncks -v 'Q.?'              # Variables that contain Q
ncks -v '^Q.?'             # Variables that start with Q
ncks -v '^Q+.?.'           # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..'             # Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]'     # Q01--Q99, Q100
ncks -v '^Q[[:digit:]]{2}' # Q01--Q99
ncks -v 'H2O$'             # Q_H2O, X_H2O 
ncks -v 'H2O$|CO2$'        # Q_H2O, X_H2O, Q_CO2, X_CO2 
ncks -v '^Q[0-9][0-9]$'    # Q01--Q99
ncks -v '^Q[0-6][0-9]|7[0-3]' # Q01--Q73, Q100
ncks -v '(Q[0-6][0-9]|7[0-3])$' # Q01--Q73
ncks -v '^[a-z]_[a-z]{3}$' # Q_H2O, X_H2O, Q_CO2, X_CO2
ncks -g 'g.'           # 10 Groups g0-g9
ncks -g 's.'       # 100 sub-groups g0/s0, g0/s1, ... g9/s9
ncks -g 'g.' -v 'v.'   # All variables 'v.' in groups 'g.'

Beware—two of the most frequently used repetition pattern matching operators, ‘*’ and ‘?’, are also valid pattern matching operators for filename expansion (globbing) at the shell-level. Confusingly, their meanings in extended regular expressions and in shell-level filename expansion are significantly different. In an extended regular expression, ‘*’ matches zero or more occurences of the preceding regular expression. Thus ‘Q*’ selects all variables, and ‘Q+.*’ selects all variables containing ‘Q’ (the ‘+’ ensures the preceding item matches at least once). To match zero or one occurence of the preceding regular expression, use ‘?’. Documentation for the UNIX egrep command details the extended regular expressions which NCO supports.

One must be careful to protect any special characters in the regular expression specification from being interpreted (globbed) by the shell. This is accomplish by enclosing special characters within single or double quotes

ncra -v Q??   # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc # Correct: NCO interprets, Shell globs 

The final example shows that commands may use a combination of variable wildcarding and shell filename expansion (globbing). For globbing, ‘*’ and ‘?have nothing to do with the preceding regular expression! In shell-level filename expansion, ‘*’ matches any string, including the null string and ‘?’ matches any single character. Documentation for bash and csh describe the rules of filename expansion (globbing).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.13 Subsetting Coordinate Variables

Availability: ncap2, ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-C’, ‘-c
Long options: ‘--no-coords’, ‘--no-crd’, ‘--crd’, ‘--coords

By default, coordinates variables associated with any variable appearing in the input-file will be placed in the output-file, even if they are not explicitly specified, e.g., with the ‘-v’ switch. Thus variables with a latitude coordinate lat always carry the values of lat with them into the output-file. This feature can be disabled with ‘-C’, which causes NCO to not automatically add coordinates to the variables appearing in the output-file. However, using ‘-C’ does not preclude the user from including some coordinates in the output files simply by explicitly selecting the coordinates with the -v option. The ‘-c’ option, on the other hand, is a shorthand way of automatically specifying that all coordinate variables in the input-files should appear in the output-file. Thus ‘-c’ allows the user to select all the coordinate variables without having to know their names. As of NCO version 4.4.5 (July, 2014) both ‘-c’ and ‘-C’ honor the CF ancillary_variables convention described in CF Conventions. As of NCO version 4.0.8 (April, 2011) both ‘-c’ and ‘-C’ honor the CF bounds convention described in CF Conventions. As of NCO version 4.6.4 (January, 2017) both ‘-c’ and ‘-C’ honor the CF cell_measures convention described in CF Conventions. As of NCO version 4.4.9 (May, 2015) both ‘-c’ and ‘-C’ honor the CF climatology convention described in CF Conventions. As of NCO version 3.9.6 (January, 2009) both ‘-c’ and ‘-C’ honor the CF coordinates convention described in CF Conventions. As of NCO version 4.6.4 (January, 2017) both ‘-c’ and ‘-C’ honor the CF formula_terms convention described in CF Conventions. As of NCO version 4.6.0 (May, 2016) both ‘-c’ and ‘-C’ honor the CF grid_mapping convention described in CF Conventions.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.14 Group Path Editing

Options -G gpe_dsc
Availability: ncbo, ncecat, nces, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-G
Long options: ‘--gpe

Group Path Editing, or GPE, allows the user to restructure (i.e., add, remove, and rename groups) in the output file relative to the input file based on the instructions they provide. As of NCO 4.2.3 (November, 2012), all operators that accept netCDF4 files with groups accept the ‘-G’ switch, or its long-option equivalent ‘--gpe’. To master GPE one must understand the meaning of the required gpe_dsc structure/argument that specifies the transformation of input-to-output group paths.

Each gpe_dsc contains up to three elements (two are optional) in the following order:
gpe_dsc = grp_pth:lvl_nbr or grp_pth@lvl_nbr


Group Path. This (optional) component specifies the output group path that should be appended after any editing (i.e., deletion or truncation) of the input path is performed.


The number of levels to delete (from the head) or truncate (from the tail) of the input path.

If both components of the argument are present, then a single character, either the colon or at-sign (: or @), must separate them. If only grp_pth is specifed, the separator character may be omitted, e.g., ‘-G g1’. If only lvl_nbr is specifed, the separator character is still required to indicate it is a lvl_nbr arugment and not a grp_pth, e.g., ‘-G :-1’ or ‘-G @1’.

If the at-sign separator character @ is used instead of the colon separator character :, then the following lvl_nbr arugment must be positive and it will be assumed to refer to Truncation-Mode. Hence, ‘-G :-1’ is the same as ‘-G @1’. This is simply a way of making the lvl_nbr argument positive-definite.

Next: , Previous: , Up: Group Path Editing   [Contents][Index]

3.14.1 Deletion, Truncation, and Flattening of Groups

GPE has three editing modes: Delete, Truncate, and Flatten. Select one of GPE’s three editing modes by supplying a lvl_nbr that is positive, negative, or zero for Delete-, Truncate- and Flatten-mode, respectively.

In Delete-mode, lvl_nbr is a positive integer which specifies the maximum number of group path components (i.e., groups) that GPE will try to delete from the head of grp_pth. For example lvl_nbr = 3 changes the input path /g1/g2/g3/g4/g5 to the output path /g4/g5. Input paths with lvl_nbr or fewer components (groups) are completely erased and the output path commences from the root level.

In other words, GPE is tolerant of specifying too many group components to delete. It deletes as many as possible, without complaint, and then begins to flatten the file (which fails if namespace conflicts arise).

In Truncate-mode, lvl_nbr is a negative integer which specifies the maximum number of group path components (i.e., groups) that GPE will try to truncate from the tail of grp_pth. For example lvl_nbr = -3 changes the input path /g1/g2/g3/g4/g5 to the output path /g1/g2. Input paths with lvl_nbr or fewer components (groups) are completely erased and the output path commences from the root level.

In Flatten-mode, indicated by the separator character alone or with lvl_nbr = 0, GPE removes the entire group path from the input file and constructs the output path beginning at the root level. For example -G :0 and -G : are identical and change the input path /g1/g2/g3/g4/g5 to the output path / whereas -G g1:0 and -G g1: are identical and result in the output path /g1 for all variables.

Subsequent to the alteration of the input path by the specified editing mode, if any, GPE prepends (in Delete Mode) or Appends (in Truncate-mode) any specifed grp_pth to the output path. For example -G g2 changes the input paths / and /g1 to /g2 and /g1/g2, respectively. Likewise, -G g2/g3 changes the input paths / and /g1 to /g2/g3 and /g1/g2/g3, respectively. When grp_pth and lvl_nbr are both specified, the editing actions are taken in sequence so that, e.g., -G g1/g2:2 changes the input paths / and /h1/h2/h3/h4 to /g1/g2 and /g1/g2/h3/h4, respectively. Likewise, -G g1/g2:-2 changes the input paths / and /h1/h2/h3/h4 to /g1/g2 and /h1/h2/g1/g2, respectively.

Combining GPE with subsetting (see Subsetting Files) yields powerful control over the extracted (or excluded) variables and groups and their placement in the output file as shown by the following commands. All commands below may be assumed to end with ‘’.

# Prepending paths without editing:
ncks                   # /g?/v? -> /g?/v?
ncks             -v v1 # /g?/v1 -> /g?/v1
ncks       -g g1       # /g1/v? -> /g1/v?
ncks -G o1             # /g?/v? -> /o1/g?/v?
ncks -G o1 -g g1       # /g1/v? -> /o1/g1/v?
ncks       -g g1 -v v1 # /g1/v1 -> /g1/v1
ncks -G o1       -v v1 # /g?/v1 -> /o1/g?/v1
ncks -G o1 -g g1 -v v1 # /g1/v1 -> /o1/g1/v1
ncks -G g1 -g /  -v v1 # /v1    -> /g1/v1
ncks -G g1/g2    -v v1 # /g?/v1 -> /g1/g2/g?/v1
# Delete-mode: Delete from and Prepend to path head
# Syntax: -G [ppn]:lvl_nbr = # of levels to delete
ncks -G :1    -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :1    -g g1/g1 -v v1 # /g1/g1/v1 -> /g1/v1
ncks -G :2    -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
ncks -G :2    -g g1    -v v1 # /g1/v1    -> /v1
ncks -G g2:1  -g g1    -v v1 # /g1/v1    -> /g2/v1
ncks -G g2:2  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
ncks -G g2:1  -g /     -v v1 # /v1       -> /g2/v1
ncks -G g2:1           -v v1 # /v1       -> /g2/v1
ncks -G g2:1  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/g1/v1
# Flatten-mode: Remove all input path components
# Syntax: -G [apn]: colon without numerical argument
ncks -G :            -v v1 # /g?/v1    -> /v1
ncks -G :   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :   -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
ncks -G g2:          -v v1 # /g?/v1    -> /g2/v1
ncks -G g2:                # /g?/v?    -> /g2/v?
ncks -G g2: -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
# Truncate-mode: Truncate from and Append to path tail
# Syntax: -G [apn]:-lvl_nbr = # of levels to truncate
# NB: -G [apn]:-lvl_nbr is equivalent to -G [apn]@lvl_nbr
ncks -G :-1   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G :-1   -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/v1
ncks -G :-2   -g g1/g2 -v v1 # /g1/g2/v1 -> /v1
ncks -G :-2   -g g1    -v v1 # /g1/v1    -> /v1
ncks -G g2:-1          -v v1 # /g?/v1    -> /g2/v1
ncks -G g2:-1 -g g1    -v v1 # /g1/v1    -> /g2/v1
ncks -G g1:-1 -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/g1/v1

Next: , Previous: , Up: Group Path Editing   [Contents][Index]

3.14.2 Moving Groups

Until fall 2013 (netCDF version 4.3.1-pre1), netCDF contained no library function for renaming groups, and therefore ncrename cannot rename groups. However, NCO built on earlier versions of netCDF than 4.3.1 can use a GPE-based workaround mechanism to “rename” groups. The GPE mechanism actually moves (i.e., copies to a new location) groups, a more arduous procedure than simply renaming them. GPE applies to all selected groups, so, in the general case, one must move only the desired group to a new file, and then merge that new file with the original to obtain a file where the desired group has been “renamed” and all else is unchanged. Here is how to “rename” group /g4 to group /f4 with GPE instead of ncrename

ncks -O -G f4:1 -g g4 ~/nco/data/ ~/ # Move /g4 to /f4
ncks -O -x -g g4 ~/nco/data/ ~/ # Excise /g4
ncks -A ~/ ~/ # Add /f4 to new file

If the original group g4 is not excised from (step two above), then the final output file would contain both g4 and a copy named f4. Thus GPE can be used to both “rename” and copy groups. The recommended way to rename groups when when netCDF version 4.3.1 is availale is to use ncrename (see ncrename netCDF Renamer).

One may wish to flatten hierarchical group files for many reasons. These include 1. To obtain flat netCDF3 files for use with tools that do not work with netCDF4 files, 2. To split-apart hierarchies to re-assemble into different hierarchies, and 3. To provide a subset of a hierarchical file with the simplest possible storage structure.

ncks -O -G : -g cesm -3 ~/nco/data/ ~/ # Extract /cesm to /

The -3 switch 34 specifies the output dataset should be in netCDF3 format, the -G : option flattens all extracted groups, and the -g cesm option extracts only the cesm group and leaves all other groups (e.g., ecmwf, giss).

Next: , Previous: , Up: Group Path Editing   [Contents][Index]

3.14.3 Dismembering Files

Let us show how to completely disaggregate (or, more memorably) dismember a hierarchical dataset. For now we take this to mean: store each group as a standalone flat dataset in netCDF3 format. This can be accomplished by looping the previous example over all groups. This script ncdismember dismembers the input file fl_in specified in the first argument and places the resulting files in the directory drc_out specified by the second argument:

cat > ~/ncdismember << 'EOF'

# Purpose: Dismember netCDF4/HDF5 hierarchical files. CF-check them.
# Place each input file group in separate netCDF3 output file
# Described in NCO User Guide at
# Requirements: NCO 4.3.x+, UNIX shell utilities awk, grep, sed
# Optional: Decker CFchecker

# Usage:
# ncdismember <fl_in> <drc_out> [cf_chk] [cf_vrs] [opt]
# where fl_in is input file/URL to dismember, drc_out is output directory
# CF-compliance check is performed when optional third argument is not '0'
# Default checker is Decker's cfchecker installed locally
# Specify cf_chk=nerc for smallified uploads to NERC checker
# Optional fourth argument cf_vrs is CF version to check
# Optional fifth argument opt passes straight-through to ncks
# Arguments must not use shell expansion/globbing
# NB: ncdismember does not clean-up output directory, so user must
# chmod a+x ~/sh/ncdismember
# Examples:
# ncdismember ~/nco/data/ /data/zender/tmp
# ncdismember /tmp
# ncdismember /tmp
# ncdismember ~/nco/data/ /data/zender/nco/tmp cf
# ncdismember ~/nco/data/ /data/zender/nco/tmp nerc
# ncdismember ~/nco/data/ /data/zender/nco/tmp cf 1.3
# ncdismember ~/nco/data/ /data/zender/nco/tmp cf 1.5 --fix_rec_dmn=all

# Command-line argument defaults
fl_in="${HOME}/nco/data/" # [sng] Input file to dismember/check
drc_out="${DATA}/nco/tmp" # [sng] Output directory
cf_chk='0' # [flg] Perform CF-compliance check? Which checker?
cf_vrs='1.5' # [sng] Compliance-check this CF version (e.g., '1.5')
opt='' # [flg] Additional ncks options (e.g., '--fix_rec_dmn=all')
# Use single quotes to pass multiple arguments to opt=${5}
# Otherwise arguments would be seen as ${5}, ${6}, ${7} ...

# Command-line argument option parsing
if [ -n "${1}" ]; then fl_in=${1}; fi
if [ -n "${2}" ]; then drc_out=${2}; fi
if [ -n "${3}" ]; then cf_chk=${3}; fi
if [ -n "${4}" ]; then cf_vrs=${4}; fi
if [ -n "${5}" ]; then opt=${5}; fi

# Prepare output directory
echo "NCO dismembering file ${fl_in}"
fl_stb=$(basename ${fl_in})
mkdir -p ${drc_out}
cd ${drc_out}
if [ ${cf_chk} = 'nerc' ]; then
fi # chk_nrc
if [ ${cf_chk} != '0' ] && [ ${cf_chk} != 'nerc' ]; then
    hash cfchecker 2>/dev/null || { echo >&2 "Local cfchecker command not found, will smallify and upload to NERC checker instead"; chk_nrc='y'; chk_dck='n'; }
fi # !cf_chk
# Obtain group list
grp_lst=`ncks -m ${fl_in} | grep '// group' | awk '{$1=$2=$3="";sub(/^  */,"",$0);print}'`
IFS=$'\n' # Change Internal-Field-Separator from <Space><Tab><Newline> to <Newline>
for grp_in in ${grp_lst} ; do
    # Replace slashes by dots for output group filenames
    grp_out=`echo ${grp_in} | sed 's/\///' | sed 's/\//./g'`
    if [ "${grp_out}" = '' ]; then grp_out='root' ; fi
    # Tell older NCO/netCDF if HDF4 with --hdf4 switch (signified by .hdf/.HDF suffix)
    hdf4=`echo ${fl_in} | awk '{if(match(tolower($1),".hdf$")) hdf4="--hdf4"; print hdf4}'`
    # Flatten to netCDF3, anchor, no history, no temporary file, padding, HDF4 flag, options
    cmd="ncks -O -3 -G : -g ${grp_in}/ -h --no_tmp_fl --hdr_pad=40 ${hdf4} ${opt} ${fl_in} ${drc_out}/${grp_out}.nc"
    # Use eval in case ${opt} contains multiple arguments separated by whitespace
    eval ${cmd}
    if [ ${chk_dck} = 'y' ]; then
       # Decker checker needs Conventions <= 1.6
       no_bck_sls=`echo ${drc_out}/${grp_out} | sed 's/\\\ / /g'`
       ncatted -h -a Conventions,global,o,c,CF-${cf_vrs} ${no_bck_sls}.nc
    else # !chk_dck
       echo ${drc_out}/${grp_out}.nc
    fi # !chk_dck
if [ ${chk_dck} = 'y' ]; then
    echo 'Decker CFchecker reports CF-compliance of each group in flat netCDF3 format'
    cfchecker -c ${cf_vrs} *.nc
if [ ${chk_nrc} = 'y' ]; then
    # Smallification and NERC upload from qdcf script by Phil Rasch (PJR)
    echo 'Using remote CFchecker'
    for fl in ${drc_out}/*.nc ; do
	dmns=`ncdump -h ${fl_in} | sed -n -e '/dimensions/,/variables/p' | grep = | sed -e 's/=.*//'`
	for dmn in ${dmns}; do
	    dmn_lc=`echo ${dmn} | tr "[:upper:]" "[:lower:]"`
	    if [ ${dmn_lc} = 'lat' ] || [ ${dmn_lc} = 'latitude' ] || [ ${dmn_lc} = 'lon' ] || [ ${dmn_lc} = 'longitude' ] || [ ${dmn_lc} = 'time' ]; then
		hyp_sml=`echo ${hyp_sml}" -d ${dmn},0"`
	    fi # !dmn_lc
	# Create small version of input file by sampling only first element of lat, lon, time
	ncks -O ${hyp_sml} ${fl} ${fl_sml}
	# Send small file to NERC checker
	curl --form cfversion=1.6 --form upload=@${fl_sml} --form press="Check%20file" ${cf_lcn} -o ${cf_out}
	# Strip most HTML to improve readability
	cat ${cf_out} | sed -e "s/<[^>]*>//g" -e "/DOCTYPE/,/\]\]/d" -e "s/CF-Convention//g" -e "s/Output of//g" -e "s/Compliance Checker//g" -e "s/Check another//g" -e "s/CF-Checker follows//g" -e "s/Received//g" -e "s/for NetCDF//g" -e "s/NetCDF format//g" -e "s/against CF version 1//g" -e "s/\.\.\.//g"
	echo "Full NERC compliance-check log for ${fl} in ${cf_out}"
fi # !nerc
chmod 755 ~/ncdismember # Make command executable
/bin/mv -f ~/ncdismember ~/sh # Store in location on $PATH, e.g., /usr/local/bin

zender@roulee:~$ ncdismember ~/nco/data/ ${DATA}/nco/tmp
NCO dismembering file /home/zender/nco/data/

A (potentially more portable) binary executable could be written to dismember all groups with a single invocation, yet dismembering without loss of information is possible now with this simple script on all platforms with UNIXy utilities. Note that all dimensions inherited by groups in the input file are correctly placed by ncdismember into the flat files. Moreover, each output file preserves the group metadata of all ancestor groups, including the global metadata from the input file. As written, the script could fail on groups that contain advanced netCDF4 features because the user requests (with the ‘-3’ switch) that output be netCDF3 classic format. However, ncks detects many format incompatibilities in advance and works around them. For example, ncks autoconverts netCDF4-only atomic-types (such as NC_STRING and NC_UBYTE) to corresponding netCDF3 atomic types (NC_CHAR and NC_SHORT) when the output format is netCDF3.

Previous: , Up: Group Path Editing   [Contents][Index]

3.14.4 Checking CF-compliance

One application of dismembering is to check the CF-compliance of each group in a file. When invoked with the optional third argumnt ‘cf’, ncdismember passes each file it generates to freely available compliance checkers, such as cfchecker 35.

zender@roulee:~$ ncdismember ~/nco/data/ /data/zender/nco/tmp cf
NCO dismembering file /home/zender/nco/data/
CFchecker reports CF-compliance of each group in flat netCDF3 format
WARNING: Using the default (non-CF) Udunits database 
INFO: INIT:     running CFchecker version 1.5.15
INFO: INIT:     checking compliance with convention CF-1.5
INFO: INIT:     using standard name table version: 25, last modified: 2013-07-05T05:40:30Z
INFO: INIT:     using area type table version: 2, date: 10 July 2013
INFO: 2.4:      no axis information found in dimension variables, not checking dimension order
WARNING: 3:     variable "tas1" contains neither long_name nor standard_name attribute
WARNING: 3:     variable "tas2" contains neither long_name nor standard_name attribute
INFO: 3.1:      variable "tas1" does not contain units attribute
INFO: 3.1:      variable "tas2" does not contain units attribute

By default the CF version checked is determined automatically by cfchecker. The user can override this default by supplying a supported CF version, e.g., ‘1.3’, as an optional fourth argument to ncdismember. Current valid CF options are ‘1.0’, ‘1.1’, ‘1.2’, ‘1.3’, ‘1.4’, and ‘1.5’.

Our development and testing of ncdismember is funded by our involvement in NASA’s Dataset Interoperability Working Group (DIWG), though our interest extends beyond NASA datasets. Taken together, NCO’s features (autoconversion to netCDF3 atomic types, fixing multiple record dimensions, autosensing HDF4 input, scoping rules for CF conventions) make ncdismember reliable and friendly for both dismembering hierarchical files and for CF-compliance checks. Most HDF4 and HDF5 datasets can be checked for CF-compliance with a one-line command. Example compliance checks of common NASA datasets are at Our long-term goal is to enrich the hierarchical data model with the expressivity and syntactic power of CF conventions.

NASA asked the DIWG to prepare a one-page summary of the procedure necessary to check HDF files for CF-compliance:

cat > ~/ncdismember.txt << 'EOF'
    Preparing an RPM-based OS to Test HDF & netCDF Files for CF-Compliance

By Charlie Zender, UCI & NASA Dataset Interoperability Working Group (DIWG)

Installation Summary:
1. HDF4 [with internal netCDF support _disabled_]
2. HDF5
3. netCDF [with external HDF4 support _enabled_]
4. NCO
5. numpy
6. netcdf4-python
7. python-lxml
8. CFunits-python
9. CFChecker
10. ncdismember

All 10 packages can use default installs _except_ HDF4 and netCDF.
Following instructions for Fedora Core 20 (FC20), an RPM-based Linux OS
Feedback and changes for other Linux-based OS's welcome to zender at
${H4DIR}, ${H5DIR}, ${NETCDFDIR}, ${NCODIR}, may all be different
For simplicity CZ sets them all to /usr/local

# 1. HDF4. Build in non-default manner. Turn-off its own netCDF support.
# Per
# HDF4 support not necessary though it makes ncdismember more comprehensive
wget -c
tar xvzf hdf-4.2.9.tar.gz
cd hdf-4.2.9
./configure --enable-shared --disable-netcdf --disable-fortran --prefix=${H4DIR}
make && make check && make install

# 2. HDF5. Build normally. RPM may work too. Please let me know if so.
# HDF5 is a necessary pre-requisite for netCDF4
wget -c
tar xvzf hdf5-1.8.11.tar.gz
cd hdf5-1.8.11
./configure --enable-shared --prefix=${H5DIR}
make && make check && make install

# 3. netCDF version 4.3.1 or later. Build in non-default manner with HDF4.
# Per
# Earlier versions of netCDF may fail checking some HDF4 files
wget -c
tar xvzf netcdf-4.3.2.tar.gz
cd netcdf-4.3.2
CPPFLAGS="-I${H5DIR}/include -I${H4DIR}/include" \
LDFLAGS="-L${H5DIR}/lib -L${H4DIR}/lib" \
./configure --enable-hdf4 --enable-hdf4-file-tests
make && make check && make install

# 4. NCO version 4.4.0 or later. Some RPMs available. Or install by hand.
# Later versions of NCO have much better support for ncdismember
wget .
tar xvzf nco-4.4.4.tar.gz
cd nco-4.4.4
./configure --prefix=${NCODIR}
make && make install

# 5. numpy
sudo yum install numpy -y

# 6. netcdf4-python
sudo yum install netcdf4-python -y

# 7. python-lxml
sudo yum install python-lxml -y

# 8. CFunits-python. No RPM available. Must install by hand.
wget .
tar xvzf cfunits-0.9.6.tar.gz
cd cfunits-0.9.6
sudo python install

# 9. CFChecker. No RPM available. Must install by hand.
wget . 
tar xvjf CFchecker-1.5.15.tar.bz2 
cd CFchecker
sudo python install

# 10. ncdismember. Copy script from
# Store dismembered files somewhere, e.g., ${DATA}/nco/tmp/hdf
mkdir -p ${DATA}/nco/tmp/hdf
# Many datasets work with a simpler command...
ncdismember ~/nco/data/ ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ~/nco/data/ ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/AMSR_E_L2_Rain_V10_200905312326_A.hdf \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5
ncdismember ${DATA}/hdf/HIRDLS-Aura_L3ZAD_v06-00-00-c02_2005d022-2008d077.he5 ${DATA}/nco/tmp/hdf cf 1.5
# Some datasets, typically .h5, require the --fix_rec_dmn=all argument
ncdismember_${DATA}/hdf/GATMO_npp_d20100906_t1935191_e1935505_b00012_c20110707155932065809_noaa_ops.h5 ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
ncdismember ${DATA}/hdf/mabel_l2_20130927t201800_008_1.h5 \
            ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all

A PDF version of these instructions is available here.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.15 C and Fortran Index conventions

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-F
Long options: ‘--fortran

The ‘-F’ switch changes NCO to read and write with the Fortran index convention. By default, NCO uses C-style (0-based) indices for all I/O. In C, indices count from 0 (rather than 1), and dimensions are ordered from slowest (inner-most) to fastest (outer-most) varying. In Fortran, indices count from 1 (rather than 0), and dimensions are ordered from fastest (inner-most) to slowest (outer-most) varying. Hence C and Fortran data storage conventions represent mathematical transposes of eachother. Note that record variables contain the record dimension as the most slowly varying dimension. See ncpdq netCDF Permute Dimensions Quickly for techniques to re-order (including transpose) dimensions and to reverse data storage order.

Consider a file containing 12 months of data in the record dimension time. The following hyperslab operations produce identical results, a June-July-August average of the data:

ncra -d time,5,7
ncra -F -d time,6,8

Printing variable three_dmn_var in file first with the C indexing convention, then with Fortran indexing convention results in the following output formats:

% ncks --trd -v three_dmn_var
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0 
% ncks --trd -F -v three_dmn_var
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0 

Next: , Previous: , Up: Shared features   [Contents][Index]

3.16 Hyperslabs

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-d dim,[min][,[max][,[stride]]]
Long options: ‘--dimension dim,[min][,[max][,[stride]]]’,
--dmn dim,[min][,[max][,[stride]]]

hyperslab is a subset of a variable’s data. The coordinates of a hyperslab are specified with the -d dim,[min][,[max][,[stride]]] short option (or with the same arguments to the ‘--dimension’ or ‘--dmn’ long options). At least one hyperslab argument (min, max, or stride) must be present. The bounds of the hyperslab to be extracted are specified by the associated min and max values. A half-open range is specified by omitting either the min or max parameter. The separating comma must be present to indicate the omission of one of these arguments. The unspecified limit is interpreted as the maximum or minimum value in the unspecified direction. A cross-section at a specific coordinate is extracted by specifying only the min limit and omitting a trailing comma. Dimensions not mentioned are passed with no reduction in range. The dimensionality of variables is not reduced (in the case of a cross-section, the size of the constant dimension will be one).

# First and second longitudes
ncks -F -d lon,1,2
# Second and third longitudes
ncks -d lon,1,2

As of version 4.2.1 (August, 2012), NCO allows one to extract the last N elements of a hyperslab. Negative integers as min or max elements of a hyperslab specification indicate offsets from the end (Python also uses this convention). Consistent with this convention, the value ‘-1’ (negative one) indicates the last element of a dimension, and negative zero is algebraically equivalent to zero and so indicates the first element of a dimension. Previously, for example, ‘-d time,-2,-1’ caused a domain error. Now it means select the penultimate and last timesteps, independent of the size of the time dimension. Select only the first and last timesteps, respectively, with ‘-d time,0’ and ‘-d time,-1’. Negative integers work for min and max indices, though not for stride.

# Second through penultimate longitudes
ncks -d lon,1,-2
# Second through last longitude
ncks -d lon,1,-1
# Second-to-last to last longitude
ncks -d lon,-3,-1
# Second-to-last to last longitude 
ncks -d lon,-3,

The ‘-F’ argument, if any, applies the Fortran index convention only to indices specified as positive integers:

# First through penultimate longitudes
ncks -F -d lon,1,-2 (-F affects only start index)
# First through last longitude
ncks -F -d lon,1,-1
# Second-to-last to penultimate longitude (-F has no effect)
ncks -F -d lon,-3,-1
# Second-to-last to last longitude (-F has no effect)
ncks -F -d lon,-3,

Coordinate values should be specified using real notation with a decimal point required in the value, whereas dimension indices are specified using integer notation without a decimal point. This convention serves only to differentiate coordinate values from dimension indices. It is independent of the type of any netCDF coordinate variables. For a given dimension, the specified limits must both be coordinate values (with decimal points) or dimension indices (no decimal points).

If values of a coordinate-variable are used to specify a range or cross-section, then the coordinate variable must be monotonic (values either increasing or decreasing). In this case, command-line values need not exactly match coordinate values for the specified dimension. Ranges are determined by seeking the first coordinate value to occur in the closed range [min,max] and including all subsequent values until one falls outside the range. The coordinate value for a cross-section is the coordinate-variable value closest to the specified value and must lie within the range or coordinate-variable values. The stride argument, if any, must be a dimension index, not a coordinate value. See Stride, for more information on the stride option.

# All longitude values between 1 and 2 degrees
ncks -d lon,1.0,2.0
# All longitude values between 1 and 2 degrees
ncks -F -d lon,1.0,2.0
# Every other longitude value between 0 and 90 degrees
ncks -F -d lon,0.0,90.0,2

As shown, we recommend using a full floating-point suffix of .0 instead of simply . in order to make obvious the selection of hyperslab elements based on coordinate value rather than index.

User-specified coordinate limits are promoted to double-precision values while searching for the indices which bracket the range. Thus, hyperslabs on coordinates of type NC_CHAR are computed numerically rather than lexically, so the results are unpredictable.

The relative magnitude of min and max indicate to the operator whether to expect a wrapped coordinate (see Wrapped Coordinates), such as longitude. If min > max, the NCO expects the coordinate to be wrapped, and a warning message will be printed. When this occurs, NCO selects all values outside the domain [max < min], i.e., all the values exclusive of the values which would have been selected if min and max were swapped. If this seems confusing, test your command on just the coordinate variables with ncks, and then examine the output to ensure NCO selected the hyperslab you expected (coordinate wrapping is currently only supported by ncks).

Because of the way wrapped coordinates are interpreted, it is very important to make sure you always specify hyperslabs in the monotonically increasing sense, i.e., min < max (even if the underlying coordinate variable is monotonically decreasing). The only exception to this is when you are indeed specifying a wrapped coordinate. The distinction is crucial to understand because the points selected by, e.g., -d longitude,50.,340., are exactly the complement of the points selected by -d longitude,340.,50..

Not specifying any hyperslab option is equivalent to specifying full ranges of all dimensions. This option may be specified more than once in a single command (each hyperslabbed dimension requires its own -d option).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.17 Stride

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-d dim,[min][,[max][,[stride]]]
Long options: ‘--dimension dim,[min][,[max][,[stride]]]’,
--dmn dim,[min][,[max][,[stride]]]

All data operators support specifying a stride for any and all dimensions at the same time. The stride is the spacing between consecutive points in a hyperslab. A stride of 1 picks all the elements of the hyperslab, and a stride of 2 skips every other element, etc.. ncks multislabs support strides, and are more powerful than the regular hyperslabs supported by the other operators (see Multislabs). Using the stride option for the record dimension with ncra and ncrcat makes it possible, for instance, to average or concatenate regular intervals across multi-file input data sets.

The stride is specified as the optional fourth argument to the ‘-d’ hyperslab specification: -d dim,[min][,[max][,[stride]]]. Specify stride as an integer (i.e., no decimal point) following the third comma in the ‘-d’ argument. There is no default value for stride. Thus using ‘-d time,,,2’ is valid but ‘-d time,,,2.0’ and ‘-d time,,,’ are not. When stride is specified but min is not, there is an ambiguity as to whether the extracted hyperslab should begin with (using C-style, 0-based indexes) element 0 or element ‘stride-1’. NCO must resolve this ambiguity and it chooses element 0 as the first element of the hyperslab when min is not specified. Thus ‘-d time,,,stride’ is syntactically equivalent to ‘-d time,0,,stride’. This means, for example, that specifying the operation ‘-d time,,,2’ on the array ‘1,2,3,4,5’ selects the hyperslab ‘1,3,5’. To obtain the hyperslab ‘2,4’ instead, simply explicitly specify the starting index as 1, i.e., ‘-d time,1,,2’.

For example, consider a file which contains 60 consecutive months of data. Say you wish to obtain just the March data from this file. Using 0-based subscripts (see C and Fortran Index Conventions) these data are stored in records 2, 14, … 50 so the desired stride is 12. Without the stride option, the procedure is very awkward. One could use ncks five times and then use ncrcat to concatenate the resulting files together:

for idx in 02 14 26 38 50; do # Bourne Shell
  ncks -d time,${idx} foo.${idx}
foreach idx (02 14 26 38 50) # C Shell
  ncks -d time,${idx} foo.${idx}
ncrcat foo.??
rm foo.??

With the stride option, ncks performs this hyperslab extraction in one operation:

ncks -d time,2,,12

See ncks netCDF Kitchen Sink, for more information on ncks.

Applying the stride option to the record dimension in ncra and ncrcat makes it possible, for instance, to average or concatenate regular intervals across multi-file input data sets.

ncra -F -d time,3,,12
ncrcat -F -d time,3,,12

Next: , Previous: , Up: Shared features   [Contents][Index]

3.18 Record Appending

Availability: ncra, ncrcat
Short options: None
Long options: ‘--rec_apn’, ‘--record_append

As of version 4.2.6 (March, 2013), NCO allows both Multi-File, Multi-Record operators (ncra and ncrcat) to append their output directly to the end of an existing file. This feature may be used to augment a target file, rather than construct it from scratch. This helps, for example, when a timeseries is concatenated from input data that becomes available in stages rather than all at once. In such cases this switch significantly speeds writing.

Consider the use case where one wishes to preserve the contents of, and add to them new records contained in Previously the output had to be placed in a third file, (which could also safely be named, via

ncrcat -O

Under the hood this operation copies all information in and not once but twice. The first copy is performed through the netCDF interface, as all data from and are extracted and placed in the output file. The second copy occurs (usually much) more quickly as the (by default) temporary output file is copied (sometimes a quick re-link suffices) to the final output file (see Temporary Output Files). All this copying is expensive for large files.

The ‘--record_append’ switch appends all records in to the end (after the last record) of

ncrcat --rec_apn

The ordering of the filename arguments may seem non-intuitive. If the record variable represents time in these files, then the values in precede those in, so why do the files appear in the reverse order on the command line? is the last file named because it is the pre-existing output file to which we will append all the other input files listed (in this case only The contents of are completely preserved, and only values in (and any other input files) are copied. This switch avoids the necessity of copying all of through the netCDF interface to a new output file. The ‘--rec_apn’ switch automatically puts NCO into append mode (see Appending Variables), so specifying ‘-A’ is redundant, and simultaneously specifying overwrite mode with ‘-O’ causes an error. By default, NCO works in an intermediate temporary file. Power users may combine ‘--rec_apn’ with the ‘--no_tmp_fl’ switch (see Temporary Output Files):

ncrcat --rec_apn --no_tmp_fl

This avoids creating an intermediate file, and copies only the minimal amount of data (i.e., all of Hence, it is fast. We recommend users try to understand the safety trade-offs involved.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.19 Subcycle

Availability: ncra, ncrcat
Short options: ‘-d dim,[min][,[max][,[stride][,[subcycle]]]]
Long options: ‘--mro’ ‘--dimension dim,[min][,[max][,[stride][,[subcycle]]]]
--dmn dim,[min][,[max][,[stride][,[subcycle]]]]

As of version 4.2.1 (August, 2012), NCO allows both Multi-File, Multi-Record operators, ncra and ncrcat, to extract and operate on multiple groups of records. These groups may be connected to physical sub-cycles of a periodic nature, e.g., months of a year, or hours of a day. Or they may be thought of as groups of a specifed duration. The feature and the terminology to describe it are new. For now, we call this the subcycle feature, sometimes abbreviated SSC 36.

The subcycle feature allows processing of groups of records separated by regular intervals of records. It is perhaps best illustrated by an extended example which describes how to solve the same problem both with and without the SSC feature.

The first task in climate data processing is often creating seasonal cycles. Suppose a 150-year climate simulation produces 150 output files, each comprising 12 records, each record a monthly mean:,, ... Our goal is to create a single file containing the summertime (June, July, and August, aka JJA) mean. Traditionally, we would first compute the climatological monthly mean for each month of summer. Each of these is a 150-year mean, i.e.,

# Step 1: Create climatological monthly files
for mth in {6..8}; do
  mm=`printf "%02d" $mth`
  ncra -O -F -d time,${mm},,12 -n 150,4,1 clm${mm}.nc
# Step 2: Average climatological monthly files into summertime mean
ncra -O clm06

So far, nothing is unusual and this task can be performed by any NCO version. The SSC feature makes obsolete the need for the shell loop used in Step 1 above.

The new SSC option aggregates more than one input record at a time before performing arithmetic operations, and, with an additional switch, allows us to archive those results in multiple-record output (MRO) files. This reduces the task of producing the climatological summertime mean to one step:

# Step 1: Compute climatological summertime mean
ncra -O -F -d time,6,,12,3 -n 150,4,1

The SSC option instructs ncra (or ncrcat) to process files in groups of three records. To better understand the meaning of each argument to the ‘-d’ hyperslab option, read it this way: “for the time dimension start with the sixth record, continue without end, repeat the process every twelfth record, and define a sub-cycle as three consecutive records”.

A separate option, ‘--mro’, instructs ncra to output its results from each sub-group, and to produce a Multi-Record Output (MRO) file rather than a Single-Record Output (SRO) file. Unless ‘--mro’ is specified, ncra collects together all the sub-groups, operates on their ensemble, and produces a single output record. The addition of ‘--mro’ to the above example causes ncra to archive all (150) annual summertime means to one file:

# Step 1: Archive all 150 summertime means in one file
ncra --mro -O -F -d time,6,,12,3 -n 150,4,1
# ...or all (150) annual means...
ncra --mro -O -d time,,,12,12 -n 150,4,1

These operations generate and require no intermediate files. This contrasts to previous NCO methods, which require generating, averaging, then catenating 150 files. The ‘--mro’ option only works on ncra and has no effect on (or rather is redundant for) ncrcat, since ncrcat always outputs all selected records.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.20 Multislabs

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat
Short options: ‘-d dim,[min][,[max][,[stride]]]
Long options: ‘--dimension dim,[min][,[max][,[stride]]]’,
--dmn dim,[min][,[max][,[stride]]]
--msa_usr_rdr’, ‘--msa_user_order

A multislab is a union of one or more hyperslabs. One defines multislabs by chaining together hyperslab commands, i.e., -d options (see Hyperslabs). Support for specifying a multi-hyperslab or multislab for any variable was first added to ncks in late 2002. The other operators received these capabilities in April 2008. Multi-slabbing is often referred to by the acronym MSA, which stands for “Multi-Slabbing Algorithm”. As explained below, the user may additionally request that the multislabs be returned in the user-specified order, rather than the on-disk storage order. Although MSA user-ordering has been available in all operators since 2008, most users were unaware of it since the documentation (below, and in the man pages) was not written until July 2013.

Multislabs overcome many restraints that limit simple hyperslabs. A single -d option can only specify a contiguous and/or a regularly spaced multi-dimensional data array. Multislabs are constructed from multiple -d options and may therefore have non-regularly spaced arrays. For example, suppose it is desired to operate on all longitudes from 10.0 to 20.0 and from 80.0 to 90.0 degrees. The combined range of longitudes is not selectable in a single hyperslab specfication of the form ‘-d dimension,min,max’ or ‘-d dimension,min,max,stride’ because its elements are irregularly spaced in coordinate space (and presumably in index space too). The multislab specification for obtaining these values is simply the union of the hyperslabs specifications that comprise the multislab, i.e.,

ncks -d lon,10.,20. -d lon,80.,90.
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90.

Any number of hyperslabs specifications may be chained together to specify the multislab. MSA creates an output dimension equal in size to the sum of the sizes of the multislabs. This can be used to extend and or pad coordinate grids.

Users may specify redundant ranges of indices in a multislab, e.g.,

ncks -d lon,0,4 -d lon,2,9,2

This command retrieves the first five longitudes, and then every other longitude value up to the tenth. Elements 0, 2, and 4 are specified by both hyperslab arguments (hence this is redundant) but will count only once if an arithmetic operation is being performed. This example uses index-based (not coordinate-based) multislabs because the stride option only supports index-based hyper-slabbing. See Stride, for more information on the stride option.

Multislabs are more efficient than the alternative of sequentially performing hyperslab operations and concatenating the results. This is because NCO employs a novel multislab algorithm to minimize the number of I/O operations when retrieving irregularly spaced data from disk. The NCO multislab algorithm retrieves each element from disk once and only once. Thus users may take some shortcuts in specifying multislabs and the algorithm will obtain the intended values. Specifying redundant ranges is not encouraged, but may be useful on occasion and will not result in unintended consequences.

Suppose the Q variable contains three dimensional arrays of distinct chemical constituents in no particular order. We are interested in the NOy species in a certain geographic range. Say that NO, NO2, and N2O5 are elements 0, 1, and 5 of the species dimension of Q. The multislab specification might look something like

ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2

Multislabs are powerful because they may be specified for every dimension at the same time. Thus multislabs obsolete the need to execute multiple ncks commands to gather the desired range of data.

The MSA user-order switch ‘--msa_usr_rdr’ (or ‘--msa_user_order’, both of which shorten to ‘--msa’) requests that the multislabs be output in the user-specified order from the command-line, rather than in the input-file on-disk storage order. This allows the user to perform complex data re-ordering in one operation that would otherwise require cumbersome steps of hyperslabbing, concatenating, and permuting. Consider the example of converting datasets stored with the longitude coordinate Lon ranging from [-180,180) to datasets that follow the [0,360) convention.

% ncks -H -v Lon

What is needed is a simple way to rotate longitudes. Although simple in theory, this task requires both mathematics to change the numerical value of the longitude coordinate, data hyperslabbing to split the input on-disk arrays at Greenwich, and data re-ordering within to stitch the western hemisphere onto the eastern hemisphere at the date-line. The ‘--msa’ user-order switch overrides the default that data are output in the same order in which they are stored on-disk in the input file, and instead stores them in the same order as the multi-slabs are given to the command line. This default is intuitive and is not important in most uses. However, the MSA user-order switch allows users to meet their output order needs by specifying multi-slabs in a certain order. Compare the results of default ordering to user-ordering for longitude:

% ncks -O -H       -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0
% ncks -O -H --msa -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0

The two multi-slabs are the same but they can be presented to screen, or to an output file, in either order. The second example shows how to place the western hemisphere after the eastern hemisphere, although they are stored in the opposite order in the input file.

With this background, one sees that the following commands suffice to rotate the input file by 180 degrees longitude:

% ncks -O -v LatLon --msa -d Lon,0.,180. -d Lon,-180.,-1.0
% ncap2 -O -s 'where(Lon < 0) Lon=Lon+360'
% ncks -C -H -v LatLon ~/nco/data/
Lat[0]=-45 Lon[0]=-180 LatLon[0]=0 
Lat[0]=-45 Lon[1]=-90 LatLon[1]=1 
Lat[0]=-45 Lon[2]=0 LatLon[2]=2 
Lat[0]=-45 Lon[3]=90 LatLon[3]=3 
Lat[1]=45 Lon[0]=-180 LatLon[4]=4 
Lat[1]=45 Lon[1]=-90 LatLon[5]=5 
Lat[1]=45 Lon[2]=0 LatLon[6]=6 
Lat[1]=45 Lon[3]=90 LatLon[7]=7 
% ncks -C -H -v LatLon ~/
Lat[0]=-45 Lon[0]=0 LatLon[0]=2 
Lat[0]=-45 Lon[1]=90 LatLon[1]=3 
Lat[0]=-45 Lon[2]=180 LatLon[2]=0 
Lat[0]=-45 Lon[3]=270 LatLon[3]=1 
Lat[1]=45 Lon[0]=0 LatLon[4]=6 
Lat[1]=45 Lon[1]=90 LatLon[5]=7 
Lat[1]=45 Lon[2]=180 LatLon[6]=4 
Lat[1]=45 Lon[3]=270 LatLon[7]=5 

There are other workable, valid methods to accomplish this rotation, yet none are simpler nor more efficient than utilizing MSA user-ordering. Some final comments on applying this algorithm: Be careful to specify hemispheres that do not overlap, e.g., by inadvertently specifying coordinate ranges that both include Greenwich. Some users will find using index-based rather than coordinate-based hyperslabs makes this clearer.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.21 Wrapped Coordinates

Availability: ncks
Short options: ‘-d dim,[min][,[max][,[stride]]]
Long options: ‘--dimension dim,[min][,[max][,[stride]]]’,
--dmn dim,[min][,[max][,[stride]]]

wrapped coordinate is a coordinate whose values increase or decrease monotonically (nothing unusual so far), but which represents a dimension that ends where it begins (i.e., wraps around on itself). Longitude (i.e., degrees on a circle) is a familiar example of a wrapped coordinate. Longitude increases to the East of Greenwich, England, where it is defined to be zero. Halfway around the globe, the longitude is 180 degrees East (or West). Continuing eastward, longitude increases to 360 degrees East at Greenwich. The longitude values of most geophysical data are either in the range [0,360), or [-180,180). In either case, the Westernmost and Easternmost longitudes are numerically separated by 360 degrees, but represent contiguous regions on the globe. For example, the Saharan desert stretches from roughly 340 to 50 degrees East. Extracting the hyperslab of data representing the Sahara from a global dataset presents special problems when the global dataset is stored consecutively in longitude from 0 to 360 degrees. This is because the data for the Sahara will not be contiguous in the input-file but is expected by the user to be contiguous in the output-file. In this case, ncks must invoke special software routines to assemble the desired output hyperslab from multiple reads of the input-file.

Assume the domain of the monotonically increasing longitude coordinate lon is 0 < lon < 360. ncks will extract a hyperslab which crosses the Greenwich meridian simply by specifying the westernmost longitude as min and the easternmost longitude as max. The following commands extract a hyperslab containing the Saharan desert:

ncks -d lon,340.,50.
ncks -d lon,340.,50. -d lat,10.,35.

The first example selects data in the same longitude range as the Sahara. The second example further constrains the data to having the same latitude as the Sahara. The coordinate lon in the output-file,, will no longer be monotonic! The values of lon will be, e.g., ‘340, 350, 0, 10, 20, 30, 40, 50’. This can have serious implications should you run through another operation which expects the lon coordinate to be monotonically increasing. Fortunately, the chances of this happening are slim, since lon has already been hyperslabbed, there should be no reason to hyperslab lon again. Should you need to hyperslab lon again, be sure to give dimensional indices as the hyperslab arguments, rather than coordinate values (see Hyperslabs).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.22 Auxiliary Coordinates

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat
Short options: ‘-X lon_min,lon_max,lat_min,lat_max
Long options: ‘--auxiliary lon_min,lon_max,lat_min,lat_max

Utilize auxiliary coordinates specified in values of the coordinate variable’s standard_name attributes, if any, when interpreting hyperslab and multi-slab options. Also ‘--auxiliary’. This switch supports hyperslabbing cell-based grids (aka unstructured grids) over coordinate ranges. When these grids are stored as 1D-arrays of cell data, this feature is helpful at hyperslabbing and/or performing arithmetic on selected geographic regions. This feature cannot be used to select regions of 2D grids (instead use the ncap2 where statement for such grids Where statement). This feature works on datasets that associate coordinate variables to grid-mappings using the CF-convention (see CF Conventions) coordinates and standard_name attributes described here. Currently, NCO understands auxiliary coordinate variables pointed to by the standard_name attributes for latitude and longitude. Cells that contain a value within the user-specified range [lon_min,lon_max,lat_min,lat_max] are included in the output hyperslab.

A cell-based or unstructured grid collapses the horizontal spatial information (latitude and longitude) and stores it along a one-dimensional coordinate that has a one-to-one mapping to both latitude and longitude coordinates. Rectangular (in longitude and latitude) horizontal hyperslabs cannot be selected using the typical procedure (see Hyperslabs) of separately specifying ‘-d’ arguments for longitude and latitude. Instead, when the ‘-X’ is used, NCO learns the names of the latitude and longitude coordinates by searching the standard_name attribute of all variables until it finds the two variables whose standard_name’s are “latitude” and “longitude”, respectively. This standard_name attribute for latitude and longitude coordinates follows the CF-convention (see CF Conventions).

Putting it all together, consider a variable gds_3dvar output from simulations on a cell-based geodesic grid. Although the variable contains three dimensions of data (time, latitude, and longitude), it is stored in the netCDF file with only two dimensions, time and gds_crd.

% ncks -m -C -v gds_3dvar ~/nco/data/
gds_3dvar: type NC_FLOAT, 2 dimensions, 4 attributes, chunked? no, \
 compressed? no, packed? no, ID = 41
gds_3dvar RAM size is 10*8*sizeof(NC_FLOAT) = 80*4 = 320 bytes
gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 20 \ 
gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = \ 
 Geodesic variable
gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = \
 lat_gds lon_gds
gds_3dvar attribute 3: purpose, size = 64 NC_CHAR, value = \ 
 Test auxiliary coordinates like those that define geodesic grids

The coordinates attribute lists the names of the latitude and longitude coordinates, lat_gds and lon_gds, respectively. The coordinates attribute is recommended though optional. With it, the user can immediately identify which variables contain the latitude and longitude coordinates. Without a coordinates attribute it would be unclear at first glance whether a variable resides on a cell-based grid. In this example, time is a normal record dimension and gds_crd is the cell-based dimension.

The cell-based grid file must contain two variables whose standard_name attributes are “latitude”, and “longitude”:

% ncks -m -C -v lat_gds,lon_gds ~/nco/data/
lat_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 37
lat_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lat_gds attribute 3: purpose, size = 62 NC_CHAR, value = \ 
 1-D latitude coordinate referred to by geodesic grid variables

lon_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
 chunked? no, compressed? no, packed? no, ID = 38
lon_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lon_gds attribute 3: purpose, size = 63 NC_CHAR, value = \
 1-D longitude coordinate referred to by geodesic grid variables

In this example lat_gds and lon_gds represent the latitude or longitude, respectively, of cell-based variables. These coordinates (must) have the same single dimension (gds_crd, in this case) as the cell-based variables. And the coordinates must be one-dimensional—multidimensional coordinates will not work.

This infrastructure allows NCO to identify, interpret, and process (e.g., hyperslab) the variables on cell-based grids as easily as it works with regular grids. To time-average all the values between zero and 180 degrees longitude and between plus and minus 30 degress latitude, we use

ncra -O -X 0.,180.,-30.,30. -v gds_3dvar

NCO accepts multiple ‘-X’ arguments for cell-based grid multi-slabs, just as it accepts multiple ‘-d’ arguments for multi-slabs of regular coordinates.

ncra -O -X 0.,180.,-30.,30. -X 270.,315.,45.,90.

The arguments to ‘-X’ are always interpreted as floating-point numbers, i.e., as coordinate values rather than dimension indices so that these two commands produce identical results

ncra -X 0.,180.,-30.,30.
ncra -X 0,180,-30,30

In contrast, arguments to ‘-d’ require decimal places to be recognized as coordinates not indices (see Hyperslabs). We recommend always using decimal points with ‘-X’ arguments to avoid confusion.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.23 Grid Generation

Availability: ncks
Short options: None
Long options: ‘--rgr key=val’ (multiple invocations allowed)

As of NCO version 4.5.2 (August, 2015), ncks generates accurate and complete SCRIP-format gridfiles for select grid types, including uniform, capped and Gaussian rectangular latitude/longitude grids. The grids are stored in an external grid-file.

All options pertinent to the grid geometry and metadata are passed to NCO via key-value pairs prefixed by the ‘--rgr’ option, or its synonym, ‘--regridding’. The option ‘--rgr’ (and its long option equivalents such as ‘--regridding’) indicates the argument syntax will be key=val. As such, ‘--rgr’ and its synonyms are indicator options that accept arguments supplied one-by-one like ‘--rgr key1=val1 --rgr key2=val2’, or aggregated together in multi-argument format like ‘--rgr key1=val1#key2=val2’ (see Multi-arguments).

The text strings that describe the grid and name the file are important aids to convey the grid geometry to other users. These arguments, and their corresponding keys, are the grid title (grd_ttl), and grid filename (grid), respectively. The numbers of latitudes (lat_nbr) and longitudes (lon_nbr) are independent, and together determine the grid storage size. These four options should be considered mandatory, although NCO provides defaults for any arguments omitted.

The remaining arguments depend on the whether the grid is global or regional. For global grids, one should specify only two more arguments, the latitude (lat_typ) and longitude (lon_typ) grid-types. These types are chosen as described below from a small selection of options that together define the most common rectangular global grids. For regional grids, one must specify the bounding box, i.e., the edges of the rectangular grid on the North (lat_nrt), South (lat_sth), East (lat_est), and West (lat_nrt) sides. Specifying a bounding box for global grids is redundant and will cause an error to ensure the user intends a global grid. NCO assumes that regional grids are uniform, though it will attempt to produce regional grids of other types if the user specifies other latitude (lat_typ) and longitude (lon_typ) grid-types, e.g., Gaussian or Cap. Edges of a regional bounding box may be specified individually, or in the single-argument forms.

The full description of grid-generation arguments, and their corresponding keys, is:

Grid Title: grd_ttl

It is surprisingly difficult to discern the geometric configuration of a grid from the coordinates of a SCRIP-format gridfile. A human-readable grid description should be placed in grd_ttl. Examples include “CAM-FV scalar grid 129x256” and “T42 Gaussian grid”.

Grid File: scrip_grid

The grid-generation API was bolted-on to NCO and contains some temporary kludges. For example, the output grid filename is distinct from the output filename of the host ncks command. Specify the output gridfile name scrip_grid with keywords grid or scrip, e.g., ‘--rgr grid=scrip_grid’ or ‘--rgr’. It is conventional to include a datestamp in the gridfile name. This helps users identify up-to-date and out-of-date grids. Any valid netCDF file may be named as the source (e.g., It will not be altered. The destination file (e.g., will be overwritten. Its contents are immaterial.

Grid Types: lat_typ, lon_typ

The keys that hold the longitude and latitude gridtypes (which are, by the way, independent of eachother) are lon_typ and lat_typ. The lat_typ options for global grids are ‘uni’ for Uniform, ‘cap’ (or ‘fv’) for Capped (equivalent to FV), and ‘gss’ for Gaussian. These values are all case-independent, so ‘Gss’ and ‘gss’ both work.

As its name suggests, the latitudes in a Uniform-latitude grid are uniformly spaced 37. The Uniform-latitude grid may have any number of latitudes. NCO can only generate longitude grids (below) that are uniformly spaced, so the Uniform-latitude grids we describe are also uniform in the 2D sense. Uniform grids are intuitive, easy to visualize, and simple to program. Hence their popularity in data exchange, visualization, and archives. Moreover, regional grids (unless they include the poles), are free of polar singularities, and thus are well-suited to storage on Uniform grids. Theoretically, a Uniform-latitude grid could have non-uniform longitudes, but NCO currently does not implement non-uniform longitude grids.

Their mathematical properties (convergence and excessive resolution at the poles, which can appear as singularities) make Uniform grids fraught for use in global models. One purpose Uniform grids serve in modeling is as “offset” or “staggered” grids, meaning grids whose centers are the interfaces of another grid. The Finite-Volume (FV) method is often used to represent and solve the equations of motion in climate-related fields. Many FV solutions (including the popular Lin-Rood method as used in the CESM CAM-FV atmospheric model) evaluate scalar (i.e., non-vector) fields (e.g., temperature, water vapor) at gridcell centers of what is therefore called the scalar grid. FV methods (like Lin-Rood) that employ an Arakawa C-grid or D-grid formulation define velocities on the edges of the scalar grid. This CAM-FV velocity grid is therefore “staggered” or “offset” from the CAM-FV scalar grid by one-half gridcell. The CAM-FV scalar latitude grid has gridpoints (the “caps”) centered on each pole to avoid singularities. The offset of a Cap-grid is a Uniform-grid, so the Uniform grid is often called an FV-”offset” or “staggered” grid. Hence an NCO Uniform grid is equivalent to an NCL “Fixed Offset” grid. For example, a 128x256 Uniform grid is the offset or staggered version of a 129x256 Cap grid (aka FV-grid).

Referring the saucer-like cap-points at the poles, NCO uses the term “Cap grid” to describe the latitude portion of the FV-scalar grid as used by the CAM-FV Lin-Rood dynamics formulation. NCO accepts the shorthand FV, and the more descriptive “Yarmulke”, as synonyms for Cap. A Cap-latitude grid differs from a Uniform-latitude grid in many ways:

Most importantly, Cap grids are 2D-representations of numerical grids with cap-midpoints instead of zonal-teeth convergence at the poles. The rectangular 2D-representation of each cap contains gridcells shaped like sharp teeth that converge at the poles similar to the Uniform grid, but the Cap gridcells are meant to be aggregated into a single cell centered at the pole in a dynamical transport algorithm. In other words, the polar teeth are a convenient way to encode a non-rectangular grid in memory into a rectangular array on disk. Hence Cap grids have the unusual property that the poles are labeled as being both the centers and the outer interfaces of all polar gridcells. Second, Cap grids are uniform in angle except at the poles, where the latitudes span half the meridional range of the rest of the gridcells. Even though in the host dynamical model the Cap grid polar points are melded into caps uniform (in angle) with the rest of the grid, the disk representation on disk is not uniform. Nevertheless, some call the Cap grid a uniform-angle grid because the information contained at the poles is aggregated in memory to span twice the range of a single polar gridcell (which has half the normal width). NCL uses the term “Fixed grid” for a Cap grid. The “Fixed” terminology seems broken.

Finally, Gaussian grids are the Cartesian representation of global spectral transform models. Gaussian grids do not have points at the poles, and typically have an even number of latitudes. All three latitude grid-type supported by NCO (Uniform, Cap, and Gaussian) are Regular grids in that they are monotonic.

The lon_typ options for global grids are ‘grn_ctr’ and ‘180_ctr’ for the first gridcell centered at Greenwich or 180 degrees, respecitvely. And ‘grn_wst’ and ‘180_wst’ for Greenwich or 180 degress lying on the western edge of the first gridcell. Many global models use the ‘grn_ctr’ longitude grid as their “scalar grid” (where, e.g., temperature, humidity, and other scalars are defined). The “staggered” or “offset” grid (where often the dynamics variables are defined) then must have the ‘grn_wst’ longitude convention. That way the centers of the scalar grid are the vertices of the offset grid, and visa versa.

Grid Resolution: lat_nbr, lon_nbr

The number of gridcells in the horizontal spatial dimensions are lat_nbr and lon_nbr, respectively. There are no restrictions on lon_nbr for any gridtype. Latitude grids do place some restrictions on lat_nbr (see above). As of NCO version 4.5.3, released in October, 2015, the ‘--rgr latlon=lat_nbr,lon_nbr’ switch may be used to simultaneously specify both latitude and longitude, e.g., ‘--rgr latlon=180,360’.

Grid Edges: lon_wst, lon_est, lat_sth, lat_nrt

The outer edges of a regional rectangular grid are specified by the North (lat_nrt), South (lat_sth), East (lat_est), and West (lat_nrt) sides. Latitudes and longigudes must be specified in degrees (not radians). Latitude edges must be between -90 and 90. Longitude edges may be positive or negative and separated by no more than 360 degrees. The edges may be specified individually with four arguments, consecutively separated by the multi-argument delimiter (‘#’ by default), or together in a short list to the pre-ordered options ‘wesn’ or ‘snwe’. These three specifications are equivalent:

ncks ... --rgr lat_sth=30.0 --rgr lat_nrt=70.0 --rgr lon_wst=-120.0 --rgr lon_est=-90.0 ...
ncks ... --rgr lat_sth=30.0#lat_nrt=70.0#lon_wst=-120.0#lon_est=-90.0 ...
ncks ... --rgr snwe=30.0,70.0,-120.0,-90.0 ...

The first example above supplies the bounding box with four key=val pairs. The second example above supplies the bounding box with a single option in multi-argument format (see Multi-arguments). The third example uses a convenience switch introduced to reduce typing.

Generating common grids:

# 180x360 (1x1 degree) Equi-Angular grid, first longitude centered at Greenwich
ncks --rgr grd_ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 180x360 (1x1 degree) Equi-Angular grid, first longitude west edge at Greenwich
ncks --rgr grd_ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_wst \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 129x256 CAM-FV grid, first longitude centered at Greenwich
ncks --rgr grd_ttl='CAM-FV scalar grid 129x256'#latlon=129,256#lat_typ=fv#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 192x288 CAM-FV grid, first longitude centered at Greenwich
ncks --rgr grd_ttl='CAM-FV scalar grid 192x288'#latlon=192,288#lat_typ=fv#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 1441x2880 CAM-FV grid, first longitude centered at Greenwich
ncks --rgr grd_ttl='CAM-FV scalar grid 1441x2880'#latlon=1441,2880#lat_typ=fv#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 91x180 CAM-FV grid, first longitude centered at Greenwich (2 degree grid)
ncks --rgr grd_ttl='CAM-FV scalar grid 91x180'#latlon=91,180#lat_typ=fv#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 25x48 CAM-FV grid, first longitude centered at Greenwich (7.5 degree grid)
ncks --rgr grd_ttl='CAM-FV scalar grid 25x48'#latlon=25,48#lat_typ=fv#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 128x256 Equi-Angular grid, Greenwich west edge of first longitude
# This is the CAM-FV offset grid for the 129x256 CAM-FV scalar grid above
ncks --rgr grd_ttl='Equi-Angular grid 128x256'#latlon=128,256#lat_typ=uni#lon_typ=grn_wst \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# T42 Gaussian grid, first longitude centered at Greenwich
ncks --rgr grd_ttl='T42 Gaussian grid'#latlon=64,128#lat_typ=gss#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# NASA Climate Modeling Grid (CMG) 3600x7200 (0.05x0.05 degree) Equi-Angular grid
# Date-line west edge of first longitude, east edge of last longitude
# Write to compressed netCDF4-classic file to reduce filesize ~140x from 2.2 GB to 16 MB
ncks -7 -L 1 \
     --rgr grd_ttl='Equi-Angular grid 3600x7200 (NASA CMG)'#latlon=3600,7200#lat_typ=uni#lon_typ=180_wst \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# DOE E3SM/ACME High Resolution Topography (1 x 1 km grid) for Elevation Classes
# Write to compressed netCDF4-classic file to reduce filesize from ~85 GB to 607 MB
ncks -O -7 -L 1 \
     --rgr grd_ttl='Global latxlon = 18000x36000 ~ 1 x 1 km'#latlon=18000,36000#lat_typ=uni#lon_typ=grn_ctr \
     --rgr scrip=${DATA}/grids/ ~zender/nco/data/ ~/

# 1x1 degree Equi-Angular Regional grid over Greenland, centered longitudes
ncks --rgr grd_ttl='Equi-Angular Greenland grid'#latlon=30,90#snwe=55.0,85.0,-90.0,0.0#lat_typ=uni#lon_typ=grn_ctr \
     --rgr scrip=${HOME}/ ~zender/nco/data/ ~/

Often researchers face the problem not of generating a known, idealized grid but of understanding an unknown, possibly irregular or curvilinear grid underlying a dataset produced elsewhere. NCO will infer the grid of a datafile by examining its coordinates (and boundaries, if available), reformat that information as necessary to diagnose gridcell areas, and output the results in SCRIP format. As of NCO version 4.5.3, released in October, 2015, the ‘--rgr infer’ flag activates the machinery to infer the grid rather than construct the grid from other user-specified switches. To infer the grid properties, NCO interrogates input-file for horizontal coordinate information, such as the presence of dimension names rooted in latitude/longitude-naming traditions and conventions. Once NCO identifies the likely horizontal dimensions it looks for horizontal coordinates and bounds. If bounds are not found, NCO assumes the underlying grid comprises quadrilateral cells whose edges are midway between cell centers, for both rectilinear and curvilinear grids.

# Infer AIRS swath grid from input, write it to
ncks --rgr infer --rgr scrip=${DATA}/sld/rgr/ \
     ${DATA}/sld/raw/ ~/

When inferring grids, the grid file ( is written in SCRIP format, the input file ( is read, and the output file ( is overwritten (its contents are immaterial).

As of NCO version 4.6.6, released in April, 2017, inferred 2D rectangular grids may also be written in UGRID-format (defined here). Request a UGRID mesh with the option ‘--rgr ugrid=fl_ugrid’. Currently both UGRID and SCRIP grids must be requested in order to produce the UGRID output, e.g.,

ncks --rgr infer --rgr ugrid=${HOME}/ \
     --rgr scrip=${HOME}/ ~/ ~/

The SCRIP gridfile and UGRID meshfile metadata produced for the equiangular 1-by-1 degree global grid are:

zender@aerosol:~$ ncks -m ~/ 
netcdf grd_scrip {
    grid_corners = 4 ;
    grid_rank = 2 ;
    grid_size = 64800 ;

    double grid_area(grid_size) ;
      grid_area:units = "steradian" ;

    double grid_center_lat(grid_size) ;
      grid_center_lat:units = "degrees" ;

    double grid_center_lon(grid_size) ;
      grid_center_lon:units = "degrees" ;

    double grid_corner_lat(grid_size,grid_corners) ;
      grid_corner_lat:units = "degrees" ;

    double grid_corner_lon(grid_size,grid_corners) ;
      grid_corner_lon:units = "degrees" ;

    int grid_dims(grid_rank) ;

    int grid_imask(grid_size) ;
} // group /

zender@aerosol:~$ ncks -m ~/ 
netcdf grd_ugrid {
    maxNodesPerFace = 4 ;
    nEdges = 129240 ;
    nFaces = 64800 ;
    nNodes = 64442 ;
    two = 2 ;

    int mesh ;
      mesh:cf_role = "mesh_topology" ;
      mesh:standard_name = "mesh_topology" ;
      mesh:long_name = "Topology data" ;
      mesh:topology_dimension = 2 ;
      mesh:node_coordinates = "mesh_node_x mesh_node_y" ;
      mesh:face_node_connectivity = "mesh_face_nodes" ;
      mesh:face_coordinates = "mesh_face_x mesh_face_y" ;
      mesh:face_dimension = "nFaces" ;
      mesh:edge_node_connectivity = "mesh_edge_nodes" ;
      mesh:edge_coordinates = "mesh_edge_x mesh_edge_y" ;
      mesh:edge_dimension = "nEdges" ;

    int mesh_edge_nodes(nEdges,two) ;
      mesh_edge_nodes:cf_role = "edge_node_connectivity" ;
      mesh_edge_nodes:long_name = "Maps every edge to the two nodes that it connects" ;
      mesh_edge_nodes:start_index = 0 ;

    double mesh_edge_x(nEdges) ;
      mesh_edge_x:standard_name = "longitude" ;
      mesh_edge_x:long_name = "Characteristic longitude of 2D mesh face" ;
      mesh_edge_x:units = "degrees_east" ;

    double mesh_edge_y(nEdges) ;
      mesh_edge_y:standard_name = "latitude" ;
      mesh_edge_y:long_name = "Characteristic latitude of 2D mesh face" ;
      mesh_edge_y:units = "degrees_north" ;

    int mesh_face_nodes(nFaces,maxNodesPerFace) ;
      mesh_face_nodes:cf_role = "face_node_connectivity" ;
      mesh_face_nodes:long_name = "Maps every face to its corner nodes" ;
      mesh_face_nodes:start_index = 0 ;
      mesh_face_nodes:_FillValue = -2147483648 ;

    double mesh_face_x(nFaces) ;
      mesh_face_x:standard_name = "longitude" ;
      mesh_face_x:long_name = "Characteristic longitude of 2D mesh edge" ;
      mesh_face_x:units = "degrees_east" ;

    double mesh_face_y(nFaces) ;
      mesh_face_y:standard_name = "latitude" ;
      mesh_face_y:long_name = "Characteristic latitude of 2D mesh edge" ;
      mesh_face_y:units = "degrees_north" ;

    double mesh_node_x(nNodes) ;
      mesh_node_x:standard_name = "longitude" ;
      mesh_node_x:long_name = "Longitude of mesh nodes" ;
      mesh_node_x:units = "degrees_east" ;

    double mesh_node_y(nNodes) ;
      mesh_node_y:standard_name = "latitude" ;
      mesh_node_y:long_name = "Latitude of mesh nodes" ;
      mesh_node_y:units = "degrees_north" ;
} // group /

Another task that arises in regridding is characterizing new grids. In such cases it can be helpful to have a “skeleton” version of a dataset on the grid, so that grid center and interfaces locations can be assessed, continental outlines can be examined, or the skeleton can be manually populated with data rather than relying on a model. SCRIP files can be difficult to visualize and manipulate, so NCO will provide, if requested, a so-called skeleton file on the user-specified grid. As of NCO version 4.5.3, released in October, 2015, the ‘--rgr skl=fl_skl’ switch outputs the skeleton file to fl_skl. The skeleton file may then be examined in a dataset viewer, populated with data, and generally serve as a template for what to expect from datasets of the same geometry.

# Generate T42 Gaussian grid file and skeleton file
ncks --rgr skl=${DATA}/grids/ --rgr scrip=${DATA}/grids/ \
     --rgr latlon=64,128#lat_typ=gss#lon_typ=Grn_ctr \
     ~zender/nco/data/ ~/

When generating skeleton files, both the grid file ( and the skeleton file ( are written, the input file ( is ignored, and the output file ( is overwritten (its contents are immaterial).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.24 Regridding

Availability: ncclimo, ncks, ncremap
Short options: None
Long options: ‘--map map-file’ or ‘--rgr_map map-file
--rgr key=val’ (multiple invocations allowed)
--rnr=wgt_thr’ or ‘--rgr_rnr=wgt_thr’ or ‘--renormalize=wgt_thr

NCO includes extensive regridding features in ncclimo (as of version 4.6.0 in May, 2016), ncremap (as of version 4.5.4 in November, 2015) and ncks (since version 4.5.0 in June, 2015). Regridding can involve many choices, options, inputs, and outputs. The appropriate operator for this workflow is the ncremap script which automatically handles many details of regridding and passes the required commands to ncks and external programs. Occasionally users need access to lower-level remapping functionality present in ncks and not exposed to direct manipulation through ncremap or ncclimo. This section describes the lower-level functionality and switches as implemented in ncks. Knowing what these features are will help ncremap and ncclimo users understand the full potential of these operators.

ncks supports horizontal regridding of datasets where the grids and weights are all stored in an external map-file. Use the ‘--map’ or ‘--rgr_map’ options to specify the map-file, and NCO will regrid the input-file to a new (or possibly the same, aka, an identity mapping) horizontal grid in the output-file, using the input and output grids and mapping weights specified in the ESMF- or SCRIP-format map-file. Currently NCO understands only the mapfile format pioneered by SCRIP ( and later extended by ESMF (, and adopted (along with Exodus) by TempestRemap ( See those references for documentation on map formats, grid specification, and weight generation.

The regridding currently supported by NCO could equally well be called weight-application. NCO reads-in pre-stored weights from the map-file and applies them to (almost) every variable, thereby creating a regridded output-file. Specify regridding with a standard ncks command and options along with the additional specification of a map-file:

# Regrid entire file, same output format as input:
# Entire file, netCDF4 output:
ncks -4
# Deflated netCDF4 output
ncks -4 -L 1
# Selected variables
ncks -v FS.?,T
# Threading
ncks -t 8
# Deflated netCDF4 output, threading, selected variables:
ncks -4 -L 1 -t 8 -v FS.?,T

OpenMP threading works well with regridding large datasets. Threading improves throughput of regridding 1–10 GB files by factors of 2–5. Options specific to regridding are described below.

NCO supports 1D⇒1D, 1D⇒2D, 2D⇒1D, and 2D⇒2D regridding for any unstructured 1D-grid and any rectangular 2D-grid. This has been tested by converting among and between Gaussian, equiangular, FV, unstructured cubed-sphere grids, and regionally refined grids. Support for irregular 2D- and regional grids (e.g., swath-like data) is planned.


Conservative regridding is, for first-order accurate algorithms, a straightforward procedure of identifying gridcell overlap and apportioning values correctly from source to destination. The presence of missing values forces a decision on how to handle destination gridcells where some but not all source cells are valid. NCO allows the user to choose between two distinct algorithms: “conservative” and “renormalized”. The “conservative” algorithm uses all valid data from the input grid on the output grid once and only once. Destination cells receive the weighted valid values of the source cells. This is conservative because the global integrals of the source and destination fields are equal. The “renormalized” algorithm divides the destination value by the sum of the valid weights. This produces values equal to the mean of the valid input values, but extended to the entire destination gridcell. Thus renormalization is equivalent to extrapolating valid data to missing regions. Input and output integrals are unequal and renormalized regridding is not conservative. Both algorithms produce identical answers when no missing data maps to the destination gridcell.

The renormalized algorithm is useful because it solves some problems, like producing physically unrealistic temperature values, at the expense of incurring others, like non-conservation. Many land and ocean modelers eschew unrealistic gridpoint values, and conservative regridding often produces “weird” values along coastlines or missing data gaps where state variables are regridded to/from small fractions of a gridcell. Renormalization ensures the output values are physically consistent, although the integral of their value times area is not conservative.

By default, NCO implements the “conservative” algorithm because it has useful properties, is simpler to understand, and requires no additional parameters. To employ the “renormalized” algorithm instead, use the ‘--rnr’, ‘--rgr_rnr’, or ‘--renormalize’ options to supply wgt_thr, the threshold weight for valid destination values. Valid values must cover at least the fraction wgt_thr of the destination gridcell to meet the threshold for a non-missing destination value. When wgt_thr is exceeded, the mean valid value is renormalized by the valid area and placed in the destination gridcell. If the valid area covers less than wgt_thr, then the destination gridcell is assigned the missing value. Valid values of wgt_thr range from zero to one. Keep in mind though, that this threshold is potentially a divisor, and values of zero or very near to zero can lead to floating-point underflow and divide-by-zero errors. For convenience NCO permits users to specify a wgt_thr = 0.0 threshold weight. This indicates that any valid data should be represented and renormalized on the output grid.

ncks  # Conservative regridding
ncks --rnr=0.1 # Renormalized regridding

The first example uses the default conservative algorithm. The second example specifies that valid values must cover at least 10% of the destination gridcell to meet the threshold for a non-missing destination value. With valid destination areas of, say 25% or 50%, the renormalized algorithm would produce destination values greater than the conservative algorithm by factors of four or two, respectively.

In practice, it may make sense to use the default “conservative” algorithm when performing conservative regridding, and the “renormalized” algorithm when performing other regridding such as bilinear interpolation or nearest-neighbor. Another consideration is whether the fields being regridded are fluxes or state variables. For example, temperature (unlike heat) and concentrations (amount per unit volume) are not physically conserved quantities under areal-regridding so it often makes sense to interpolate them in a non-conservative fashion, to preserve their fine-scale structure. Few researchers can digest the unphysical values of temperature that the “conservative” option will produce in regions rife with missing values. A counter-example is fluxes, which should be physically conserved under areal-regridding. One should consider both the type of field and its conservation properties when choosing a regridding strategy.

NCO automatically annotates the output with relevant metadata such as coordinate bounds, axes, and vertices (à la CF). These annotations include

Horizontal Dimension Names: lat, lon

The name of the horizontal spatial dimensions assumed to represent latitude and longitude in 2D rectangular input files are lat_nm and lon_nm, which default to lat and lon, respectively. Variables that contain a lat_nm-dimension and a lon_nm-dimension on a 2D-rectangular input grid will be regridded, and variables regridded to a 2D-rectangular output grid will all contain the lat_nm- and lon_nm-dimensions, and variables regridded to a 1D-unstructured output grid will have lat_nm and lon_nm as auxiliary coordinate variables. To treat different dimensions and variables as latitude and longitude, use the options ‘--rgr lat_nm=lat_nm’ and ‘--rgr lon_nm=lon_nm’. Note that, for now at least, lat_nm and lon_nm indicate both the variable names associated and, where applicable (i.e., on 2D-grids), the dimensions of the horizontal coordinates.

Unstructured Dimension Name: col

The name of the horizontal spatial dimension assumed to delineate an unstructured grid is col_nm, which defaults to ncol (number of columns), the name CAM employs. Other common names for the columns in an unstructured grid include lndgrid (used by CLM), and nCells (used by MPAS-O). Variables that contain the col_nm-dimension on an unstructured input grid will be regridded, and regridded variables written to an unstructured output grid will all contain the col_nm-dimension. To treat a different dimension as unstructured, use the option ‘--rgr col_nm=col_nm’. Note: Often there is no coordinate variable for the col_nm-dimension, i.e., there is no variable named col_nm, although such a coordinate could contain useful information about the unstructured grid.

Structured Grid Standard Names and Units

Longitude and latitude coordinates (both regular and auxiliary, i.e., for unstructured grids) receive CF standard_name values of latitude and longitude, CF axes attributes with values X and Y, and units attributes with values degrees_east and degrees_north, respectively.

Unstructured Grid Auxiliary Coordinates

Unstructured grid auxiliary coordinates for longitude and latitude receive CF coordinates attributes with values lon and lat, respectively.

Structured Grid Bounds Variables: bnd, lat_bnd, lon_bnd

Structured grids with 1D-coordinates use the dimension bnd_nm (which defaults to nbnd) with the spatial bounds variables in lat_bnd_nm and lon_bnd_nm which default to lon_bnds and lat_bnds, respectively. By default spatial bounds for such structured grids parallel the oft-used temporal bounds dimension (nbnd=2) and variable (time_bnds). Bounds are attached to the horizontal spatial dimensions via their bounds attributes. Change the spatial bounds dimension with the option ‘--rgr bnd_nm=bnd_nm’. Rename the spatial bounds variables with the options ‘--rgr lat_bnd_nm=lat_bnd_nm’ and ‘--rgr lon_bnd_nm=lon_bnd_nm’.

Unstructured Grid Bounds Variables: bnd, lat_bnd, lon_bnd

Unstructured grids with 1D-coordinates use the dimension bnd_nm (which defaults to nv, number of vertices) for the spatial bounds variables lat_bnd_nm and lon_bnd_nm which default to lat_vertices and lon_vertices, respectively. It may be impossible to re-use the temporal bounds dimension (often nbnd) for unstructure grids, because the gridcells are not rectangles, and thus require specification of all vertices for each gridpoint, rather than only two parallel interfaces per dimension. These bounds are attached to the horizontal spatial dimensions via their bounds attributes. Change the spatial bounds dimension with the option ‘--rgr bnd_nm=bnd_nm’. Rename the spatial bounds variables with the options ‘--rgr lat_bnd_nm=lat_bnd_nm’ and ‘--rgr lon_bnd_nm=lon_bnd_nm’. The temporal bounds dimension in unstructured grid output remains as in the input-file, usually nbnd.

Gridcell Area: area

The variable area_nm (which defaults to area) is, by default, (re-)created in the output_file to hold the gridcell area in steradians. To store the area in a different variable, use the option ‘--rgr area=area_nm’. The area_nm variable receives a standard_name attribute of cell_area, a units attribute of steradian (the SI unit of solid angle), and a cell_methods attribute with value lat, lon: sum, which indicates that area_nm is extensive, meaning that its value depends on the gridcell boundaries. Since area_nm is a property of the grid, it is read directly from the map-file rather than regridded itself. To omit the area variable from the output file, set the no_area_out flag. The --no_cll_msr switch to ncremap and ncclimo does this automatically.

Gridcell Fraction: frc

The variable frc_nm (which defaults to frac_b) is automatically copied to the output_file to hold the valid fraction of each gridcell when certain conditions are met. First, the regridding method must be conservative. Second, at least one value of frc_nm must be non-unity. These conditions ensure that whenever fractional gridcells affect the regridding, they are also placed in the output file. To store the fraction in a different variable, use the option ‘--rgr frc_nm=frc_nm’. The frc_nm variable receives a cell_methods attribute with value lat, lon: sum, which indicates that frc_nm is extensive, meaning that its value depends on the gridcell boundaries. Since frc_nm is a property of the grid, it is read directly from the map-file rather than regridded itself.

Gridcell Mask: mask

The variable msk_nm (which defaults to mask) can, if present, be copied from the map-file to hold the gridcell mask on the destination grid in output-file. To store the mask in a different variable, use the option ‘--rgr msk_nm=msk_nm’. Since msk_nm is a property of the grid, it is read directly from the map-file rather than regridded itself. To include the mask variable in the output file, set the msk_out flag. To omit the mask variable from the output file, set the no_msk_out flag. In grid inferral and map-generation modes, this option tells the regridder to generate an integer mask map from the variable msk_nm. The mask will be one (i.e., points at that location will contribute to regridding weights) where msk_nm has valid values. The mask will be zero (i.e., points at that location will not contribute to regridding weights) where msk_nm has a missing value. This feature is useful when creating weights between masked grids, e.g., ocean-only points or land-only points.

Latitude weights: lat_wgt

Rectangular 2D-grids use the variable lat_wgt_nm, which defaults to gw (originally for “Gaussian weight”), to store the 1D-weight appropriate for area-weighting the latitude grid. To store the latitude weight in a different variable, use the option ‘--rgr lat_wgt=lat_wgt_nm’. The lat_wgt_nm variable will not appear in 1D-grid output. Weighting statistics by latitude (i.e., by lat_wgt_nm will produce the same answers (up-to round-off error) as weighting by area (i.e., by area_nm) in grids that have both variables. The former requires less memory because lat_wgt_nm is 1D), whereas the latter is more general because area_nm works on any grid.

Provenance Attributes

The map-file and input-file names are stored in the output-file global attributes mapping_file and source_file, respectively.

Staggered Grid Coordinates and Weights

Owing to its heritage as an early CCM analysis tool, NCO tries to create output interoperable with other CESM analysis tools. Like many models, CAM computes and archives thermodynamic state variables on gridcell centers, while dynamics variables (U, V) are on gridcell edges (interfaces). The dual-grid, sometimes called the “staggered grid”, formed by connecting edge centers is thus the natural location for storing output dynamics variables. Some analysis packages, such as the AMWG diagnostics, require access to these dual-grid coordinates with the names slat and slon (for “staggered” latitude and longitude). By default the NCO regridder outputs these coordinates, along with the latitude weights (called w_stag), when the input is on a cap (aka FV) grid so that the result can be processed by AMWG diagnostics. Turn-off archiving the staggered grid (i.e., slat, slon, and w_stag) by setting the no_stagger flag. The --no_stg_grd flag in ncremap and ncclimo sets this --no_stagger flag.

One may supply muliple ‘--rgr key=value’ options to simultaneously customize multiple grid-field names. The following examples may all be assumed to end with the standard options ‘’.

ncks --rgr lat_nm=latitude --rgr lon_nm=longitude
ncks --rgr col_nm=column --rgr lat_wgt=lat_wgt
ncks --rgr bnd_nm=bounds --rgr lat_bnd_nm=lat_bounds --rgr lon_bnd_nm=lon_bounds
ncks --rgr bnd_nm=vertices --rgr lat_bnd_nm=lat_vrt --rgr lon_bnd_nm=lon_vrt

The first command causes the regridder to associate the latitude and longitude dimensions with the dimension names latitude and longitude (instead of the defaults, lat and lon). The second command causes the regridder to associate the independent columns in an unstructured grid with the dimension name column (instead of the default, ncol) and the variable containing latitude weights to be named lat_wgt (instead of the default, gw). The third command associates the latitude and longitude bounds with the dimension bounds (instead of the default, nbnd) and the variables lat_bounds and lon_bounds (instead of the defaults, lat_bnds and lon_bnds, respectively). The fourth command associates the latitude and longitude bounds with the dimension vertices (instead of the default, nv) and the variables lat_vrt and lon_vrt (instead of the defaults, lat_vertices and lon_vertices, respectively).

When used with an identity remapping files, regridding can signficantly enhance the metadata and therefore the dataset usability. Consider these selected metadata (those unchanged are not shown for brevity) associated with the variable FSNT from typical unstructured grid (CAM-SE cubed-sphere) output before and after an identity regridding:

# Raw model output before regridding
netcdf ne30_FSNT {
    nbnd = 2 ;
    ncol = 48602 ;
    time = UNLIMITED ; // (1 currently)

    float FSNT(time,ncol) ;
      FSNT:long_name = "Net solar flux at top of model" ;

    double time(time) ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;

    double time_bnds(time,nbnd) ;
      time_bnds:long_name = "time interval endpoints" ;
} // group /

# Same model output after identity regridding
netcdf dogfood {
    nbnd = 2 ;
    ncol = 48602 ;
    nv = 5 ;
    time = 1 ;

    float FSNT(time,ncol) ;
      FSNT:long_name = "Net solar flux at top of model" ;
      FSNT:coordinates = "lat lon" ;

    double lat(ncol) ;
      lat:long_name = "latitude" ;
      lat:standard_name = "latitude" ;
      lat:units = "degrees_north" ;
      lat:axis = "Y" ;
      lat:bounds = "lat_vertices" ;
      lat:coordinates = "lat lon" ;

    double lat_vertices(ncol,nv) ;
      lat_vertices:long_name = "gridcell latitude vertices" ;

    double lon(ncol) ;
      lon:long_name = "longitude" ;
      lon:standard_name = "longitude" ;
      lon:units = "degrees_east" ;
      lon:axis = "X" ;
      lon:bounds = "lon_vertices" ;
      lon:coordinates = "lat lon" ;

    double lon_vertices(ncol,nv) ;
      lon_vertices:long_name = "gridcell longitude vertices" ;

    double time(time) ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;

    double time_bnds(time,nbnd) ;
      time_bnds:long_name = "time interval endpoints" ;
} // group /

The raw model output lacks the CF coordinates and bounds attributes that the regridder adds. The metadata turns lat and lon into auxiliary coordinate variables (see Auxiliary Coordinates) which can then be hyperslabbed (with ‘-X’) using latitude/longitude coordinates bounding the region of interest:

% ncks -u -H -X 314.6,315.3,-35.6,-35.1 -v FSNT
time[0]=31 ncol[0] FSNT[0]=344.575 W/m2

ncol[0] lat[0]=-35.2643896828 degrees_north

ncol[0] nv[0] lat_vertices[0]=-35.5977213708 
ncol[0] nv[1] lat_vertices[1]=-35.5977213708 
ncol[0] nv[2] lat_vertices[2]=-35.0972113817 
ncol[0] nv[3] lat_vertices[3]=-35.0972113817 
ncol[0] nv[4] lat_vertices[4]=-35.0972113817 

ncol[0] lon[0]=315 degrees_east

ncol[0] nv[0] lon_vertices[0]=315 
ncol[0] nv[1] lon_vertices[1]=315 
ncol[0] nv[2] lon_vertices[2]=315.352825437 
ncol[0] nv[3] lon_vertices[3]=314.647174563 
ncol[0] nv[4] lon_vertices[4]=314.647174563 

time[0]=31 days since 1979-01-01 00:00:00

time[0]=31 nbnd[0] time_bnds[0]=0 
time[0]=31 nbnd[1] time_bnds[1]=31 

Thus auxiliary coordinate variables help to structure unstructured grids. The expanded metadata annotations from an identity regridding may obviate the need to place unstructured data on a rectangular grid. For example, statistics for regions that can be expressed as unions of rectangular regions can now be performed on the native (unstructured) grid.

Here are some quick examples of regridding from common models. All examples require ‘’ at the end.

# Identity re-map E3SM/ACME CAM-SE Cubed-Sphere output (to improve metadata)
ncks --map=${DATA}/maps/
# Convert E3SM/ACME CAM-SE Cubed Sphere output to rectangular lat/lon
ncks --map=${DATA}/maps/
# Convert CAM3 T42 output to Cubed-Sphere grid
ncks --map=${DATA}/maps/

Next: , Previous: , Up: Shared features   [Contents][Index]

3.25 UDUnits Support

Availability: ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-d dim,[min][,[max][,[stride]]]
Long options: ‘--dimension dim,[min][,[max][,[stride]]]’,
--dmn dim,[min][,[max][,[stride]]]

There is more than one way to hyperskin a cat. The UDUnits package provides a library which, if present, NCO uses to translate user-specified physical dimensions into the physical dimensions of data stored in netCDF files. Unidata provides UDUnits under the same terms as netCDF, so sites should install both. Compiling NCO with UDUnits support is currently optional but may become required in a future version of NCO.

Two examples suffice to demonstrate the power and convenience of UDUnits support. First, consider extraction of a variable containing non-record coordinates with physical dimensions stored in MKS units. In the following example, the user extracts all wavelengths in the visible portion of the spectrum in terms of the units very frequently used in visible spectroscopy, microns:

% ncks -C -H -v wvl -d wvl,"0.4 micron","0.7 micron"
wvl[0]=5e-07 meter

The hyperslab returns the correct values because the wvl variable is stored on disk with a length dimension that UDUnits recognizes in the units attribute. The automagical algorithm that implements this functionality is worth describing since understanding it helps one avoid some potential pitfalls. First, the user includes the physical units of the hyperslab dimensions she supplies, separated by a simple space from the numerical values of the hyperslab limits. She encloses each coordinate specifications in quotes so that the shell does not break the value-space-unit string into separate arguments before passing them to NCO. Double quotes ("foo") or single quotes ('foo') are equally valid for this purpose. Second, NCO recognizes that units translation is requested because each hyperslab argument contains text characters and non-initial spaces. Third, NCO determines whether the wvl is dimensioned with a coordinate variable that has a units attribute. In this case, wvl itself is a coordinate variable. The value of its units attribute is meter. Thus wvl passes this test so UDUnits conversion is attempted. If the coordinate associated with the variable does not contain a units attribute, then NCO aborts. Fourth, NCO passes the specified and desired dimension strings (microns are specified by the user, meters are required by NCO) to the UDUnits library. Fifth, the UDUnits library that these dimension are commensurate and it returns the appropriate linear scaling factors to convert from microns to meters to NCO. If the units are incommensurate (i.e., not expressible in the same fundamental MKS units), or are not listed in the UDUnits database, then NCO aborts since it cannot determine the user’s intent. Finally, NCO uses the scaling information to convert the user-specified hyperslab limits into the same physical dimensions as those of the corresponding cooridinate variable on disk. At this point, NCO can perform a coordinate hyperslab using the same algorithm as if the user had specified the hyperslab without requesting units conversion.

The translation and dimensional interpretation of time coordinates shows a more powerful, and probably more common, UDUnits application. In this example, the user prints all data between 4 PM and 7 PM on December 8, 1999, from a variable whose time dimension is hours since the year 1900:

% ncks -u -H -C -v time_udunits -d time_udunits,"1999-12-08 \
  16:00:0.0","1999-12-08 19:00:0.0"
time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0

Here, the user invokes the stride (see Stride) capability to obtain every other timeslice. This is possible because the UDUnits feature is additive, not exclusive—it works in conjunction with all other hyperslabbing (see Hyperslabs) options and in all operators which support hyperslabbing. The following example shows how one might average data in a time period spread across multiple input files

ncra -d time,"1939-09-09 12:00:0.0","1945-05-08 00:00:0.0" \

Note that there is no excess whitespace before or after the individual elements of the ‘-d’ argument. This is important since, as far as the shell knows, ‘-d’ takes only one command-line argument. Parsing this argument into its component dim,[min][,[max][,[stride]]] elements (see Hyperslabs) is the job of NCO. When unquoted whitespace is present between these elements, the shell passes NCO arugment fragments which will not parse as intended.

NCO implemented support for the UDUnits2 library with version 3.9.2 (August, 2007). The UDUnits2 package supports non-ASCII characters and logarithmic units. We are interested in user-feedback on these features.

One aspect that deserves mention is that UDUnits, and thus NCO, supports run-time definition of the location of the relevant UDUnits databases. With UDUnits version 1, users may specify the directory which contains the UDUnits database, udunits.dat, via the UDUNITS_PATH environment variable. With UDUnits version 2, users may specify the UDUnits database file itself, udunits2.xml, via the UDUNITS2_XML_PATH environment variable.

# UDUnits1
export UDUNITS_PATH='/unusual/location/share/udunits'
# UDUnits2
export UDUNITS2_XML_PATH='/unusual/location/share/udunits/udunits2.xml'

This run-time flexibility can enable the full functionality of pre-built binaries on machines with libraries in different locations.

The UDUnits package documentation describes the supported formats of time dimensions. Among the metadata conventions that adhere to these formats are the Climate and Forecast (CF) Conventions and the Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions. The following ‘-d arguments’ extract the same data using commonly encountered time dimension formats:

-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11 00:00:0.0','1939-09-09 00:00:0.0'
-d time,'1918-11-11T00:00:0.0Z','1939-09-09T00:00:0.0Z'
-d time,'1918-11-11','1939-09-09'
-d time,'1918-11-11','1939-9-9'

All of these formats include at least one dash - in a non-leading character position (a dash in a leading character position is a negative sign). NCO assumes that a space, colon, or non-leading dash in a limit string indicates that a UDUnits units conversion is requested. Some date formats like YYYYMMDD that are valid in UDUnits are ambiguous to NCO because it cannot distinguish a purely numerical date (i.e., no dashes or text characters in it) from a coordinate or index value:

-d time,1918-11-11 # Interpreted as the date November 11, 1918
-d time,19181111   # Interpreted as time-dimension index 19181111
-d time,19181111.  # Interpreted as time-coordinate value 19181111.0

Hence, use the YYYY-MM-DD format rather than YYYYMMDD for dates.

As of version 4.0.0 (January, 2010), NCO supports some calendar attributes specified by the CF conventions.

Supported types:

"365_day"/"noleap", "360_day", "gregorian", "standard"

Unsupported types:


Unsupported types default to mixed Gregorian/Julian as defined by UDUnits.

An Example: Consider the following netCDF variable

  double lon_cal(lon_cal) ;
    lon_cal:long_name = "lon_cal" ;
    lon_cal:units = "days since 1964-2-28 0:0:0" ;
    lon_cal:calendar = "365_day" ;
  lon_cal = 1,2,3,4,5,6,7,8,9,10;

ncks -v lon_cal -d lon_cal,'1964-3-1 0:00:0.0','1964-3-4 00:00:0.0'’ results in lon_cal=1,2,3,4.

netCDF variables should always be stored with MKS (i.e., God’s) units, so that application programs may assume MKS dimensions apply to all input variables. The UDUnits feature is intended to alleviate some of the NCO user’s pain when handling MKS units. It connects users who think in human-friendly units (e.g., miles, millibars, days) to extract data which are always stored in God’s units, MKS (e.g., meters, Pascals, seconds). The feature is not intended to encourage writers to store data in esoteric units (e.g., furlongs, pounds per square inch, fortnights).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.26 Rebasing Time Coordinate

Availability: ncra, ncrcat Short options: None

Time rebasing is invoked when numerous files share a common record coordinate, and the record coordinate basetime (not the time increment, e.g., days or hours) changes among input files. The rebasing is performed automatically if and only if UDUnits is installed. Rebasing occurs when the record coordinate is a time-based variable, and times are recorded in units of a time-since-basetime, and the basetime changes from file to file. Since the output file can have only one unit (i.e., one basetime) for the record coordinate, NCO, in such cases, chooses the units of the first input file to be the units of the output file. It is necessary to “rebase” all the input record variables to this output time unit in order for the output file to have the correct values.

For example suppose the time coordinate is in hours and each day in January is stored in its own daily file. Each daily file records the temperature variable tpt(time) with an (unadjusted) time coordinate value between 0–23 hours, and uses the units attribute to advance the base time: time:units="hours since 1990-1-1" time:units="hours since 1990-1-2"   
... time:units="hours since 1990-1-31"   
// Mean noontime temperature in January
ncra -v tpt -d time,"1990-1-1 12:00:00","1990-1-31 23:59:59",24 \

// Concatenate day2 noon through day3 noon records
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \    

// Results: time is "re-based" to the time units in ""
time=36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, \
     51, 52, 53, 54, 55, 56, 57, 58, 59 ;
// If we repeat the above command but with only two input files...
ncrcat -v tpt -d time,"1990-1-2 12:00:00","1990-1-3 11:59:59" \ file03    

// ...then output time coordinate is based on time units in ""
time = 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, \ 
     26, 27, 28, 29, 30, 31, 32, 33, 34, 35 ;

As of NCO version 4.2.1 (August, 2012), NCO automatically rebases not only the record coordinate (time, here) but also any cell boundaries associated with the record coordinate (e.g., time_bnds) (see CF Conventions).

As of NCO version 4.4.9 (May, 2015), NCO also rebases any climatology boundaries associated with the record coordinate (e.g., climatology_bounds) (see CF Conventions).

As of NCO version 4.6.3 (December, 2016), NCO also rebases the time coordinate when the unit between files differ For example the first file may have units="days since 2014-03-01" and the second file units="hours since 2014-03-10 00:00".

Next: , Previous: , Up: Shared features   [Contents][Index]

3.27 Multiple Record Dimensions

Availability: ncecat, ncpdq Short options: None
Long options: ‘--mrd

The netCDF3 file format allows only one record dimension, and that dimension must be the first dimension (i.e., the least rapidly varying dimension) of any variable in which it appears. This imposes certain rules on how operators must perform operations that alter the ordering of dimensions or the number of record variables. The netCDF4 file format has no such restrictions. Files and variables may have any number of record dimensions in any order. This additional flexibility of netCDF4 can only be realized by selectively abandoning the constraints that would make operations behave completely consistently between netCDF3 and netCDF4 files.

NCO chooses, by default, to impose netCDF3-based constraints on netCDF4 files. This reduces the number of unanticipated consequences and keeps the operators functioning in a familiar way. Put another way, NCO limits production of additional record dimensions so processing netCDF4 files leads to the same results as processing netCDF3 files. Users can override this default with the ‘--mrd’ (or ‘--multiple_record_dimension’) switch, which enables netCDF4 variables to accumulate additional record dimensions.

How can additional record dimensions be produced? Most commonly ncecat (in record-aggregate mode) defines a new leading record dimension. In netCDF4 files this becomes an additional record dimension unless the original record dimension is changed to a fixed dimension (as must be done in netCDF3 files). Also when ncpdq reorders dimensions it can preserve the “record” property of record variables. ncpdq tries to define as a record dimension whichever dimension ends up first in a record variable, and, in netCDF4 files, this becomes an additional record dimension unless the original record dimension is changed to a fixed dimension (as must be done in netCDF3 files). It it easier if ncpdq and ncecat do not increase the number of record dimensions in a variable so that is the default. Use ‘--mrd’ to override this.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.28 Missing values

Availability: ncap2, ncbo, ncclimo, nces, ncflint, ncpdq, ncra, ncremap, ncwa
Short options: None

The phrase missing data refers to data points that are missing, invalid, or for any reason not intended to be arithmetically processed in the same fashion as valid data. All NCO arithmetic operators attempt to handle missing data in an intelligent fashion. There are four steps in the NCO treatment of missing data:

  1. Identifying variables that may contain missing data.

    NCO follows the convention that missing data should be stored with the _FillValue specified in the variable’s _FillValue attributes. The only way NCO recognizes that a variable may contain missing data is if the variable has a _FillValue attribute. In this case, any elements of the variable which are numerically equal to the _FillValue are treated as missing data.

    NCO adopted the behavior that the default attribute name, if any, assumed to specify the value of data to ignore is _FillValue with version 3.9.2 (August, 2007). Prior to that, the missing_value attribute, if any, was assumed to specify the value of data to ignore. Supporting both of these attributes simultaneously is not practical. Hence the behavior NCO once applied to missing_value it now applies to any _FillValue. NCO now treats any missing_value as normal data 38.

    It has been and remains most advisable to create both _FillValue and missing_value attributes with identical values in datasets. Many legacy datasets contain only missing_value attributes. NCO can help migrating datasets between these conventions. One may use ncrename (see ncrename netCDF Renamer) to rename all missing_value attributes to _FillValue:

    ncrename -a .missing_value,_FillValue

    Alternatively, one may use ncatted (see ncatted netCDF Attribute Editor) to add a _FillValue attribute to all variables

    ncatted -O -a _FillValue,,o,f,1.0e36
  2. Converting the _FillValue to the type of the variable, if neccessary.

    Consider a variable var of type var_type with a _FillValue attribute of type att_type containing the value _FillValue. As a guideline, the type of the _FillValue attribute should be the same as the type of the variable it is attached to. If var_type equals att_type then NCO straightforwardly compares each value of var to _FillValue to determine which elements of var are to be treated as missing data. If not, then NCO converts _FillValue from att_type to var_type by using the implicit conversion rules of C, or, if att_type is NC_CHAR 39, by typecasting the results of the C function strtod(_FillValue). You may use the NCO operator ncatted to change the _FillValue attribute and all data whose data is _FillValue to a new value (see ncatted netCDF Attribute Editor).

  3. Identifying missing data during arithmetic operations.

    When an NCO arithmetic operator processes a variable var with a _FillValue attribute, it compares each value of var to _FillValue before performing an operation. Note the _FillValue comparison imposes a performance penalty on the operator. Arithmetic processing of variables which contain the _FillValue attribute always incurs this penalty, even when none of the data are missing. Conversely, arithmetic processing of variables which do not contain the _FillValue attribute never incurs this penalty. In other words, do not attach a _FillValue attribute to a variable which does not contain missing data. This exhortation can usually be obeyed for model generated data, but it may be harder to know in advance whether all observational data will be valid or not.

  4. Treatment of any data identified as missing in arithmetic operators.

    NCO averagers (ncra, nces, ncwa) do not count any element with the value _FillValue towards the average. ncbo and ncflint define a _FillValue result when either of the input values is a _FillValue. Sometimes the _FillValue may change from file to file in a multi-file operator, e.g., ncra. NCO is written to account for this (it always compares a variable to the _FillValue assigned to that variable in the current file). Suffice it to say that, in all known cases, NCO does “the right thing”.

    It is impossible to determine and store the correct result of a binary operation in a single variable. One such corner case occurs when both operands have differing _FillValue attributes, i.e., attributes with different numerical values. Since the output (result) of the operation can only have one _FillValue, some information may be lost. In this case, NCO always defines the output variable to have the same _FillValue as the first input variable. Prior to performing the arithmetic operation, all values of the second operand equal to the second _FillValue are replaced with the first _FillValue. Then the arithmetic operation proceeds as normal, comparing each element of each operand to a single _FillValue. Comparing each element to two distinct _FillValue’s would be much slower and would be no likelier to yield a more satisfactory answer. In practice, judicious choice of _FillValue values prevents any important information from being lost.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.29 Chunking

Availability: ncap2, ncbo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: none
Long options: ‘--cnk_byt sz_byt’, ‘--chunk_byte sz_byt
--cnk_csh sz_byt’, ‘--chunk_cache sz_byt
--cnk_dmn dmn_nm,sz_lmn’, ‘--chunk_dimension dmn_nm,sz_lmn
, ‘--cnk_map cnk_map’, ‘--chunk_map cnk_map’,
--cnk_min sz_byt’, ‘--chunk_min sz_byt’,
--cnk_plc cnk_plc’, ‘--chunk_policy cnk_plc’,
--cnk_scl sz_lmn’, ‘--chunk_scalar sz_lmn

All netCDF4-enabled NCO operators that define variables support a plethora of chunksize options. Chunking can significantly accelerate or degrade read/write access to large datasets. Dataset chunking issues are described by THG and Unidata here, here, and here. NCO authors are working on generalized algorithms and applications of chunking strategies (stay tuned for more in 2018).

As of NCO version 4.6.5 (March, 2017), NCO supports run-time alteration of the chunk cache size. By default, the cache size is set (by the --with-chunk-cache-size option to configure) at netCDF compile time. The --cnk_csh sz option sets the cache size to sz bytes for all variables. When the debugging level is set (with -D dbg_lvl) to three or higher, NCO prints the current value of the cache settings for informational purposes. Also ‘--chunk_cache’.

Increasing cache size from the default can dramatically accelerate time to aggregate and rechunk multiple large input datasets, e.g.,

ncrcat -4 -L 1 --cnk_csh=1000000000 --cnk_plc=g3d --cnk_dmn=time,365 \
       --cnk_dmn=lat,1800 --cnk_dmn=lon,3600 in*.nc4

In this example all 3D variables the input datasets (which may or may not be chunked already) are re-chunked to a size of 365 along the time dimension. Because the default chunk cache size of about 4 MB is too small to manipulate the large chunks, we reset the cache to 1 GB. The operation completes much faster, and subsequent reads along the time dimension will be much more rapid.

The NCO chunking implementation is designed to be flexible. Users control four aspects of the chunking implementation. These are the chunking policy, chunking map, chunksize, and minimum chunksize. The chunking policy determines which variables to chunk, and the chunking map determines how (with what exact sizes) to chunk those variables. These are high-level mechanisms that apply to an entire file and all variables and dimensions. The chunksize option allows per-dimension specification of sizes that will override the selected (or default) chunking map.

The distinction between elements and bytes is subtle yet crucial to understand. Elements refers to values of an array, whereas bytes refers to the memory size required to hold the elements. These measures differ by a factor of four or eight for NC_FLOAT or NC_DOUBLE, respectively. The option ‘--cnk_scl’ takes an argument sz_lmn measured in elements. The options ‘--cnk_byt’, ‘--cnk_csh’, and ‘--cnk_min’ take arguments sz_byt measured in bytes.

Use the ‘--cnk_min=sz_byt’ option to set the minimum size in bytes (not elements) of variables to chunk. This threshold is intended to restrict use of chunking to variables for which it is efficient. By default this minimum variable size for chunking is twice the system blocksize (when available) and is 8192 bytes otherwise. Users may set this to any value with the ‘--cnk_min=sz_byt’ switch. To guarantee that chunking is performed on all arrays, regardless of size, set the minimum size to one byte (not to zero bytes).

The chunking implementation is similar to a hybrid of the ncpdq packing policies (see ncpdq netCDF Permute Dimensions Quickly) and hyperslab specifications (see Hyperslabs). Each aspect is intended to have a sensible default, so that many users only need to set one switch to obtain sensible chunking. Power users can tune chunking with the three switches in tandem to obtain optimal performance.

By default, NCO preserves the chunking characteristics of the input file in the output file 40. In other words, preserving chunking requires no switches or user intervention.

Users specify the desired chunking policy with the ‘-P’ switch (or its long option equivalents, ‘--cnk_plc’ and ‘--chunk_policy’) and its cnk_plc argument. As of August, 2014, six chunking policies are implemented:

Chunk All Variables

Definition: Chunk all variables possible. For obvious reasons, scalar variables cannot be chunked.
Alternate invocation: ncchunk
cnk_plc key values: ‘all’, ‘cnk_all’, ‘plc_all
Mnemonic: All

Chunk Variables with at least Two Dimensions [default]

Definition: Chunk all variables possible with at least two dimensions
Alternate invocation: none
cnk_plc key values: ‘g2d’, ‘cnk_g2d’, ‘plc_g2d
Mnemonic: Greater than or equal to 2 Dimensions

Chunk Variables with at least Three Dimensions

Definition: Chunk all variables possible with at least three dimensions
Alternate invocation: none
cnk_plc key values: ‘g3d’, ‘cnk_g3d’, ‘plc_g3d
Mnemonic: Greater than or equal to 3 Dimensions

Chunk One-Dimensional Record Variables

Definition: Chunk all 1-D record variables
Alternate invocation: none
Any specified (with ‘--cnk_dmn’) record dimension chunksizes will be applied only to 1-D record variables (and to no other variables). Other dimensions may be chunked with their own ‘--cnk_dmn’ options that will apply to all variables. cnk_plc key values: ‘r1d’, ‘cnk_r1d’, ‘plc_r1d
Mnemonic: Record 1-D variables

Chunk Variables Containing Explicitly Chunked Dimensions

Definition: Chunk all variables possible that contain at least one dimension whose chunksize was explicitly set with the ‘--cnk_dmn’ option. Alternate invocation: none
cnk_plc key values: ‘xpl’, ‘cnk_xpl’, ‘plc_xpl
Mnemonic: EXPLicitly specified dimensions

Chunk Variables that are already Chunked

Definition: Chunk only variables that are already chunked in the input file. When used in conjunction with ‘cnk_map=xst’ this option preserves and copies the chunking parameters from the input to the output file. Alternate invocation: none
cnk_plc key values: ‘xst’, ‘cnk_xst’, ‘plc_xst
Mnemonic: EXiSTing chunked variables

Chunk Variables with NCO recommendations

Definition: Chunk all variables according to NCO best practices. This is a virtual option that ensures the chunking policy is (in the subjective opinion of the authors) the best policy for typical usage. As of NCO version 4.4.8 (February, 2015), this virtual policy implements ‘map_rew’ for 3-D variables and ‘map_lfp’ for all other variables.
Alternate invocation: none
cnk_plc key values: ‘nco’, ‘cnk_nco’, ‘plc_nco
Mnemonic: NetCDFOperator


Definition: Unchunk all variables possible. The HDF5 storge layer requires that record variables (i.e., variables that contain at least one record dimension) must be chunked. Also variables that are compressed or use checksums must be chunked. Such variables cannot be unchunked.
Alternate invocation: ncunchunk
cnk_plc key values: ‘uck’, ‘cnk_uck’, ‘plc_uck’, ‘none’, ‘unchunk
Mnemonic: UnChunK

Equivalent key values are fully interchangeable. Multiple equivalent options are provided to satisfy disparate needs and tastes of NCO users working with scripts and from the command line.

The chunking algorithms must know the chunksizes of each dimension of each variable to be chunked. The correspondence between the input variable shape and the chunksizes is called the chunking map. The user specifies the desired chunking map with the ‘-M’ switch (or its long option equivalents, ‘--cnk_map’ and ‘--chunk_map’) and its cnk_map argument. Nine chunking maps are currently implemented:

Chunksize Equals Dimension Size

Definition: Chunksize defaults to dimension size. Explicitly specify chunksizes for particular dimensions with ‘--cnk_dmn’ option.
cnk_map key values: ‘dmn’, ‘cnk_dmn’, ‘map_dmn
Mnemonic: DiMeNsion

Chunksize Equals Dimension Size except Record Dimension

Definition: Chunksize equals dimension size except record dimension has size one. Explicitly specify chunksizes for particular dimensions with ‘--cnk_dmn’ option.
cnk_map key values: ‘rd1’, ‘cnk_rd1’, ‘map_rd1
Mnemonic: Record Dimension size 1

Chunksize Equals Scalar Size Specified

Definition: Chunksize for all dimensions is set with the ‘--cnk_scl=sz_lmn’ option. For this map sz_lmn itself becomes the chunksize of each dimension. This is in contrast to the cnk_prd map, where the rth root of sz_lmn) becomes the chunksize of each dimension.
cnk_map key values: ‘scl’, ‘cnk_scl’, ‘map_scl
Mnemonic: SCaLar
cnk_map key values: ‘xpl’, ‘cnk_xpl’, ‘map_xpl
Mnemonic: EXPLicitly specified dimensions

Chunksize Product Matches Scalar Size Specified

Definition: The product of the chunksizes for each variable matches (approximately equals) the size specified with the ‘--cnk_scl=sz_lmn’ option. A dimension of size one is said to be degenerate. For a variable of rank R (i.e., with R non-degenerate dimensions), the chunksize in each non-degenerate dimension is (approximately) the Rth root of sz_lmn. This is in contrast to the cnk_scl map, where sz_lmn itself becomes the chunksize of each dimension.
cnk_map key values: ‘prd’, ‘cnk_prd’, ‘map_prd
Mnemonic: PRoDuct

Chunksize Lefter Product Matches Scalar Size Specified

Definition: The product of the chunksizes for each variable (approximately) equals the size specified with the ‘--cnk_byt=sz_byt’ (not ‘--cnk_dfl’) option. This is accomplished by using dimension sizes as chunksizes for the rightmost (most rapidly varying) dimensions, and then “flexing” the chunksize of the leftmost (least rapidly varying) dimensions such that the product of all chunksizes matches the specified size. All L-dimensions to the left of and including the first record dimension define the left-hand side. To be precise, if the total size (in bytes) of the variable is var_sz, and if the specified (with ‘--cnk_byt’) product of the R “righter” dimensions (those that vary more rapidly than the first record dimension) is sz_byt, then chunksize (in bytes) of each of the L lefter dimensions is (approximately) the Lth root of var_sz/sz_byt. This map was first proposed by Chris Barker.
cnk_map key values: ‘lfp’, ‘cnk_lfp’, ‘map_lfp
Mnemonic: LeFter Product

Chunksize Equals Existing Chunksize

Definition: Chunksizes are copied from the input to the output file for every variable that is chunked in the input file. Variables not chunked in the input file will be chunked with default mappings.
cnk_map key values: ‘xst’, ‘cnk_xst’, ‘map_xst
Mnemonic: EXiST

Chunksize Balances 1D and (N-1)-D Access to N-D Variable [default for netCDF4 input]

Definition: Chunksizes are chosen so that 1-D and (N-1)-D hyperslabs of 3-D variables (e.g., point-timeseries orn latitude/longitude surfaces of 3-D fields) both require approximately the number of chunks. Hence their access time should be balanced. Russ Rew explains the motivation and derivation for this strategy here.
cnk_map key values: ‘rew’, ‘cnk_rew’, ‘map_rew
Mnemonic: Russ REW

Chunksizes use netCDF4 defaults

Definition: Chunksizes are determined by the underlying netCDF library. All variables selected by the current chunking policy have their chunksizes determined by netCDF library defaults. The default algorithm netCDF uses to determine chunksizes has changed through the years, and thus depends on the netCDF library version. This map can be used to reset (portions of) previously chunked files to default chunking values.
cnk_map key values: ‘nc4’, ‘cnk_nc4’, ‘map_nc4
Mnemonic: NetCDF4

Chunksizes use NCO recommendations [default for netCDF3 input]

Definition: Chunksizes are determined by the currently recommended NCO map. This is a virtual option that ensures the chunking map is (in the subjective opinion of the authors) the best map for typical usage. As of NCO version 4.4.9 (May, 2015), this virtual map calls ‘map_lfp’.
cnk_map key values: ‘nco’, ‘cnk_nco’, ‘map_nco
Mnemonic: NetCDFOperator

It is possible to combine the above chunking map algorithms with user-specified per-dimension (though not per-variable) chunksizes that override specific chunksizes determined by the maps above. The user specifies the per-dimension chunksizes with the (equivalent) long options ‘--cnk_dmn’ or ‘--chunk_dimension’). The option takes two comma-separated arguments, dmn_nm,sz_lmn, which are the dimension name and its chunksize (in elements, not bytes), respectively. The ‘--cnk_dmn’ option may be used as many times as necessary.

The default behavior of chunking depends on several factors. As mentioned above, when no chunking options are explicitly specified by the user, then NCO preserves the chunking characteristics of the input file in the output file. This is equivalent to specifying both cnk_plc and cnk_map as “existing”, i.e., ‘--cnk_plc=xst --cnk_map=xst’. If output netCDF4 files are chunked with the default behavior of the netCDF4 library.

When any chunking parameter exceptcnk_plc’ or ‘cnk_map’ is specified (such as ‘cnk_dmn’ or ‘cnk_scl’), then the “existing” policy and map are retained and the output chunksizes are modified where necessary in accord with the user-specified parameter. When ‘cnk_map’ is specified and ‘cnk_plc’ is not, then NCO picks (what it thinks is) the optimal chunking policy. This has always been policy ‘map_g2d’. When ‘cnk_plc’ is specified and ‘cnk_map’ is not, then NCO picks (what it thinks is) the optimal chunking map. This has always been map ‘map_rd1’.

To start afresh and return to netCDF4 chunking defaults, select ‘cnk_map=nc4’.

# Simple chunking and unchunking
ncks -O -4 --cnk_plc=all # Chunk
ncks -O -4 --cnk_plc=unchunk # Unchunk

# Chunk data then unchunk it, printing informative metadata
ncks -O -4 -D 4 --cnk_plc=all ~/nco/data/ ~/
ncks -O -4 -D 4 --cnk_plc=uck ~/ ~/

# Set total chunksize to 8192 B
ncks -O -4 -D 4 --cnk_plc=all --cnk_byt=8192 ~/nco/data/ ~/

# More complex chunking procedures, with informative metadata
ncks -O -4 -D 4 --cnk_scl=8 ~/nco/data/ ~/
ncks -O -4 -D 4 --cnk_scl=8 ~/
ncks -O -4 -D 4 --cnk_dmn lat,64 --cnk_dmn lon,128 \ 
ncks -O -4 -D 4 --cnk_plc=uck ~/ ~/
ncks -O -4 -D 4 --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \
 --cnk_dmn lon,128 ~/

# Chunking works with all operators...
ncap2 -O -4 -D 4 --cnk_scl=8 -S ~/nco/data/ncap2_tst.nco \ 
 ~/nco/data/ ~/
ncbo -O -4 -D 4 --cnk_scl=8 -p ~/nco/data ~/
ncecat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 ~/
ncflint -O -4 -D 4 --cnk_scl=8 ~/nco/data/ ~/
ncpdq -O -4 -D 4 -P all_new --cnk_scl=8 -L 5 ~/nco/data/ ~/
ncrcat -O -4 -D 4 -n 12,2,1 --cnk_dmn lat,32 \ 
 -p /data/zender/dstmch90 ~/
ncwa -O -4 -D 4 -a time --cnk_plc=g2d --cnk_map=rd1 --cnk_dmn lat,32 \ 
 --cnk_dmn lon,128 ~/

Chunking policy ‘r1d’ changes the chunksize of 1-D record variables (and no other variables) to that specified (with ‘--cnk_dmn’) chunksize. Any specified record dimension chunksizes will be applied to 1-D record variables only. Other dimensions may be chunked with their own ‘--cnk_dmn’ options that will apply to all variables. For example,

ncks --cnk_plc=r1d --cnk_dmn=time,1000.

This sets time chunks to 1000 only in 1-D record variables. Without the ‘r1d’ policy, time chunks would change in all variables.

It is appropriate to conclude by informing users about an aspect of chunking that may not be expected. Three types of variables are always chunked: Record variables, Deflated (compressed) variables, and Checksummed variables. Hence all variables that contain a record dimension are also chunked (since data must be chunked in all dimensions, not just one). Unless otherwise specified by the user, the other (fixed, non-record) dimensions of record variables are assigned default chunk sizes. The HDF5 layer does all this automatically to optimize the on-disk variable/file storage geometry of record variables. Do not be surprised to learn that files created without any explicit instructions to activate chunking nevertheless contain chunked variables.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.30 Compression

Availability: ncbo, ncecat, nces, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: None
Long options: ‘--ppc var1[,var2[,...]]=prc’,
--precision_preserving_compression var1[,var2[,...]]=prc’,
--quantize var1[,var2[,...]]=prc

NCO implements or accesses four different compression algorithms, the standard lossless DEFLATE algorithm and three lossy compression algorithms. All four algorithms reduce the on-disk size of a dataset while sacrificing no (lossless) or a tolerable amount (lossy) of precision. First, NCO can access the lossless DEFLATE algorithm, a combination of Lempel-Ziv encoding and Huffman coding, algorithm on any netCDF4 dataset (see Deflation). Because it is lossless, this algorithm re-inflates deflated data to their full original precision. This algorithm is accessed via the HDF5 library layer (which itself calls the zlib library also used by gzip), and is unavailable with netCDF3.

Next: , Previous: , Up: Compression   [Contents][Index]

3.30.1 Linear Packing

The three lossy compression algorithms are Linear Packing (see Packed data), and two precision-preserving algorithms. Linear packing quantizes data of a higher precision type into a lower precision type (often NC_SHORT) that thus stores a fewer (though constant) number of bytes per value. Linearly packed data unpacks into a (much) smaller dynamic range than the floating-point data can represent. The type-conversion and reduced dynamic range of the data allows packing to eliminate bits typically used to store an exponent, thus improving its packing efficiency. Packed data also can also be deflated for additional space savings.

A limitation of linear packing is that unpacking data stored as integers into the linear range defined by scale_factor and add_offset rapidly loses precision outside of a narrow range of floating-point values. Variables packed as NC_SHORT, for example, can represent only about 64000 discrete values in the range -32768*scale_factor+add_offset to 32767*scale_factor+add_offset. The precision of packed data equals the value of scale_factor, and scale_factor is usually chosen to span the range of valid data, not to represent the intrinsic precision of the variable. In other words, the precision of packed data cannot be specified in advance because it depends on the range of values to quantize.

Previous: , Up: Compression   [Contents][Index]

3.30.2 Precision-Preserving Compression

NCO implemented the final two lossy compression algorithms in version 4.4.8 (February, 2015). These are both Precision-Preserving Compression (PPC) algorithms and since standard terminology for precision is remarkably imprecise, so is our nomenclature. The operational definition of “significant digit” in our precision preserving algorithms is that the exact value, before rounding or quantization, is within one-half the value of the decimal place occupied by the Least Significant Digit (LSD) of the rounded value. For example, the value pi = 3.14 correctly represents the exact mathematical constant pi to three significant digits because the LSD of the rounded value (i.e., 4) is in the one-hundredths digit place, and the difference between the exact value and the rounded value is less than one-half of one one-hundredth, i.e., (3.14159265358979323844 - 3.14 = 0.00159 < 0.005).

One PPC algorithm preserves the specified total Number of Signifcant Digits (NSD) of the value. For example there is only one significant digit in the weight of most “eight-hundred pound gorillas” that you will encounter, i.e., so nsd=1. This is the most straightforward measure of precision, and thus NSD is the default PPC algorithm.

The other PPC algorithm preserves the number of Decimal Significant Digits (DSD), i.e., the number of significant digits following (positive, by convention) or preceding (negative) the decimal point. For example, ‘0.008’ and ‘800’ have, respectively, three and negative two digits digits following the decimal point, corresponding to dsd=3 and dsd=-2.

The only justifiable NSD for a given value depends on intrinsic accuracy and error characteristics of the model or measurements, and not on the units with which the value is stored. The appropriate DSD for a given value depends on these intrinsic characteristics and, in addition, the units of storage. This is the fundamental difference between the NSD and DSD approaches. The eight-hundred pound gorilla always has nsd=1 regardless of whether the value is stored in pounds or in some other unit. DSD corresponding to this weight is dsd=-2 if the value is stored in pounds, dsd=4 if stored in megapounds.

Users may wish to express the precision to be preserved as either NSD or DSD. Invoke PPC with the long option ‘--ppc var=prc’, or give the same arguments to the synonyms ‘--precision_preserving_compression’, or to ‘--quantize’. Here var is the variable to quantize, and prc is its precision. The option ‘--ppc’ (and its long option equivalents such as ‘--quantize’) indicates the argument syntax will be key=val. As such, ‘--ppc’ and its synonyms are indicator options that accept arguments supplied one-by-one like ‘--ppc key1=val1 --ppc key2=val2’, or aggregated together in multi-argument format like ‘--ppc key1=val1#key2=val2’ (see Multi-arguments). The default algorithm assumes prc specifies NSD precision, e.g., ‘T=2’ means nsd=2. Prepend prc with a decimal point to specify DSD precision, e.g., ‘T=.2’ means dsd=2. NSD precision must be specified as a positive integer. DSD precision may be a positive or negative integer; and is specified as the negative base 10 logarithm of the desired precision, in accord with common usage. For example, specifying ‘T=.3’ or ‘T=.-2’ tells the DSD algorithm to store only enough bits to preserve the value of T rounded to the nearest thousandth or hundred, respectively.

Setting var to default has the special meaning of applying the associated NSD or DSD algorithm to all floating point variables except coordinate variables. Variables not affected by default include integer and non-numeric atomic types, coordinates, and variables mentioned in the bounds, climatology, or coordinates attribute of any variable. NCO applies PPC to coordinate variables only if those variables are explicitly specified (i.e., not with the ‘default=prc’ mechanism. NCO applies PPC to integer-type variables only if those variables are explicitly specified (i.e., not with the ‘default=prc’, and only if the DSD algorithm is invoked with a negative prc. To prevent PPC from applying to certain non-coordinate variables (e.g., gridcell_area or gaussian_weight), explicitly specify a precision exceeding 7 (for NC_FLOAT) or 15 (for NC_DOUBLE) for those variables. Since these are the maximum representable precisions in decimal digits, NCO turns-off PPC (i.e., does nothing) when more precision is requested.

The time-penalty for compressing and uncompressing data varies according to the algorithm. The Number of Significant Digit (NSD) algorithm quantizes by bitmasking, and employs no floating-point math. The Decimal Significant Digit (DSD) algorithm quantizes by rounding, which does require floating-point math. Hence NSD is likely faster than DSD, though the difference has not been measured. NSD creates a bitmask to alter the significand of IEEE 754 floating-point data. The bitmask is one for all bits to be retained and zero or one for all bits to be ignored. The algorithm assumes that the number of binary digits (i.e., bits) necessary to represent a single base-10 digit is ln(10)/ln(2) = 3.32. The exact numbers of bits Nbit retained for single and double precision values are ceil(3.32*nsd)+1 and ceil(3.32*nsd)+2, respectively. Once these reach 23 and 53, respectively, bitmasking is completely ineffective. This occurs at nsd=6.3 and 15.4, respectively.

The DSD algorithm, by contrast, uses rounding to remove undesired precision. The rounding 41 zeroes the greatest number of significand bits consistent with the desired precision.

To demonstrate the change in IEEE representation caused by PPC rounding algorithms, consider again the case of pi, represented as an NC_FLOAT. The IEEE 754 single precision representations of the exact value (3.141592...), the value with only three significant digits treated as exact (3.140000...), and the value as stored (3.140625) after PPC-rounding with either the NSD (prc=3) or DSD (prc=2) algorithm are, respectively,

S Exponent  Fraction (Significand)   Decimal    Notes
0 100000001 0010010000111111011011 # 3.14159265 Exact
0 100000001 0010001111010111000011 # 3.14000000
0 100000001 0010010000000000000000 # 3.14062500 NSD = 3
0 100000001 0010010000000000000000 # 3.14062500 DSD = 2

The string of trailing zero-bits in the rounded values facilitates byte-stream compression. Note that the NSD and DSD algorithms do not always produce results that are bit-for-bit identical, although they do in this particular case.

Reducing the preserved precision of NSD-rounding produces increasingly long strings of identical-bits amenable to compression:

S Exponent  Fraction (Significand)   Decimal    Notes
0 100000001 0010010000111111011011 # 3.14159265 Exact
0 100000001 0010010000111111011011 # 3.14159265 NSD = 8
0 100000001 0010010000111111011010 # 3.14159262 NSD = 7
0 100000001 0010010000111111011000 # 3.14159203 NSD = 6
0 100000001 0010010000111111000000 # 3.14158630 NSD = 5
0 100000001 0010010000111100000000 # 3.14154053 NSD = 4
0 100000001 0010010000000000000000 # 3.14062500 NSD = 3
0 100000001 0010010000000000000000 # 3.14062500 NSD = 2
0 100000001 0010000000000000000000 # 3.12500000 NSD = 1

The consumption of about 3 bits per digit of base-10 precision is evident, as is the coincidence of a quantized value that greatly exceeds the mandated precision for NSD = 2. Although the NSD algorithm generally masks some bits for all nsd <= 7 (for NC_FLOAT), compression algorithms like DEFLATE may need byte-size-or-greater (i.e., at least eight-bit) bit patterns before their algorithms can take advantage of of encoding such patterns for compression. Do not expect significantly enhanced compression from nsd > 5 (for NC_FLOAT) or nsd > 14 (for NC_DOUBLE). Clearly values stored as NC_DOUBLE (i.e., eight-bytes) are susceptible to much greater compression than NC_FLOAT for a given precision because their significands explicitly contain 53 bits rather than 23 bits.

Maintaining non-biased statistical properties during lossy compression requires special attention. The DSD algorithm uses rint(), which rounds toward the nearest even integer. Thus DSD has no systematic bias. However, the NSD algorithm uses a bitmask technique susceptible to statistical bias. Zeroing all non-significant bits is guaranteed to produce numbers quantized to the specified tolerance, i.e., half of the decimal value of the position occupied by the LSD. However, always zeroing the non-significant bits results in quantized numbers that never exceed the exact number. This would produce a negative bias in statistical quantities (e.g., the average) subsequently derived from the quantized numbers. To avoid this bias, our NSD implementation rounds non-significant bits down (to zero) or up (to one) in an alternating fashion when processing array data. In general, the first element is rounded down, the second up, and so on. This results in a mean bias quite close to zero. The only exception is that the floating-point value of zero is never quantized upwards. For simplicity, NSD always rounds scalars downwards.

Although NSD or DSD are different algorithms under the hood, they both replace the (unwanted) least siginificant bits of the IEEE significand with a string of consecutive zeroes. Byte-stream compression techniques, such as the gzip DEFLATE algorithm compression available in HDF5, always compress zero-strings more efficiently than random digits. The net result is netCDF files that utilize compression can be significantly reduced in size. This feature only works when the data are compressed, either internally (by netCDF) or externally (by another user-supplied mechanism). It is most straightfoward to compress data internally using the built-in compression and decompression supported by netCDF4. For convenience, NCO automatically activates file-wide Lempel-Ziv deflation (see Deflation) level one (i.e., ‘-L 1’) when PPC is invoked on any variable in a netCDF4 output file. This makes PPC easier to use effectively, since the user need not explicitly specify deflation. Any explicitly specified deflation (including no deflation, ‘-L 0’) will override the PPC deflation default. If the output file is a netCDF3 format, NCO will emit a message suggesting internal netCDF4 or external netCDF3 compression. netCDF3 files compressed by an external utility such as gzip accrue approximately the same benefits (shrinkage) as netCDF4, although with netCDF3 the user or provider must uncompress (e.g., gunzip) the file before accessing the data. There is no benefit to rounding numbers and storing them in netCDF3 files unless such custom compression/decompression is employed. Without that, one may as well maintain the undesired precision.

The user accesses PPC through a single switch, ‘--ppc’, repeated as many times as necessary. To apply the NSD algorithm to variable u use, e.g.,

ncks -7 --ppc u=2

The output file will preserve only two significant digits of u. The options ‘-4’ or ‘-7’ ensure a netCDF4-format output (regardless of the input file format) to support internal compression. It is recommended though not required to write netCDF4 files after PPC. For clarity the ‘-4/-7’ switches are omitted in subsequent examples. NCO attaches attributes that indicate the algorithm used and degree of precision retained for each variable affected by PPC. The NSD and DSD algorithms store the attributes number_of_significant_digits and least_significant_digit 42, respectively.

It is safe to attempt PPC on input that has already been rounded. Variables can be made rounder, not sharper, i.e., variables cannot be “un-rounded”. Thus PPC attempted on an input variable with an existing PPC attribute proceeds only if the new rounding level exceeds the old, otherwise no new rounding occurs (i.e., a “no-op”), and the original PPC attribute is retained rather than replaced with the newer value of prc.

To request, say, five significant digits (nsd=5) for all fields, except, say, wind speeds which are only known to integer values (dsd=0) in the supplied units, requires ‘--ppc’ twice:

ncks -4 --ppc default=5 --ppc u,v=.0

To preserve five digits in all variables except coordinate variables and u and v, use the ‘default’ option and separately specify the exceptions:

ncks --ppc default=5 --ppc u,v=20

The ‘--ppc’ option may be specified any number of times to support varying precision types and levels, and each option may aggregate all the variables with the same precision

ncks --ppc p,w,z=5 --ppc q,RH=4 --ppc T,u,v=3
ncks --ppc p,w,z=5#q,RH=4#T,u,v=3 # Multi-argument format

Any var argument may be a regular expression. This simplifies generating lists of related variables:

ncks --ppc Q.?=5 --ppc FS.?,FL.?=4 --ppc RH=.3
ncks --ppc Q.?=5#FS.?,FL.?=4#RH=.3 # Multi-argument format

Although PPC-rounding instantly reduces data precision, on-disk storage reduction only occurs once the data are compressed.

How can one be sure the lossy data are sufficiently precise? PPC preserves all significant digits of every value. The DSD algorithm uses floating-point math to round each value optimally so that it has the maximum number of zeroed bits that preserve the specified precision. The NSD algorithm uses a theoretical approach (3.2 bits per base-10 digit), tuned and tested to ensure the worst case quantization error is less than half the value of the minimum increment in the least significant digit.

Note for HTML users:
The definition of error metrics relies heavily on mathematical expressions that cannot easily be represented in HTML. See the printed manual for much more detailed and complete documentation of this subject.

All three metrics are expressed in terms of the fraction of the ten’s place occupied by the LSD. If the LSD is the hundreds digit or the thousandths digit, then the metrics are fractions of 100, or of 1/100, respectively. PPC algorithms should produce maximum absolute errors no greater than 0.5 in these units. If the LSD is the hundreds digit, then quantized versions of true values will be within fifty of the true value. It is much easier to satisfy this tolerance for a true value of 100 (only 50% accuracy required) than for 999 (5% accuracy required). Thus the minimum accuracy guaranteed for nsd=1 ranges from 5–50%. For this reason, the best and worst cast performance usually occurs for true values whose LSD value is close to one and nine, respectively. Of course most users prefer prc > 1 because accuracies increase exponentially with prc. Continuing the previous example to prc=2, quantized versions of true values from 1000–9999 will also be within 50 of the true value, i.e., have accuracies from 0.5–5%. In other words, only two significant digits are necessary to guarantee better than 5% accuracy in quantization. We recommend that dataset producers and users consider quantizing datasets with nsd=3. This guarantees accuracy of 0.05–0.5% for individual values. Statistics computed from ensembles of quantized values will, assuming the mean error Emean is small, have much better accuracy than 0.5%. This accuracy is the most that can be justified for many applications.

To demonstrate these principles we conduct error analyses on an artificial, reproducible dataset, and on an actual dataset of observational analysis values. 43 The table summarizes quantization accuracy based on the three metrics.


Number of Significant Digits.


Maximum absolute error.


Mean absolute error.


Mean error.

Artificial Data: N=1000000 values in [1.0,2.0) in steps of 1.0e-6
Single-Precision        Double-Precision   Single-Precision
NSD Emabs Emebs Emean   Emabs Emebs Emean  DSD Emabs Emebs Emean
 1  0.31  0.11  4.1e-4  0.31  0.11  4.0e-4  1  0.30  0.11 -8.1e-4  
 2  0.39  0.14  6.8e-5  0.39  0.14  5.5e-5  2  0.39  0.14 -1.3e-4
 3  0.49  0.17  1.0e-6  0.49  0.17 -5.5e-7  3  0.49  0.17 -2.0e-5
 4  0.30  0.11  3.2e-7  0.30  0.11 -6.1e-6  4  0.30  0.11  5.1e-8
 5  0.37  0.13  3.1e-7  0.38  0.13 -5.6e-6  5  0.38  0.13  2.6e-6
 6  0.36  0.12 -4.4e-7  0.48  0.17 -4.1e-7  6  0.48  0.17  7.2e-6
 7  0.00  0.00  0.0     0.30  0.10  1.5e-7  7  0.00  0.00  0.0     

Observational Analysis: N=13934592 values MERRA Temperature 20130601
NSD Emabs Emebs Emean   
 1  0.31  0.11  2.4e-3
 2  0.39  0.14  3.8e-4
 3  0.49  0.17 -9.6e-5 
 4  0.30  0.11  2.3e-3
 5  0.37  0.13  2.2e-3
 6  0.36  0.13  1.7e-2
 7  0.00  0.00  0.0     

All results show that PPC quantization performs as expected. Absolute maximum errors Emabs < 0.5 for all prc. For 1 <= prc <= 6, quantization results in comparable maximum absolute and mean absolute errors Emabs and Emebs, respectively. Mean errors Emean are orders of magnitude smaller because quantization produces over- and under-estimated values in balance. When prc=7, quantization of single-precision values is ineffective, because all available bits are used to represent the maximum precision of seven digits. The maximum and mean absolute errors Emabs and Emebs are nearly identical across algorithms, precisions, and dataset types. This is consistent with both the artificial data and empirical data being random, and thus exercising equally strengths and weaknesses of the algorithms over the course of millions of input values. We generated artificial arrays with many different starting values and interval spacing and all gave qualitatively similar results. The results presented are the worst obtained.

The artificial data has much smaller mean error Emean than the observational analysis. The reason why is unclear. It may be because the temperature field is concentrated in particular ranges of values (and associated quantization errors) prevalent on Earth, e.g., 200 < T < 320. It is worth noting that the mean error Emean < 0.01 for 1 <= prc < 6, and that Emean is typically at least two or more orders of magnitude less than Emabs. Thus quantized values with precisions as low as prc=1 still yield highly significant statistics by contemporary scientific standards.

Testing shows that PPC quantization enhances compression of typical climate datasets. The degree of enhancement depends, of course, on the required precision. Model results are often computed as NC_DOUBLE then archived as NC_FLOAT to save space. This table summarizes the performance of lossless and lossy compression on two typical, or at least random, netCDF data files. The files were taken from representative model-simulated and satellite-retrieved datasets. Only floating-point data were compressed. No attempt was made to compress integer-type variables as they occupy an insignificant fraction of every dataset. The columns are


File-type: N3 for netCDF CLASSIC, N4 for NETCDF4, N7 for NETCDF4_CLASSIC (which comprises netCDF3 data types and structures with netCDF4 storage features like compression), H4 for HDF4, and H5 for HDF5. N4/7 means results apply to both N4 and N7 filetypes.


Type of lossless compression employed, if any. Bare numbers refer to the strength of the DEFLATE algorithm employed internally by netCDF4/HDF5, while numbers prefixed with B refer to the block size employed by the Burrows-Wheeler algorithm in bzip2.


Number of significant digits retained by the precision-preserving compression NSD algorithm.


Y if the default ncpdq packing algorithm (convert floating-point types to NC_SHORT) was employed.


Resulting filesize in MB.


Compression ratio, i.e., resulting filesize relative to original size, in percent. In some cases the original files is already losslessly compressed. The compression ratios reported are relative to the size of the original file as distributed, not as optimally losslessly compressed.

A dash (-) indicates the associated compression feature was not employed.

Type LLC PPC Pck  Size   %    Flags and Notes
  N3   -   -  -   34.7 100.0  Original is not compressed
  N3  B1   -  -   28.9  83.2  bzip2 -1
  N3  B9   -  -   29.3  84.4  bzip2 -9
  N7   -   -  -   35.0 101.0     
  N7   1   -  -   28.2  81.3  -L 1
  N7   9   -  -   28.0  80.8  -L 9
  N7   -   -  Y   17.6  50.9  ncpdq -L 0
  N7   1   -  Y    7.9  22.8  ncpdq -L 1
  N7   1   7  -   28.2  81.3  --ppc default=7
  N7   1   6  -   27.9  80.6  --ppc default=6
  N7   1   5  -   25.9  74.6  --ppc default=5
  N7   1   4  -   22.3  64.3  --ppc default=4
  N7   1   3  -   18.9  54.6  --ppc default=3
  N7   1   2  -   14.5  43.2  --ppc default=2
  N7   1   1  -   10.0  29.0  --ppc default=1

Type LLC PPC Pck  Size   %    Flags and Notes
  N3   -   -  -  119.8 100.0  Original is not compressed
  N3  B1   -  -   84.2  70.3  bzip2 -1
  N3  B9   -  -   84.8  70.9  bzip2 -9
  N7   -   -  -  120.5 100.7     
  N7   1   -  -   82.6  69.0  -L 1
  N7   9   -  -   82.1  68.6  -L 9
  N7   -   -  Y   60.7  50.7  ncpdq -L 0
  N7   1   -  Y   26.0  21.8  ncpdq -L 1
  N7   1   7  -   82.6  69.0  --ppc default=7
  N7   1   6  -   81.9  68.4  --ppc default=6
  N7   1   5  -   77.2  64.5  --ppc default=5
  N7   1   4  -   69.0  57.6  --ppc default=4
  N7   1   3  -   59.3  49.5  --ppc default=3
  N7   1   2  -   49.5  41.3  --ppc default=2
  N7   1   1  -   38.2  31.9  --ppc default=1

Type LLC PPC Pck  Size   %    Flags and Notes
  H4   5   -  -  244.3 100.0  Original is compressed
  H4  B1   -  -  244.7 100.1  bzip2 -1
  N4   5   -  -  214.5  87.8
  N7   5   -  -  210.6  86.2  
  N4  B1   -  -  215.4  88.2  bzip2 -1
  N4  B9   -  -  214.8  87.9  bzip2 -9
  N3   -   -  -  617.1 252.6
N4/7   -   -  -  694.0 284.0  -L 0
N4/7   1   -  -  223.2  91.3  -L 1
N4/7   9   -  -  207.3  84.9  -L 9
N4/7   -   -  Y  347.1 142.1  ncpdq -L 0
N4/7   1   -  Y  133.6  54.7  ncpdq -L 1
N4/7   1   7  -  223.1  91.3  --ppc default=7
N4/7   1   6  -  225.1  92.1  --ppc default=6
N4/7   1   5  -  221.4  90.6  --ppc default=5
N4/7   1   4  -  201.4  82.4  --ppc default=4
N4/7   1   3  -  185.3  75.9  --ppc default=3
N4/7   1   2  -  150.0  61.4  --ppc default=2
N4/7   1   1  -  100.8  41.3  --ppc default=1

# OMI-Aura_L2-OMIAuraSO2_2012m1222-o44888_v01-00-2014m0107t114720.h5
Type LLC PPC Pck  Size   %    Flags and Notes
  H5   5   -  -   29.5 100.0  Original is compressed
  H5  B1   -  -   29.3  99.6  bzip2 -1
  N4   5   -  -   29.5 100.0
  N4  B1   -  -   29.3  99.6  bzip2 -1
  N4  B9   -  -   29.3  99.4  bzip2 -9
  N4   -   -  -   50.7 172.3  -L 0
  N4   1   -  -   29.8 101.3  -L 1
  N4   9   -  -   29.4  99.8  -L 9
  N4   -   -  Y   27.7  94.0  ncpdq -L 0
  N4   1   -  Y   12.9  43.9  ncpdq -L 1
  N4   1   7  -   29.7 100.7  --ppc default=7
  N4   1   6  -   29.7 100.8  --ppc default=6
  N4   1   5  -   27.3  92.8  --ppc default=5
  N4   1   4  -   23.8  80.7  --ppc default=4
  N4   1   3  -   20.3  69.0  --ppc default=3
  N4   1   2  -   15.1  51.2  --ppc default=2
  N4   1   1  -    9.9  33.6  --ppc default=1

A selective, per-variable approach to PPC yields the best balance of precision and compression yet requires the dataset producer to understand the intrinsic precision of each variable. Such a specification for a GCM dataset might look like this (using names for the NCAR CAM model):

# Be conservative on non-explicit quantities, so default=5
# Some quantities deserve four significant digits
# Many quantities, such as aerosol optical depths and burdens, are 
# highly uncertain and only useful to three significant digits.
ncks -7 -O \
--ppc default=5 \
--ppc AN.?,AQ.?=4 \
--ppc AER.?,AOD.?,ARE.?,AW.?,BURDEN.?=3 \ ~/

Next: , Previous: , Up: Shared features   [Contents][Index]

3.31 Deflation

Availability: ncap2, ncbo, ncclimo, nces, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncremap, ncwa
Short options: ‘-L
Long options: ‘--dfl_lvl’, ‘--deflate

All NCO operators that define variables support the netCDF4 feature of storing variables compressed with the lossless DEFLATE compression algorithm. DEFLATE combines the Lempel-Ziv encoding with Huffman coding. The specific version used by netCDF4/HDF5 is that implemented in the zlib library used by gzip. Activate deflation with the -L dfl_lvl short option (or with the same argument to the ‘--dfl_lvl’ or ‘--deflate’ long options). Specify the deflation level dfl_lvl on a scale from no deflation (dfl_lvl = 0) to maximum deflation (dfl_lvl = 9). Under the hood, this selects the compression blocksize. Minimal deflation (dfl_lvl = 1) achieves considerable storage compression with little time penalty. Higher deflation levels require more time for compression. File sizes resulting from minimal (dfl_lvl = 1) and maximal (dfl_lvl = 9) deflation levels typically differ by less than 10% in size.

To compress an entire file using deflation, use

ncks -4 -L 0 # No deflation (fast, no time penalty)
ncks -4 -L 1 # Minimal deflation (little time penalty)
ncks -4 -L 9 # Maximal deflation (much slower)

Unscientific testing shows that deflation compresses typical climate datasets by 30-60%. Packing, a lossy compression technique available for all netCDF files (see Packed data), can easily compress files by 50%. Packed data may be deflated to squeeze datasets by about 80%:

ncks  -4 -L 1 # Minimal deflation (~30-60% compression)
ncks  -4 -L 9 # Maximal deflation (~31-63% compression)
ncpdq # Standard packing  (~50% compression)
ncpdq -4 -L 9 # Deflated packing  (~80% compression)

ncks prints deflation parameters, if any, to screen (see ncks netCDF Kitchen Sink).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.32 MD5 digests

Availability: ncecat, ncks, ncrcat
Short options:
Long options: ‘--md5_dgs’, ‘--md5_digest’, ‘--md5_wrt_att’, ‘--md5_write_attribute

As of NCO version 4.1.0 (April, 2012), NCO supports data integrity verification using the MD5 digest algorithm. This support is currently implemented in ncks and in the multifile concantenators ncecat and ncrcat. Activate it with the ‘--md5_dgs’ or ‘--md5_digest’ long options. As of NCO version 4.3.3 (July, 2013), NCO will write the MD5 digest of each variable as an NC_CHAR attribute named MD5. This support is currently implemented in ncks and in the multifile concantenators ncecat and ncrcat. Activate it with the ‘--md5_wrt_att’ or ‘--md5_write_attribute’ long options.

The behavior and verbosity of the MD5 digest is operator-dependent. When activating MD5 digests with ncks it is assumed that the user simply wishes to see the digest of every variable and this is done when the debugging level exceeds one. This incurs only the minor overhead of performing the hash algorithm for each variable read. MD5 digests may be activated in both ncks invocation type, the one-filename argument form for printing and the two-filename argument form for sub-setting. The MD5 digests are shown as a 32-character hexadecimal string in which each two characters represent one byte of the 16-byte digest:

> ncks --trd -D 2 -C --md5 -v md5_a,md5_abc ~/nco/data/
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 900150983cd24fb0d6963f7d28e17f72
lev[0]=100 md5_abc[0--2]='abc' 
> ncks --trd -D 2 -C -d lev,0 --md5 -v md5_a,md5_abc ~/nco/data/
ncks: INFO MD5(md5_a) = 0cc175b9c0f1b6a831c399e269772661
md5_a = 'a' 
ncks: INFO MD5(md5_abc) = 0cc175b9c0f1b6a831c399e269772661
lev[0]=100 md5_abc[0--0]='a' 

In fact these examples demonstrate the validity of the hash algorithm since the MD5 hashes of the strings “a” and “abc” are widely known. The second example shows that the hyperslab of variable md5_abc (= “abc”) consisting of only its first letter (= “a”) has the same hash as the variable md5_a (“a”). This illustrates that MD5 digests act only on variable data, not on metadata.

When activating MD5 digests with ncecat or ncrcat it is assumed that the user wishes to verify that every variable written to disk has the same MD5 digest as when it is subsequently read from disk. This incurs the major additional overhead of reading in each variable after it is written and performing the hash algorithm again on that to compare to the original hash. Moreover, it is assumed that such operations are generally done in “production mode” where the user is not interested in actually examining the digests herself. The digests proceed silently unless the debugging level exceeds three:

> ncecat -O -D 4 --md5 -p ~/nco/data ~/ | grep MD5
ncecat: INFO MD5(wnd_spd) = bec190dd944f2ce2794a7a4abf224b28
ncecat: INFO MD5 digests of RAM and disk contents for wnd_spd agree
> ncrcat -O -D 4 --md5 -p ~/nco/data ~/ | grep MD5
ncrcat: INFO MD5(wnd_spd) = 74699bb0a72b7f16456badb2c995f1a1
ncrcat: INFO MD5 digests of RAM and disk contents for wnd_spd agree

Regardless of the debugging level, an error is returned when the digests of the variable read from the source file and from the output file disagree.

These rules are evolving and as NCO pays more attention to data integrity. We welcome feedback and suggestions from users.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.33 Buffer sizes

Availability: All operators
Short options:
Long options: ‘--bfr_sz_hnt’, ‘--buffer_size_hint

As of NCO version 4.2.0 (May, 2012), NCO allows the user to request specific buffer sizes to allocate for reading and writing files. This buffer size determines how many system calls the netCDF layer must invoke to read and write files. By default, netCDF uses the preferred I/O block size returned as the ‘st_blksize’ member of the ‘stat’ structure returned by the stat() system call 44. Otherwise, netCDF uses twice the system pagesize. Larger sizes can increase access speed by reducing the number of system calls netCDF makes to read/write data from/to disk. Because netCDF cannot guarantee the buffer size request will be met, the actual buffer size granted by the system is printed as an INFO statement.

# Request 2 MB file buffer instead of default 8 kB buffer
> ncks -O -D 3 --bfr_sz=2097152 ~/nco/data/ ~/
ncks: INFO nc__open() will request file buffer size = 2097152 bytes
ncks: INFO nc__open() opened file with buffer size = 2097152 bytes

Next: , Previous: , Up: Shared features   [Contents][Index]

3.34 RAM disks

Availability: All operators
Short options:
Long options: ‘--ram_all’, ‘--create_ram’, ‘--open_ram’, ‘--diskless_all

As of NCO version 4.2.1 (August, 2012), NCO supports the use of diskless files, aka RAM disks, for file access and creation. Two independent switches, ‘--open_ram’ and ‘--create_ram’, control this feature. Before describing the specifics of these switches, we describe why many NCO operations will not benefit from them. Essentially, reading/writing from/to RAM rather than disk only hastens the task when reads/writes to disk are avoided. Most NCO operations are simple enough that they require a single read-from/write-to disk for every block of input/output. Diskless access does not change this, but it does add an extra read-from/write-to RAM. However this extra RAM write/read does avoid contention for limited system resources like disk-head access. Operators which may benefit from RAM disks include ncwa, which may need to read weighting variables multiple times, the multi-file operators ncra, ncrcat, and ncecat, which may try to write output at least once per input file, and ncap2 scripts which may be arbitrarily long and convoluted.

The ‘--open_ram’ switch causes input files to copied to RAM when opened. All further metadata and data access occurs in RAM and thus avoids access time delays caused by disk-head movement. Usually input data is read at most once so it is unlikely that requesting input files be stored in RAM will save much time. The likeliest exceptions are files that are accessed numerous times, such as those analyzed extensively analyzed by ncap2.

Invoking ‘--open_ram’, ‘--ram_all’, or ‘--diskless_all’ uses much more system memory. To copy the input file to RAM increases the sustained memory use by exactly the on-disk filesize of the input file, i.e., MS += FT. For large input files this can be a huge memory burden that starves the rest of the NCO analysis of sufficient RAM. To be safe, use ‘--open_ram’, ‘--ram_all’, or ‘--diskless_all’ only on files that are much (say at least a factor of four) smaller than your available system RAM. See Memory Requirements for further details.

The ‘--create_ram’ switch causes output files to be created in RAM, rather than on disk. These files are copied to disk only when closed, i.e., when the operator completes. Creating files in RAM may save time, especially with ncap2 computations that are iterative, e.g., loops, and for multi-file operators that write output every record (timestep) or file. RAM files provide many of the same benefits as RAM variables in such cases (see RAM variables).

Two switches, ‘--ram_all’ and ‘--diskless_all’, are convenient shortcuts for specifying both ‘--create_ram’ and ‘--diskless_ram’. Thus

ncks # Default: Open on disk, write to disk
ncks --open_ram # Open in RAM, write to disk
ncks --create_ram # Create in RAM, write to disk
# Open in RAM, create in RAM, then write to disk
ncks --open_ram --create_ram
ncks --ram_all # Same as above
ncks --diskless_all # Same as above

It is straightforward to demonstrate the efficacy of RAM disks. For NASA we constructed a test that employs ncecat an arbitrary number (set to one hundred thousand) of files are all symbolically linked to the same file. Everything is on the local filesystem (not DAP).

# Create symbolic links for benchmark
cd ${DATA}/nco # Do all work here
for idx in {1..99999}; do
  idx_fmt=`printf "%05d" ${idx}`
  /bin/ln -s ${DATA}/nco/ \
# Benchmark time to ncecat one hundred thousand files
time ncecat --create_ram -O -u time -v ts -d Latitude,40.0 \ 
 -d Longitude,-105.0 -p ${DATA}/nco -n 99999,5,1 ~/

Run normally on a laptop in 201303, this completes in 21 seconds. The ‘--create_ram’ reduces the elapsed time to 9 seconds. Some of this speed may be due to using symlinks and caching. However, the efficacy of ‘--create_ram’ is clear. Placing the output file in RAM avoids thousands of disk writes. It is not unreasonable to for NCO to process a million files like this in a few minutes. However, there is no substitute for benchmarking with real files.

A completely independent way to reduce time spent writing files is to refrain from writing temporary output files. This is accomplished with the ‘--no_tmp_fl’ switch (see Temporary Output Files).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.35 Packed data

Availability: ncap2, ncbo, nces, ncflint, ncpdq, ncra, ncwa
Short options: None
Long options: ‘--hdf_upk’, ‘--hdf_unpack

The phrase packed data refers to data which are stored in the standard netCDF3 lossy linear packing format. See ncks netCDF Kitchen Sink for a description of deflation, a lossless compression technique available with netCDF4 only. Packed data may be deflated to save additional space.

Standard Packing Algorithm

Packing The standard netCDF linear packing algorithm (described here) produces packed data with the same dynamic range as the original but which requires no more than half the space to store. NCO will always use this algorithm for packing. Like all packing algorithms, linear packing is lossy. Just how lossy depends on the values themselves, especially their range. The packed variable is stored (usually) as type NC_SHORT with the two attributes required to unpack the variable, scale_factor and add_offset, stored at the original (unpacked) precision of the variable 45. Let min and max be the minimum and maximum values of x.

scale_factor = (max-min)/ndrv
add_offset = 0.5*(min+max)
pck = (upk-add_offset)/scale_factor = (upk-0.5*(min+max))*ndrv/(max-min)

where ndrv is the number of discrete representable values for given type of packed variable. The theoretical maximum value for ndrv is two raised to the number of bits used to store the packed variable. Thus if the variable is packed into type NC_SHORT, a two-byte datatype, then there are at most 2^{16} = 65536 distinct values representable. In practice, the number of discretely representible values is taken to be two less than the theoretical maximum. This leaves space for a missing value and solves potential problems with rounding that may occur during the unpacking of the variable. Thus for NC_SHORT, ndrv = 65536 - 2 = 65534. Less often, the variable may be packed into type NC_CHAR, where ndrv = 2^{8} - 2 = 256 - 2 = 254, or type NC_INT where where ndrv = 2^{32} - 2 = 4294967295 - 2 = 4294967293. One useful feature of the (lossy) netCDF packing algorithm is that lossless packing algorithms perform well on top of it.

Standard (Default) Unpacking Algorithm

Unpacking The unpacking algorithm depends on the presence of two attributes, scale_factor and add_offset. If scale_factor is present for a variable, the data are multiplied by the value scale_factor after the data are read. If add_offset is present for a variable, then the add_offset value is added to the data after the data are read. If both scale_factor and add_offset attributes are present, the data are first scaled by scale_factor before the offset add_offset is added.

upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)

NCO will use this algorithm for unpacking unless told otherwise as described below. When scale_factor and add_offset are used for packing, the associated variable (containing the packed data) is typically of type byte or short, whereas the unpacked values are intended to be of type int, float, or double. An attribute’s scale_factor and add_offset and _FillValue, if any, should all be of the type intended for the unpacked data, i.e., int, float or double.

Non-Standard Packing and Unpacking Algorithms

Many (most?) files originally written in HDF4 format use poorly documented packing/unpacking algorithms that are incompatible and easily confused with the netCDF packing algorithm described above. The unpacking component of the “conventional” HDF algorithm (described here and in Section 3.10.6 of the HDF4 Users Guide here, and in the FAQ for MODIS MOD08 data here) is

upk = scale_factor*(pck - add_offset)

The unpacking component of the HDF algorithm employed for MODIS MOD13 data is

upk = (pck - add_offset)/scale_factor

The unpacking component of the HDF algorithm employed for MODIS MOD04 data is the same as the netCDF algorithm.

Confusingly, the (incompatible) netCDF and HDF algorithms both store their parameters in attributes with the same names (scale_factor and add_offset). Data packed with one algorithm should never be unpacked with the other; doing so will result in incorrect answers. Unfortunately, few users are aware that their datasets may be packed, and fewer know the details of the packing algorithm employed. This is what we in the “bizness” call an interoperability issue because it hampers data analysis performed on heterogeneous systems.

As described below, NCO automatically unpacks data before performing arithmetic. This automatic unpacking occurs silently since there is usually no reason to bother users with these details. There is as yet no generic way for NCO to know which packing convention was used, so NCO assumes the netCDF convention was used. NCO uses the same convention for unpacking unless explicitly told otherwise with the ‘--hdf_upk’ (also ‘--hdf_unpack’) switch. Until and unless a method of automatically detecting the packing method is devised, it must remain the user’s responsibility to tell NCO when to use the HDF convention instead of the netCDF convention to unpack.

If your data originally came from an HDF file (e.g., NASA EOS) then it was likely packed with the HDF convention and must be unpacked with the same convention. Our recommendation is to only request HDF unpacking when you are certain. Most packed datasets encountered by NCO will have used the netCDF convention. Those that were not will hopefully produce noticeably weird values when unpacked by the wrong algorithm. Before or after panicking, treat this as a clue to re-try your commands with the ‘--hdf_upk’ switch. See ncpdq netCDF Permute Dimensions Quickly for an easy technique to unpack data packed with the HDF convention, and then re-pack it with the netCDF convention.

Handling of Packed Data by Other Operators

All NCO arithmetic operators understand packed data. The operators automatically unpack any packed variable in the input file which will be arithmetically processed. For example, ncra unpacks all record variables, and ncwa unpacks all variable which contain a dimension to be averaged. These variables are stored unpacked in the output file.

On the other hand, arithmetic operators do not unpack non-processed variables. For example, ncra leaves all non-record variables packed, and ncwa leaves packed all variables lacking an averaged dimension. These variables (called fixed variables) are passed unaltered from the input to the output file. Hence fixed variables which are packed in input files remain packed in output files. Completely packing and unpacking files is easily accomplished with ncpdq (see ncpdq netCDF Permute Dimensions Quickly). Pack and unpack individual variables with ncpdq and the ncap2 pack() and unpack() functions (see Methods and functions).

Next: , Previous: , Up: Shared features   [Contents][Index]

3.36 Operation Types

Availability: ncap2, ncra, nces, ncwa
Short options: ‘-y
Long options: ‘--operation’, ‘--op_typ

The ‘-y op_typ’ switch allows specification of many different types of operations Set op_typ to the abbreviated key for the corresponding operation:


Mean value


Square of the mean


Mean of sum of squares


Maximum value


Minimum value


Maximum absolute value


Mean absolute value


Minimum absolute value


Root-mean-square (normalized by N)


Root-mean square (normalized by N-1)


Square root of the mean


Sum of absolute values


Sum of values

NCO assumes coordinate variables represent grid axes, e.g., longitude. The only rank-reduction which makes sense for coordinate variables is averaging. Hence NCO implements the operation type requested with ‘-y’ on all non-coordinate variables, not on coordinate variables. When an operation requires a coordinate variable to be reduced in rank, i.e., from one dimension to a scalar or from one dimension to a degenerate (single value) array, then NCO always averages the coordinate variable regardless of the arithmetic operation type performed on the non-coordinate variables.

The mathematical definition of each arithmetic operation is given below. See ncwa netCDF Weighted Averager, for additional information on masks and normalization. If an operation type is not specified with ‘-y’ then the operator performs an arithmetic average by default. Averaging is described first so the terminology for the other operations is familiar.

Note for HTML users:
The definition of mathematical operations involving rank reduction (e.g., averaging) relies heavily on mathematical expressions which cannot easily be represented in HTML. See the printed manual for much more detailed and complete documentation of this subject.

The definitions of some of these operations are not universally useful. Mostly they were chosen to facilitate standard statistical computations within the NCO framework. We are open to redefining and or adding to the above. If you are interested in having other statistical quantities defined in NCO please contact the NCO project (see Help Requests and Bug Reports).


Suppose you wish to examine the variable prs_sfc(time,lat,lon) which contains a time series of the surface pressure as a function of latitude and longitude. Find the minimum value of prs_sfc over all dimensions:

ncwa -y min -v prs_sfc 

Find the maximum value of prs_sfc at each time interval for each latitude:

ncwa -y max -v prs_sfc -a lon

Find the root-mean-square value of the time-series of prs_sfc at every gridpoint:

ncra -y rms -v prs_sfc
ncwa -y rms -v prs_sfc -a time

The previous two commands give the same answer but ncra is preferred because it has a smaller memory footprint. A dimension of size one is said to be degenerate. By default, ncra leaves the (degenerate) time dimension in the output file (which is usually useful) whereas ncwa removes the time dimension (unless ‘-b’ is given).

These operations work as expected in multi-file operators. Suppose that prs_sfc is stored in multiple timesteps per file across multiple files, say,, We can now find the three month maximum surface pressure at every point.

nces -y max -v prs_sfc

It is possible to use a combination of these operations to compute the variance and standard deviation of a field stored in a single file or across multiple files. The procedure to compute the temporal standard deviation of the surface pressure at all points in a single file involves three steps.

ncwa -O -v prs_sfc -a time
ncbo -O -v prs_sfc 
ncra -O -y rmssdn

First construct the temporal mean of prs_sfc in the file Next overwrite with the anomaly (deviation from the mean). Finally overwrite with the root-mean-square of itself. Note the use of ‘-y rmssdn’ (rather than ‘-y rms’) in the final step. This ensures the standard deviation is correctly normalized by one fewer than the number of time samples. The procedure to compute the variance is identical except for the use of ‘-y avgsqr’ instead of ‘-y rmssdn’ in the final step.

ncap2 can also compute statistics like standard deviations. Brute-force implementation of formulae is one option, e.g.,

ncap2 -s 'prs_sfc_sdn=sqrt((prs_sfc-prs_sfc.avg($time)^2). \

The operation may, of course, be broken into multiple steps in order to archive intermediate quantities, such as the time-anomalies

ncap2 -s 'prs_sfc_anm=prs_sfc-prs_sfc.avg($time)' \
      -s 'prs_sfc_sdn=sqrt((prs_sfc_anm^2).total($time)/($time.size-1))' \

ncap2 supports intrinsic standard deviation functions (see Operation Types) which simplify the above expression to

ncap2 -s 'prs_sfc_sdn=(prs_sfc-prs_sfc.avg($time)).rmssdn($time)'

These instrinsic functions compute the answer quickly and concisely.

The procedure to compute the spatial standard deviation of a field in a single file involves three steps.

ncwa -O -v prs_sfc,gw -a lat,lon -w gw
ncbo -O -v prs_sfc,gw
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw

First the spatially weighted (by ‘-w gw’) mean values are written to the output file, as are the mean weights. The initial output file is then overwritten with the gridpoint deviations from the spatial mean. It is important that the output file after the second line contain the original, non-averaged weights. This will be the case if the weights are named so that NCO treats them like a coordinate (see CF Conventions). One such name is gw, and any variable whose name begins with msk_ (for “mask”) or wgt_ (for “weight”) will likewise be treated as a coordinate, and will be copied (not differenced) straight from to in the second step. When using weights to compute standard deviations one must remember to include the weights in the initial output files so that they may be used again in the final step. Finally the root-mean-square of the appropriately weighted spatial deviations is taken.

No elegant ncap2 solution exists to compute weighted standard deviations. Those brave of heart may try to formulate one. A general formula should allow weights to have fewer than and variables to have more than the minimal spatial dimensions (latitude and longitude).

The procedure to compute the standard deviation of a time-series across multiple files involves one extra step since all the input must first be collected into one file.

ncrcat -O -v tpt
ncwa -O -a time
ncbo -O -v tpt
ncra -O -y rmssdn

The first step assembles all the data into a single file. Though this may consume a lot of temporary disk space, it is more or less required by the ncbo operation in the third step.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.37 Type Conversion

Availability (automatic type conversion): ncap2, ncbo, nces, ncflint, ncra, ncwa
Short options: None (it’s automatic)
Availability (manual type conversion): nces, ncra, ncwa
Short options: None
Long options: ‘--dbl’, ‘--flt’, ‘--rth_dbl’, ‘--rth_flt

Type conversion refers to the casting or coercion of one fundamental or atomic data type to another, e.g., converting NC_SHORT (two bytes) to NC_DOUBLE (eight bytes). Type conversion always promotes or demotes the range and/or precision of the values a variable can hold. Type conversion is automatic when the language carries out this promotion according to an internal set of rules without explicit user intervention. In contrast, manual type conversion refers to explicit user commands to change the type of a variable or attribute. Most type conversion happens automatically, yet there are situations in which manual type conversion is advantageous.

Next: , Previous: , Up: Type Conversion   [Contents][Index]

3.37.1 Automatic type conversion

There are at least two reasons to avoid type conversions. First, type conversions are expensive since they require creating (temporary) buffers and casting each element of a variable from its storage type to some other type and then, often, converting it back. Second, a dataset’s creator perhaps had a good reason for storing data as, say, NC_FLOAT rather than NC_DOUBLE. In a scientific framework there is no reason to store data with more precision than the observations merit. Normally this is single-precision, which guarantees 6–9 digits of precision. Reasons to engage in type conversion include avoiding rounding errors and out-of-range limitations of less-precise types. This is the case with most integers. Thus NCO defaults to automatically promote integer types to floating-point when performing lengthy arithmetic, yet NCO defaults to not promoting single to double-precision floats.

Before discussing the more subtle floating-point issues, we first examine integer promotion. We will show how following parsimonious conversion rules dogmatically can cause problems, and what NCO does about that. That said, there are situations in which implicit conversion of single- to double-precision is also warranted. Understanding the narrowness of these situations takes time, and we hope the reader appreciates the following detailed discussion.

Consider the average of the two NC_SHORTs 17000s and 17000s. A straightforward average without promotion results in garbage since the intermediate value which holds their sum is also of type NC_SHORT and thus overflows on (i.e., cannot represent) values greater than 32,767 46. There are valid reasons for expecting this operation to succeed and the NCO philosophy is to make operators do what you want, not what is purest. Thus, unlike C and Fortran, but like many other higher level interpreted languages, NCO arithmetic operators will perform automatic type conversion on integers when all the following conditions are met 47:

  1. The requested operation is arithmetic. This is why type conversion is limited to the operators ncap2, ncbo, nces, ncflint, ncra, and ncwa.
  2. The arithmetic operation could benefit from type conversion. Operations that could benefit include averaging, summation, or any “hard” arithmetic that could overflow or underflow. Larger representable sums help avoid overflow, and more precision helps to avoid underflow. Type conversion does not benefit searching for minima and maxima (‘-y min’, or ‘-y max’).
  3. The variable on disk is of type NC_BYTE, NC_CHAR, NC_SHORT, or NC_INT. Type NC_DOUBLE is not promoted because there is no type of higher precision. Conversion of type NC_FLOAT is discussed in detail below. When it occurs, it follows the same procedure (promotion then arithmetic then demotion) as conversion of integer types.

When these criteria are all met, the operator promotes the variable in question to type NC_DOUBLE, performs all the arithmetic operations, casts the NC_DOUBLE type back to the original type, and finally writes the result to disk. The result written to disk may not be what you expect, because of incommensurate ranges represented by different types, and because of (lack of) rounding. First, continuing the above example, the average (e.g., ‘-y avg’) of 17000s and 17000s is written to disk as 17000s. The type conversion feature of NCO makes this possible since the arithmetic and intermediate values are stored as NC_DOUBLEs, i.e., 34000.0d and only the final result must be represented as an NC_SHORT. Without the type conversion feature of NCO, the average would have been garbage (albeit predictable garbage near -15768s). Similarly, the total (e.g., ‘-y ttl’) of 17000s and 17000s written to disk is garbage (actually -31536s) since the final result (the true total) of 34000 is outside the range of type NC_SHORT.

After arithmetic is computed in double-precision for promoted variables, the intermediate double-precision values must be demoted to the variables’ original storage type (e.g., from NC_DOUBLE to NC_SHORT). NCO has handled this demotion in three ways in its history. Prior to October, 2011 (version 4.0.8), NCO employed the C library truncate function, trunc() 48. Truncation rounds x to the nearest integer not larger in absolute value. For example, truncation rounds 1.0d, 1.5d, and 1.8d to the same value, 1s. Clearly, truncation does not round floating-point numbers to the nearest integer! Yet truncation is how the C language performs implicit conversion of real numbers to integers.

NCO stopped using truncation for demotion when an alert user (Neil Davis) informed us that this caused a small bias in the packing algorithm employed by ncpdq. This led to NCO adopting rounding functions for demotion. Rounding functions eliminated the small bias in the packing algorithm.

From February, 2012 through March, 2013 (versions 4.0.9–4.2.6), NCO employed the C library family of rounding functions, lround(). These functions round x to the nearest integer, halfway cases away from zero. The problem with lround() is that it always rounds real values ending in .5 away from zero. This rounds, for example, 1.5d and 2.5d to 2s and 3s, respectively.

Since April, 2013 (version 4.3.0), NCO has employed the other C library family of rounding functions, lrint(). This algorithm rounds x to the nearest integer, using the current rounding direction. Halfway cases are rounded to the nearest even integer. This rounds, for example, both 1.5d and 2.5d to the same value, 2s, as recommended by the IEEE. This rounding is symmetric: up half the time, down half the time. This is the current and hopefully final demotion algorithm employed by NCO.

Hence because of automatic conversion, NCO will compute the average of 2s and 3s in double-precision arithmetic as (2.0d + 3.0d)/2.0d) = 2.5d. It then demotes this intermediate result back to NC_SHORT and stores it on disk as trunc(2.5d) = 2s (versions up to 4.0.8), lround(2.5d) = 3s (versions 4.0.9–4.2.6), and lrint(2.5d) = 2s (versions 4.3.0 and later).

Next: , Previous: , Up: Type Conversion   [Contents][Index]

3.37.2 Promoting Single-precision to Double

Promotion of real numbers from single- to double-precision is fundamental to scientific computing. When it should occur depends on the precision of the inputs and the number of operations. Single-precision (four-byte) numbers contain about seven significant figures, while double-precision contain about sixteen. More, err, precisely, the IEEE single-precision representation gives from 6 to 9 significant decimal digits precision 49. And the IEEE double-precision representation gives from 15 to 17 significant decimal digits precision 50. Hence double-precision numbers represent about nine digits more precision than single-precision numbers.

Given these properties, there are at least two possible arithmetic conventions for the treatment of real numbers:

  1. Conservative, aka Fortran Convention Automatic type conversion during arithmetic in the Fortran language is, by default, performed only when necessary. All operands in an operation are converted to the most precise type involved the operation before the arithmetic operation. Expressions which involve only single-precision numbers are computed entirely in single-precision. Expressions involving mixed precision types are computed in the type of higher precision. NCO by default employs the Fortan Convention for promotion.
  2. Aggressive, aka C Convention The C language is by default much more aggressive (and thus wasteful) than Fortran, and will always implicitly convert single- to double-precision numbers, even when there is no good reason. All real-number standard C library functions are double-precision, and C programmers must take extra steps to only utilize single precision arithmetic. The high-level interpreted data analysis languages IDL, Matlab, and NCL all adopt the C Convention.

NCO does not automatically promote NC_FLOAT because, in our judgement, the performance penalty of always doing so would outweigh the potential benefits. The now-classic text “Numerical Recipes in C” discusses this point under the section “Implicit Conversion of Float to Double” 51. That said, such promotion is warranted in some circumstances.

For example, rounding errors can accumulate to worrisome levels during arithmetic performed on large arrays of single-precision floats. This use-case occurs often in geoscientific studies of climate where thousands-to-millions of gridpoints may contribute to a single average. If the inputs are all single-precision, then so should be the output. However the intermediate results where running sums are accumulated may suffer from too much rounding or from underflow unless computed in double-precision.

The order of operations matters to floating-point math even when the analytic expressions are equal. Cautious users feel disquieted when results from equally valid analyses differ in the final bits instead of agreeing bit-for-bit. For example, averaging arrays in multiple stages produces different answers than averaging them in one step. This is easily seen in the computation of ensemble averages by two different methods. The NCO test file contains single- and double-precision representations of the same temperature timeseries as tpt_flt and tpt_dbl. Pretend each datapoint in this timeseries represents a monthly-mean temperature. We will mimic the derivation of a fifteen-year ensemble-mean January temperature by concatenating the input file five times, and then averaging the datapoints representing January two different ways. In Method 1 we derive the 15-year ensemble January average in two steps, as the average of three five-year averages. This method is naturally used when each input file contains multiple years and multiple input files are needed 52. In Method 2 we obtain 15-year ensemble January average in a single step, by averaging all 15 Januaries at one time:

# tpt_flt and tpt_dbl are identical except for precision
ncks -C -v tpt_flt,tpt_dbl ~/nco/data/
# tpt_dbl = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# tpt_flt = 273.1, 273.2, 273.3, 273.4, 273.5, 273.6, 273.7, 273.8, 273.9, 274
# Create file with five "ten-month years" (i.e., 50 timesteps) of temperature data
ncrcat -O -v tpt_flt,tpt_dbl -p ~/nco/data ~/
# Average 1st five "Januaries" (elements 1, 11, 21, 31, 41)
ncra --flt -O -F -d time,1,,10 ~/ ~/
# Average 2nd five "Januaries" (elements 2, 12, 22, 32, 42)
ncra --flt -O -F -d time,2,,10 ~/ ~/
# Average 3rd five "Januaries" (elements 3, 13, 23, 33, 43)
ncra --flt -O -F -d time,3,,10 ~/ ~/
# Method 1: Obtain ensemble January average by averaging the averages
ncra --flt -O ~/ ~/ ~/ ~/
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's "subcycle" feature (
ncra --flt -O -F -d time,1,,10,3 ~/ ~/
# Difference the two methods
ncbo -O ~/ ~/ ~/
ncks ~/
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = -3.051758e-05 ;

Although the two methods are arithmetically equivalent, they produce slightly different answers due to the different order of operations. Moreover, it appears at first glance that the single-precision answers suffer from greater error than the double-precision answers. In fact both precisions suffer from non-zero rounding errors. The answers differ negligibly to machine precision, which is about seven significant figures for single precision floats (tpt_flt), and sixteen significant figures for double precision (tpt_dbl). The input precision determines the answer precision.

IEEE arithmetic guarantees that two methods will produce bit-for-bit identical answers only if they compute the same operations in the same order. Bit-for-bit identical answers may also occur by happenstance when rounding errors exactly compensate one another. This is demonstrated by repeating the example above with the ‘--dbl’ (or ‘--rth_dbl’ for clarity) option which forces conversion of single-precision numbers to double-precision prior to arithmetic. Now ncra will treat the first value of tpt_flt, 273.1000f, as 273.1000000000000d. Arithmetic on tpt_flt then proceeds in double-precision until the final answer, which is converted back to single-precision for final storage.

# Average 1st five "Januaries" (elements 1, 11, 21, 31, 41)
ncra --dbl -O -F -d time,1,,10 ~/ ~/
# Average 2nd five "Januaries" (elements 2, 12, 22, 32, 42)
ncra --dbl -O -F -d time,2,,10 ~/ ~/
# Average 3rd five "Januaries" (elements 3, 13, 23, 33, 43)
ncra --dbl -O -F -d time,3,,10 ~/ ~/
# Method 1: Obtain ensemble January average by averaging the averages
ncra --dbl -O ~/ ~/ ~/ ~/
# Method 2: Obtain ensemble January average by averaging the raw data
# Employ ncra's "subcycle" feature (
ncra --dbl -O -F -d time,1,,10,3 ~/ ~/
# Difference the two methods
ncbo -O ~/ ~/ ~/
# Show differences
ncks ~/
# tpt_dbl = 5.6843418860808e-14 ;
# tpt_flt = 0 ;

The ‘--dbl’ switch has no effect on the results computed from double-precision inputs. But now the two methods produce bit-for-bit identical results from the single-precision inputs! This is due to the happenstance of rounding along with the effects of the ‘--dbl’ switch. The ‘--flt’ and ‘--rth_flt’ switches are provided for symmetry. They enforce the traditional NCO and Fortran convention of keeping single-precision arithmetic in single-precision unless a double-precision number is explicitly involved.

We have shown that forced promotion of single- to double-precision prior to arithmetic has advantages and disadvantages. The primary disadvantages are speed and size. Double-precision arithmetic is 10–60% slower than, and requires twice the memory of single-precision arithmetic. The primary advantage is that rounding errors in double-precision are much less likely to accumulate to values near the precision of the underlying geophysical variable.

For example, if we know temperature to five significant digits, then a rounding error of 1-bit could affect the least precise digit of temperature after 1,000–10,000 consecutive one-sided rounding errors under the worst possible scenario. Many geophysical grids have tens-of-thousands to millions of points that must be summed prior to normalization to compute an average. It is possible for single-precision rouding errors to accumulate and degrade the precision in such situtations. Double-precision arithmetic mititgates this problem, so ‘--dbl’ would be warranted.

This can be seen with another example, averaging a global surface temperature field with ncwa. The input contains a single-precision global temperature field (stored in TREFHT) produced by the CAM3 general circulation model (GCM) run and stored at 1.9 by 2.5 degrees resolution. This requires 94 latitudes and 144 longitudes, or 13,824 total surface gridpoints, a typical GCM resolution in 2008–2013. These input characteristics are provided only to show the context to the interested reader, equivalent results would be found in statistics of any dataset of comparable size. Models often represent Earth on a spherical grid where global averages must be created by weighting each gridcell by its latitude-dependent weight (e.g., a Gaussian weight stored in gw), or by the surface area of each contributing gridpoint (stored in area).

Like many geophysical models and most GCMs, CAM3 runs completely in double-precision yet stores its archival output in single-precision to save space. In practice such models usually save multi-dimensional prognostic and diagnostic fields (like TREFHT(lat,lon)) as single-precision, while saving all one-dimensional coordinates and weights (here lat, lon, and gw(lon)) as double-precision. The gridcell area area(lat,lon) is an extensive grid property that should be, but often is not, stored as double-precision. To obtain pure double-precision arithmetic and storage of the globla mean temperature, we first create and store double-precision versions of the single-precision fields:

ncap2 -O -s 'TREFHT_dbl=double(TREFHT);area_dbl=double(area)'

The single- and double-precision temperatures may each be averaged globally using four permutations for the precision of the weight and of the intermediate arithmetic representation:

  1. Single-precision weight (area), single-precision arithmetic
  2. Double-precision weight (gw), single-precision arithmetic
  3. Single-precision weight (area), double-precision arithmetic
  4. Double-precision weight (gw), double-precision arithmetic
# NB: Values below are printed with C-format %5.6f using
# ncks -H -C -s '%5.6f' -v TREFHT,TREFHT_dbl
# Single-precision weight (area), single-precision arithmetic
ncwa --flt -O -a lat,lon -w area
# TREFHT     = 289.246735 
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   single-precision arithmetic
ncwa --flt -O -a lat,lon -w gw
# TREFHT     = 289.226135
# TREFHT_dbl = 289.239964
# Single-precision weight (area), double-precision arithmetic
ncwa --dbl -O -a lat,lon -w area
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic
ncwa --dbl -O -a lat,lon -w gw
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964

First note that the TREFHT_dbl average never changes because TREFHT_dbl(lat,lon) is double-precision in the input file. As described above, NCO automatically converts all operands involving to the highest precision involved in the operation. So specifying ‘--dbl’ is redundant for double-precision inputs.

Second, the single-precision arithmetic averages of the single-precision input TREFHT differ by 289.246735 - 289.226135 = 0.0206 from eachother, and, more importantly, by as much as 289.239964 - 289.226135 = 0.013829 from the correct (double-precision) answer. These averages differ in the fifth digit, i.e., they agree only to four significant figures! Given that climate scientists are concerned about global temperature variations of a tenth of a degree or less, this difference is large. Global mean temperature changes significant to climate scientists are comparable in size to the numerical artifacts produced by the averaging procedure.

Why are the single-precision numerical artifacts so large? Each global average is the result of multiplying almost 15,000 elements each by its weight, summing those, and then dividing by the summed weights. Thus about 50,000 single-precision floating-point operations caused the loss of two to three significant digits of precision. The net error of a series of independent rounding errors is a random walk phenomena 53. Successive rounding errors displace the answer further from the truth. An ensemble of such averages will, on average, have no net bias. In other words, the expectation value of a series of IEEE rounding errors is zero. And the error of any given sequence of rounding errors obeys, for large series, a Gaussian distribution centered on zero.

Single-precision numbers use three of their four eight-bit bytes to represent the mantissa so the smallest representable single-precision mantissa is \epsilon \equiv 2^{-23} = 1.19209 \times 10^{-7}. This \epsilon is the smallest x such that 1.0 + x \ne 1.0. This is the rounding error for non-exact precision-numbers. Applying random walk theory to rounding, it can be shown that the expected rounding error after n inexact operations is \sqrt{2n/\pi} for large n. The expected (i.e., mean absolute) rounding error in our example with 13,824 additions is about \sqrt{2 \times 13824 / \pi} = 91.96. Hence, addition alone of about fifteen thousand single-precision floats is expected to consume about two significant digits of precision. This neglects the error due to the inner product (weights times values) and normalization (division by tally) aspects of a weighted average. The ratio of two numbers each containing a numerical bias can magnify the size of the bias. In summary, a global mean number computed from about 15,000 gridpoints each with weights can be expected to lose up to three significant digits. Since single-precision starts with about seven significant digits, we should not expect to retain more than four significant digits after computing weighted averages in single-precision. The above example with TREFHT shows the expected four digits of agreement.

The NCO results have been independently validated to the extent possible in three other languages: C, Matlab, and NCL. C and NCO are the only languages that permit single-precision numbers to be treated with single precision arithmetic:

# Double-precision weight (gw),   single-precision arithmetic (C)
# TREFHT     = 289.240112
# Double-precision weight (gw),   double-precision arithmetic (C)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (Matlab)
# TREFHT     = 289.239964
# Single-precision weight (area), double-precision arithmetic (NCL)
ncl < ncwa_3528514.ncl
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964
# Double-precision weight (gw),   double-precision arithmetic (NCL)
# TREFHT     = 289.239960
# TREFHT_dbl = 289.239964

All languages tested (C, Matlab, NCL, and NCO) agree to machine precision with double-precision arithmetic. Users are fortunate to have a variety of high quality software that liberates them from the drudgery of coding their own. Many packages are free (as in beer)! As shown above NCO permits one to shift to their float-promotion preferences as desired. No other language allows this with a simple switch.

To summarize, until version 4.3.6 (September, 2013), the default arithmetic convention of NCO adhered to Fortran behavior, and automatically promoted single-precision to double-precision in all mixed-precision expressions, and left-alone pure single-precision expressions. This is faster and more memory efficient than other conventions. However, pure single-precision arithmetic can lose too much precision when used to condense (e.g., average) large arrays. Statistics involving about n = 10,000 single-precision inputs will lose about 2–3 digits if not promoted to double-precision prior to arithmetic. The loss scales with the squareroot of n. For larger n, users should promote floats with the ‘--dbl’ option if they want to preserve more than four significant digits in their results.

The ‘--dbl’ and ‘--flt’ switches are only available with the NCO arithmetic operators that could potentially perform more than a few single-precision floating-point operations per result. These are nces, ncra, and ncwa. Each is capable of thousands to millions or more operations per result. By contrast, the arithmetic operators ncbo and ncflint perform at most one floating-point operation per result. Providing the ‘--dbl’ option for such trivial operations makes little sense, so the option is not currently made available.

We are interested in users’ opinions on these matters. The default behavior was changed from ‘--flt’ to ‘--dbl’ with the release of NCO version 4.3.6 (October 2013). We will change the default back to ‘--flt’ if users prefer. Or we could set a threshold (e.g., n \ge 10000) after which single- to double-precision promotion is automatically invoked. Or we could make the default promotion convention settable via an environment variable (GSL does this a lot). Please let us know what you think of the selected defaults and options.

Previous: , Up: Type Conversion   [Contents][Index]

3.37.3 Manual type conversion

ncap2 provides intrinsic functions for performing manual type conversions. This, for example, converts variable tpt to external type NC_SHORT (a C-type short), and variable prs to external type NC_DOUBLE (a C-type double).

ncap2 -s 'tpt=short(tpt);prs=double(prs)'

With ncap2 there also is the convert() method that takes an integer argument. For example the above statements become:

ncap2 -s 'tpt=tpt.convert(NC_SHORT);prs=prs.convert(NC_DOUBLE)'

Can also use convert() in combination with type() so to make variable ilev_new the same type as ilev just do:

ncap2 -s 'ilev_new=ilev_new.convert(ilev.type())'

See ncap2 netCDF Arithmetic Processor, for more details.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.38 Batch Mode

Availability: All operators
Short options: ‘-O’, ‘-A
Long options: ‘--ovr’, ‘--overwrite’, ‘--apn’, ‘--append

If the output-file specified for a command is a pre-existing file, then the operator will prompt the user whether to overwrite (erase) the existing output-file, attempt to append to it, or abort the operation. However, interactive questions reduce productivity when processing large amounts of data. Therefore NCO also implements two ways to override its own safety features, the ‘-O’ and ‘-A’ switches. Specifying ‘-O’ tells the operator to overwrite any existing output-file without prompting the user interactively. Specifying ‘-A’ tells the operator to attempt to append to any existing output-file without prompting the user interactively. These switches are useful in batch environments because they suppress interactive keyboard input. NB: As of 20120515, ncap2 is unable to append to files that already contain the appended dimensions.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.39 Global Attribute Addition

Availability: All operators
Short options: None
Long options: ‘--glb’, ‘--gaa’, ‘--glb_att_add
--glb att_nm=att_val’ (multiple invocations allowed)

All operators can add user-specified global attributes to output files. As of NCO version 4.5.2 (July, 2015), NCO supports multiple uses of the ‘--glb’ (or equivalent ‘--gaa’ or ‘--glb_att_add’) switch. The option ‘--gaa’ (and its long option equivalents such as ‘--glb_att_add’) indicates the argument syntax will be key=val. As such, ‘--gaa’ and its synonyms are indicator options that accept arguments supplied one-by-one like ‘--gaa key1=val1 --gaa key2=val2’, or aggregated together in multi-argument format like ‘--gaa key1=val1#key2=val2’ (see Multi-arguments).

The switch takes mandatory arguments ‘--glb att_nm=att_val’ where att_nm is the desired name of the global attribute to add, and att_val is its value. Currently only text attributes are supported (recorded as type NC_CHAR), and regular expressions are not allowed (unlike see ncatted netCDF Attribute Editor). Attributes are added in “Append” mode, meaning that values are appended to pre-existing values, if any. Multiple invocations can simplify the annotation of output file at creation (or modification) time:

ncra --glb machine=${HOSTNAME} --glb created_by=${USER} in*.nc

As of NCO version 4.6.2 (October, 2016), one may instead combine the separate invocations into a single list of invocations separated by colons:

ncra --glb machine=${HOSTNAME}:created_by=${USER} in*.nc

The list may contain any number of key-value pairs. Special care must be taken should a key or value contain a delimiter (i.e., a colon) otherwise NCO will interpret the colon as a delimiter and will attempt to create a new attribute. To protect a colon from being interpreted as an argument delimiter, precede it with a backslash.

The global attribution addition feature helps to avoid the performance penalty incurred by using ncatted separately to annotate large files. Should users emit a loud hue and cry, we will consider ading the the functionality of ncatted to the front-end of all operators, i.e., accepting valid ncatted arguments to modify attributes of any type and to apply regular expressions.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.40 History Attribute

Availability: All operators
Short options: ‘-h
Long options: ‘--hst’, ‘--history

All operators automatically append a history global attribute to any file they create or modify. The history attribute consists of a timestamp and the full string of the invocation command to the operator, e.g., ‘Mon May 26 20:10:24 1997: ncks’. The full contents of an existing history attribute are copied from the first input-file to the output-file. The timestamps appear in reverse chronological order, with the most recent timestamp appearing first in the history attribute. Since NCO adheres to the history convention, the entire data processing path of a given netCDF file may often be deduced from examination of its history attribute. As of May, 2002, NCO is case-insensitive to the spelling of the history attribute name. Thus attributes named History or HISTORY (which are non-standard and not recommended) will be treated as valid history attributes. When more than one global attribute fits the case-insensitive search for “history”, the first one found is used. To avoid information overkill, all operators have an optional switch (‘-h’, ‘--hst’, or ‘--history’) to override automatically appending the history attribute (see ncatted netCDF Attribute Editor). Note that the ‘-h’ switch also turns off writing the nco_input_file_list-attribute for multi-file operators (see File List Attributes).

As of NCO version 4.5.0 (June, 2015), NCO supports its own convention to retain the history-attribute contents of all files that were appended to a file 54. This convention stores those contents in the history_of_appended_files attribute, which complements the history-attribute to provide a more complete provenance. These attributes may appear something like this in output:

// global attributes:
:history = "Thu Jun  4 14:19:04 2015: ncks -A /home/zender/ /home/zender/\n",
  "Thu Jun  4 14:19:04 2015: ncks -A /home/zender/ /home/zender/\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att1,global,o,c,global metadata only in foo1 /home/zender/\n",
  "original history from the ur-file serving as the basis for subsequent appends." ;
:history_of_appended_files = "Thu Jun  4 14:19:04 2015: Appended file \
  /home/zender/ had following \"history\" attribute:\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att2,global,o,c,global metadata only in foo3 /home/zender/\n",
  "history from foo3 from which data was appended to foo1 after data from foo2 was appended\n",
  "Thu Jun  4 14:19:04 2015: Appended file /home/zender/ had following \"history\" attribute:\n",
  "Thu Jun  4 14:19:04 2015: ncatted -O -a att2,global,o,c,global metadata only in foo2 /home/zender/\n",
  "history of some totally different file foo2 from which data was appended to foo1 before foo3 was appended\n",
:att1 = "global metadata only in foo1" ;

Note that the history_of_appended_files-attribute is only created, and will only exist, in a file that is, or descends from a file that was, appended to. The optional switch ‘-h’ (or ‘--hst’ or ‘--history’) also overrides automatically appending the history_of_appended_files attribute.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.41 File List Attributes

Availability: nces, ncecat, ncra, ncrcat
Short options: ‘-H
Long options: ‘--fl_lst_in’, ‘--file_list

Many methods of specifying large numbers of input file names pass these names via pipes, encodings, or argument transfer programs (see Large Numbers of Files). When these methods are used, the input file list is not explicitly passed on the command line. This results in a loss of information since the history attribute no longer contains the exact command by which the file was created.

NCO solves this dilemma by archiving input file list attributes. When the input file list to a multi-file operator is specified via stdin, the operator, by default, attaches two global attributes to any file they create or modify. The nco_input_file_number global attribute contains the number of input files, and nco_input_file_list contains the file names, specified as standard input to the multi-file operator. This information helps to verify that all input files the user thinks were piped through stdin actually arrived. Without the nco_input_file_list attribute, the information is lost forever and the “chain of evidence” would be broken.

The ‘-H’ switch overrides (turns off) the default behavior of writing the input file list global attributes when input is from stdin. The ‘-h’ switch does this too, and turns off the history attribute as well (see History Attribute). Hence both switches allows space-conscious users to avoid storing what may amount to many thousands of filenames in a metadata attribute.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.42 CF Conventions

Availability: ncbo, nces, ncecat, ncflint, ncpdq, ncra, ncwa
Short options: None

NCO recognizes some Climate and Forecast (CF) metadata conventions, and applies special rules to such data. NCO was contemporaneous with COARDS and still contains some rules to handle older model datasets that pre-date CF, such as NCAR CCM and early CCSM datasets. Such datasets may not contain an explicit Conventions attribute (e.g., ‘CF-1.0’). Nevertheless, we refer to all such metadata collectively as CF metadata. Skip this section if you never work with CF metadata.

The latest CF netCDF conventions are described here. Most CF netCDF conventions are transparent to NCO. There are no known pitfalls associated with using any NCO operator on files adhering to these conventions. NCO applies some rules that are not in CF, or anywhere else, because experience shows that they simplify data analysis, and stay true to the NCO mantra to do what users want.

Here is a general sense of NCO’s CF-support:

Finally, a main use of NCO is to “produce CF”, i.e., to improve CF-compliance by annotating metadata, renaming objects (attributes, variables, and dimensions), permuting and inverting dimensions, recomputing values, and data compression.

Currently, NCO determines whether a datafile is a CF output datafile simply by checking (case-insensitively) whether the value of the global attribute Conventions (if any) equals ‘CF-1.0’ or ‘NCAR-CSM’ Should Conventions equal either of these in the (first) input-file, NCO will apply special rules to certain variables because of their usual meaning in CF files. NCO will not average the following variables often found in CF files: ntrm, ntrn, ntrk, ndbase, nsbase, nbdate, nbsec, mdt, mhisf. These variables contain scalar metadata such as the resolution of the host geophysical model and it makes no sense to change their values.

Furthermore, the size and rank-preserving arithmetic operators try not to operate on certain grid properties. These operators are ncap2, ncbo, nces, ncflint, and ncpdq (when used for packing, not for permutation). These operators do not operate, by default, on (i.e., add, subtract, pack, etc.) the following variables: ORO, area, datesec, date, gw, hyai, hyam, hybi. hybm, lat_bnds, lon_bnds, msk_*, wgt_*. These variables represent Gaussian weights, land/sea masks, time fields, hybrid pressure coefficients, and latititude/longitude boundaries. We call these fields non-coordinate grid properties. Coordinate grid properties are easy to identify because they are coordinate variables such as latitude and longitude.

Users usually want all grid properties to remain unaltered in the output file. To be treated as a grid property, the variable name must exactly match a name in the above list, or be a coordinate variable. Handling of msk_* and wgt_* is exceptional in that any variable whose name starts with msk_ or wgt_ is considered to be a “mask” or a “weight” and is thus preserved (not operated on when arithmetic can be avoided).

As of NCO version 4.5.0 (June, 2015), NCO began to support behavior required for the DOE E3SM/ACME program, and we refer to these rules collectively as the E3SM/ACME convention. The first E3SM/ACME rule implemented is that the contents of input-file variables named date_written and time_written, if any, will be updated to the current system-supplied (with gmtime()) GMT-time as the variables are copied to the output-file.

You must spoof NCO if you would like any grid properties or other special CF fields processed normally. For example rename the variables first with ncrename, or alter the Conventions attribute.

As of NCO version 4.0.8 (April, 2011), NCO supports the CF bounds convention for cell boundaries described here. This convention allows coordinate variables (including multidimensional coordinates) to describe the boundaries of their cells. This is done by naming the variable which contains the bounds in in the bounds attribute. Note that coordinates of rank N have bounds of rank N+1. NCO-generated subsets of CF-compliant files with bounds attributes will include the coordinates specified by the bounds attribute, if any. Hence the subsets will themselves be CF-compliant. Bounds are subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables.

As of NCO version 4.4.9 (May, 2015), NCO supports the CF climatology convention for climatological statistics described here. This convention allows coordinate variables (including multidimensional coordinates) to describe the (possibly nested) periods and statistical methods of their associated statistics. This is done by naming the variable which contains the periods and methods in the climatology attribute. Note that coordinates of rank N have climatology bounds of rank N+1. NCO-generated subsets of CF-compliant files with climatology attributes will include the variables specified by the climatology attribute, if any. Hence the subsets will themselves be CF-compliant. Climatology variables are subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables.

As of NCO version 4.4.5 (July, 2014), NCO supports the CF ancillary_variables convention for described here. This convention allows ancillary variables to be associated with one or more primary variables. NCO attaches any such variables to the extraction list along with the primary variable and its usual (one-dimensional) coordinates, if any. Ancillary variables are subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables.

As of NCO version 4.6.4 (January, 2017), NCO supports the CF cell_measures convention described here. This convention allows variables to indicate which other variable or variables contains area or volume information about a gridcell. These measures variables are pointed to by the cell_measures attribute. The CDL specification of a measures variable for area looks like

orog:cell_measures = "area: areacella"

where areacella is the name of the measures variable. Unless the default behavior is overridden, NCO attaches any measures variables to the extraction list along with the primary variable and other associated variables. By definition, measures variables are a subset of the rank of the variable they measure. The most common case is that the measures variable for area is the same size as 2D fields (like surface air temperature) and much smaller than 3D fields (like full air temperature). In such cases the measures variable might occupy 50% of the space of a dataset consisting of only one 2D field. Extraction of measures variables is subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables. To conserve space without sacrificing too much metadata, NCO makes it possible to override the extraction of measures variables independent of extracting other associated variables. Override the default with ‘--no_cell_measures’ or ‘--no_cll_msr’. These options are available in all operators that perform subsetting (i.e., all operators except ncatted and ncrename).

As of NCO version 4.6.4 (January, 2017), NCO supports the CF formula_terms convention described here. This convention encodes formulas used to construct (usually vertical) coordinate grids. The CDL specification of a vertical coordinate formula for looks like

lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate"
lev:formula_terms = "a: hyam b: hybm p0: P0 ps: PS"

where standard_name contains the standardized name of the formula variable and formula_terms contains a list of the variables used, called formula variables. Above the formula variables are hyam, hybm, P0, and PS. Unless the default behavior is overridden, NCO attaches any formula variables to the extraction list along with the primary variable and other associated variables. By definition, formula variables are a subset of the rank of the variable they define. One common case is that the formula variables for constructing a 3D height grid involves a 2D variable (like surface pressure, or elevation). In such cases the formula variables typically constitute only a small fraction of a dataset consisting of one 3D field. Extraction of formula variables is subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables. To conserve space without sacrificing too much metadata, NCO makes it possible to override the extraction of formula variables independent of extracting other associated variables. Override the default with ‘--no_formula_terms’ or ‘--no_frm_trm’. These options are available in all operators that perform subsetting (i.e., all operators except ncatted and ncrename).

As of NCO version 4.6.0 (May, 2016), NCO supports the CF grid_mapping convention for described here. This convention allows descriptions of map-projections to be associated with variables. NCO attaches any such map-projection variables to the extraction list along with the primary variable and its usual (one-dimensional) coordinates, if any. Map-projection variables are subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables.

As of NCO version 3.9.6 (January, 2009), NCO supports the CF coordinates convention described here. This convention allows variables to specify additional coordinates (including mult-idimensional coordinates) in a space-separated string attribute named coordinates. NCO attaches any such coordinates to the extraction list along with the variable and its usual (one-dimensional) coordinates, if any. These auxiliary coordinates are subject to the user-specified override switches (including ‘-c’ and ‘-C’) described in Subsetting Coordinate Variables.

Elimination of reduced dimensions from the coordinates attribute helps ensure that rank-reduced variables become completely independent from their former dimensions. As of NCO version 4.4.9 (May, 2015), NCO may modify the coordinates attribute to assist this. In particular, ncwa eliminates from the coordinates attribute any dimension that it collapses, e.g., by averaging. The former presence of this dimension will usually be indicated by the CF cell_methods convention described here. Hence the CF cell_methods and coordinates conventions can be said to work in tandem to characterize the state and history of a variable’s analysis.

As of NCO version 4.4.2 (February, 2014), NCO supports some of the CF cell_methods convention to describe the analysis procedures that have been applied to data. The convention creates (or appends to an existing) cell_methods attribute a space-separated list of couplets of the form dmn: op where dmn is a comma-separated list of dimensions previously contained in the variable that have been reduced by the arithmetic operation op. For example, the cell_methods value time: mean says that the variable in question was averaged over the time dimension. In such cases time will either be a scalar variable or a degenerate dimension or coordinate. This simply means that it has been averaged-over. The value time, lon: mean lat: max says that the variable in question is the maximum zonal mean of the time averaged original variable. Which is to say that the variable was first averaged over time and longitude, and then the residual latitudinal array was reduced by choosing the maximum value. Since the cell methods convention may alter metadata in an undesirable (or possibly incorrect) fashion, we provide switches to ensure it is always or never used. Use long-options ‘--cll_mth’ or ‘--cell_methods’ to invoke the algorithm (true by default), and options ‘--no_cll_mth’ or ‘--no_cell_methods’ to turn it off. These options are only available in the operators ncwa and ncra.

Next: , Previous: , Up: Shared features   [Contents][Index]

3.43 ARM Conventions

Availability: ncrcat
Short options: None

ncrcat has been programmed to correctly handle data files which utilize the Atmospheric Radiation Measurement (ARM) Program convention for time and time offsets. If you do not work with ARM data then you may skip this section. ARM data files store time information in two variables, a scalar, base_time, and a record variable, time_offset. Subtle but serious problems can arise when these type of files are blindly concatenated without CF or ARM support. NCO implements rebasing (see Rebasing Time Coordinate) as necessary on both CF and ARM files. Rebasing chains together consecutive input-files and produces an output-file which contains the correct time information. For ARM files this is expecially complex because the time coordinates are often stored as type NC_CHAR. Currently, ncrcat determines whether a datafile is an ARM datafile simply by testing for the existence of the variables base_time, time_offset, and the dimension time. If these are found in the input-file then ncrcat will automatically perform two non-standard, but hopefully useful, procedures. First, ncrcat will ensure that values of time_offset appearing in the output-file are relative to the base_time appearing in the first input-file (and presumably, though not necessarily, also appearing in the output-file). Second, if a coordinate variable named time is not found in the input-files, then ncrcat automatically creates the time coordinate in the output-file. The values of time are defined by the ARM conventions time = base_time + time_offset. Thus, if output-file contains the time_offset variable, it will also contain the time coordinate. A short message is added to the history global attribute whenever these ARM-specific procedures are executed.

Previous: , Up: Shared features   [Contents][Index]

3.44 Operator Version

Availability: All operators
Short options: ‘-r
Long options: ‘--revision’, ‘--version’, or ‘--vrs

All operators can be told to print their version information, library version, copyright notice, and compile-time configuration with the ‘-r’ switch, or its long-option equivalent ‘revision’. The ‘--version’ or ‘--vrs’ switches print the operator version information only. The internal version number varies between operators, and indicates the most recent change to a particular operator’s source code. This is useful in making sure you are working with the most recent operators. The version of NCO you are using might be, e.g., 3.9.5. Using ‘-r’ on, say, ncks, produces something like ‘NCO netCDF Operators version "3.9.5" last modified 2008/05/11 built May 12 2008 on neige by zender Copyright (C) 1995--2008 Charlie Zender ncks version 20090918’. This tells you that ncks contains all patches up to version 3.9.5, which dates from May 11, 2008.

Next: , Previous: , Up: Top   [Contents][Index]

4 Reference Manual

This chapter presents reference pages for each of the operators individually. The operators are presented in alphabetical order. All valid command line switches are included in the syntax statement. Recall that descriptions of many of these command line switches are provided only in Shared features, to avoid redundancy. Only options specific to, or most useful with, a particular operator are described in any detail in the sections below.

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.1 ncap2 netCDF Arithmetic Processor

ncap2 understands a relatively full-featured language of operations, including loops, conditionals, arrays, and math functions. ncap2 is the most rapidly changing NCO operator and its documentation is incomplete. The distribution file data/ncap2_tst.nco contains an up-to-date overview of its syntax and capabilities. The data/*.nco distribution files (especially bin_cnt.nco, psd_wrf.nco, and rgr.nco) contain in-depth examples of ncap2 solutions to complex problems.


ncap2 [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] 
[-D dbg] [-F] [-f] [--glb ...] [-h] [--hdf] [--hdr_pad nbr] [-L dfl_lvl] [-l path]
[--no_tmp_fl] [-O] [-o output-file] [-p path] [-R] [-r] [--ram_all]
[-s algebra] [-S fl.nco] [-t thr_nbr] [-v]
input-file [output-file]  


ncap2 arithmetically processes netCDF files 55. The processing instructions are contained either in the NCO script file fl.nco or in a sequence of command line arguments. The options ‘-s’ (or long options ‘--spt’ or ‘--script’) are used for in-line scripts and ‘-S’ (or long options ‘--fl_spt’ or ‘--script-file’) are used to provide the filename where (usually multiple) scripting commands are pre-stored. ncap2 was written to perform arbitrary algebraic transformations of data and archive the results as easily as possible. See Missing Values, for treatment of missing values. The results of the algebraic manipulations are called derived fields.

Unlike the other operators, ncap2 does not accept a list of variables to be operated on as an argument to ‘-v’ (see Subsetting Files). Rather, the ‘-v’ switch takes no arguments and indicates that ncap2 should output only user-defined variables. ncap2 neither accepts nor understands the -x switch. NB: As of 20120515, ncap2 is unable to append to files that already contain the appended dimensions.

Defining new variables in terms of existing variables is a powerful feature of ncap2. Derived fields inherit the metadata (i.e., attributes) of their ancestors, if any, in the script or input file. When the derived field is completely new (no identically-named ancestors exist), then it inherits the metadata (if any) of the left-most variable on the right hand side of the defining expression. This metadata inheritance is called attribute propagation. Attribute propagation is intended to facilitate well-documented data analysis, and we welcome suggestions to improve this feature.

The only exception to this rule of attribute propagation is in cases of left hand casting (see Left hand casting). The user must manually define the proper metadata for variables defined using left hand casting.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.1 Syntax of ncap2 statements

Mastering ncap2 is relatively simple. Each valid statement statement consists of standard forward algebraic expression. The fl.nco, if present, is simply a list of such statements, whitespace, and comments. The syntax of statements is most like the computer language C. The following characteristics of C are preserved:

Array syntax

Arrays elements are placed within [] characters;

Array indexing

Arrays are 0-based;

Array storage

Last dimension is most rapidly varying;

Assignment statements

A semi-colon ‘;’ indicates the end of an assignment statement.


Multi-line comments are enclosed within /* */ characters. Single line comments are preceded by // characters.


Files may be nested in scripts using #include script. The #include command is not followed by a semi-colon because it is a pre-processor directive, not an assignment statement. The filename script is interpreted relative to the run directory.

Attribute syntax

The at-sign @ is used to delineate an attribute name from a variable name.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.2 Expressions

Expressions are the fundamental building block of ncap2. Expressions are composed of variables, numbers, literals, and attributes. The following C operators are “overloaded” and work with scalars and multi-dimensional arrays:

Arithmetic Operators: * / % + - ^
Binary Operators:     > >= < <= == != == || && >> <<
Unary Operators:      + - ++ -- !
Conditional Operator: exp1 ? exp2 : exp3
Assign Operators:     = += -= /= *=

In the following section a variable also refers to a number literal which is read in as a scalar variable:

Arithmetic and Binary Operators

Consider var1 ’op’ var2



Even though the logical operators return True(1) or False(0) they are treated in the same way as the arithmetic operators with regard to precision and rank.

dimensions: time=10, lat=2, lon=4
Suppose we have the two variables:

double  P(time,lat,lon);
float   PZ0(lon,lat);  // PZ0=1,2,3,4,5,6,7,8;

Consider now the expression:

PZ0 is made to conform to P and the result is
PZ0 =

Once the expression is evaluated then PZ will be of type double;

Consider now 
 start=four-att_var@double_att;  // start =-69  and is of type intger;
 four_pow=four^3.0f               // four_pow=64 and is of type float  
 three_nw=three_dmn_var_sht*1.0f; // type is now float
                                  // start@n1=5329 and is type int 

Binary Operators
Unlike C the binary operators return an array of values. There is no such thing as short circuiting with the AND/OR operators. Missing values are carried into the result in the same way they are with the arithmetic operators. When an expression is evaluated in an if() the missing values are treated as true.
The binary operators are, in order of precedence:

!   Logical Not
<<  Less Than Selection
>>  Greater Than Selection
>   Greater than
>=  Greater than or equal to
<   Less than
<=  Less than or equal to
==  Equal to
!=  Not equal to
&&  Logical AND
||  Logical OR

To see all operators: see Operator precedence and associativity Examples:

tm1=time>2 && time <7;  // tm1=0, 0, 1, 1, 1, 1, 0, 0, 0, 0 double
tm2=time==3 || time>=6; // tm2=0, 0, 1, 0, 0, 1, 1, 1, 1, 1 double
tm3=int(!tm1);          // tm3=1, 1, 0, 0, 0, 0, 1, 1, 1, 1 int
tm4=tm1 && tm2;         // tm4=0, 0, 1, 0, 0, 1, 0, 0, 0, 0 double
tm5=!tm4;               // tm5=1, 1, 0, 1, 1, 0, 1, 1, 1, 1 double

Regular Assign Operator
var1 ’=’ exp1
If var1 does not already exist in Input or Output then var1 is written to Output with the values, type and dimensions from expr1. If var1 is in Input only it is copied to Output first. Once the var is in Ouptut then the only reqirement on expr1 is that the number of elements must match the number already on disk. The type of expr1 is converted as necessary to the disk type.

If you wish to change the type or shape of a variable in Input then you must cast the variable. See see Left hand casting


Other Assign Operators +=,-=,*=./=
var1 ’ass_op’ exp1
if exp1 is a variable and it doesn’t conform to var1 then an attempt is made to make it conform to var1. If exp1 is an attribute it must have unity size or else have the same number of elements as var1. If expr1 has a different type to var1 the it is converted to the var1 type.

z1=four+=one*=10 // z1=14 four=14 one=10;	
time-=2          // time= -1,0,1,2,3,4,5,6,7,8

Increment/Decrement Operators
These work in a similar fashion to their regular C counterparts. If say the variable four is input only then the statement ++four effectively means read four from input increment each element by one, then write the new values to Output;


n2=++four;   n2=5, four=5 
n3=one--+20; n3=21  one=0;	 
n4=--time;   n4=time=0.,1.,2.,3.,4.,5.,6.,7.,8.,9.;

Conditional Operator ?:
exp1 ? exp2 : exp3
The conditional operator (or ternary Operator) is a succinct way of writing an if/then/else. If exp1 evaluates to true then exp2 is returned else exp3 is returned.


weight_avg@units= (weight_avg == 1 ? "kilo" : "kilos");  
PS_nw=PS-(PS.min() > 100000 ? 100000 : 0);

Clipping Operators

<< Less-than Clipping

For arrays, the less-than selection operator selects all values in the left operand that are less than the corresponding value in the right operand. If the value of the left side is greater than or equal to the corresponding value of the right side, then the right side value is placed in the result

>> Greater-than Clipping

For arrays, the greater-than selection operator selects all values in the left operand that are greater than the corresponding value in the right operand. If the value of the left side is less than or equal to the corresponding value of the right side, then the right side value is placed in the result.


RDM2=RDM >> 100.0 // 100,100,100,100,126,126,100,100,100,100 double
RDM2=RDM <<  90s  // 1, 9, 36, 84, 90, 90, 84, 36, 9, 1 int

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.3 Dimensions

Dimensions are defined in Output using the defdim() function.

defdim("cnt",10); # Dimension size is fixed by default
defdim("cnt",10,NC_UNLIMITED); # Dimension is unlimited (record dimension)
defdim("cnt",10,0); # Dimension is unlimited (record dimension)
defdim("cnt",10,1); # Dimension size is fixed
defdim("cnt",10,737); # All non-zero values indicate dimension size is fixed

This dimension name must then be prefixed with a dollar-sign ‘$’ when referred to in method arguments or left-hand-casting, e.g.,


The size method allows the dimension size to be used in an arithmetic expression: / $time.size;

Increase the size of a new variable by one and set new member to zero:


To define an unlimited dimension simply set the size to zero


Dimension Abbreviations
It is possible to use dimension abbreviations as method arguments:
$0 is the first dimension of a variable
$1 is the second dimension of a variable
$n is the n+1 dimension of a variable

float four_dmn_rec_var(time,lat,lev,lon);
double three_dmn_var_dbl(time,lat,lon);




ID Quoting
If the dimension name contains non-regular characters use ID quoting: See see ID Quoting


It is not possible to manually define in Output any dimensions that exist in Input. When a variable from Input appears in an expression or statement its dimensions in Input are automagically copied to Output (if they are not already present)

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.4 Left hand casting

The following examples demonstrate the utility of the left hand casting ability of ncap2. Consider first this simple, artificial, example. If lat and lon are one dimensional coordinates of dimensions lat and lon, respectively, then addition of these two one-dimensional arrays is intrinsically ill-defined because whether lat_lon should be dimensioned lat by lon or lon by lat is ambiguous (assuming that addition is to remain a commutative procedure, i.e., one that does not depend on the order of its arguments). Differing dimensions are said to be orthogonal to one another, and sets of dimensions which are mutually exclusive are orthogonal as a set and any arithmetic operation between variables in orthogonal dimensional spaces is ambiguous without further information.

The ambiguity may be resolved by enumerating the desired dimension ordering of the output expression inside square brackets on the left hand side (LHS) of the equals sign. This is called left hand casting because the user resolves the dimensional ordering of the RHS of the expression by specifying the desired ordering on the LHS.

ncap2 -s 'lat_lon[lat,lon]=lat+lon'
ncap2 -s 'lon_lat[lon,lat]=lat+lon'

The explicit list of dimensions on the LHS, [lat,lon] resolves the otherwise ambiguous ordering of dimensions in lat_lon. In effect, the LHS casts its rank properties onto the RHS. Without LHS casting, the dimensional ordering of lat_lon would be undefined and, hopefully, ncap2 would print an error message.

Consider now a slightly more complex example. In geophysical models, a coordinate system based on a blend of terrain-following and density-following surfaces is called a hybrid coordinate system. In this coordinate system, four variables must be manipulated to obtain the pressure of the vertical coordinate: PO is the domain-mean surface pressure offset (a scalar), PS is the local (time-varying) surface pressure (usually two horizontal spatial dimensions, i.e. latitude by longitude), hyam is the weight given to surfaces of constant density (one spatial dimension, pressure, which is orthogonal to the horizontal dimensions), and hybm is the weight given to surfaces of constant elevation (also one spatial dimension). This command constructs a four-dimensional pressure prs_mdp from the four input variables of mixed rank and orthogonality:

ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm'

Manipulating the four fields which define the pressure in a hybrid coordinate system is easy with left hand casting.

Finally, we show how to use interface quantities to define midpoint quantities. In particular, we will define interface pressures using the standard CESM output hybrid coordinate parameters, and then difference those interface pressures to obtain the pressure difference between the interfaces. The pressure difference is necessary obtain gridcell mass path and density (which are midpoint quantities). Definitions are as in the above example, with new variables hyai and hybi defined at grid cell vertical interfaces (rather than midpoints like hyam and hybm). The approach naturally fits into two lines:

cat > ~/pdel.nco << 'EOF'
// Requires NCO 4.5.4 and later:
// Derived variable that require pressure thickness:
// Divide by gravity to obtain total mass path in layer aka mpl [kg m-2] 
// Multiply by mass mixing ratio to obtain mass path of constituent
ncap2 -O -v -S ~/pdel.nco ~/nco/data/ ~/
ncks -O -C -v prs_dlt ~/

The first line defines the four-dimensional interface pressures prs_ntf as a RAM variable because those are not desired in the output file. The second differences each pressure level from the pressure above it to obtain the pressure difference. This line employs both left-hand casting and array hyperslabbing. However, this syntax only works with NCO version 4.5.4 (November, 2015) and later because earlier versions require that LHS and RHS dimension names (not just sizes) match. From the pressure differences, one can obtain the mass path in each layer as shown.

Another reason to cast a variable is to modify the shape or type of a variable already in Input


Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.5 Arrays and hyperslabs

Generating a regularly spaced n-dimensional array with ncap2 is simple with the array() function. The function comes in three (overloaded) forms

(A) var_out=array(val_srt, val_inc, $dmn_nm);           // One-dimensional output
(B) var_out=array(val_srt, val_inc, var_tpl);           // Multi-dimensional output
(C) var_out=array(val_srt, val_inc, /$dmn1,$dmn2...,$dmnN/); // Multi-dimensional output

Starting value of the array. The type of the array will be the type of this starting value.


Spacing (or increment) between elements.


Variable from which the array can derive its shape 1D or nD

One-Dimensional Arrays
Use form (A) or (B) for 1D arrays

# var_out will be NC_DOUBLE:
var_out=array(10.0,2,$time) // 10.5,12.5,14.5,16.5,18.5,20.5,22.5,24.5,26.5,28.5

// var_out will be NC_UINT, and "shape" will duplicate "ilev"
var_out=array(0ul,2,ilev) // 0,2,4,6

// var_out will be NC_FLOAT
var_out=array(99.0f,2.5,$lon) // 99,101.5,104,106.5

// Create an array of zeros 
var_out=array(0,0,$time) // 0,0,0,0,0,0,0,0,0,0 

// Create array of ones
var_out=array(1.0,0.0,$lon) // 1.0,1.0,1.0,1.0 

n-Dimensional Arrays
Use form (B) or (C) for creating n-D arrays.
NB: In (C) the final argument is a list of dimensions

// These are equivalent
var_out=array(1.0, 2.0, three_dmn_var)
var_out=array(1.0, 2.0,/$lat, $lev, $lon/)

// var_out is NC_BYTE
var_out=array(20b, -4, /$lat,$lon/)  // 20,16,12,8,4,0,-4,-8  

srt=3.14159f; inc=srt/2.0f
var_out(srt,inc, var_2D_rrg)  
// 3.14159, 4.712385, 6.28318, 7.853975, 9.42477, 10.99557, 12.56636, 14.13716 ; 

Hyperslabs in ncap2 are more limited than hyperslabs with the other NCO operators. ncap2 does not understand the shell command-line syntax used to specify multi-slabs, wrapped co-ordinates, negative stride or coordinate value limits. However with a bit of syntactic magic they are all are possible. ncap2 accepts (in fact, it requires) N-hyperslab arguments for a variable of rank N:

var1(arg1,arg2 ... argN);

where each hyperslab argument is of the form


and the arguments for different dimensions are separated by commas. If start is omitted, it defaults to zero. If end is omitted, it defaults to dimension size minus one. If stride is omitted, it defaults to one.

If a single value is present then it is assumed that that dimension collapses to a single value (i.e., a cross-section). The number of hyperslab arguments MUST equal the variable’s rank.

Hyperslabs on the Right Hand Side of an assign

A simple 1D example:


od(7);     // 34
od(7:);    // 34,36,38
od(:7);    // 20,22,24,26,28,30,32,34 
od(::4);   // 20,28,36
od(1:6:2)  // 22,26,30
od(:)      // 20,22,24,26,28,30,32,34,36,38 

A more complex three dimensional example:

                          {1, 2, 3, 4, 5, 6, 7, 8,
                          -99,74,75,76,77,78,79,-99 };

th(1,1,3);        // 16
th(2,0,:);        // 17, 18, 19, 20
th(:,1,3);        // 8, 16, 24, -99, 40, 48, 56, 64, 72, -99 
th(::5,:,0:3:2); // 1, 3, 5, 7, 41, 43, 45, 47

If hyperslab arguments collapse to a single value (a cross-section has been specified), then that dimension is removed from the returned variable. If all the values collapse then a scalar variable is returned. So, for example, the following is valid:

// th_nw has dimensions $lon,$lat 
// NB: the time dimension has become degenerate

The following is invalid:


because the $lon dimension now only has two elements. The above can be calculated by using a LHS cast with $lon_nw as replacement dim for $lon:

th_nw[$lat,$lon_nw]=th(0,:,0:1) +th(9,:,0:1);

Hyperslabs on the Left Hand Side of an assign
When hyperslabing on the LHS, the expression on the RHS must evaluate to a scalar or a variable/attribute with the same number of elements as the LHS hyperslab. Set all elements of the last record to zero:


Set first element of each lon element to 1.0:


One may hyperslab on both sides of an assign. For example, this sets the last record to the first record:


Say th0 represents pressure at height=0 and th1 represents pressure at height=1. Then it is possible to insert these hyperslabs into the records


Reverse method
Use the reverse() method to reverse a dimension’s elements in a variable with at least one dimension. This is equivalent to a negative stride, e.g.,

th_rv=th(1 ,:,:).reverse($lon); // {12,11,10,9 }, {16,15,14,13}
od_rv=od.reverse($time);        // {38,36,34,32,30,28,26,24,22,20}

Permute methodp
Use the permute() method to swap the dimensions of a variable. The number and names of dimension arguments must match the dimensions in the variable. If the first dimension in the variable is of record type then this must remain the first dimension. If you want to change the record dimension then consider using ncpdq.

Consider the variable:

float three_dmn_var(lat,lev,lon);
// The permuted values are

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.6 Attributes

Attributes are referred to by var_nm@att_nm
All the following are valid statements:

global@text="Test Attributes"; /* Assign a global variable attribute */

The netCDF specification allows all attribute types to have a size greater than one. The maximum is defined by NC_MAX_ATTRS. The following is an ncdump of the metadata for variable a1

double a1(time) ;
  a1:long_name = "Kelvin" ;
  a1:max = 199. ;
  a1:min = 21. ;
  a1:trip1 = 1, 2, 3 ;
  a1:triplet = 21., 110., 199. ;

The following basic methods size(), type(), exists() can be used with attributes. For example, to save an attribute text string in a variable,

sng_arr[$sng_len]=a1@long_name; // sng_arr now contains "Kelvin" 

Attributes defined in a script are stored in memory and are written to Output after script completion. To stop the attribute being written use the ram_delete() method or use a bogus variable name.

Attribute Propagation and Inheritance

// prs_mdp inherits attributes from P0:
// th_min inherits attributes from three_dmn_var_dbl:
th_min=1.0 + 2*three_dmn_var_dbl.min($time);

Attribute Concatenation

The push() function concatenates attributes, or appends an “expression” to a pre-existing attribute. It comes in two forms

(A) att_new=push(att_exp, expr)
(B) att_size=push(&att_nm,expr)

In form (A) The first argument should be an attribute identifier or an expression that evaluates to an attribute. The second argument can evalute to an attribute or a variable. The second argument is then converted to the the type of att_exp; and appended to att_exp ; and the resulting attribute is returned.

In form (B) the first argument is a call-by-reference attribute identifier (which may not yet exist). The second argument is then evaluated (and type-converted as needed) and appended to the call-by-reference atttribute. The final size of the attribute is then returned.

push(&temp@range,12.0); // temp@range=-10.0,12.0

push(&number@squares,25ull); // numbers@squares=1,4,9,16,25  

Now some text examples.
Remember, an atttribute identifier that begins with @ implies a global attribute. For example, ’@institution’ is short for ’global@institution’.

global@greetings=push("hello"," world !!");
// Append an NC_STRING
// Pushing an NC_CHAR to a NC_STRING attribute is allowed, it is converted to an an NC_CHAR

// Pushing a single NC_STRING to an NC_CHAR is not allowed
push(&@h," again"s); // BAD PUSH

If the attribute name contains non-regular characters use ID quoting:


See see ID Quoting.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.7 Value List

A value list is a special type of attribute. It can only be used on the RHS of the assign family of statements.
That is =, +=, -=, *=, /=
A value list CANNOT be involved in any logical, binary, or arithmetical operations (except those above).
A value list CANNOT be used as a function argument.
A value list CANNOT have nested value lists.
The type of a value list is the type of the member with the highest type.

a1@trip+={3,2,1}; // 4,4,4
dlon[lon]={1b,2s,3ull,4.0f}; // final type NC_FLOAT

a1@ind={1,2,3}+{4,4,4}; // BAD
a1@s=sin({1.0,16.0}); // BAD

One can also use a value_list to create an attribute of type NC_STRING. Remember, a literal string of type NC_STRING has a postfix ’s’. A value list of NC_CHAR has no semantic meaning and is plain wrong.

array@numbers={"one"s, "two"s, "four"s, "seven"s}; // GOOD

ar@numbers={"zero","twenty"}; // BAD

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.8 Number literals

The table below lists the postfix character(s) to add to a number literal (aka, a naked constant) for explicit type specification. The same type-specification rules are used for variables and attributes. A floating-point number without a postfix defaults to NC_DOUBLE, while an integer without a postfix defaults to type NC_INT:

var[$rlev]=0.1;     // Variable will be type @code{NC_DOUBLE}
var[$lon_grd]=2.0;  // Variable will be type @code{NC_DOUBLE}
var[$gds_crd]=2e3;  // Variable will be type @code{NC_DOUBLE}
var[$gds_crd]=2.0f; // Variable will be type @code{NC_FLOAT} (note "f")
var[$gds_crd]=2e3f; // Variable will be type @code{NC_FLOAT} (note "f")
var[$gds_crd]=2;    // Variable will be type @code{NC_INT}
var[$gds_crd]=-3;   // Variable will be type @code{NC_INT}
var[$gds_crd]=2s;   // Variable will be type @code{NC_SHORT}
var[$gds_crd]=-3s;  // Variable will be type @code{NC_SHORT}
var@att=41.;        // Attribute will be type @code{NC_DOUBLE}
var@att=41.f;       // Attribute will be type @code{NC_FLOAT}
var@att=41;         // Attribute will be type @code{NC_INT}
var@att=-21s;       // Attribute will be type @code{NC_SHORT}  
var@units="kelvin"; // Attribute will be type @code{NC_CHAR}

There is no postfix for characters, use a quoted string instead for NC_CHAR. ncap2 interprets a standard double-quoted string as a value of type NC_CHAR. In this case, any receiving variable must be dimensioned as an array of NC_CHAR long enough to hold the value.

To use the newer netCDF4 types NCO must be compiled/linked to the netCDF4 library and the output file must be of type NETCDF4:

var[$time]=1UL;    // Variable will be type @code{NC_UINT}
var[$lon]=4b;      // Variable will be type @code{NC_BYTE}
var[$lat]=5ull;    // Variable will be type @code{NC_UINT64}  
var[$lat]=5ll;     // Variable will be type @code{NC_INT64}  
var@att=6.0d;      // Attribute will be type @code{NC_DOUBLE}
var@att=-666L;     // Attribute will be type @code{NC_INT}
var@att="kelvin"s; // Attribute will be type @code{NC_STRING} (note the "s")

Use a post-quote ‘s’ for NC_STRING. Place the letter ‘s’ immediately following the double-quoted string to indicate that the value is of type NC_STRING. In this case, the receiving variable need not have any memory allocated to hold the string because netCDF4 handles that memory allocation.

Suppose one creates a file containing an ensemble of model results, and wishes to label the record coordinate with the name of each model. The NC_STRING type is well-suited to this because it facilitates storing arrays of strings of arbitrary length. This is sophisticated, though easy with ncap2:

% ncecat -O -u model
% ncap2 -4 -O -s 'model[$model]={"cesm"s,"ecmwf"s,"giss"s}'

The key here to place an ‘s’ character after each double-quoted string value to indicate an NC_STRING type. The ‘-4’ ensures the output filetype is netCDF4 in case the input filetype is not.

netCDF3/4 Types

NC_BYTE, a signed 1-byte integer


NC_CHAR, an ISO/ASCII character


NC_SHORT, a signed 2-byte integer


NC_INT, a signed 4-byte integer


NC_FLOAT, a single-precision (4-byte) floating-point number


NC_DOUBLE, a double-precision (8-byte) floating-point number

netCDF4 Types

NC_UBYTE, an unsigned 1-byte integer


NC_USHORT, an unsigned 2-byte integer


NC_UINT, an unsigned 4-byte integer


NC_INT64, a signed 8-byte integer


NC_UINT64, an unsigned 8-byte integer


NC_STRING, a string of arbitrary length

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.9 if statement

The syntax of the if statement is similar to its C counterpart. The Conditional Operator (ternary operator) has also been implemented.

else if(exp2)     

# Can use code blocks as well:
}else if(exp2)     

For a variable or attribute expression to be logically true all its non-missing value elements must be logically true, i.e., non-zero. The expression can be of any type. Unlike C there is no short-circuiting of an expression with the OR (||) and AND (&&) operators. The whole expression is evaluated regardless if one of the AND/OR operands are True/False.

# Simple example
  print("All values of time are greater than zero\n");
else if(time<0)
  print("All values of time are less than zero\n");   
else {
  print("min value of time=");print(time_min,"%f");
  print("max value of time=");print(time_max,"%f");

# Example from ddra.nco
  lmn_nbr=1.0*var_nbr_apx*varsz_gcm_4D; /* [nbr] Variable size */
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_gcm_4D; // Block size
    lmn_nbr_wgt=dmnsz_gcm_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
}else if(fl_typ==fl_typ_stl){
  lmn_nbr=1.0*var_nbr_apx*varsz_stl_2D; /* [nbr] Variable size */
    lmn_nbr_avg=1.0*var_nbr_apx*varsz_stl_2D; // Block size
    lmn_nbr_wgt=dmnsz_stl_lat; /* [nbr] Weight size */
  } // !nco_op_typ_avg
} // !fl_typ

Conditional Operator

// netCDF4 needed for this example
th_nw=(three_dmn_var_sht >= 0 ? three_dmn_var_sht.uint() : \; 

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.10 Print & String methods

The print statement comes in a variety of forms

(A)   print(variable_name, format string?);
(A1)  print(expression/string, format string?);

(B)   sprint(expression/string, format string?);
(B1)  sprint4(expression/string, format string?);


If the variable exists in I/O then it is printed in a similar fashion to ncks -H.


lat[0]=-90 lon[0]=0 byt_2D[0]=0 
lat[0]=-90 lon[1]=90 byt_2D[1]=1 
lat[0]=-90 lon[2]=180 byt_2D[2]=2 
lat[0]=-90 lon[3]=270 byt_2D[3]=3 
lat[1]=90 lon[0]=0 byt_2D[4]=4 
lat[1]=90 lon[1]=90 byt_2D[5]=5 
lat[1]=90 lon[2]=180 byt_2D[6]=6 
lat[1]=90 lon[3]=270 byt_2D[7]=7 

If the first argument is NOT a variable the form (A1) is invoked.

mss_val_fst@_FillValue, size = 1 NC_FLOAT, value = -999

print("This function \t is monotonic\n");
This function is 	  monotonic

att_var@float_att, size = 7 NC_FLOAT, value = 73, 72, 71, 70.01, 69.001, 68.01, 67.01

lon, size = 4 NC_DOUBLE, value = 0, 900, 1800, 2700

If the format string is specified then the results from (A) and (A1) forms are the same


print(lon*10.0, "%g,")

print(att_var@float_att,"%g," )

sprint() & sprint4()

These functions work in an identical fashion to (A1) except that sprint() outputs a regular netCDF3 NC_CHAR attribute and sprint4() outputs a netCDF4 NC_STRING attribute

time@units=sprint(nDays, "%d days since 1970-1-1") 
bnd@num=sprint4(bnd_idx,"Band number=%d") 

time@arr=sprint4(time,"%.2f,")  // "1.00,2.00,3.00,4.00,5.00,6.00,7.00,8.00,9.00,10.00,"

You can also use sprint4() to convert a NC_CHAR string to a NC_STRING string and sprint() to convert a NC_STRING to a NC_CHAR

lat_1D_rct@long_name = "Latitude for 2D rectangular grid stored as 1D arrays"; // 

// convert to NC_STRING
lat_1D_rct@long_name = sprint4(lat_1D_rct@long_name) 

hyperslab a netCDF string

Its possible to index-into a NC_CHAR string. Just like a C-String. Remember an NC_CHAR string is has no terminating null. You CANNOT index into a NC_STRING. You have to convert to an NC_CHAR first.

global@greeting="hello world!!!"
@h=@greeting(0:4);  // "hello"
@w=@greeting(6:11); // "world"

// can use negative inidices
@x=@greeting(-3:-1);  // "!!!"

// can  use stride
@n=@greeting(::2);  // "hlowrd!"

// concatenation
global@new_greeting=push(@h, " users !!!"); // "hello users!!!"

@institution="hotel california"s; 
@h=@institution(0:4); // BAD 

// convert NC_STRING to NC_CHAR
@h=@is(0:4);  // "hotel"

// convert NC_CHAR to NC_STRING

get_vars_in() & get_vars_out()


These functions are used to create a list of vars in Input or Output. The optional arg ’att_regexp’. Can be an NC_CHAR att or a NC_STRING att. If NC_CHAR then only a single reg-exp can be specified. If NC_STRING then multiple reg-exp can be specified. The output is allways an NC_STRING att. The matching works in an identical fashion to the -v switch in ncks. if there is no arg then all vars are returned.

@slist=get_vars_in("^time");  // "time", "time_bnds", "time_lon", "time_udunits"
@slist=get_vars_in(@regExp);  // "lat_bnd", "lat_grd", "lev_bnd", "lon_grd", "time_bnds", "cnv_CF_grd"

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.11 Missing values ncap2

Missing values operate slightly differently in ncap2 Consider the expression where op is any of the following operators (excluding ’=’)

Arithmetic operators ( * / % + - ^ )
Binary Operators     ( >, >= <, <= ==, !=,==,||,&&, >>,<< ) 
Assign Operators     ( +=,-=,/=, *= ) 

var1 'op' var2

If var1 has a missing value then this is the value used in the operation, otherwise the missing value for var2 is used. If during the element-by-element operation an element from either operand is equal to the missing value then the missing value is carried through. In this way missing values ’percolate’ or propagate through an expression.
Missing values associated with Output variables are stored in memory and are written to disk after the script finishes. During script execution its possible (and legal) for the missing value of a variable to take on several different values.

# Consider the variable:
int rec_var_int_mss_val_int(time); =-999,2,3,4,5,6,7,8,-999,-999;
rec_var_int_mss_val_int:_FillValue = -999;

n2=rec_var_int_mss_val_int + rec_var_int_mss_val_int.reverse($time); 


The following methods query or manipulate missing value information associated with a variable. The methods that “manipulate” will only succeed on variables in Output


The numeric argument expr becomes the new missing value, overwriting the old missing value, if any. The argument given is converted if necessary to the variable’s type. NB: This only changes the missing value attribute. Missing values in the original variable remain unchanged, and thus are no long considered missing values. They are effectively “orphaned”. Thus set_miss() is normally used only when creating new variables. The intrinsic function change_miss() (see below) is typically used to edit values of existing variables.


Sets or changes (any pre-existing) missing value attribute and missing data values to expr. NB: This is an expensive function since all values must be examined. Use this function when changing missing values for pre-existing variables.


Returns the missing value of a variable. If the variable exists in Input and Output then the missing value of the variable in Output is returned. If the variable has no missing value then an error is returned.


Delete the missing value associated with a variable.


Count the number of missing values a variable contains.


Returns 1 (True) if the variable has a missing value associated with it. else returns 0 (False)


This function creates a True/False mask array of where the missing value is set. It is syntatically equivalent to ( var_in == var_in.get_miss()). except that cant do this unless you delete the missing value before-hand!!

/* Set values less than 0 or greater than 50 to missing value */
where(th < 0.0 || th > 50.0) th=th.get_miss();

# Another example:

// Extract only elements divisible by 3
where (three_dmn_var_dbl%3 == 0)

// Print missing value and variable summary
print("Number of missing values in three_dmn_var_dbl: ");

// find the total number of missing values along dims $lat and $lon
print(mss_ttl); // 0, 0, 0, 8, 0, 0, 0, 1, 0, 2 ;

This function takes a variable and attempts to fill missing values using an average of up to the 4 nearest neighbour grid points. The method used is iterative (up to 1000 cycles). For very large areas of missing values results can be unpredictable. The given variable must be at least 2D; and the algorithm assumes that the last two dims are lat/lon or y/x


Weighted_fill_miss is more sophisticated. Up to 8 nearest neighbours are used to calculate a weighted average. The weighting used is the inverse square of distance. Again the method is iterative (up to 1000 cycles). The area filled is defined by the final two dims of the variable. In addition this function assumes the existance of coordinate vars the same name as the last two dims. if it doesn’t find these dims it will gently exit with warning.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.12 Methods and functions

The convention within this document is that methods can be used as functions. However, functions are not and cannot be used as methods. Methods can be daisy-chained d and their syntax is cleaner than functions. Method names are reserved words and CANNOT be used as variable names. The command ncap2 -f shows the complete list of methods available on your build.

n2=sin(theta)^2 + cos(theta)^2 
n2=theta.sin().pow(2) + theta.cos()^2

This statement chains together methods to convert three_dmn_var_sht to type double, average it, then convert this back to type short:


Aggregate Methods
These methods mirror the averaging types available in ncwa. The arguments to the methods are the dimensions to average over. Specifying no dimensions is equivalent to specifying all dimensions i.e., averaging over all dimensions. A masking variable and a weighting variable can be manually created and applied as needed.


Mean value


Square of the mean


Mean of sum of squares


Maximum value


Minimum value


Maximum absolute value


Mean absolute value


Minimum absolute value


Root-mean-square (normalize by N)


Root-mean square (normalize by N-1)

tabs() or ttlabs()

Sum of absolute values

ttl() or total() or sum()

Sum of values

// Average a variable over time

Packing Methods
For more information see see Packed data and see ncpdq netCDF Permute Dimensions Quickly

pack() & pack_short()

The default packing algorithm is applied and variable is packed to NC_SHORT


Variable is packed to NC_BYTE


Variable is packed to NC_SHORT


Variable is packed to NC_INT


The standard unpacking algorithm is applied.

NCO automatically unpacks packed data before arithmetically modifying it. After modification NCO stores the unpacked data. To store it as packed data again, repack it with, e.g., the pack() function. To ensure that temperature is packed in the output file, regardless of whether it is packed in the input file, one uses, e.g.,

ncap2 -s 'temperature=pack(temperature-273.15)'

Basic Methods
These methods work with variables and attributes. They have no arguments


Total number of elements


Number of dimensions in variable


Returns the netcdf type (see previous section)


Return 1 (true) if var or att is present in I/O else return 0 (false)


Returns an NC_STRING attribute of all the dim names of a variable

Utility Methods
These functions are used to manipulate missing values and RAM variables. see Missing values ncap2


Takes one argument the missing value. Sets or overwrites the existing missing value. The argument given is converted if necessary to the variable type


Changes the missing value elements of the variable to the new missing value (n.b. an expensive function).


Returns the missing value of a variable in Input or Output


Deletes the missing value associated with a variable.


Returns 1 (True) if the variable has a missing else returns 0 (False)


Returns the number of missing values a variable contains


Writes a RAM variable to disk i.e., converts it to a regular disk type variable


Deletes a RAM variable or an attribute

PDQ Methods
See see ncpdq netCDF Permute Dimensions Quickly

reverse(dim args)

Reverse the dimension ordering of elements in a variable.

permute(dim args)

Re-shape variables by re-ordering the dimensions. All the dimensions of the variable must be specified in the arguments. A limitation of this permute (unlike ncpdq) is that the record dimension cannot be re-assigned.

// Swap dimensions about and reorder along lon


Type Conversion Methods and Functions
These methods allow ncap2 to convert variables and attributes to the different netCDF types. For more details on automatic and manual type conversion see (see Type Conversion). netCDF4 types are only available if you have compiled/links NCO with the netCDF4 library and the Output file is HDF5.

netCDF3/4 Types

convert to NC_BYTE, a signed 1-byte integer


convert to NC_CHAR, an ISO/ASCII character


convert to NC_SHORT, a signed 2-byte integer


convert to NC_INT, a signed 4-byte integer


convert to NC_FLOAT, a single-precision (4-byte) floating-point number


convert to NC_DOUBLE, a double-precision (8-byte) floating-point number

netCDF4 Types

convert to NC_UBYTE, an unsigned 1-byte integer


convert to NC_USHORT, an unsigned 2-byte integer


convert to NC_UINT, an unsigned 4-byte integer


convert to NC_INT64, a signed 8-byte integer


convert to NC_UINT64, an unsigned 8-byte integer

You can also use the convert() method to do type conversion. This takes an integer agument. For convenience, ncap2 defines the netCDF pre-processor tokens as RAM variables. For example you may wish to convert a non-floating point variable to the same type as another variable.

if(time.type() != NC_DOUBLE && time.type() != NC_FLOAT) 

Intrinsic Mathematical Methods
The list of mathematical methods is system dependant. For the full list see Intrinsic mathematical methods

All the mathematical methods take a single argument except atan2() and pow() which take two. If the operand type is less than float then the result will be of type float. Arguments of type double yield results of type double. Like the other methods, you are free to use the mathematical methods as functions.

n1=pow(2,3.0f)    // n1 type float
n2=atan2(2,3.0)   // n2 type double
n3=1/(three_dmn_var_dbl.cos().pow(2))-tan(three_dmn_var_dbl)^2; // n3 type double

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.13 RAM variables

Unlike regular variables, RAM variables are never written to disk. Hence using RAM variables in place of regular variables (especially within loops) significantly increases execution speed. Variables that are frequently accessed within for or where clauses provide the greatest opportunities for optimization. To declare and define a RAM variable simply prefix the variable name with an asterisk (*) when the variable is declared/initialized. To delete RAM variables (and recover their memory) use the ram_delete() method. To write a RAM variable to disk (like a regular variable) use ram_write().

*temp[$time,$lat,$lon]=10.0;     // Cast
*temp_avg=temp.avg($time);      // Regular assign
temp.ram_delete();              // Delete RAM variable
temp_avg.ram_write();           // Write Variable to output

// Create and increment a RAM variable from "one" in Input
// Create RAM variables from the variables three and four in Input.
// Multiply three by 10 and add it to four. 
*four+=*three*=10; // three=30, four=34 

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.14 Where statement

The where() statement combines the definition and application of a mask and can lead to succinct code. The syntax of a where() statement is:

// Single assign ('elsewhere' is optional)

// Multiple assigns

Consider the variables float lon_2D_rct(lat,lon); and float var_msk(lat,lon);. Suppose we wish to multiply by two the elements for which var_msk equals 1:

where(var_msk == 1) lon_2D_rct=2*lon_2D_rct;

Suppose that we have the variable int RDM(time) and that we want to set its values less than 8 or greater than 80 to 0:

where(RDM < 8 || RDM > 80) RDM=0;          

To use where on a variable subset, define and use a temporary variable, e.g.,

where (var1 < 0.5) var_tmp=1234; 

Consider irregularly gridded data, described using rank 2 coordinates: double lat(south_north,east_west), double lon(south_north,east_west), double temperature(south_north,east_west). This type of structure is often found in regional weather/climate model (such as WRF) output, and in satellite swath data. For this reason we call it “Swath-like Data”, or SLD. To find the average temperature in a region bounded by [lat_min,lat_max] and [lon_min,lon_max]:

where((lat >= lat_min && lat <= lat_max) && (lon >= lon_min && lon <= lon_max))


For North American Regional Reanalysis (NARR) data (example dataset) the procedure looks like this

ncap2 -O -v -S ~/narr.nco ${DATA}/hdf/ ~/

where narr.nco is an ncap2 script like this:

/* North American Regional Reanalysis (NARR) Statistics
   NARR stores grids with 2-D latitude and longitude, aka Swath-like Data (SLD) 
   Here we work with three variables:
   lat(y,x), lon(y,x), and uwnd(time,level,y,x);
   To study sub-regions of SLD, we use masking techniques:
   1. Define mask as zero times variable to be masked
      Then mask automatically inherits variable attributes
      And average below will inherit mask attributes
   2. Optionally, create mask as RAM variable (as below with asterisk *)
      NCO does not write RAM variable to output
      Masks are often unwanted, and can be big, so this speeds execution
   3. Example could be extended to preserve mean lat and lon of sub-region
      Follow uwnd example to do this: lat_msk=0.0*lat ... lat_avg=lat.avg($y,$x) */
where((lat >= 35.6 && lat <= 37.0) && (lon >= -100.5 && lon <= -99.0))

// Average only over horizontal dimensions x and y (preserve level and time)

Stripped of comments and formatting, this example is a three-statement script executed by a one-line command. NCO needs only this meagre input to unpack and copy the input data and attributes, compute the statistics, and then define and write the output file. Unless the comments pointed out that wind variable (uwnd) was four-dimensional and the latitude/longitude grid variables were both two-dimensional, there would be no way to tell. This shows how NCO hides from the user the complexity of analyzing multi-dimensional SLD. We plan to extend such SLD features to more operators soon.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.15 Loops

ncap2 supplies for() loops and while() loops. They are completely unoptimized so use them only with RAM variables unless you want thrash your disk to death. To break out of a loop use the break command. To iterate to the next cycle use the continue command.

// Set elements in variable double temp(time,lat) 
// If element < 0 set to 0, if element > 100 set to 100

    if(temp(idx,jdx) > 100) temp(idx,jdx)=100.0; 
      else if(temp(idx,jdx) < 0) temp(idx,jdx)=0.0;

// Are values of co-ordinate variable double lat(lat) monotonic?

  if(lat(idx)-lat(idx-1) < 0.0) break;

if(idx == sz) print("lat co-ordinate is monotonic\n");
   else print("lat co-ordinate is NOT monotonic\n");

// Sum odd elements	

  if(lat(idx)%2) sum+=lat(idx);

print("Total of odd elements ");print(sum);print("\n"); 

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.16 Include files

The syntax of an include-file is:

#include "script.nco"
#include "/opt/SOURCES/nco/data/tst.nco"

If the filename is relative and not absolute then the directory searched is relative to the run-time directory. It is possible to nest include files to an arbitrary depth. A handy use of inlcude files is to store often used constants. Use RAM variables if you do not want these constants written to nc-file.


// script.nco
// Sample file to #include in ncap2 script
*pi=3.1415926535; // RAM variable, not written to output
*h=6.62607095e-34; // RAM variable, not written to output
e=2.71828; // Regular (disk) variable, written to output

As of NCO version 4.6.3 (December, 2016), The user can specify the directory(s) to be searched by specifing them in the UNIX environment var NCO_PATH. The format used is identical to the UNIX PATH. The directory(s) are only searched if the include filename is relative.

export NCO_PATH=":/home/henryb/bin/:/usr/local/scripts:/opt/SOURCES/nco/data:"

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.17 sort methods

In ncap2 there are multiple ways to sort data. Beginning with NCO 4.1.0 (March, 2012), ncap2 support six sorting functions:

var_out=sort(var_in,&srt_map); // Ascending sort
var_out=asort(var_in,&srt_map); // Accending sort 
var_out=dsort(var_in,&srt_map); // Desending sort     
var_out=remap(var_in,srt_map); // Apply srt_map to var_in
var_out=unmap(var_in,srt_map); // Reverse what srt_map did to var_in
dsr_map=invert_map(srt_map); // Produce "de-sort" map that inverts srt_map

The first two functions, sort() and asort() sort, in ascending order, all the elements of var_in (which can be a variable or attribute) without regard to any dimensions. The third function, dsort() does the same but sorts in descending order. Remember that ascending and descending sorts are specified by asort() and dsort(), respectively.

These three functions are overloaded to take a second, optional argument called the sort map srt_map, which should be supplied as a call-by-reference variable, i.e., preceded with an ampersand. If the sort map does not yet exist, then it will be created and returned as an integer type the same shape as the input variable.

The output var_out of each sort function is a sorted version of the input, var_in. The output var_out of the two mapping functions the result of applying (with remap() or un-applying (with unmap()) the sort map srt_map to the input var_in. To apply the sort map with remap() the size of the variable must be exactly divisible by the size of the sort map.

The final function invert_map() returns the so-called de-sorting map dsr_map which is inverse map of the input map srt_map. This gives the user access to both the forward and inverse sorting maps which can be useful in special situations.

// 1, 2, 3, 3, 4, 4, 5, 6, 7, 10;

// 1, 2, 3, 4
// 1, 0, 3, 2;

If the map variable does not exist prior to the sort() call, then it will be created with the same shape as the input variable and be of type NC_INT. If the map variable already exists, then the only restriction is that it be of at least the same size as the input variable. To apply a map use remap(var_in,srt_map).


 1, 2, 3, 4, 
 5, 6, 7, 8,

// 0, 2, 1, 3;

// 1, 3, 2, 4,
// 5, 7, 6, 8,
// 9,11,10,12,
// 13,15,14,16,
// 17,19,18,20;


// 3, 5, 4, 2, 1
// 8, 10, 9,7, 6, 
// 13,15,14,12,11, 
// 18,20,19,17,16

As in the above example you may create your own sort map. To sort in descending order, apply the reverse() method after the sort().

Here is an extended example of how to use ncap2 features to hyperslab an irregular region based on the values of a variable not a coordinate. The distinction is crucial: hyperslabbing based on dimensional indices or coordinate values is straightforward. Using the values of single or multi-dimensional variable to define a hyperslab is quite different.

cat > ~/ncap2_foo.nco << 'EOF'
// Purpose: Save irregular 1-D regions based on variable values

// Included in NCO User Guide at

/* NB: Single quotes around EOF above turn off shell parameter 
    expansion in "here documents". This in turn prevents the
    need for protecting dollarsign characters in NCO scripts with
    backslashes when the script is cut-and-pasted (aka "moused") 
    from an editor or e-mail into a shell console window */

/* Copy coordinates and variable(s) of interest into RAM variable(s)
   1. ncap2 defines writes all variables on LHS of expression to disk
      Only exception is RAM variables, which are stored in RAM only
      Repeated operations on regular variables takes more time, 
      because changes are written to disk copy after every change.
      RAM variables are only changed in RAM so script works faster
      RAM variables can be written to disk at end with ram_write()
   2. Script permutes variables of interest during processing
      Safer to work with copies that have different names
      This discourages accidental, mistaken use of permuted versions
   3. Makes this script a more generic template:
      var_in instead of specific variable names everywhere */
*dmn_in_sz=$time.size; // [nbr] Size of input arrays

/* Create all other "intermediate" variables as RAM variables 
   to prevent them from cluttering the output file.
   Mask flag and sort map are same size as variable of interest */

/* In this example we mask for all values evenly divisible by 3
   This is the key, problem-specific portion of the template
   Replace this where() condition by that for your problem
   Mask variable is Boolean: 1=Meets condition, 0=Fails condition */
where(var_in % 3 == 0) msk_flg=1; elsewhere msk_flg=0;

// print("msk_flg = ");print(msk_flg); // For debugging...

/* The sort() routine is overloaded, and takes one or two arguments
   The second argument (optional) is the "sort map" (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input
   msk_flg is not needed in its original order after sort()
   Hence we use msk_flg as both input to and output from sort()
   Doing this prevents the need to define a new, unneeded variable */

// Count number of valid points in mask by summing the one's

// Define output dimension equal in size to number of valid points

/* Now sort the variable of interest using the sort map and remap()
   The output, on the LHS, is the input re-arranged so that all points
   meeting the mask condition are contiguous at the end of the array
   Use same srt_map to hyperslab multiple variables of the same shape
   Remember to apply srt_map to the coordinate variables */

/* Hyperslab last msk_nbr values of variable(s) of interest */

/* NB: Even though we created all variables possible as RAM variables,
   the original coordinate of interest, time, is written to the ouput.
   I'm not exactly sure why. For now, delete it from the output with: 
   ncks -O -x -v time ~/ ~/
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/ ~/
ncks -O -x -v time ~/ ~/
ncks ~/

Here is an extended example of how to use ncap2 features to sort multi-dimensional arrays based on the coordinate values along a single dimension.

cat > ~/ncap2_foo.nco << 'EOF'
/* Purpose: Sort multi-dimensional array based on coordinate values
   This example sorts the variable three_dmn_rec_var(time,lat,lon)
   based on the values of the time coordinate. */

// Included in NCO User Guide at

// Randomize the time coordinate
//print("original randomized time =\n");print(time);

/* The sort() routine is overloaded, and takes one or two arguments
   The first argument is a one dimensional array
   The second argument (optional) is the "sort map" (srt_map below)
   Pass the sort map by reference, i.e., prefix with an ampersand
   If the sort map does not yet exist, then it will be created and 
   returned as an integer type the same shape as the input variable.
   The output of sort(), on the LHS, is a sorted version of the input */

//print("sorted time (ascending order) and associated sort map =\n");print(time);print(srt_map);

/* sort() always sorts in ascending order
   The associated sort map therefore re-arranges the original,
   randomized time array into ascending order.
   There are two methods to obtain the descending order the user wants
   1) We could solve the problem in ascending order (the default)
   and then apply the reverse() method to re-arrange the results.
   2) We could change the sort map to return things in descending
   order of time and solve the problem directly in descending order. */

// Following shows how to do method one:

/* Expand the sort map to srt_map_3d, the size of the data array
   1. Use data array to provide right shape for the expanded sort map
   2. Coerce data array into an integer so srt_map_3d is an integer
   3. Multiply data array by zero so 3-d map elements are all zero
   4. Add the 1-d sort map to the 3-d sort map (NCO automatically resizes)
   5. Add the spatial (lat,lon) offsets to each time index 
   6. de-sort using the srt_map_3d
   7. Use reverse to obtain descending in time order
   Loops could accomplish the same thing (exercise left for reader)
   However, loops are slow for large datasets */

/* Following index manipulation requires understanding correspondence
   between 1-d (unrolled, memory order of storage) and access into that
   memory as a multidimensional (3-d, in this case) rectangular array.
   Key idea to understand is how dimensionality affects offsets */ 
// Copy 1-d sort map into 3-d sort map
// Multiply base offset by factorial of lesser dimensions

print("sort map 3d =\n");print(srt_map_3d);

// Use remap() to re-map the data

// Finally, reverse data so time coordinate is descending
//print("sorted time (descending order) =\n");print(time);

// Method two: Key difference is srt_map=$time.size-srt_map-1;
ncap2 -O -v -S ~/ncap2_foo.nco ~/nco/data/ ~/

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.18 UDUnits script

As of NCO version 4.6.3 (December, 2016), ncap2 includes support for UDUnits conversions. The function is called udunits. Its syntax is

varOut=udunits(varIn, "UnitsOutString")

The udunits() function looks for the attribute of varIn@units and fails if it is not found. A quirk of this function that due to attribute propagation varOut@units will be overwritten by varIn@units. It is best to re-initialize this attribute AFTER the call. In addition if varIn@units is of the form "time_interval since basetime" then the calendar attribute varIn@calendar will read it. If it does not exist then the calendar used defaults to mixed Gregorian/Julian as defined by UDUnits.

If varIn is not a floating point type then it is promoted to NC_DOUBLE for the system call in the Udunits library; and then demoted back to to its original type after.

// Overwrite variable
// 273.15, 373.15, 423.15, 473.15 ;

// Rebase coordinate days to hours 
// 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ;
timeOld@units="days since 2012-01-30";

@units="hours since 2012-02-01 01:00";
timeNew=udunits(timeOld, @units);
// -25, -1, 23, 47, 71, 95, 119, 143, 167, 191 ;

// nb in this calendar NO Leap year
tOld@units="minutes since 2012-02-28 23:58:00.00";

@units="seconds since 2012-03-01 00:00";
tNew=udunits(tOld, @units);
// -60, 0, 60, 120, 180, 240, 300, 360, 420, 480 


The var_str=strtime(var_time, fmt_sng ) method take time based variable and a format string and returns a NC_STRING variable ( of the same shape as var_time) of time-stamps in the form specified by ’fmt_sng’. In order to run this command output type must be NetCDF-4

ncap2 -4  -v -O -s 'time_str=strftime(time,"%Y-%m-%d");'

time_str="1964-03-13", "1964-03-14", "1964-03-15", "1964-03-16", 
         "1964-03-17", "1964-03-18", "1964-03-19", "1964-03-20", 
         "1964-03-21", "1964-03-22" ;

Under the hood there are a few steps invoved.
First the method reads ’var_time@units’ and ’var_time@calendar’ (if present) then converts var-time to ’seconds since 1970-01-01’.
It then converts these possibly UTC seconds to the standard struture struct *tm.
Finally strftime is called with fmt_sng and the *tm struct.
The c-standard strftime is used as defined in ’time.h’
If the method is called without fmt_sng then the following default is used: "%Y-%m-%d %H:%M:%S"

ncap2 -4  -v -O -s 'time_str=strftime(time);'

time_str = "1964-03-13 21:09:00", "1964-03-14 21:09:00", "1964-03-15 21:09:00", 
           "1964-03-16 21:09:00", "1964-03-17 21:09:00", "1964-03-18 21:09:00", 
           "1964-03-19 21:09:00", "1964-03-20 21:09:00", "1964-03-21 21:09:00", 
           "1964-03-22 21:09:00" ;

Another working example

ncap2 -v -O -s 'ts=strftime(frametime(0),"%Y-%m-%d/");'

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.19 Vpointer

A variable-pointer or vpointer is a pointer to a variable or attribute. It is most usefull when one needs to apply a set of operations on a list of variables. For example After regular processing one may wish to set the _FillVlaue of all NC_FLOAT variables to a particular value. Or say after processing, create min/max attributes for all 3D variables of type NC_DOUBLE. A vpointer is not a ’pointer’ to a memory location in the normal C/C++ sense. Rather the vpointer is a text attribute that contains the name of a variable. To use the pointer simply prefix the pointer with ’*’. Most places where you use VAR_ID you can use *vpointer_nm. There is a variety of ways to maintain a list of strings in ncap2. The easiest method is to use a NC_STRING attribute.

Below is a simple illustration using NC_CHAR as the vpointer.
Remember an attribute starting with @ implies ’global’. eg ’@vpx’ is short for ’global@vpx’



// Increment idx by one

// multiply by 5
*@vpx*=5;  // idx now 50

// Add 200 (long method)
*@vpx=*@vpx+200;  //idx now 250


// Add together idx and idy
idz=*@vpx+*@vpy;  // idz == 270

// we can also reference variables in  the input file
// can use an existing att pointer as atts are not written
// to the netcdf file until after script has finished

// we can convert the above var to type NC_DOUBLE and 
// write  it to ouptut all in one go 

The following script writes to Output all vars that are of type NC_DOUBLE and and have at least 2 dimesions. It then changes their _FillValue to ’1.0 E-9’. The function get_vars_in() creates an NC_STRING attribue that contains all the var names in Input. It is important to note that that a vpointer must be a plain attribute and NOT an a attribute expression. So in the below script using *all(idx) would be a fundamental mistake. In the above example the vpointer var_nm is of type NC_STRING.



  // @var_nm is of type NC_STRING
  if(*@var_nm.type() == NC_DOUBLE && *@var_nm.ndims() >= 2){

The following script writes to Output all 3D/4D vars as type NC_FLOAT. Then for each var it calculates a range att that contains min & max; and a total att that is the sum of all elements. Note that in this example vpointer are used to ’point’ to attributes


  if(*@var_nm.ndims() >= 3){
    // The push function also takes a call-by-ref att -if it  doesnt already exist then it is created
    // the call below is pushing a NC_STRING to an att so the end result is a list of NC_STRINGS   



  // we can work with att pointers as well 
  // sprint - ouptut is of type NC_CHAR

  // if you are still confused then print out the atts 
  *@att_total= *;
  *@att_range={ min(*@var_nm), max(*@var_nm)}; 

This is an ncdump of one of the variables that has been processed by the above script

float three_dmn_var_int(time, lat, lon) ;
three_dmn_var_int:_FillValue = -99.f ;
three_dmn_var_int:long_name = "three dimensional record variable of type int" ;
three_dmn_var_int:range = 1.f, 80.f ;
three_dmn_var_int:total = 2701.f ;
three_dmn_var_int:units = "watt meter-2" ;

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.20 Irregular Grids

NCO is capable of analyzing datasets for many different underlying coordinate grid types. netCDF was developed for and initially used with grids comprised of orthogonal dimensions forming a rectangular coordinate system. We call such grids standard grids. It is increasingly common for datasets to use metadata to describe much more complex grids. Let us first define three important coordinate grid properties: rectangularity, regularity, and fxm.

Grids are regular if the spacing between adjacent is constant. For example, a 4-by-5 degree latitude-longitude grid is regular because the spacings between adjacent latitudes (4 degrees) are constant as are the (5 degrees) spacings between adjacent longitudes. Spacing in irregular grids depends on the location along the coordinate. Grids such as Gaussian grids have uneven spacing in latitude (points cluster near the equator) and so are irregular.

Grids are rectangular if the number of elements in any dimension is not a function of any other dimension. For example, a T42 Gaussian latitude-longitude grid is rectangular because there are the same number of longitudes (128) for each of the (64) latitudes. Grids are non-rectangular if the elements in any dimension depend on another dimension. Non-rectangular grids present many special challenges to analysis software like NCO.

Wrapped coordinates (see Wrapped Coordinates), such as longitude, are independent of these grid properties (regularity, rectangularity).

The preferred NCO technique to analyze data on non-standard coordinate grids is to create a region mask with ncap2, and then to use the mask within ncap2 for variable-specific processing, and/or with other operators (e.g., ncwa, ncdiff) for entire file processing.

Before describing the construction of masks, let us review how irregularly gridded geoscience data are described. Say that latitude and longitude are stored as R-dimensional arrays and the product of the dimension sizes is the total number of elements N in the other variables. Geoscience applications tend to use R=1, R=2, and R=3.

If the grid is has no simple representation (e.g., discontinuous) then it makes sense to store all coordinates as 1D arrays with the same size as the number of grid points. These gridpoints can be completely independent of all the other (own weight, area, etc.).

R=1: lat(number_of_gridpoints) and lon(number_of_gridpoints)

If the horizontal grid is time-invariant then R=2 is common:

R=2: lat(south_north,east_west) and lon(south_north,east_west)

The Weather and Research Forecast (WRF) model uses R=3:

R=3: lat(time,south_north,east_west), lon(time,south_north,east_west)

and so supports grids that change with time.

Grids with R > 1 often use missing values to indicated empty points. For example, so-called “staggered grids” will use fewer east_west points near the poles and more near the equator. netCDF only accepts rectangular arrays so space must be allocated for the maximum number of east_west points at all latitudes. Then the application writes missing values into the unused points near the poles.

We demonstrate the ncap2 analysis technique for irregular regions by constructing a mask for an R=2 grid. We wish to find, say, the mean temperature within [lat_min,lat_max] and [lon_min,lon_max]:

ncap2 -s 'mask_var= (lat >= lat_min && lat <= lat_max) && \
                    (lon >= lon_min && lon <= lon_max);'

Arbitrarily shaped regions can be defined by more complex conditional statements. Once defined, masks can be applied to specific variables, and to entire files:

ncap2 -s 'temperature_avg=(temperature*mask_var).avg()'
ncwa -a lat,lon -m mask_var -w area

Crafting such commands on the command line is possible though unwieldy. In such cases, a script is often cleaner and allows you to document the procedure:

cat > << 'EOF'
mask_var = (lat >= lat_min && lat <= lat_max) && (lon >= lon_min && > lon <= lon_max);
if( > 0){ // Check that mask contains some valid values
  temperature_avg=(temperature*mask_var).avg(); // Average temperature
  temperature_max=(temperature*mask_var).max(); // Maximum temperature
ncap2 -S

Grids like those produced by the WRF model are complex because one must use global metadata to determine the grid staggering and offsets to translate XLAT and XLONG into real latitudes, longitudes, and missing points. The WRF grid documentation should describe this. For WRF files creating regional masks looks, in general, like

mask_var = (XLAT >= lat_min && XLAT <= lat_max) && (XLONG >= lon_min && XLONG <= lon_max);

A few notes: Irregular regions are the union of arrays lat/lon_min/max’s. The mask procedure is identical for all R.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.21 Bilinear interpolation

As of version 4.0.0 NCO has internal routines to perform bilinear interpolation on gridded data sets. In mathematics, bilinear interpolation is an extension of linear interpolation for interpolating functions of two variables on a regular grid. The idea is to perform linear interpolation first in one direction, and then again in the other direction.

Suppose we have an irregular grid of data temperature[lat,lon], with co-ordinate vars lat[lat], lon[lon]. We wish to find the temperature at an arbitary point [X,Y] within the grid. If we can locate lat_min,lat_max and lon_min,lon_max such that lat_min <= X <= lat_max and lon_min <= Y <= lon_max then we can interpolate in two dimensions the temperature at [X,Y].

The general form of the ncap2 interpolation function is




Input function data. Usually a two dimensional variable. It must be of size grid_in_x.size()*grid_in_y.size()


This variable is the shape of var_out. Usually a two dimensional variable. It must be of size grid_out_x.size()*grid_out_y.size()


X output values


Y output values


X input values values. Must be monotonic (increasing or decreasing).


Y input values values. Must be monotonic (increasing or decreasing).

Prior to calculations all arguments are converted to type NC_DOUBLE. After calculations var_out is converted to the input type of grid_in.

Suppose the first part of an ncap2 script is


// Temperature
 {100, 200, 300, 400, 500,
  101, 202, 303, 404, 505,
  102, 204, 306, 408, 510,
  103, 206, 309, 412, 515.0 };

// Coordinate variables

Now we interpolate with the following variables:


// 110, 200, 300, 400,
// 110.022, 200.04, 300.06, 400.08,
// 113.3, 206, 309, 412 ;

It is possible to interpolate a single point:

// 513.920594059406

Wrapping and Extrapolation
The function bilinear_interp_wrap() takes the same arguments as bilinear_interp() but performs wrapping (Y) and extrapolation (X) for points off the edge of the grid. If the given range of longitude is say (25-335) and we have a point at 20 degrees, then the endpoints of the range are used for the interpolation. This is what wrapping means. For wrapping to occur Y must be longitude and must be in the range (0,360) or (-180,180). There are no restrictions on the longitude (X) values, though typically these are in the range (-90,90). This ncap2 script illustrates both wrapping and extrapolation of end points:


// Coordinate input vars
lon_in[$lon_in]={30, 110, 190, 270, 350.0};

    20,55,60,35,20.0 };

// Coordinate variables


// 13.4375, 49.5, 14.09375,
// 16.25, 54, 16.625,
// 19.25, 58.8, 19.325,
// 20.15, 60.24, 20.135 ;

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.22 GSL special functions

As of version 3.9.6 (released January, 2009), NCO can link to the GNU Scientific Library (GSL). ncap2 can access most GSL special functions including Airy, Bessel, error, gamma, beta, hypergeometric, and Legendre functions and elliptical integrals. GSL must be version 1.4 or later. To list the GSL functions available with your NCO build, use ncap2 -f | grep ^gsl.

The function names used by ncap2 mirror their GSL names. The NCO wrappers for GSL functions automatically call the error-handling version of the GSL function when available 56. This allows NCO to return a missing value when the GSL library encounters a domain error or a floating-point exception. The slow-down due to calling the error-handling version of the GSL numerical functions was found to be negligible (please let us know if you find otherwise).

Consider the gamma function.
The GSL function prototype is
int gsl_sf_gamma_e(const double x, gsl_sf_result * result) The ncap2 script would be:

lon_out= _, 9.5135, 4.5908, 2.9915 

The first value is set to _FillValue since the gamma function is undefined for negative integers. If the input variable has a missing value then this value is used. Otherwise, the default double fill value is used (defined in the netCDF header netcdf.h as NC_FILL_DOUBLE = 9.969e+36).

Consider a call to a Bessel function with GSL prototype
int gsl_sf_bessel_Jn_e(int n, double x, gsl_sf_result * result)

An ncap2 script would be

lon_out=0.11490, 0.0012, 0.00498, 0.011165

This computes the Bessel function of order n=2 for every value in lon_in. The Bessel order argument, an integer, can also be a non-scalar variable, i.e., an array.

lon_out= 0.93846, 0.24226, 0.03060, 0.00256

Arguments to GSL wrapper functions in ncap2 must conform to one another, i.e., they must share the same sub-set of dimensions. For example: three_out=gsl_sf_bessel_Jn(n_in,three_dmn_var_dbl) is valid because the variable three_dmn_var_dbl has a lon dimension, so n_in in can be broadcast to conform to three_dmn_var_dbl. However time_out=gsl_sf_bessel_Jn(n_in,time) is invalid.

Consider the elliptical integral with prototype int gsl_sf_ellint_RD_e(double x, double y, double z, gsl_mode_t mode, gsl_sf_result * result)


The three arguments are all conformable so the above ncap2 call is valid. The mode argument in the function prototype controls the convergence of the algorithm. It also appears in the Airy Function prototypes. It can be set by defining the environment variable GSL_PREC_MODE. If unset it defaults to the value GSL_PREC_DOUBLE. See the GSL manual for more details.


The ncap2 wrappers to the array functions are slightly different. Consider the following GSL prototype
int gsl_sf_bessel_Jn_array(int nmin, int nmax, double x, double *result_array)


This calculates the Bessel function of x=0.5 for n=1 to 4. The first three arguments are scalar values. If a non-scalar variable is supplied as an argument then only the first value is used. The final argument is the variable where the results are stored (NB: the & indicates this is a call by reference). This final argument must be of type double and must be of least size nmax-nmin+1. If either of these conditions is not met then then the function returns an error message. The function/wrapper returns a status flag. Zero indicates success.

Consider another array function
int gsl_sf_legendre_Pl_array(int lmax, double x, double *result_array);

status=gsl_sf_legendre_Pl_array(a1.size()-1, x,&a1);  

This call calculates P_l(0.3) for l=0..9. Note that |x|<=1, otherwise there will be a domain error. See the GSL documentation for more details.

The GSL functions implemented in NCO are listed in the table below. This table is correct for GSL version 1.10. To see what functions are available on your build run the command ncap2 -f |grep ^gsl . To see this table along with the GSL C-function prototypes look at the spreadsheet doc/nco_gsl.ods.


Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.23 GSL interpolation

As of version 3.9.9 (released July, 2009), NCO has wrappers to the GSL interpolation functions.

Given a set of data points (x1,y1)...(xn, yn) the GSL functions computes a continuous interpolating function Y(x) such that Y(xi) = yi. The interpolation is piecewise smooth, and its behavior at the end-points is determined by the type of interpolation used. For more information consult the GSL manual.

Interpolation with ncap2 is a two stage process. In the first stage, a RAM variable is created from the chosen interpolating function and the data set. This RAM variable holds in memory a GSL interpolation object. In the second stage, points along the interpolating function are calculated. If you have a very large data set or are interpolating many sets then consider deleting the RAM variable when it is redundant. Use the command ram_delete(var_nm).

A simple example


// Ram variable is declared and defined here 



print(y_out); // 1.10472, 1.2, 1.4, 1.42658, 1.69680002 
print(y2);    // 1.12454 
print(y3);    // '_' 

Note in the above example y3 is set to ’missing value’ because 0.0 isn’t within the input X range.

GSL Interpolation Types
All the interpolation functions have been implemented. These are:

Evaluation of Interpolating Types

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.24 GSL least-squares fitting

Least Squares fitting is a method of calculating a straight line through a set of experimental data points in the XY plane. The data maybe weighted or unweighted. For more information please refer to the GSL manual.

These GSL functions fall into three categories:
A) Fitting data to Y=c0+c1*X
B) Fitting data (through the origin) Y=c1*X
C) Multi-parameter fitting (not yet implemented)

Section A

status=gsl_fit_linear (data_x,stride_x,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&sumsq)

Input variables: data_x, stride_x, data_y, stride_y, n
From the above variables an X and Y vector both of length ’n’ are derived. If data_x or data_y is less than type double then it is converted to type double. It is up to you to do bounds checking on the input data. For example if stride_x=3 and n=8 then the size of data_x must be at least 24

Output variables: c0, c1, cov00, cov01, cov11,sumsq
The ’&’ prefix indicates that these are call-by-reference variables. If any of the output variables don’t exist prior to the call then they are created on the fly as scalar variables of type double. If they already exist then their existing value is overwritten. If the function call is successful then status=0.

status= gsl_fit_wlinear(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&chisq)

Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n


This function calculates y values along the line Y=c0+c1*X

Section B


Input variables: data_x, stride_x, data_y, stride_y, n
From the above variables an X and Y vector both of length ’n’ are derived. If data_x or data_y is less than type double then it is converted to type double.

Output variables: c1,cov11,sumsq

status= gsl_fit_wmul(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&c1,&cov11,&sumsq)

Similar to the above call except it creates an additional weighting vector from the variables data_w, stride_w, n


This function calculates y values along the line Y=c1*X

The below example shows gsl_fit_linear() in action

print(c0);  // 0.2
print(c1);  // 2.98545454545


yout=gsl_fit_linear_est(xout, c0,c1, cov00,cov01, cov11, sumsq);

print(yout);  // 3.18545454545 ,9.15636363636, ,12.1418181818 ,33.04

The following code does linear regression of sst(time,lat,lon) for each time-step

// Declare variables
c0[$lat, $lon]=0.; // Intercept
c1[$lat, $lon]=0.; // Slope
sdv[$lat, $lon]=0.; // Standard deviation
covxy[$lat, $lon]=0.; // Covariance
for (i=0;i<$lat.size;i++) // Loop over lat
  for (j=0;j<$lon.size;j++) // Loop over lon
      // Linear regression function
      gsl_fit_linear(time,1,sst(:, i, j),1, $time.size, &tc0, &tc1, &cov00, &cov01,&cov11,&sumsq); 
      c0(i,j) = tc0; // Output results
      c1(i,j) = tc1; // Output results
      // Covariance function
      covxy(i,j) = gsl_stats_covariance(time,1,$time.size,double(sst(:,i,j)),1,$time.size); 
      // Standard deviation function
      sdv(i,j) = gsl_stats_sd(sst(:,i,j), 1, $time.size); 

// slope (c1) missing values are set to '0', change to -999. (variable c0 intercept value)
where( c0 == -999 ) c1 = -999;

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.25 GSL statistics

Wrappers for most of the GSL Statistical functions have been implemented. The GSL function names include a type specifier (except for type double functions). To obtain the equivalent NCO name simply remove the type specifier; then depending on the data type the appropriate GSL function is called. The weighed statistical functions e.g., gsl_stats_wvariance() are only defined in GSL for floating-point types; so your data must of type float or double otherwise ncap2 will emit an error message. To view the implemented functions use the shell command ncap2 -f|grep _stats

GSL Functions

short gsl_stats_max (short data[], size_t stride, size_t n);
double gsl_stats_int_mean (int data[], size_t stride, size_t n);
double gsl_stats_short_sd_with_fixed_mean (short data[], size_t stride, size_t n, double mean);
double gsl_stats_wmean (double w[], size_t wstride, double data[], size_t stride, size_t n);
double gsl_stats_quantile_from_sorted_data (double sorted_data[], size_t stride, size_t n, double f) ;

Equivalent ncap2 wrapper functions

short gsl_stats_max (var_data, data_stride, n);
double gsl_stats_mean (var_data, data_stride, n);
double gsl_stats_sd_with_fixed_mean (var_data, data_stride, n, var_mean);
double gsl_stats_wmean (var_weight, weight_stride, var_data, data_stride, n, var_mean);
double gsl_stats_quantile_from_sorted_data (var_sorted_data, data_stride, n, var_f) ;

GSL has no notion of missing values or dimensionality beyond one. If your data has missing values which you want ignored in the calculations then use the ncap2 built in aggregate functions(Methods and functions). The GSL functions operate on a vector of values created from the var_data/stride/n arguments. The ncap wrappers check that there is no bounding error with regard to the size of the data and the final value in the vector.


print(a1_avg); // 5.5

print(a1_var); // 16.0

// bounding error, vector attempts to access element a1(10)

For functions with the signature func_nm(var_data,data_stride,n), one may omit the second or third arguments. The default value for stride is 1. The default value for n is 1+(data.size()-1)/stride.

// Following statements are equvalent

// Following statements are equvalent

// Following statements are NOT equvalent
n4=gsl_stats_kurtosis(a1,3); //default n=4

The following example illustrates some of the weighted functions. The data are randomly generated. In this case the value of the weight for each datum is either 0.0 or 1.0


// Fill with random numbers [0.0,10.0)

// Create a weighting variable



// number of values in data that are greater than 4;

// print min/max of data 

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.26 GSL random number generation

The GSL library has a large number of random number generators. In addition there are a large set of functions for turning uniform random numbers into discrete or continuous probabilty distributions. The random number generator algorithms vary in terms of quality numbers output, speed of execution and maximum number output. For more information see the GSL documentation. The algorithm and seed are set via environment variables, these are picked up by the ncap2 code.

The number algorithm is set by the environment variable GSL_RNG_TYPE. If this variable isn’t set then the default rng algorithm is gsl_rng_19937. The seed is set with the environment variable GSL_RNG_SEED. The following wrapper functions in ncap2 provide information about the chosen algorithm.


the minimum value returned by the rng algorithm.


the maximum value returned by the rng algorithm.

Uniformly Distributed Random Numbers


This function returns var_in with integers from the chosen rng algorithm. The min and max values depend uoon the chosen rng algorthm.


This function returns var_in with random integers from 0 to n-1. The value n must be less than or equal to the maximum value of the chosen rng algorithm.


This function returns var_in with double-precision numbers in the range [0.0,1). The range includes 0.0 and excludes 1.0.


This function returns var_in with double-precision numbers in the range (0.0,1), excluding both 0.0 and 1.0.

Below are examples of gsl_rng_get() and gsl_rng_uniform_int() in action.

export GSL_RNG_TYPE=ranlux
export GSL_RNG_SEED=10
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' 
// 10 random numbers from the range 0 - 16777215
// a2=9056646, 12776696, 1011656, 13354708, 5139066, 1388751, 11163902, 7730127, 15531355, 10387694 ;

ncap2 -v -O -s 'a1[time]=21;a2=gsl_rng_uniform_int(a1).sort();'
// 10 random numbers from the range 0 - 20
a2 = 1, 1, 6, 9, 11, 13, 13, 15, 16, 19 ;

The following example produces an ncap2 runtime error. This is because the chose rng algorithm has a maximum value greater than NC_MAX_INT=2147483647 ; the wrapper functions to gsl_rng_get() and gsl_rng_uniform_int() return variable of type NC_INT. Please be aware of this when using random number distribution functions functions from the GSL library which return unsigned int. Examples of these are gsl_ran_geometric() and gsl_ran_pascal().

export GSL_RNG_TYPE=mt19937
ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' 

To find the maximum value of the chosen rng algorithm use the following code snippet.

ncap2 -v -O -s 'rng_max=gsl_rng_max();print(rng_max)'

Random Number Distributions
The GSL library has a rich set of random number disribution functions. The library also provides cumulative distribution functions and inverse cumulative distribution functions sometimes referred to a quantile functions. To see whats available on your build use the shell command ncap2 -f|grep -e _ran -e _cdf.

The following examples all return variables of type NC_INT

//a2 = 10, 11, 12, 12, 13, 14, 14, 15, 15, 16, 16, 16, 16, 17, 22 ;
//a2 = 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5 ;
//a5 = 37, 40, 40, 42, 43, 45, 46, 49, 52, 58, 60, 62, 62, 65, 67 ;

The following all return variables of type NC_DOUBLE;

// b2_avg = 0.756047976787

// b3_avg = -0.00903446534258;
// b3_rms = 0.81162979889;

// b6_avg=15.0588129413

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.27 Examples ncap2

See the and scripts released with NCO for more complete demonstrations of ncap2 functionality (script available on-line at

Define new attribute new for existing variable one as twice the existing attribute double_att of variable att_var:

ncap2 -s 'one@new=2*att_var@double_att'

Average variables of mixed types (result is of type double):

ncap2 -s 'average=(var_float+var_double+var_int)/3' 

Multiple commands may be given to ncap2 in three ways. First, the commands may be placed in a script which is executed, e.g., tst.nco. Second, the commands may be individually specified with multiple ‘-s’ arguments to the same ncap2 invocation. Third, the commands may be chained into a single ‘-s’ argument to ncap2. Assuming the file tst.nco contains the commands a=3;b=4;c=sqrt(a^2+b^2);, then the following ncap2 invocations produce identical results:

ncap2 -v -S tst.nco
ncap2 -v -s 'a=3' -s 'b=4' -s 'c=sqrt(a^2+b^2)'
ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)'

The second and third examples show that ncap2 does not require that a trailing semi-colon ‘;’ be placed at the end of a ‘-s’ argument, although a trailing semi-colon ‘;’ is always allowed. However, semi-colons are required to separate individual assignment statements chained together as a single ‘-s’ argument.

ncap2 may be used to “grow” dimensions, i.e., to increase dimension sizes without altering existing data. Say has ORO(lat,lon) and the user wishes a new file with new_ORO(new_lat,new_lon) that contains zeros in the undefined portions of the new grid.

defdim("new_lat",$lat.size+1); // Define new dimension sizes
new_ORO[$new_lat,$new_lon]=0.0f; // Initialize to zero
new_ORO(0:$lat.size-1,0:$lon.size-1)=ORO; // Fill valid data

The commands to define new coordinate variables new_lat and new_lon in the output file follow a similar pattern. One would might store these commands in a script grow.nco and then execute the script with

ncap2 -v -S grow.nco

Imagine you wish to create a binary flag based on the value of an array. The flag should have value 1.0 where the array exceeds 1.0, and value 0.0 elsewhere. This example creates the binary flag ORO_flg in from the continuous array named ORO in

ncap2 -s 'ORO_flg=(ORO > 1.0)'

Suppose your task is to change all values of ORO which equal 2.0 to the new value 3.0:

ncap2 -s 'ORO_msk=(ORO==2.0);ORO=ORO_msk*3.0+!ORO_msk*ORO'

This creates and uses ORO_msk to mask the subsequent arithmetic operation. Values of ORO are only changed where ORO_msk is true, i.e., where ORO equals 2.0
Using the where statement the above code simplifies to :

ncap2 -s 'where(ORO==2.0) ORO=3.0;'

This example uses ncap2 to compute the covariance of two variables. Let the variables u and v be the horizontal wind components. The covariance of u and v is defined as the time mean product of the deviations of u and v from their respective time means. Symbolically, the covariance [u'v'] = [uv]-[u][v] where [x] denotes the time-average of x and x' denotes the deviation from the time-mean. The covariance tells us how much of the correlation of two signals arises from the signal fluctuations versus the mean signals. Sometimes this is called the eddy covariance. We will store the covariance in the variable uprmvprm.

ncwa -O -a time -v u,v # Compute time mean of u,v
ncrename -O -v u,uavg -v v,vavg # Rename to avoid conflict
ncks -A -v uavg,vavg # Place time means with originals
ncap2 -O -s 'uprmvprm=u*v-uavg*vavg' # Covariance
ncra -O -v uprmvprm # Time-mean covariance

The mathematically inclined will note that the same covariance would be obtained by replacing the step involving ncap2 with

ncap2 -O -s 'uprmvprm=(u-uavg)*(v-vavg)' # Covariance

As of NCO version 3.1.8 (December, 2006), ncap2 can compute averages, and thus covariances, by itself:

ncap2 -s 'uavg=u.avg($time);vavg=v.avg($time);uprmvprm=u*v-uavg*vavg' \
      -s 'uprmvrpmavg=uprmvprm.avg($time)'

We have not seen a simpler method to script and execute powerful arithmetic than ncap2.

ncap2 utilizes many meta-characters (e.g., ‘$’, ‘?’, ‘;’, ‘()’, ‘[]’) that can confuse the command-line shell if not quoted properly. The issues are the same as those which arise in utilizing extended regular expressions to subset variables (see Subsetting Files). The example above will fail with no quotes and with double quotes. This is because shell globbing tries to interpolate the value of $time from the shell environment unless it is quoted:

ncap2 -s 'uavg=u.avg($time)' # Correct (recommended)
ncap2 -s  uavg=u.avg('$time') # Correct (and dangerous)
ncap2 -s  uavg=u.avg($time) # Wrong ($time = '')
ncap2 -s "uavg=u.avg($time)" # Wrong ($time = '')

Without the single quotes, the shell replaces $time with an empty string. The command ncap2 receives from the shell is uavg=u.avg(). This causes ncap2 to average over all dimensions rather than just the time dimension, and unintended consequence.

We recommend using single quotes to protect ncap2 command-line scripts from the shell, even when such protection is not strictly necessary. Expert users may violate this rule to exploit the ability to use shell variables in ncap2 command-line scripts (see CCSM Example). In such cases it may be necessary to use the shell backslash character ‘\’ to protect the ncap2 meta-character.

A dimension of size one is said to be degenerate. Whether a degenerate record dimension is desirable or not depends on the application. Often a degenerate time dimension is useful, e.g., for concatentating, but it may cause problems with arithmetic. Such is the case in the above example, where the first step employs ncwa rather than ncra for the time-averaging. Of course the numerical results are the same with both operators. The difference is that, unless ‘-b’ is specified, ncwa writes no time dimension to the output file, while ncra defaults to keeping time as a degenerate (size 1) dimension. Appending u and v to the output file would cause ncks to try to expand the degenerate time axis of uavg and vavg to the size of the non-degenerate time dimension in the input file. Thus the append (ncks -A) command would be undefined (and should fail) in this case. Equally important is the ‘-C’ argument (see Subsetting Coordinate Variables) to ncwa to prevent any scalar time variable from being written to the output file. Knowing when to use ncwa -a time rather than the default ncra for time-averaging takes, well, time.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.28 Intrinsic mathematical methods

ncap2 supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:


Absolute value Absolute value of x. Example: abs(-1) = 1


Arc-cosine Arc-cosine of x where x is specified in radians. Example: acos(1.0) = 0.0


Hyperbolic arc-cosine Hyperbolic arc-cosine of x where x is specified in radians. Example: acosh(1.0) = 0.0


Arc-sine Arc-sine of x where x is specified in radians. Example: asin(1.0) = 1.57079632679489661922


Hyperbolic arc-sine Hyperbolic arc-sine of x where x is specified in radians. Example: asinh(1.0) = 0.88137358702


Arc-tangent Arc-tangent of x where x is specified in radians between -pi/2 and pi/2. Example: atan(1.0) = 0.78539816339744830961


Arc-tangent2 Arc-tangent of y/x :Example atan2(1,3) = 0.321689857


Hyperbolic arc-tangent Hyperbolic arc-tangent of x where x is specified in radians between -pi/2 and pi/2. Example: atanh(3.14159265358979323844) = 1.0


Ceil Ceiling of x. Smallest integral value not less than argument. Example: ceil(0.1) = 1.0


Cosine Cosine of x where x is specified in radians. Example: cos(0.0) = 1.0


Hyperbolic cosine Hyperbolic cosine of x where x is specified in radians. Example: cosh(0.0) = 1.0


Error function Error function of x where x is specified between -1 and 1. Example: erf(1.0) = 0.842701


Complementary error function Complementary error function of x where x is specified between -1 and 1. Example: erfc(1.0) = 0.15729920705


Exponential Exponential of x, e^x. Example: exp(1.0) = 2.71828182845904523536


Floor Floor of x. Largest integral value not greater than argument. Example: floor(1.9) = 1


Gamma function Gamma function of x, Gamma(x). The well-known and loved continuous factorial function. Example: gamma(0.5) = sqrt(pi)


Incomplete Gamma function Incomplete Gamma function of parameter a and variable x, gamma_inc_P(a,x). One of the four incomplete gamma functions. Example: gamma_inc_P(1,1) = 1-1/e


Natural Logarithm Natural logarithm of x, ln(x). Example: ln(2.71828182845904523536) = 1.0


Natural Logarithm Exact synonym for ln(x).


Base 10 Logarithm Base 10 logarithm of x, log10(x). Example: log(10.0) = 1.0


Round inexactly Nearest integer to x is returned in floating-point format. No exceptions are raised for inexact conversions. Example: nearbyint(0.1) = 0.0


Power Value of x is raised to the power of y. Exceptions are raised for domain errors. Due to type-limitations in the C language pow function, integer arguments are promoted (see Type Conversion) to type NC_FLOAT before evaluation. Example: pow(2,3) = 8


Round exactly Nearest integer to x is returned in floating-point format. Exceptions are raised for inexact conversions. Example: rint(0.1) = 0


Round Nearest integer to x is returned in floating-point format. Round halfway cases away from zero, regardless of current IEEE rounding direction. Example: round(0.5) = 1.0


Sine Sine of x where x is specified in radians. Example: sin(1.57079632679489661922) = 1.0


Hyperbolic sine Hyperbolic sine of x where x is specified in radians. Example: sinh(1.0) = 1.1752


Square Root Square Root of x, sqrt(x). Example: sqrt(4.0) = 2.0


Tangent Tangent of x where x is specified in radians. Example: tan(0.78539816339744830961) = 1.0


Hyperbolic tangent Hyperbolic tangent of x where x is specified in radians. Example: tanh(1.0) = 0.761594155956


Truncate Nearest integer to x is returned in floating-point format. Round halfway cases toward zero, regardless of current IEEE rounding direction. Example: trunc(0.5) = 0.0

The complete list of mathematical functions supported is platform-specific. Functions mandated by ANSI C are guaranteed to be present and are indicated with an asterisk 57. and are indicated with an asterisk. Use the ‘-f’ (or ‘fnc_tbl’ or ‘prn_fnc_tbl’) switch to print a complete list of functions supported on your platform. 58

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.29 Operator precedence and associativity

This page lists the ncap2 operators in order of precedence (highest to lowest). Their associativity indicates in what order operators of equal precedence in an expression are applied.

++ --Postfix Increment/DecrementRight to Left
()Parentheses (function call)
.Method call
++ --Prefix Increment/DecrementRight to Left
+ -Unary Plus/Minus
!Logical Not
^Power of OperatorRight to Left
* / %Multiply/Divide/ModulusLeft To Right
+ -Addition/SubtractionLeft To Right
>> <<Fortran style array clippingLeft to Right
< <=Less than/Less than or equal toLeft to Right
> >=Greater than/Greater than or equal to
== !=Equal to/Not equal toLeft to Right
&&Logical ANDLeft to Right
||Logical ORLeft to Right
?:Ternary OperatorRight to Left
=AssignmentRight to Left
+= -=Addition/subtraction assignment
*= /=Multiplication/division assignment

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.30 ID Quoting

In this section when I refer to a name I mean a variable name, attribute name or a dimension name The allowed characters in a valid netCDF name vary from release to release. (See end section). If you want to use metacharacters in a name or use a method name as a variable name then the name has to be quoted wherever it occurs.

The default NCO name is specified by the regular expressions:

DGT:     ('0'..'9');
LPH:     ( 'a'..'z' | 'A'..'Z' | '_' );
name:    (LPH)(LPH|DGT)+

The first character of a valid name must be alphabetic or the underscore. Any subsequent characters must be alphanumeric or underscore. ( e.g., a1,_23, hell_is_666 )

The valid characters in a quoted name are specified by the regular expressions:

LPHDGT:  ( 'a'..'z' | 'A'..'Z' | '_' | '0'..'9');
name:    (LPHDGT|'-'|'+'|'.'|'('|')'|':' )+  ;      

Quote a variable:
’avg’ , ’10_+10’,’set_miss’ ’+-90field’ , ’–test’=10.0d

Quote a attribute:
’three@10’, ’set_mss@+10’, ’666@hell’, ’t1@+units’="kelvin"

Quote a dimension:
’$10’, ’$t1–’, ’$–odd’, c1[’$10’,’$t1–’]=23.0d

The following comments are from the netCDF library definitions and detail the naming conventions for each release. netcdf-3.5.1

 * ( [a-zA-Z]|[0-9]|'_'|'-'|'+'|'.'|'|':'|'@'|'('|')' )+
 * Verify that name string is valid CDL syntax, i.e., all characters are
 * alphanumeric, '-', '_', '+', or '.'.
 * Also permit ':', '@', '(', or ')' in names for chemists currently making 
 * use of these characters, but don't document until ncgen and ncdump can 
 * also handle these characters in names.

netcdf-4.0 Final 2008/08/28

 * Verify that a name string is valid syntax.  The allowed name
 * syntax (in RE form) is:
 * ([a-zA-Z_]|{UTF8})([^\x00-\x1F\x7F/]|{UTF8})*
 * where UTF8 represents a multibyte UTF-8 encoding.  Also, no
 * trailing spaces are permitted in names.  This definition
 * must be consistent with the one in ncgen.l.  We do not allow '/'
 * because HDF5 does not permit slashes in names as slash is used as a
 * group separator.  If UTF-8 is supported, then a multi-byte UTF-8
 * character can occur anywhere within an identifier.  We later
 * normalize UTF-8 strings to NFC to facilitate matching and queries.

Next: , Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.31 create_bounds() function

The ncap2 custom function ’create_bounds()’ takes any monotonic 1D coordinate variable with regular or irregular (e.g., Gaussian) spacing and creates a bounds variable.

<bounds_var_out>=create_bounds( <coordinate_var_in>, <dim in>, <string>)

1st Argument

The name of the input coordinate variable.

2nd Argument

The dimension name of the second dimension of the output variable. The size of this dimension should always be 2. If the dimension does not exist create it using defdim().

3rd Argument

The string value of a "bounds" attribute that is created in the input coordinate variable. This must be the variable name to contain the bounds.

Typical usage:


Another common CF convention:


Previous: , Up: ncap2 netCDF Arithmetic Processor   [Contents][Index]

4.1.32 solar_zenith_angle function

<zenith_out>=solar_zenith_angle( <time_in>, <latitude in>)

This function takes two arguments, mean local solar time and latitude. Calculation and output is done with type NC_DOUBLE. The calendar attribute for <time_in> in is NOT read and is assumed to be Gregorian (this is the calendar that UDUnits uses). As part of the calculation <time_in> is converted to days since start of year. For some input units e.g., seconds, this function may produce gobbledygook. The output <zenith_out> is in degrees. For more details of the algorithm used please examine the function solar_geometry() in Note that this routine does not account for the equation of time, and so can be in error by the angular equivalent of up to about fifteen minutes time depending on the day of year.

my_time[time]={10.50, 11.0, 11.50, 12.0, 12.5, 13.0, 13.5, 14.0, 14.50, 15.00};  
my_time@units="hours since 2017-06-21";

// Assume we are at Equator

// 32.05428, 27.61159, 24.55934, 23.45467, 24.55947, 27.61184, 32.05458, 37.39353, 43.29914, 49.55782 ;

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.2 ncatted netCDF Attribute Editor


ncatted [-a att_dsc] [-a …] [-D dbg] [-h] [--hdr_pad nbr]
[-l path] [-O] [-o output-file] [-p path] [-R] [-r]
[--ram_all] [-t] input-file [[output-file]]


ncatted edits attributes in a netCDF file. If you are editing attributes then you are spending too much time in the world of metadata, and ncatted was written to get you back out as quickly and painlessly as possible. ncatted can append, create, delete, modify, and overwrite attributes (all explained below). ncatted allows each editing operation to be applied to every variable in a file. This saves time when changing attribute conventions throughout a file. ncatted is for writing attributes. To read attribute values in plain text, use ncks -m -M, or define something like ncattget as a shell command (see Filters for ncks).

Because repeated use of ncatted can considerably increase the size of the history global attribute (see History Attribute), the ‘-h’ switch is provided to override automatically appending the command to the history global attribute in the output-file.

According to the netCDF User Guide, altering metadata in netCDF files does not incur the penalty of recopying the entire file when the new metadata occupies less space than the old metadata. Thus ncatted may run much faster (at least on netCDF3 files) if judicious use of header padding (see Metadata Optimization) was made when producing the input-file. Similarly, using the ‘--hdr_pad’ option with ncatted helps ensure that future metadata changes to output-file occur as swiftly as possible.

When ncatted is used to change the _FillValue attribute, it changes the associated missing data self-consistently. If the internal floating-point representation of a missing value, e.g., 1.0e36, differs between two machines then netCDF files produced on those machines will have incompatible missing values. This allows ncatted to change the missing values in files from different machines to a single value so that the files may then be concatenated, e.g., by ncrcat, without losing information. See Missing Values, for more information.

To master ncatted one must understand the meaning of the structure that describes the attribute modification, att_dsc specified by the required option ‘-a’ or ‘--attribute’. This option is repeatable and may be used multiple time in a single ncatted invocation to increase the efficiency of altering multiple attributes. Each att_dsc contains five elements. This makes using ncatted somewhat complicated, though powerful. The att_dsc fields are in the following order:

att_dsc = att_nm, var_nm, mode, att_type, att_val


Attribute name. Example: units As of NCO 4.5.1 (July, 2015), ncatted accepts regular expressions (see Subsetting Files) for attribute names (it has “always” accepted regular expressions for variable names). Regular expressions will select all matching attribute names.


Variable name. Example: pressure, '^H2O'. Regular expressions (see Subsetting Files) are accepted and will select all matching variable (and/or group) names. The names global and group have special meaning.


Edit mode abbreviation. Example: a. See below for complete listing of valid values of mode.


Attribute type abbreviation. Example: c. See below for complete listing of valid values of att_type.


Attribute value. Example: pascal.

There should be no empty space between these five consecutive arguments. The description of these arguments follows in their order of appearance.

The value of att_nm is the name of the attribute to edit. The meaning of this should be clear to all ncatted users. Both att_nm and var_nm may be specified as regular expressions. If att_nm is omitted (i.e., left blank) and Delete mode is selected, then all attributes associated with the specified variable will be deleted.

The value of var_nm is the name of the variable containing the attribute (named att_nm) that you want to edit. There are three very important and useful exceptions to this rule. The value of var_nm can also be used to direct ncatted to edit global attributes, or to repeat the editing operation for every group or variable in a file. A value of var_nm of global indicates that att_nm refers to a global (i.e., root-level) attribute, rather than to a particular variable’s attribute. This is the method ncatted supports for editing global attributes. A value of var_nm of group indicates that att_nm refers to all groups, rather than to a particular variable’s or group’s attribute. The operation will proceed to edit group metadata for every group. Finally, if var_nm is left blank, then ncatted attempts to perform the editing operation on every variable in the file. This option may be convenient to use if you decide to change the conventions you use for describing the data. As of NCO 4.6.0 (May, 2016), ncatted accepts the ‘-t’ (or long-option equivalent ‘--typ_mch’ or ‘--type_match’) option. This causes ncatted to perform the editing operation only on variables that are the same type as the specified attribute.

The value of mode is a single character abbreviation (a, c, d, m, n, or o) standing for one of five editing modes:


Append. Append value att_val to current var_nm attribute att_nm value att_val, if any. If var_nm does not already have an existing attribute att_nm, it is created with the value att_val.


Create. Create variable var_nm attribute att_nm with att_val if att_nm does not yet exist. If var_nm already has an attribute att_nm, there is no effect, so the existing attribute is preserved without change.


Delete. Delete current var_nm attribute att_nm. If var_nm does not have an attribute att_nm, there is no effect. If att_nm is omitted (left blank), then all attributes associated with the specified variable are automatically deleted. When Delete mode is selected, the att_type and att_val arguments are superfluous and may be left blank.


Modify. Change value of current var_nm attribute att_nm to value att_val. If var_nm does not have an attribute att_nm, there is no effect.


Nappend. Append value att_val to var_nm attribute att_nm value att_val if att_nm already exists. If var_nm does not have an attribute att_nm, there is no effect. In other words, if att_nm already exist, Nappend behaves like Append otherwise it does nothing. The mnenomic is “non-create append”. Nappend mode was added to ncatted in version 4.6.0 (May, 2016).


Overwrite. Write attribute att_nm with value att_val to variable var_nm, overwriting existing attribute att_nm, if any. This is the default mode.

The value of att_type is a single character abbreviation (f, d, l, i, s, c, b, u) or a short string standing for one of the twelve primitive netCDF data types:


Float. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_FLOAT.


Double. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_DOUBLE.

i, l

Integer or (its now deprecated synonym) Long. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_INT.


Short. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_SHORT.


Char. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_CHAR.


Byte. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_BYTE.


Unsigned Byte. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_UBYTE.


Unsigned Short. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_USHORT.

u, ui, ul

Unsigned Int. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_UINT.

ll, int64

Int64. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_INT64.

ull, uint64

Uint64. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_UINT64.

sng, string

String. Value(s) specified in att_val will be stored as netCDF intrinsic type NC_STRING. Note that ncatted handles type NC_STRING attributes correctly beginning with version 4.3.3 released in July, 2013. Earlier versions fail when asked to handle NC_STRING attributes.

In Delete mode the specification of att_type is optional (and is ignored if supplied).

The value of att_val is what you want to change attribute att_nm to contain. The specification of att_val is optional in Delete (and is ignored) mode. Attribute values for all types besides NC_CHAR must have an attribute length of at least one. Thus att_val may be a single value or one-dimensional array of elements of type att_type. If the att_val is not set or is set to empty space, and the att_type is NC_CHAR, e.g., -a units,T,o,c,"" or -a units,T,o,c,, then the corresponding attribute is set to have zero length. When specifying an array of values, it is safest to enclose att_val in single or double quotes, e.g., -a levels,T,o,s,"1,2,3,4" or -a levels,T,o,s,'1,2,3,4'. The quotes are strictly unnecessary around att_val except when att_val contains characters which would confuse the calling shell, such as spaces, commas, and wildcard characters.

NCO processing of NC_CHAR attributes is a bit like Perl in that it attempts to do what you want by default (but this sometimes causes unexpected results if you want unusual data storage). If the att_type is NC_CHAR then the argument is interpreted as a string and it may contain C-language escape sequences, e.g., \n, which NCO will interpret before writing anything to disk. NCO translates valid escape sequences and stores the appropriate ASCII code instead. Since two byte escape sequences, e.g., \n, represent one-byte ASCII codes, e.g., ASCII 10 (decimal), the stored string attribute is one byte shorter than the input string length for each embedded escape sequence. The most frequently used C-language escape sequences are \n (for linefeed) and \t (for horizontal tab). These sequences in particular allow convenient editing of formatted text attributes. The other valid ASCII codes are \a, \b, \f, \r, \v, and \\. See ncks netCDF Kitchen Sink, for more examples of string formatting (with the ncks-s’ option) with special characters.

Analogous to printf, other special characters are also allowed by ncatted if they are “protected” by a backslash. The characters ", ', ?, and \ may be input to the shell as \", \', \?, and \\. NCO simply strips away the leading backslash from these characters before editing the attribute. No other characters require protection by a backslash. Backslashes which precede any other character (e.g., 3, m, $, |, &, @, %, {, and }) will not be filtered and will be included in the attribute.

Note that the NUL character \0 which terminates C language strings is assumed and need not be explicitly specified. If \0 is input, it is translated to the NUL character. However, this will make the subsequent portion of the string, if any, invisible to C standard library string functions. And that may cause unintended consequences. Because of these context-sensitive rules, one must use ncatted with care in order to store data, rather than text strings, in an attribute of type NC_CHAR.

Note that ncatted interprets character attributes (i.e., attributes of type NC_CHAR) as strings. EXAMPLES

Append the string Data version 2.0.\n to the global attribute history:

ncatted -a history,global,a,c,'Data version 2.0\n' 

Note the use of embedded C language printf()-style escape sequences.

Change the value of the long_name attribute for variable T from whatever it currently is to “temperature”:

ncatted -a long_name,T,o,c,temperature

Many model and observational datasets use missing values that are not annotated in the standard manner. For example, at the time (2015–2016) of this writing, the MPAS ocean and ice models use -9.99999979021476795361e+33 as the missing value, yet do not store a _FillValue attribute with any variables. To prevent arithmetic from treating these values as normal, designate this value as the _FillValue attribute:

ncatted    -a _FillValue,,o,d,-9.99999979021476795361e+33
ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33
ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33 \
           -a _FillValue,,o,f,1.0e36 -a _FillValue,,o,i,-999

The first example adds the attribute to all variables. The ‘-t’ switch causes the second example to add the attribute only to double precision variables. This is often more useful, and can be used to provide distinct missing value attributes to each numeric type, as in the third example.

NCO arithmetic operators may not work as expected on IEEE NaN (short for Not-a-Number) and NaN-like numbers such as positive infinity and negative infinity 59. One way to work-around this problem is to change IEEE NaNs to normal missing values. As of NCO 4.1.0 (March, 2012), ncatted works with NaNs (though none of the arithmetic operators do). This limited support enables users to change NaN to a normal number before performing arithmetic or propagating a NaN-tainted dataset. First set the missing value (i.e., the value of the _FillValue attribute) for the variable(s) in question to the IEEE NaN value.

ncatted -a _FillValue,,o,f,NaN

Then change the missing value from the IEEE NaN value to a normal IEEE number, like 1.0e36 (or to whatever the original missing value was).

ncatted -a _FillValue,,m,f,1.0e36

Some NASA MODIS datasets provide a real-world example.

ncatted -O -a _FillValue,,m,d,1.0e36 -a missing_value,,m,d,1.0e36 \

Delete all existing units attributes:

ncatted -a units,,d,,

The value of var_nm was left blank in order to select all variables in the file. The values of att_type and att_val were left blank because they are superfluous in Delete mode.

Delete all attributes associated with the tpt variable, and delete all global attributes

ncatted -a ,tpt,d,, -a ,global,d,,

The value of att_nm was left blank in order to select all attributes associated with the variable. To delete all global attributes, simply replace tpt with global in the above.

Modify all existing units attributes to meter second-1:

ncatted -a units,,m,c,'meter second-1'

Add a units attribute of kilogram kilogram-1 to all variables whose first three characters are ‘H2O’:

ncatted -a units,'^H2O',c,c,'kilogram kilogram-1'

Overwrite the quanta attribute of variable energy to an array of four integers.

ncatted -a quanta,energy,o,s,'010,101,111,121'

As of NCO 3.9.6 (January, 2009), ncatted accepts extended regular expressions as arguments for variable names, and, since NCO 4.5.1 (July, 2015), for attribute names.

ncatted -a isotope,'^H2O*',c,s,'18'
ncatted -a '.?_iso19115$','^H2O*',d,,

The first example creates isotope attributes for all variables whose names contain ‘H2O’. The second deletes all attributes whose names end in _iso19115 from all variables whose names contain ‘H2O’. See Subsetting Files for more details on using regular expressions.

As of NCO 4.3.8 (November, 2013), ncatted accepts full and partial group paths in names of attributes, variables, dimensions, and groups.

# Overwrite units attribute of specific 'lon' variable
ncatted -O -a units,/g1/lon,o,c,'degrees_west'
# Overwrite units attribute of all 'lon' variables
ncatted -O -a units,lon,o,c,'degrees_west'
# Delete units attribute of all 'lon' variables
ncatted -O -a units,lon,d,,
# Overwrite units attribute with new type for specific 'lon' variable
ncatted -O -a units,/g1/lon,o,sng,'degrees_west'
# Add new_att attribute to all variables
ncatted -O -a new_att,,c,sng,'new variable attribute'
# Add new_grp_att group attribute to all groups
ncatted -O -a new_grp_att,group,c,sng,'new group attribute'
# Add new_grp_att group attribute to single group
ncatted -O -a g1_grp_att,g1,c,sng,'new group attribute'
# Add new_glb_att global attribute to root group
ncatted -O -a new_glb_att,global,c,sng,'new global attribute'

Note that regular expressions work well in conjuction with group path support. In other words, the variable name (including group path component) and the attribute names may both be extended regular expressions.

Demonstrate input of C-language escape sequences (e.g., \n) and other special characters (e.g., \")

ncatted -h -a special,global,o,c,
'\nDouble quote: \"\nTwo consecutive double quotes: \"\"\n
Single quote: Beyond my shell abilities!\nBackslash: \\\n
Two consecutive backslashes: \\\\\nQuestion mark: \?\n'

Note that the entire attribute is protected from the shell by single quotes. These outer single quotes are necessary for interactive use, but may be omitted in batch scripts.

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.3 ncbo netCDF Binary Operator


ncbo [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]
[--cnk_byt sz_byt] [--cnk_csh sz_byt] [--cnk_dmn nm,sz_lmn]
[--cnk_map map] [--cnk_min sz_byt] [--cnk_plc plc] [--cnk_scl sz_lmn]
[-D dbg] [-d dim,[min][,[max][,[stride]]] [-F] [--fl_fmt fl_fmt]
[-G gpe_dsc] [-g grp[,…]] [--glb ...] [-h] [--hdr_pad nbr]
[-L dfl_lvl] [-l path] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o file_3] [-p path] [-R] [-r] [--ram_all]
[-t thr_nbr] [--unn] [-v var[,…]] [-X ...] [-x] [-y op_typ]
file_1 file_2 [file_3]


ncbo performs binary operations on variables in file_1 and the corresponding variables (those with the same name) in file_2 and stores the results in file_3. The binary operation operates on the entire files (modulo any excluded variables). See Missing Values, for treatment of missing values. One of the four standard arithmetic binary operations currently supported must be selected with the ‘-y op_typ’ switch (or long options ‘--op_typ’ or ‘--operation’). The valid binary operations for ncbo, their definitions, corresponding values of the op_typ key, and alternate invocations are:


Definition: file_3 = file_1 + file_2
Alternate invocation: ncadd
op_typ key values: ‘add’, ‘+’, ‘addition
Examples: ‘ncbo --op_typ=add’, ‘ncadd


Definition: file_3 = file_1 - file_2
Alternate invocations: ncdiff, ncsub, ncsubtract
op_typ key values: ‘sbt’, ‘-’, ‘dff’, ‘diff’, ‘sub’, ‘subtract’, ‘subtraction
Examples: ‘ncbo --op_typ=-’, ‘ncdiff


Definition: file_3 = file_1 * file_2
Alternate invocations: ncmult, ncmultiply
op_typ key values: ‘mlt’, ‘*’, ‘mult’, ‘multiply’, ‘multiplication
Examples: ‘ncbo --op_typ=mlt’, ‘ncmult


Definition: file_3 = file_1 / file_2
Alternate invocation: ncdivide
op_typ key values: ‘dvd’, ‘/’, ‘divide’, ‘division
Examples: ‘ncbo --op_typ=/’, ‘ncdivide

Care should be taken when using the shortest form of key values, i.e., ‘+’, ‘-’, ‘*’, and ‘/. Some of these single characters may have special meanings to the shell 60. Place these characters inside quotes to keep them from being interpreted (globbed) by the shell 61. For example, the following commands are equivalent

ncbo --op_typ=* # Dangerous (shell may try to glob)
ncbo --op_typ='*' # Safe ('*' protected from shell)
ncbo --op_typ="*" # Safe ('*' protected from shell)
ncbo --op_typ=mlt
ncbo --op_typ=mult
ncbo --op_typ=multiply
ncbo --op_typ=multiplication
ncmult # First do 'ln -s ncbo ncmult'
ncmultiply # First do 'ln -s ncbo ncmultiply'

No particular argument or invocation form is preferred. Users are encouraged to use the forms which are most intuitive to them.

Normally, ncbo will fail unless an operation type is specified with ‘-y’ (equivalent to ‘--op_typ’). You may create exceptions to this rule to suit your particular tastes, in conformance with your site’s policy on symbolic links to executables (files of a different name point to the actual executable). For many years, ncdiff was the main binary file operator. As a result, many users prefer to continue invoking ncdiff rather than memorizing a new command (‘ncbo -y sbt’) which behaves identically to the original ncdiff command. However, from a software maintenance standpoint, maintaining a distinct executable for each binary operation (e.g., ncadd) is untenable, and a single executable, ncbo, is desirable. To maintain backward compatibility, therefore, NCO automatically creates a symbolic link from ncbo to ncdiff. Thus ncdiff is called an alternate invocation of ncbo. ncbo supports many additional alternate invocations which must be manually activated. Should users or system adminitrators decide to activate them, the procedure is simple. For example, to use ‘ncadd’ instead of ‘ncbo --op_typ=add’, simply create a symbolic link from ncbo to ncadd 62. The alternatate invocations supported for each operation type are listed above. Alternatively, users may always define ‘ncadd’ as an alias to ‘ncbo --op_typ=add63.

It is important to maintain portability in NCO scripts. Therefore we recommend that site-specfic invocations (e.g., ‘ncadd’) be used only in interactive sessions from the command-line. For scripts, we recommend using the full invocation (e.g., ‘ncbo --op_typ=add’). This ensures portability of scripts between users and sites.

ncbo operates (e.g., adds) variables in file_2 with the corresponding variables (those with the same name) in file_1 and stores the results in file_3. Variables in file_1 or file_2 are broadcast to conform to the corresponding variable in the other input file if necessary64. Now ncbo is completely symmetric with respect to file_1 and file_2, i.e., file_1 - file_2 = - (file_2 - file_1.

Broadcasting a variable means creating data in non-existing dimensions by copying data in existing dimensions. For example, a two dimensional variable in file_2 can be subtracted from a four, three, or two (not one or zero) dimensional variable (of the same name) in file_1. This functionality allows the user to compute anomalies from the mean. In the future, we will broadcast variables in file_1, if necessary to conform to their counterparts in file_2. Thus, presently, the number of dimensions, or rank, of any processed variable in file_1 must be greater than or equal to the rank of the same variable in file_2. Of course, the size of all dimensions common to both file_1 and file_2 must be equal.

When computing anomalies from the mean it is often the case that file_2 was created by applying an averaging operator to a file with initially the same dimensions as file_1 (often file_1 itself). In these cases, creating file_2 with ncra rather than ncwa will cause the ncbo operation to fail. For concreteness say the record dimension in file_1 is time. If file_2 was created by averaging file_1 over the time dimension with the ncra operator (rather than with the ncwa operator), then file_2 will have a time dimension of size 1 rather than having no time dimension at all 65. In this case the input files to ncbo, file_1 and file_2, will have unequally sized time dimensions which causes ncbo to fail. To prevent this from occurring, use ncwa to remove the time dimension from file_2. See the example below.

ncbo never operates on coordinate variables or variables of type NC_CHAR or NC_STRING. This ensures that coordinates like (e.g., latitude and longitude) are physically meaningful in the output file, file_3. This behavior is hardcoded. ncbo applies special rules to some CF-defined (and/or NCAR CCSM or NCAR CCM fields) such as ORO. See CF Conventions for a complete description. Finally, we note that ncflint (see ncflint netCDF File Interpolator) is designed for file interpolation. As such, it also performs file subtraction, addition, multiplication, albeit in a more convoluted way than ncbo.

Beginning with NCO version 4.3.1 (May, 2013), ncbo supports group broadcasting. Group broadcasting means processing data based on group patterns in the input file(s) and automatically transferring or transforming groups to the output file. Consider the case where file_1 contains multiple groups each with the variable v1, while file_2 contains v1 only in its top-level (i.e., root) group. Then ncbo will replicate the group structure of file_1 in the output file, file_3. Each group in file_3 contains the output of the corresponding group in file_1 operating on the data in the single group in file_2. An example is provided below.


Say files and each contain 12 months of data. Compute the change in the monthly averages from 1985 to 1986:

ncbo --op_typ=sub
ncbo --op_typ='-'

These commands are all different ways of expressing the same thing.

The following examples demonstrate the broadcasting feature of ncbo. Say we wish to compute the monthly anomalies of T from the yearly average of T for the year 1985. First we create the 1985 average from the monthly data, which is stored with the record dimension time.

ncwa -O -a time

The second command, ncwa, gets rid of the time dimension of size 1 that ncra left in Now none of the variables in has a time dimension. A quicker way to accomplish this is to use ncwa from the beginning:

ncwa -a time

We are now ready to use ncbo to compute the anomalies for 1985:

ncdiff -v T

Each of the 12 records in now contains the monthly deviation of T from the annual mean of T for each gridpoint.

Say we wish to compute the monthly gridpoint anomalies from the zonal annual mean. A zonal mean is a quantity that has been averaged over the longitudinal (or x) direction. First we use ncwa to average over longitudinal direction lon, creating, the zonal mean of Then we use ncbo to subtract the zonal annual means from the monthly gridpoint data:

ncwa -a lon

This examples works assuming has dimensions time and lon, and that has no time or lon dimension.

Group broadcasting simplifies evaluation of multiple models against observations. Consider the input file which contains multiple top-level groups cesm, ecmwf, and giss, each of which contains the surface air temperature field tas. We wish to compare these models to observations stored in which contains tas only in its top-level (i.e., root) group. It is often the case that many models and/or model simulations exist, whereas only one observational dataset does. We evaluate the models and obtain the bias (difference) between models and observations by subtracting from Then ncbo “broadcasts” (i.e., replicates) the observational data to match the group structure of, subtracts, and then stores the results in the output file, which has the same group structure as

% ncbo -O
% ncks -H -v tas -d time,3
time[3] tas[3]=-1 
time[3] tas[3]=0 
time[3] tas[3]=1 

As a final example, say we have five years of monthly data (i.e., 60 months) stored in and we wish to create a file which contains the twelve month seasonal cycle of the average monthly anomaly from the five-year mean of this data. The following method is just one permutation of many which will accomplish the same result. First use ncwa to create the five-year mean:

ncwa -a time

Next use ncbo to create a file containing the difference of each month’s data from the five-year mean:


Now use ncks to group together the five January anomalies in one file, and use ncra to create the average anomaly for all five Januarys. These commands are embedded in a shell loop so they are repeated for all twelve months:

for idx in {1..12}; do # Bash Shell (version 3.0+) 
  idx=`printf "%02d" ${idx}` # Zero-pad to preserve order
  ncks -F -d time,${idx},,12 foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
for idx in 01 02 03 04 05 06 07 08 09 10 11 12; do # Bourne Shell
  ncks -F -d time,${idx},,12 foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc
foreach idx (01 02 03 04 05 06 07 08 09 10 11 12) # C Shell
  ncks -F -d time,${idx},,12 foo.${idx}
  ncra foo.${idx} t_anm_8589_${idx}.nc

Note that ncra understands the stride argument so the two commands inside the loop may be combined into the single command

ncra -F -d time,${idx},,12 foo.${idx}

Finally, use ncrcat to concatenate the 12 average monthly anomaly files into one twelve-record file which contains the entire seasonal cycle of the monthly anomalies:

ncrcat t_anm_8589_??.nc

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.4 ncclimo netCDF Climatology Generator


ncclimo [-3] [-4] [-5] [-6] [-7] 
[-a dec_md] [-C clm_md] [-c caseid] [-d dbg_lvl]
[-E yr_prv] [-e yr_end] [-f fml_nm] [--fl_fmt=fl_fmt] [-h hst_nm] [-i drc_in]
[-j job_nbr] [-L dfl_lvl] [-l lnk_flg] [-m mdl_nm] [-n nco_opt] 
[--no_cll_msr] [--no_frm_trm] [--no_ntv_tms] [--no_stg_grd]
[-O drc_rgr] [-o drc_out] [-p par_typ] [--ppc=ppc_prc]
[-R rgr_opt] [-r rgr_map]
[-S yr_prv] [-s yr_srt] [--seasons=csn_lst] [--stdin] 
[-t thr_nbr] [--tpd=tpd_dly] [-v var_lst] [--version] 
[-x cf_flg] [-X drc_xtn] [-x drc_prv] 
[-Y rgr_xtn] [-y rgr_prv] [--ypf=ypf_max]


In climatology generation mode, ncclimo ingests “raw” data consisting of a monthly or annual timeseries of files and from these produces climatological monthly means, seasonal means, and/or annual means. Alternatively, in timeseries reshaping (aka “splitter”) mode, ncclimo will subset and temporally split the input raw data timeseries into per-variable files spanning the entire period. ncclimo can optionally regrid all output files in either mode.

There are five required options (‘-c’, ‘-s’, ‘-e’, ‘-i’, and ‘-o’)) to generate climatologies, and many more options are available to customize the processing. Options are similar to ncremap options. Standard ncclimo usage for climatology generation looks like

ncclimo            -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo -m mdl_nm  -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo -v var_lst -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
ncclimo --case=caseid --start=srt_yr --end=end_yr --input=drc_in --output=drc_out

In climatology generation mode, ncclimo constructs the list of input filenames from the argument to the date and model-type options. ncclimo automatically switches to timeseries reshaping mode if it receives a list of files from stdin, or, alternatively, placed as positional arguments (after the last command-line option), or if neither of these is done and no caseid is specified, in which case it assumes all *.nc files in drc_in constitute the input file list.

Options come in both short (single-letter) and long forms. The handful of long-option synonyms for each option allows the user to imbue the commands with a level of verbosity and precision that suits her taste. A complete description of all options is given below, in alphabetical order of the short option letter. Long option synonyms are given just after the letter. When invoked without options, ncclimo prints a succinct table of all options and some examples.

-a dec_md (--dec_md, --december_mode, --dec_mode)

December mode determines the type of DJF average. The two valid options are scd (default) and sdd. SCD-mode stands for “Seasonally Continuous December”. The first month used is December of the year before the start year specified with ‘-s’. The last month is November of the end year specified with ‘-e’. SDD-mode stands for “Seasonally Discontinuous December”. The first month used is January of the specified start year. The last month is December of the end year specified with ‘-e’.

-C clm_md (--clm_md, --climatology_mode, --mode, --climatology)

Climatology mode. Valid values are for clm_md are ‘ann’, ‘mth’, and ‘dly’. The value indicates the timespan of each input file for annual and monthly climatologies. The default mode is ‘mth’, which means input files are monthly averages. Use ‘ann’ when the input files are a series of annual means (a common temporal resolution for ice-sheet simulations). The value ‘dly’ is used for all input files whose temporal resolution is daily or finer. The data could be a daily average, or diurnally-resolved, e.g., 3-hourly.

The climatology generator and splitter do not require that daily-mode input files begin or end on daily boundaries. These tools hyperslab the input files using the date information required to performed their analysis. This facilitates analyzing datasets with varying numbers of days per input file.

Explicitly specifying ‘--clm_md=mth’ serves a secondary purpose, namely invoking the default setting on systems that control stdin. When ncclimo detects that stdin is not attached to the terminal (keyboard) it automatically expects a list of files on stdin. Some environments, however, hijack stdin for their purposes and thereby confuse ncclimo into expecting a list argument. Users have encountered this issue when attempting to run ncclimo in Python parallel environments, via inclusion in crontab, and in nohup-mode (whatever that is!). In such cases, explicitly specify ‘--clm_md=mth’ (or ann or day) to persuade ncclimo to run a normal climatology.

-c caseid (--case, --caseid, --case_id)

Simulation name, or any input filename for non-CESM’ish files. The use of caseid is required in climate generation mode (unless equivalent information is provided through other options), where caseid is used to construct both input and output filenames. For CESM’ish input files like, specify ‘-c famipc5_ne30_v0.3_00001’. The ‘.cam.’ and ‘.h0.’ bits are added internally to produce the input filenames. Modify these via the -m mdl_nm and -h hst_nm switches if needed. For input files named slightly differently than standard CESM’ish names, supply the filename (excluding the path component) as the caseid and ncclimo will attempt to parse it by matching it to a database of know regular expressions common to model output. These are all of the format prefix[.-]YYYY[-]MM[-]DD.suffix. The particular formats current supported, as of NCO version 4.7.3 (March, 2018) are: prefix_YYYYMM.suffix, prefix.YYYY-MM.suffix, and prefix.YYYY-MM-01.suffix. For example, input files like (i.e., the six digits that precede the suffix are YYYYMM-format), specify ‘-c’ and the prefix (merra2) will be automatically abstracted and used to template and generate all the filenames based on the specified yr_srt and yr_end. Please tell us any dataset filename regular expressions that you would like added to ncclimo’s internal database.

-D dbg_lvl (--dbg_lvl, --dbg, --debug, --debug_level)

Specifies a debugging level similar to the rest of NCO. If dbg_lvl = 1, ncclimo prints more extensive diagnostics of its behavior. If dbg_lvl = 2, ncclimo prints the commands it would execute at any higher or lower debugging level, but does not execute these commands. If dbg_lvl > 2, ncclimo prints the diagnostic information, executes all commands, and passes-through the debugging level to the regridder (ncks) for additional diagnostics.

-e end_yr (--end_yr, --yr_end, --end_year, --year_end, --end)

End year (example: 2000). Unless the option ‘-a sdd’ is specified, the last month used is November of the specified end year. If ‘-a sdd’ is specified, the last month is December of the specified end year.

-f fml_nm (--fml_nm, --family, --family_name)

Family name (nickname) of output files. In climate generation mode, output climo file names are constructed by default with the same caseid as the input files. The fml_nm, if supplied, replaces caseid in output climo names, which are of the form where XX is the month or seasonal abbreviation. Use ‘-f fml_nm’ to simplify long names, avoid overlap, etc. Example values of fml_nm are ‘control’, ‘experiment’, and (for a single-variable climo) ‘FSNT’. In timeseries reshaping mode, fml_nm will be used, if supplied, as an additional string in the output filename. For example, specifying ‘-f control’ would cause to be instead named

-h hst_nm (--hst_nm, --history_name, --history)

History volume name of file used to generate climatologies. This referring to the hst_nm character sequence used to construct input file names: By default input climo file names are constructed from the caseid of the input files, together with the model name mdl_nm (specified with ‘-m’) and the date range. Use ‘-h hst_nm’ to specify alternative history volumes. Examples include ‘h0’ (default, works for CAM, ALM/CLM/CTSM/ELM), ‘h1’, and ‘h’ (for CISM).

-i drc_in (--drc_in, --in_drc, --dir_in, --input)

Directory containing all monthly mean files to read as input to the climatology. The use of drc_in is mandatory in climate generation mode and is optional in timeseries reshaping mode. In timeseries reshaping mode, ncclimo uses all netCDF files (meaning files with suffixes .nc, .nc3, .nc4, .nc5, .nc6, .nc7, .cdf, .hdf, .he5, or .h5) in drc_in to create the list of input files when no list is provided through stdin or as positional arguments to the command-line.

-j job_nbr (--job_nbr, --job_number, --jobs)

Specifies the number of simultaneous subsetting processes to spawn during parallel execution for both Background and MPI modes. This applies to timeseries reshaping mode only, and has no effect in climatology generation mode. In both parallel modes ncclimo spawns processes in batches of job_nbr jobs, then waits for those processes to complete. Once a batch finishes, ncclimo spawns the next batch. In Background mode, all jobs are spawned to the local node. In MPI mode, all jobs are spawned in round-robin fashion to all available nodes until job_nbr jobs are running.

If subsetting consumes so much RAM (e.g., because variables are large and/or the number of threads is large) that a single node can perform only one subsetting job at a time, then a reasonable value for job_nbr is the number of nodes, node_nbr. Often, however, nodes can usually subset (and then regrid, if requested) multiple variables simultaneously.

By default job_nbr = 2 in Background mode, and job_nbr = node_nbr in MPI mode. This helps prevent users from overloading nodes with too many jobs. Subject to the availability of adequate RAM, expand the number of jobs per node by increasing job_nbr until, ideally, each core on the node is used.

The main throughput bottleneck in timeseries reshaping mode is I/O. It is possible that each node can write to only one file at a time, in which case increasing job_nbr may have little effect. Regridding requires math that can relieve some I/O contention and allows for some throughput gains with increasing job_nbr. In general, though, increasing job_nbr is expected to improve throughput much more in ncremap than in ncclimo.

-L (--dfl_lvl, --dfl, --deflate)

Activate deflation (i.e., lossless compress, see Deflation) with the -L dfl_lvl short option (or with the same argument to the ‘--dfl_lvl’ or ‘--deflate’ long options). Specify deflation level dfl_lvl on a scale from no deflation (dfl_lvl = 0, the default) to maximum deflation (dfl_lvl = 9).

-l (--lnk_flg, --link_flag, --no_amwg_links)

This switch (which takes no argument) turns-off the default linking of E3SM/ACME-climo to AMWG-climo filenames. AMWG omits the YYYYMM components of climo filenames, resulting in shorter names. By default ncclimo symbolically links the full (E3SM/ACME) filename to the shorter (AMWG) name. AMWG diagnostics scripts can produce plots directly from these linked filenames. Use this switch to turn-off that linking and reduce filename proliferation if you do not need AMWG filenames.

-m mdl_nm (--mdl_nm, --model_name, --model)

Model name (as embedded in monthly input filenames). Default is ‘cam’. Other options are ‘clm2’, ‘ocn’, ‘ice’, ‘cism’, ‘cice’, ‘pop’.

-n nco_opt (nco_opt, nco, nco_options)

Specifies a string of options to pass-through unaltered to ncks. nco_opt defaults to ‘-O --no_tmp_fl’.

-O drc_rgr (--drc_rgr, --rgr_drc, --dir_rgr, --regrid)

Directory to hold regridded climo files. Regridded climos are placed in drc_out unless a separate directory for them is specified with ‘-O’ (NB: capital “O”).

--no_cll_msr (--no_cll_msr, --no_cll, --no_cell_measures, --no_area)

This switch (which takes no argument) controls whether ncclimo and ncremap add measures variables to the extraction list along with the primary variable and other associated variables. See CF Conventions for a detailed description.

--no_frm_trm (--no_frm_trm, --no_frm, --no_formula_terms)

This switch (which takes no argument) controls whether ncclimo and ncremap add formula variables to the extraction list along with the primary variable and other associated variables. See CF Conventions for a detailed description.

--no_ntv_tms (--no_ntv_tms, --no_ntv, --no_native, --remove_native)

This switch (which takes no argument) controls whether the splitter retains native grid split files, which it does by default, or deletes them. ncclimo can split model output from multi-variable native grid files into per-variable timeseries files and regrid those onto a so-called analysis grid. That is the typical format in which Model Intercomparison Projects (MIPs) request and disseminate contributions. When the data producer has no use for the split timeseries on the native grid, he/she can invoke this flag to cause ncclimo to delete the native grid timeseries (not the raw native grid datafiles).

--no_stg_grd (--no_stg_grd, --no_stg, --no_stagger, --no_staggered_grid)

This switch (which takes no argument) controls whether regridded output will contain the staggered grid coordinates slat, slon, and w_stag (see Regridding). By default the staggered grid is output for all files regridded from a Cap (aka FV) grid, except when the regridding is performed as part of splitting (reshaping) into timeseries.

-o drc_out (--drc_out, --out_drc, --dir_out, --output)

Directory to hold computed (output) native grid climo files. Regridded climos are also placed here unless a separate directory for them is specified with ‘-O’ (NB: capital “O”).

-p par_typ (--par_typ, --par_md, --parallel_type, --parallel_mode, --parallel)

Specifies the parallelism mode desired. The options are serial mode (‘-p nil’ or ‘-p serial’), background mode parallelism (‘-p bck’), and MPI parallelism ‘-p mpi’. The default is background-mode parallelism. The default par_typ is ‘bck’, which means ncclimo runs spawns up to twelve (one for each month) parallel processes at a time. See discussion below under Memory Considerations.

--ppc=ppc_prc (--ppc, --ppc_prc, --precision, --quantize)

Specifies the precision of the Precision-Preserving Compression algorithm (see Precision-Preserving Compression). A positive integer is interpreted as the Number of Significant Digits for the Bit-Grooming algorithm, and is equivalent to specifying ‘--ppc default=ppc_prc’ to a binary operator. A positive or negative integer preceded by a period, e.g., ‘.-2’ is interpreted as the number of Decimal Significant Digits for the rounding algorithm and is equivalent to specifying ‘--ppc default=.ppc_prc’ to a binary operator. This option applies one precision algorithm and a uniform precision for the entire file. To specify variable-by-variable precision options, pass the desired options as a quoted string directly with ‘-n nco_opt’, e.g., ‘-n '--ppc FSNT,TREFHT=4 --ppc CLOUD=2'’.

-R rgr_opt (rgr_opt, regrid_options)

Specifies a string of options to pass-through unaltered to ncks. rgr_opt defaults to ‘-O --no_tmp_fl’.

-r rgr_map (--rgr_map, --regrid_map, --map)

Regridding map. Unless ‘-r’ is specified ncclimo produces only a climatology on the native grid of the input datasets. The rgr_map specifies how to (quickly) transform the native grid into the desired analysis grid. ncclimo will (call ncremap to) apply the given map to the native grid climatology and produce a second climatology on the analysis grid. Options intended exclusively for the regridder may be passed as arguments to the ‘-R’ switch. See below the discussion on regridding.

-s srt_yr (--srt_yr, --yr_srt, --start_year, --year_start, --start)

Start year (example: 1980). Unless the option ‘-a sdd’ is specified, the first month used will be December of the year before the start year (to allow for contiguous DJF climos). If ‘-a sdd’ is specified, the first month used is January of the specified start year.

--seasons=csn_lst (--seasons, --csn_lst, --csn)

Seasons for ncclimo to compute in monthly climatology generation mode. The list of seasons, csn_lst, is a comma-separated, case-insensitive, unordered subset of the abbreviations for the eleven (so far) defined seasons: jfm, amj, jas, ond, on, fm, djf, mam, jja, son, and ann. By default csn_lst=mam,jja,son,djf. Moreover, ncclimo automatically computes the climatological annual mean, ANN, is always computed when MAM, JJA, SON, and DJF are all requested (which is the default). The ANN computed automatically is the time-weighted average of the four seasons, rather than as the time-weighted average of the twelve monthly climatologies. Users who need ANN but not DJF, MAM, JJA, and SON should instead explicitly specify ANN as a season in csn_lst. The ANN computed as a season is the time-weighted average of the twelve monthly climatologies, rather than the time-weighted average of four seasonal climatologies. Specifying the four seasons and ANN in csn_lst (e.g., csn_lst=mam,jja,son,djf,ann) is legal though redundant and wasteful. It cause ANN to be computed twice, first as the average of the twelve monthly climatologies, then as the average of the four seasons. The special value csn_lst=none turns-off computation of seasonal (and annual) climatologies.

ncclimo --seasons=none ...            # Produce only monthly climos
ncclimo --seasons=mam,jja,son,djf ... # Monthly + MAM,JJA,SON,DJF,ANN
ncclimo --seasons=jfm,jas,ann ...     # Monthly + JFM,JAS,ANN
ncclimo --seasons=fm,on ...           # Monthly + FM,ON
--stdin (--stdin, --inp_std, --std_flg, --redirect, --standard_input)

This switch (which takes no argument) explicitly indicates that input file lists are provided via stdin, i.e., standard input. In interactive environments, ncclimo can automatically (i.e., without any switch) detect whether input is provided via stdin. This switch is never required for jobs run in an interactive shell. However, non-interactive batch jobs (such as those submitted to the SLURM and PBS schedulers) make it impossible to unambiguously determine whether input has been provided via stdin. Specifically, the ‘--stdin’ switch must be used in non-interactive batch jobs on PBS when the input files are piped to stdin, and on SLURM when the input files are redirected from a file to stdin. Using this switch in any other context (e.g., interactive shells) is optional.

In some other non-interactive environments (e.g., crontab, nohup), ncclimo may mistakenly expect input to be provided on stdin simply because the environment is using stdin for other purposes. In such cases users may persuade ncclimo to ignore stdin by explicitly invoking the ‘--clm_md’ option (described above).

-t thr_nbr (--thr_nbr, --threads, --thread_number)

Specifies the number of threads used per regridding process (see OpenMP Threading). The NCO regridder scales well to 8–16 threads. However, regridding with the maximum number of threads can interfere with climatology generation in parallel climatology mode (i.e., when par_typ = mpi or bck). Hence ncclimo defaults to thr_nbr=2.

--tpd_out=tpd_out (--tpd_out, --tpd, --timesteps_per_day)

The number of timesteps-per-day in output created by ncclimo’s climatology generator in daily average mode. The climatology output from input files at daily or sub-daily resolution is, by default, averaged to daily resolution, i.e., tpd_out=1. If the number of timesteps per day in each input file is tpd_in, then the user may select any value of tpd_out that is smaller than and integrally divides tpd_in. For example, an input timeseries with tpd_in=8 (i.e., 3-hourly resolution), can be used to produce climatological output at 3, 6, or 12-hourly resolution by setting tpd_out to 8, 4, or 2, respectively. This option only takes effect in daily-average climatology mode.

-v var_lst (--var_lst, --var, --vars, --variables, --variable_list)

Variables to subset or to split. Same behavior as Subsetting Files. The use of var_lst is optional in climate generation mode. We suggest using this feature to test whether an ncclimo command, especially one that is lengthy and/or time-consuming, works as intended on one or a few variables with, e.g., ‘-v T,FSNT’ before generating the full climatology (by omitting this option). Invoking this switch was required in the original splitter released in version 4.6.5 (March, 2017), and became optional as of version 4.6.6 (May, 2017). This option is recommended in timeseries reshaping mode to prevent inadvertently copying the results of an entire model simulation. Regular expressions are allowed so, e.g., ‘PREC.?’ extracts the variables ‘PRECC,PRECL,PRECSC,PRECSL’ if present. Currently in reshaping mode all matches to a regular expression are placed in the same output file. We hope to remove this limitation in the future.

--version (--version, --vrs, --config, --configuration, --cnf)

This switch (which takes no argument) causes the operator to print its version and configuration. This includes the copyright notice, URLs to the GPL and NCO license, directories from which the NCO scripts and binaries are running, and the locations of any separate executables that may be used by the script.

--ypf_max ypf_max (--ypf, --years, --years_per_file)

Specifies the maximum number of years-per-file output by ncclimo’s splitting operation. When ncclimo subsets and splits a collection of input files spanning a timerseries, it places each subset variable in its own output file. The maximum length, in years, of each output file is ypf_max, which defaults to ypf_max=50. If an input timeseries spans 237 years and ypf_max=50, then ncclimo will generate four output files of length 50 years and one output file of length 37 years. Note that invoking this option causes ncclimo to enter timeseries reshaping mode. In fact, one must use ‘--ypf’ to turn-on splitter mode when the input files are specified by using the ‘-i drc_in’ method. Otherwise it would be ambiguous whether to generate a climatology from or to split the input files.

Timeseries Reshaping mode, aka Splitting

This section of the ncclimo documentation applies only to resphaping mode, whereas all subsequent sections apply to climatology generation mode. As mentioned above, ncclimo automatically switches to timeseries reshaping mode if it receives a list of files through stdin, or, alternatively, placed as positional arguments (after the last command-line option), or if neither of these is done and no caseid is specified, in which case it assumes all *.nc files in drc_in constitute the input file list. These examples invoke reshaping mode in the three possible ways:

# Pipe list to stdin
cd $drc_in;ls *mdl*000[1-9]*.nc | ncclimo -v T,Q,RH -s 1 -e 9 -o $drc_out
# Redirect list from file to stdin
cd $drc_in;ls *mdl*000[1-9]*.nc > foo;ncclimo -v T,Q,RH -s 1 -e 9 -o $drc_out < foo
# List as positional arguments
ncclimo -v T,Q,RH -s 1 -e 9 -o $drc_out $drc_in/*mdl*000[1-9]*.nc
# Glob directory
ncclimo -v T,Q,RH -s 1 -e 9 -i $drc_in -o $drc_out

Assuming each input file is a monthly average comprising the variables T, Q, and RH, then the output will be files,, and ncclimo reshapes the input so that the outputs are continuous timeseries of each variable taken from all input files. When necessary, the output is split into segments each containing no more than ypf_max (default 50) years of input, i.e.,,,, etc.

MPAS-O/I considerations

MPAS ocean and ice models currently have their own (non-CESM’ish) naming convention that guarantees output files have the same names for all simulations. By default ncclimo analyzes the “timeSeriesStatsMonthly” analysis member output (tell us if you want options for other analysis members). ncclimo recognizes input files as being MPAS-style when invoked with ‘-m mpaso’ or ‘-m mpascice’ like this:

ncclimo -m mpaso    -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-O
ncclimo -m mpascice -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-I

MPAS climos are unaware of missing values until/unless input files are “fixed”. We recommend that simulation producers annotate all floating point variables with the appropriate _FillValue prior to invoking ncclimo. Run something like this once in the history-file directory:

for fl in `ls hist.*` ; do
  ncatted -O -t -a _FillValue,,o,d,-9.99999979021476795361e+33 ${fl}

If/when MPAS-O/I generates the _FillValue attributes itself, this step can and should be skipped. All other ncclimo features like regridding (below) are invoked identically for MPAS as for CAM/CLM users although under-the-hood ncclimo does do some special pre-processing (dimension permutation, metadata annotation) for MPAS. A five-year oEC60to30 MPAS-O climo with regridding to T62 takes less than 10 minutes on the machine rhea.

Annual climos

Not all model or observed history files are created as monthly means. To create a climatological annual mean from a series of annual mean inputs, select ncclimo’s annual climatology mode with the ‘-C ann’ option:

ncclimo -C ann -m cism -h h -c caseid -s 1851 -e 1900 -i drc_in -o drc_out

The options ‘-m mdl_nm’ and ‘-h hst_nm’ (that default to cam and h0, respectively) tell ncclimo how to construct the input filenames. The above formula names the files,, and so on. Annual climatology mode produces a single output file (or two if regridding is selected), and in all other respects behaves the same as monthly climatology mode.

Regridding Climos and Other Files

ncclimo will (optionally) regrid during climatology generation and produce climatology files on both native and analysis grids. This regridding is virtually free, because it is performed on idle nodes/cores after monthly climatologies have been computed and while seasonal climatologies are being computed. This load-balancing can save half-an-hour on ne120 datasets. To regrid, simply pass the desired mapfile name with ‘-r’, e.g., ‘-r maps/’. Although this should not be necessary for normal use, you may pass any options specific to regridding with ‘-R opt1 opt2’.

Specifying ‘-O drc_rgr’ (NB: uppercase ‘O’) causes ncclimo to place the regridded files in the directory drc_rgr. These files have the same names as the native grid climos from which they were derived. There is no namespace conflict because they are in separate directories. These files also have symbolic links to their AMWG filenames. If ‘-O drc_rgr’ is not specified, ncclimo places all regridded files in the native grid climo output directory, drc_out, specified by ‘-o drc_out’ (NB: lowercase ‘o’). To avoid namespace conflicts when both climos are stored in the same directory, the names of regridded files are suffixed by the destination geometry string obtained from the mapfile, e.g., * These files also have symbolic links to their AMWG filenames.

ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl
ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl -O drc_rgr

The above commands perform a climatology without regridding, then with regridding (all climos stored in drc_out), then with regridding and storing regridded files separately. Paths specified by drc_in, drc_out, and drc_rgr may be relative or absolute. An alternative to regridding during climatology generation is to regrid afterwards with ncremap, which has more special features built-in for regridding. To use ncremap to regrid a climatology in drc_out and place the results in drc_rgr, use something like

ncremap -I drc_out -m -O drc_rgr
ls drc_out/*climo* | ncremap -m -O drc_rgr

See ncremap netCDF Remapper for more details (including MPAS!).

Extended Climatologies

ncclimo supports two methods for generating extended climatologies: Binary and Incremental. Both methods lengthen a climatology without requiring access to all the raw monthly data spanning the time period. The binary method combines, with appropriate weighting, two previously computed climatologies into a single climatology. No raw monthly data are employed. The incremental method computes a climatology from raw monthly data and (with appropriate weighting) combines that with a previously computed climatology that ends the month prior to raw data. The incremental method was introduced in NCO version 4.6.1 (released August, 2016), and the binary method was introduced in NCO version 4.6.3 (released December, 2016).

Both methods, binary and incremental, compute the so-called “extended climo” as a weighted mean of two shorter climatologies, called the “previous” and “current” climos. The incremental method uses the original monthly input to compute the curent climo, which must immediately follow in time the previous climo which has been pre-computed. The binary method use pre-computed climos for both the previous and current climos, and these climos need not be sequential nor chronological. Both previous and current climos for both binary and incremental methods may be of any length (in years); their weights will be automatically adjusted in computing the extended climo.

The use of pre-computed climos permits ongoing simulations (or lengthy observations) to be analyzed in shorter segments combined piecemeal, instead of requiring all raw, native-grid data to be simultaneously accessible. Without extended climatology capability, generating a one-hundred year climatology requires that one-hundred years of monthly data be available on disk. Disk-space requirements for large datasets may make this untenable. Extended climo methods permits a one-hundred year climo to be generated as the weighted mean of, say, the current ten year climatology (weighted at 10%) combined with the pre-computed climatology of the previous 90-years (weighted at 90%). The 90-year climo could itself have been generated incrementally or binary-wise, and so on. Climatologies occupy at most 17/(12N) the amount of space of N years of monthly data, so the extended methods vastly reduce disk-space requirements.

Incremental mode is selected by specifying ‘-S’, the start year of the pre-computed, previous climo. The argument to ‘-S’) is the previous climo start year. That, together with the current climo end year, determines the extended climo range. ncclimo assumes that the previous climo ends the month before the current climo begins. In incremental mode, ncclimo first generates the current climatology from the current monthly input files then weights that current climo with the previous climo to produce the extended climo.

Binary mode is selected by specifying both ‘-S’ and ‘-E’, the end year of the pre-computed, previous climo. In binary mode, the previous and current climatologies can be of any length, and from any time-period, even overlapping. Most users will run extended clmos the same way they run regular climos in terms of parallelism and regridding, although that is not required. Both climos must treat Decembers same way (or else previous climo files will not be found), and if subsetting (i.e., ‘-v var_lst’) is performed, then the subset must remain the same, and if nicknames (i.e., ‘-f fml_nm’) are employed, then the nickname must remain the same.

As of 20161129, the climatology_bounds attributes of extended climo are incorrect. This is a work in progress...


-E yr_end_prv (--yr_end_prv, --prv_yr_end, --previous_end)

The ending year of the previous climo. This argument is required to trigger binary climatologies, and should not be used for incremental climatologies.

-S yr_srt_prv (--yr_srt_prv, --prv_yr_srt, --previous_start)

The starting year of the previous climo. This argument is required to trigger incremental climatologies, and is also mandatory for binary climatologies.

-X drc_xtn (--drc_xtn, --xtn_drc, --extended)

Directory in which the extended native grid climo files will be stored for an extended climatology. Default value is drc_prv. Unless a separate directory is specified (with ‘-Y’) for the extended climo on the analysis grid, it will be stored in drc_xtn, too.

-x drc_prv (--drc_prv, --prv_drc, --previous)

Directory in which the previous native grid climo files reside for an incremental climatology. Default value is drc_out. Unless a separate directory is specified (with ‘-y’) for the previous climo on the analysis grid, it is assumed to reside in drc_prv, too.

-Y drc_rgr_xtn (--drc_rgr_xtn, --drc_xtn_rgr, --extended_regridded, --regridded_extended)

Directory in which the extended analysis grid climo files will be stored in an incremental climatology. Default value is drc_xtn.

-y drc_rgr_prv (--drc_rgr_prv, --drc_prv_rgr, --regridded_previous, --previous_regridded)

Directory in which the previous climo on the analysis grid resides in an incremental climatology. Default value is drc_prv.

Incremental method climatologies can be as simple as providing a start year for the previous climo, e.g.,

ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o clm -r
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r -S 1980

By default ncclimo stores all native and analysis grid climos in one directory so the above “just works”. There are no namespace clashes because all climos are for distinct years, and regridded files have a suffix based on their grid resolution. However, there can be only one set of AMWG filename links due to AMWG filename convention. Thus AMWG filename links, if any, point to the latest extended climo in a given directory.

Many researchers segregate (with ‘-O drc_rgr’) native-grid from analysis-grid climos. Incrementally generated climos must be consistent in this regard. In other words, all climos contributing to an extended climo must have their native-grid and analysis-grid files in the same (per-climo) directory, or all climos must segregate their native from their analysis grid files. Do not segregate the grids in one climo, and combine them in another. Such climos cannot be incrementally aggregated. Thus incrementing climos can require from zero to four additional options that specify all the previous and extended climatologies for both native and analysis grids. The example below constructs the current climo in crr, then combines the weighted average of that with the previous climo in prv, and places the resulting extended climatology in xtn. Here the native and analysis climos are combined in one directory per climo:

ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -r
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r \
        -S 1980 -x prv -X xtn

If the native and analysis grid climo directories are segregated, then those directories must be specified, too:

ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -O rgr_prv -r
ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -O rgr -r \
        -S 1980 -x prv -X xtn -y rgr_prv -Y rgr_xtn

ncclimo does not know whether a pre-computed climo is on a native grid or an analysis grid, i.e., whether it has been regridded. In binary mode, ncclimo may be pointed to two pre-computed native grid climatologies, or to two pre-computed analysis grid climatologies. In other words, it is not necessary to maintain native grid climatologies for use in creating extended climatologies. It is sufficient to generate climatologies on the analysis grid, and feed them to ncclimo in binary mode, without a mapping file:

ncclimo -c caseid -S 1980 -E 1981 -x prv -s 1980 -e 1981 -i crr -o clm 

Coupled Runs

ncclimo works on all E3SM/ACME and CESM models. It can simultaneously generate climatologies for a coupled run, where climatologies mean both native and regridded monthly, seasonal, and annual averages as per E3SM/ACME specifications (which mandate the inclusion of certain helpful metadata and provenance information). Here are template commands for a recent simulation:

ncclimo -p mpi -c $caseid -m cam  -s 2 -e 5 -i $drc_in -r $map_atm -o $drc_out/atm
ncclimo        -c $caseid -m clm2 -s 2 -e 5 -i $drc_in -r $map_lnd -o $drc_out/lnd
ncclimo -p mpi -m mpaso           -s 2 -e 5 -i $drc_in -r $map_ocn -o $drc_out/ocn 
ncclimo        -m mpascice        -s 2 -e 5 -i $drc_in -r $map_ice -o $drc_out/ice

Atmosphere and ocean model output is typically larger than land and ice model output. These commands recognize that by using different parallelization strategies that may (rhea standard queue) or may not (cooley, or rhea’s bigmem queue) be required, depending on the fatness of the analysis nodes, as explained below.

Memory Considerations

It is important to employ the optimal ncclimo parallelization strategy for your computer hardware resources. Select from the three available choices with the -p par_typ switch. The options are serial mode (‘-p nil’ or ‘-p serial’), background mode parallelism (‘-p bck’), and MPI parallelism ‘-p mpi’. The default is background-mode parallelism. This is appropriate for lower resolution (e.g., ne30L30) simulations on most nodes at high-performance computer centers. Use (or at least start with) serial mode on personal laptops/workstations. Serial mode requires twelve times less RAM than the parallel modes, and is much less likely to deadlock or cause OOM (out-of-memory) conditions on your personal computer. If the available RAM (plus swap) is < 12*4*sizeof(monthly input file), then try serial mode first (12 is the optimal number of parallel processes for monthly climos, the computational overhead is a factor of four). CAM-SE ne30L30 output is about 1 GB/month so each month requires about 4 GB of RAM. CAM-SE ne30L72 output (with LINOZ) is about 10 GB/month so each month requires about 40 GB RAM. CAM-SE ne120 output is about 12 GB/month so each month requires about 48 GB RAM. The computer does not actually use all this memory at one time, and many kernels compress RAM usage to below what top reports, so the actual physical usage is hard to pin-down, but may be a factor of 2.5–3.0 (rather than a factor of four) times the size of the input file. For instance, my 16 GB 2014 MacBookPro successfully runs an ne30L30 climatology (that requests 48 GB RAM) in background mode. However the laptop is slow and unresponsive for other uses until it finishes (in 6–8 minutes) the climos. Experiment and choose the parallelization option that performs best.

Serial-mode, as its name implies, uses one core at a time for climos, and proceeds sequentially from months to seasons to annual climatologies. Serial mode means that climos are performed serially, while regridding still employs OpenMP threading (up to 16 cores) on platforms that support it. By design each month and each season is independent of the others, so all months can be computed in parallel, then each season can be computed in parallel (using monthly climatologies), from which annual average is computed. Background parallelization mode exploits this parallelism and executes the climos in parallel as background processes on a single node, so that twelve cores are simultaneously employed for monthly climatologies, four for seasonal, and one for annual. The optional regridding will employ, by default, up to two cores per process. The MPI parallelism mode executes the climatologies on different nodes so that up to (optimally) twelve nodes compute monthly climos. The full memory of each node is available for each individual climo. The optional regridding employs, by default, up to eight cores per node in MPI-mode. MPI-mode or serial-mode must be used to process ne30L72 and ne120L30 climos on all but the fattest DOE nodes. An ne120L30 climo in background mode on rhea (i.e., on one 128 GB compute node) fails due to OOM. (Unfortunately OOM errors do not produce useful return codes so if your climo processes die without printing useful information, the cause may be OOM). However the same climo in background-mode succeeds when executed on a single big-memory (1 TB) node on rhea (use ‘-lpartition=gpu’, as shown below). Or MPI-mode can be used for any climatology. The same ne120L30 climo will also finish blazingly fast in background mode on cooley (i.e., on one 384 GB compute node), so MPI-mode is unnecessary on cooley. In general, the fatter the memory, the better the performance.

Single, Dedicated Nodes at LCFs

The basic approach above (running the script from a standard terminal window) that works well for small cases can be unpleasantly slow on login nodes of LCFs and for longer or higher resolution (e.g., ne120) climatologies. As a baseline, generating a climatology of 5 years of ne30 (~1x1 degree) CAM-SE output with ncclimo takes 1–2 minutes on rhea (at a time with little contention), and 6–8 minutes on a 2014 MacBook Pro. To make things a bit faster at LCFs, request a dedicated node (this only makes sense on supercomputers or clusters with job-schedulers). On rhea or titan, which use the PBS scheduler, do this with

# Standard node (128 GB), PBS scheduler
qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -N ncclimo
# Bigmem node (1 TB), PBS scheduler
qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -lpartition=gpu -N ncclimo

The equivalent requests on cooley or mira (Cobalt scheduler) and cori or titan (SLURM scheduler) are:

# Cooley node (384 GB) with Cobalt
qsub -I -A HiRes_EarthSys --nodecount=1 --time=00:10:00 --jobname=ncclimo 
# Cori node (128 GB) with SLURM
salloc  -A acme --nodes=1 --partition=debug --time=00:10:00 --job-name=ncclimo

Flags used and their meanings:


Submit in interactive mode. This returns a new terminal shell rather than running a program.


How long to keep this dedicated node for. Unless you kill the shell created by the qsub command, the shell will exist for this amount of time, then die suddenly. In the above examples, 10 minutes is requested.

-l nodes=1

PBS syntax (e.g., on rhea) for nodes.

--nodecount 1

Cobalt syntax (e.g., on cooley) for nodes.


SLURM syntax (e.g., on cori or edison) for nodes. These scheduler-dependent variations request a quantity of nodes. Request 1 node for Serial or Background-mode, and up to 12 nodes for MPI-mode parallelism. In all cases ncclimo will use multiple cores per node if available.


Export existing environmental variables into the new interactive shell. This may not actually be needed.

-q name

Queue name. This is needed for locations like edison that have multiple queues with no default queue.


Name of account to charge for time used.

Acquiring a dedicated node is useful for any workflow, not just creating climos. This command returns a prompt once nodes are assigned (the prompt is returned in your home directory so you may then have to cd to the location you meant to run from). Then run your code with the basic ncclimo invocation. The is faster because the node is exclusively dedicated to ncclimo. Again, ne30L30 climos only require < 2 minutes, so the 10 minutes requested in the example is excessive and conservative. Tune it with experience.

12 node MPI-mode Jobs

The above parallel approaches will fail when a single node lacks enough RAM (plus swap) to store all twelve monthly input files, plus extra RAM for computations. One should employ MPI multinode parallelism ‘-p mpi’ on nodes with less RAM than 12*3*sizeof(input). The longest an ne120 climo will take is less than half an hour (~25 minutes on edison or rhea), so the simplest method to run MPI jobs is to request 12-interactive nodes using the above commands (though remember to add ‘-p mpi’), then execute the script at the command line.

It is also possible, and sometimes preferable, to request non-interactive compute nodes in a batch queue. Executing an MPI-mode climo (on machines with job scheduling and, optimally, 12 nodes) in a batch queue can be done in two commands. First, write an executable file which calls the ncclimo script with appropriate arguments. We do this below by echoing to a file, ncclimo.pbs.

echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" > ncclimo.pbs

The only new argument here is ‘-p mpi’ that tells ncclimo to use MPI parallelism. Then execute this command file with a 12 node non-interactive job:

qsub -A CLI115 -V -l nodes=12 -l walltime=00:30:00 -j oe -m e -N ncclimo \
     -o ncclimo.out ncclimo.pbs

This script adds new flags: ‘-j oe’ (combine output and error streams into standard error), ‘-m e’ (send email to the job submitter when the job ends), ‘-o ncclimo.out’ (write all output to ncclimo.out). The above commands are meant for PBS schedulers like on rhea. Equivalent commands for cooley/mira (Cobalt) and cori/edison (SLURM) are

# Cooley (Cobalt scheduler)
/bin/rm -f ncclimo.err ncclimo.out
echo '#!/bin/bash' > ncclimo.cobalt
echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" >> ncclimo.cobalt
chmod a+x ncclimo.cobalt
qsub -A HiRes_EarthSys --nodecount=12 --time=00:30:00 --jobname ncclimo \
     --error ncclimo.err --output ncclimo.out --notify ncclimo.cobalt

# Cori/Edison (SLURM scheduler)
echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out -r $map_fl" \
      > ncclimo.pbs
chmod a+x ncclimo.slurm
sbatch -A acme --nodes=12 --time=03:00:00 --partition=regular --job-name=ncclimo \
       --mail-type=END --error=ncclimo.err --output=ncclimo.out ncclimo.slurm

Notice that Cobalt and SLURM require the introductory shebang-interpreter line (#!/bin/bash) which PBS does not need. Set only the scheduler batch queue parameters mentioned above. In MPI-mode, ncclimo determines the appropriate number of tasks-per-node based on the number of nodes available and script internals (like load-balancing for regridding). Hence do not set a tasks-per-node parameter with scheduler configuration parameters as this could cause conflicts.

What does ncclimo do?

For monthly climatologies (e.g., JAN), ncclimo passes the list of all relevant January monthly files to NCO’s ncra command, which averages each variable in these monthly files over their time-dimension (if it exists) or copies the value from the first month unchanged (if no time-axis exists). Seasonal climos are then created by taking the average of the monthly climo files using ncra. To account for differing numbers of days per month, the ncra-w’ flag is used, followed by the number of days in the relevant months. For example, the MAM climo is computed with ‘ncra -w 31,30,31’ (details about file names and other optimization flags have been stripped here to make the concept easier to follow). The annual (ANN) climo is then computed as a weighted average of the seasonal climos.

Assumptions, Approximations, and Algorithms (AAA) Employed:

A climatology embodies many algorithmic choices, and regridding from the native to the analysis grid involves still more choices. A separate method should reproduce the ncclimo and NCO answers to round-off precision if it implements the same algorithmic choices. For example, ncclimo agrees to round-off with AMWG diagnostics when making the same (sometimes questionable) choices. The most important choices have to do with converting single- to double-precision (SP and DP, respectively), treatment of missing values, and generation/application of regridding weights. For concreteness and clarity we describe the algorithmic choices made in processing a CAM-SE monthly output into a climatological annual mean (ANN) and then regridding that. Other climatologies (e.g., daily to monthly, or annual-to-climatological) involve similar choices.

E3SM/ACME (and CESM) computes fields in DP and outputs history (not restart) files as monthly means in SP. The NCO climatology generator (ncclimo) processes these data in four stages. Stage N accesses input only from stage N-1, never from stage N-2 or earlier. Thus the (on-disk) files from stage N determine the highest precision achievable by stage N+1. The general principal is to perform math (addition, weighting, normalization) in DP and output results to disk in the same precision in which they were input from disk (usually SP). In Stage 1, NCO ingests Stage 0 monthly means (raw CAM-SE output), converts SP input to DP, performs the average across all years, then converts the answer from DP to SP for storage on-disk as the climatological monthly mean. In Stage 2, NCO ingests Stage 1 climatological monthly means, converts SP input to DP, performs the average across all months in the season (e.g., DJF), then converts the answer from DP to SP for storage on-disk as the climatological seasonal mean. In Stage 3, NCO ingests Stage 2 climatological seasonal means, converts SP input to DP, performs the average across all four seasons (DJF, MAM, JJA, SON), then converts the answer from DP to SP for storage on-disk as the climatological annual mean.

Stage 2 weights each input month by its number of days (e.g., 31 for January), and Stage 3 weights each input season by its number of days (e.g., 92 for MAM). E3SM/ACME runs CAM-SE with a 365-day calendar, so these weights are independent of year and never change. The treatment of missing values in Stages 1–3 is limited by the lack of missing value tallies provided by Stage 0 (model) output. Stage 0 records a value as missing if it is missing for the entire month, and present if the value is valid for one or more timesteps. Stage 0 does not record the missing value tally (number of valid timesteps) for each spatial point. Thus a point with a single valid timestep during a month is weighted the same in Stages 1–4 as a point with 100% valid timesteps during the month. The absence of tallies inexorably degrades the accuracy of subsequent statistics by an amount that varies in time and space. On the positive side, it reduces the output size (by a factor of two) and complexity of analyzing fields that contain missing values. Due to the ambiguous nature of missing values, it is debatable whether they merit efforts to treat them more exactly.

The vast majority of fields undergo three promotion/demotion cycles between CAM-SE and ANN. No promotion/demotion cycles occur for history fields that CAM-SE outputs in DP rather than SP, nor for fields without a time dimension. Typically these fields are grid coordinates (e.g., longitude, latitude) or model constants (e.g., CO2 mixing ratio). NCO never performs any arithmetic on grid coordinates or non-time-varying input, regardless of whether they are SP or DP. Instead, NCO copies these fields directly from the first input file. Stage 4 uses a mapfile to regrid climos from the native to the desired analysis grid. E3SM/ACME currently uses mapfiles generated by ESMF_RegridWeightGen (ERWG) and by TempestRemap.

The algorithmic choices, approximations, and commands used to generate mapfiles from input gridfiles are separate issues. We mention only some of these issues here for brevity. Input gridfiles used by E3SM/ACME until ~20150901, and by CESM (then and currently, at least for Gaussian grids) contained flaws that effectively reduced their precision, especially at regional scales, and especially for Gaussian grids. E3SM/ACME (and CESM) mapfiles continue to approximate grids as connected by great circles, whereas most analysis grids (and some models) use great circles for longitude and small circles for latitude. The great circle assumption may be removed in the future. Constraints imposed by ERWG during weight-generation ensure that global integrals of fields undergoing conservative regridding are exactly conserved.

Application of weights from the mapfile to regrid the native data to the analysis grid is straightforward. Grid fields (e.g., latitude, longitude, area) are not regridded. Instead they are copied (and area is reconstructed if absent) directly from the mapfile. NCO ingests all other native grid (source) fields, converts SP to DP, and accumulates destination gridcell values as the sum of the DP weight (from the sparse matrix in the mapfile) times the (usually SP-promoted-to-DP) source values. Fields without missing values are then stored to disk in their original precision. Fields with missing values are treated (by default) with what NCO calls the “conservative” algorithm. This algorithm uses all valid data from the source grid on the destination grid once and only once. Destination cells receive the weighted valid values of the source cells. This is conservative because the global integrals of the source and destination fields are equal. See ncremap netCDF Remapper for more description of the conservative and of the optional (“renormalized”) algorithm.


How to create a climo from a collection of monthly non-CESM’ish files? This is a two-step procedure: First be sure the names are arranged with a YYYYMM-format date preceding the suffix (usually ‘.nc’). Then give any monthly input filename to ncclimo. Consider the MERRA2 collection, for example. As retrieved from NASA, MERRA2 files have names like svc_MERRA2_300.tavgM_2d_aer_Nx.200903.nc4. While the sub-string ‘200903’ is easy to recognize as a month in YYYYMM format, other parts (specifically the ‘300’ code) of the filename also change with date. We can use Bash regular expressions to extract dates and create symbolic links to simpler filenames with regularly patterned YYYYMM strings like merra2_200903.nc4:

for fl in `ls *.nc4` ; do
# Convert svc_MERRA2_300.tavgM_2d_aer_Nx.YYYYMM.nc4 to merra2_YYYYMM.nc4
    sfx_out=`expr match "${fl}" '.*_Nx.\(.*.nc4\)'`
    ln -s ${fl} ${fl_out}

Then call ncclimo with merra2_200903.nc4 as caseid:

ncclimo -c merra2_200903.nc4 -s 1980 -e 2016 -i $drc_in -o $drc_out

In the default monthly climo generation mode, ncclimo expects each input file to contain one single record that is the monthly average of all fields. Another example of of wrangling observed datasets into a CESMish format is ECMWF Integrated Forecasting System (IFS) data that contains twelve months per file, rather than the one month per file that ncclimo expects.

for yr in {1979..2016}; do
# Convert to
    yyyy=`printf "%04d" $yr`
    for mth in {1..12}; do
        mm=`printf "%02d" $mth`
        ncks -O -F -d time,${mth} ifs_${yyyy}01-${yyyy} ifs_${yyyy}${mm}.nc

Then call ncclimo with as caseid:

ncclimo -c -s 1979 -e 2016 -i $drc_in -o $drc_out

ncclimo does not recognize all combinations imaginable of records per file and files per year. However, support can be added for the most prevalent combinations so that ncclimo, rather than the user, does any necessary data wrangling. Contact us if there is a common input data format you would like supported as a custom option.

Often one wishes to create a climatology of a single variable. The ‘-f fml_nm’ option to ncclimo makes this easy. Consider a series of single-variable climos for the fields FSNT, and FLNT

ncclimo -v FSNT -f FSNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
ncclimo -v FLNT -f FLNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out

These climos use the ‘-f’ option and so there output files will have no namespace conflicts. Moreover, the climatologies can be generated in parallel.

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.5 ncecat netCDF Ensemble Concatenator


ncecat [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]
[--cnk_byt sz_byt] [--cnk_csh sz_byt] [--cnk_dmn nm,sz_lmn]
[--cnk_map map] [--cnk_min sz_byt] [--cnk_plc plc] [--cnk_scl sz_lmn]
[-D dbg] [-d dim,[min][,[max][,[stride]]] [-F] [--fl_fmt fl_fmt]
[-G gpe_dsc] [-g grp[,…]] [--gag] [--glb ...] [-h] [--hdf] [--hdr_pad nbr]
[-L dfl_lvl] [-l path] [-M] [--md5_digest] [--mrd] [-n loop]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o output-file] [-p path] [--ppc ...] [-R] [-r] [--ram_all] 
[-t thr_nbr] [-u ulm_nm] [--unn] [-v var[,…]] [-X ...] [-x] 
[input-files] [output-file]


ncecat aggregates an arbitrary number of input files into a single output file using using one of two methods. Record AGgregation (RAG), the traditional method employed on (flat) netCDF3 files and still the default method, stores input-files as consecutive records in the output-file. Group AGgregation (GAG) stores input-files as top-level groups in the netCDF4 output-file. Record Aggregation (RAG) makes numerous assumptions about the structure of input files whereas Group Aggregation (GAG) makes none. Both methods are described in detail below. Since ncecat aggregates all the contents of the input files, it can easily produce large output files so it is often helpful to invoke subsetting simultaneously (see Subsetting Files).

RAG makes each variable (except coordinate variables) in each input file into a single record of the same variable in the output file. Coordinate variables are not concatenated, they are instead simply copied from the first input file to the output-file. All input-files must contain all extracted variables (or else there would be “gaps” in the output file).

A new record dimension is the glue which binds together the input file data. The new record dimension is defined in the root group of the output file so it is visible to all sub-groups. Its name is, by default, “record”. This default name can be overridden with the ‘-u ulm_nm’ short option (or the ‘--ulm_nm’ or ‘rcd_nm’ long options).

Each extracted variable must be constant in size and rank across all input-files. The only exception is that ncecat allows files to differ in the record dimension size if the requested record hyperslab (see Hyperslabs) resolves to the same size for all files. This allows easier gluing/averaging of unequal length timeseries from simulation ensembles (e.g., the CMIP archive).

Classic (i.e., all netCDF3 and NETCDF4_CLASSIC) output files can contain only one record dimension. ncecat makes room for the new glue record dimension by changing the pre-existing record dimension, if any, in the input files into a fixed dimension in the output file. netCDF4 output files may contain any number of record dimensions, so ncecat need not and does not alter the record dimensions, if any, of the input files as it copies them to the output file.

Group AGgregation (GAG) stores input-files as top-level groups in the output-file. No assumption is made about the size or shape or type of a given object (variable or dimension or group) in the input file. The entire contents of the extracted portion of each input file is placed in its own top-level group in output-file, which is automatically made as a netCDF4-format file.

GAG has two methods to specify group names for the output-file. The ‘-G’ option, or its long-option equivalent ‘--gpe’, takes as argument a group path editing description gpe_dsc of where to place the results. Each input file needs a distinct output group name to avoid namespace conflicts in the output-file. Hence ncecat automatically creates unique output group names based on either the input filenames or the gpe_dsc arguments. When the user provides gpe_dsc (i.e., with ‘-G’), then the output groups are formed by enumerating sequential two-digit numeric suffixes starting with zero, and appending them to the specified group path (see Group Path Editing). When gpe_dsc is not provided (i.e., user requests GAG with ‘--gag’ instead of ‘-G’), then ncecat forms the output groups by stripping the input file name of any type-suffix (e.g., .nc), and all but the final component of the full filename.

ncecat --gag # Output groups 85, 86, 87
ncecat -G 85_ # Output groups 85_00, 85_01, 85_02
ncecat -G 85/ # Output groups 85/00, 85/01, 85/02

With both RAG and GAG the output-file size is the sum of the sizes of the extracted variables in the input files. See Statistics vs. Concatenation, for a description of the distinctions between the various statistics tools and concatenators. As a multi-file operator, ncecat will read the list of input-files from stdin if they are not specified as positional arguments on the command line (see Large Numbers of Files).

Suppress global metadata copying. By default NCO’s multi-file operators copy the global metadata from the first input file into output-file. This helps to preserve the provenance of the output data. However, the use of metadata is burgeoning and sometimes one encounters files with excessive amounts of extraneous metadata. Extracting small bits of data from such files leads to output files which are much larger than necessary due to the automatically copied metadata. ncecat supports turning off the default copying of global metadata via the ‘-M’ switch (or its long option equivalents, ‘--no_glb_mtd’ and ‘--suppress_global_metadata’).

Consider five realizations,,, … of 1985 predictions from the same climate model. Then ncecat 85?.nc glues together the individual realizations into the single file, If an input variable was dimensioned [lat,lon], it will by default have dimensions [record,lat,lon] in the output file. A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file. Normally this means all the input files are the same size, and contain data on different realizations of the same variables.

Concatenating a variable packed with different scales across multiple datasets is beyond the capabilities of ncecat (and ncrcat, the other concatenator (Concatenation). ncecat does not unpack data, it simply copies the data from the input-files, and the metadata from the first input-file, to the output-file. This means that data compressed with a packing convention must use the identical packing parameters (e.g., scale_factor and add_offset) for a given variable across all input files. Otherwise the concatenated dataset will not unpack correctly. The workaround for cases where the packing parameters differ across input-files requires three steps: First, unpack the data using ncpdq. Second, concatenate the unpacked data using ncecat, Third, re-pack the result with ncpdq.


Consider a model experiment which generated five realizations of one year of data, say 1985. You can imagine that the experimenter slightly perturbs the initial conditions of the problem before generating each new solution. Assume each file contains all twelve months (a seasonal cycle) of data and we want to produce a single file containing all the seasonal cycles. Here the numeric filename suffix denotes the experiment number (not the month):

ncecat 85_0[1-5].nc
ncecat -n 5,2,1

These three commands produce identical answers. See Specifying Input Files, for an explanation of the distinctions between these methods. The output file,, is five times the size as a single input-file. It contains 60 months of data.

One often prefers that the (new) record dimension have a more descriptive, context-based name than simply “record”. This is easily accomplished with the ‘-u ulm_nm’ switch. To add a new record dimension named “time” to all variables

ncecat -u time

To glue together multiple files with a new record variable named “realization”

ncecat -u realization 85_0[1-5].nc

Users are more likely to understand the data processing history when such descriptive coordinates are used.

Consider a file with an existing record dimension named time. and suppose the user wishes to convert time from a record dimension to a non-record dimension. This may be useful, for example, when the user has another use for the record variable. The simplest method is to use ‘ncks --fix_rec_dmn’, and another possibility is to use ncecat followed by ncwa:

ncecat # Convert time to non-record dimension
ncwa -a record # Remove new degenerate record dimension

The second step removes the degenerate record dimension. See ncpdq netCDF Permute Dimensions Quickly and ncks netCDF Kitchen Sink for other methods of of changing variable dimensionality, including the record dimension.

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.6 nces netCDF Ensemble Statistics


nces [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]
[--cnk_byt sz_byt] [--cnk_csh sz_byt] [--cnk_dmn nm,sz_lmn]
[--cnk_map map] [--cnk_min sz_byt] [--cnk_plc plc] [--cnk_scl sz_lmn]
[-D dbg] [-d dim,[min][,[max][,[stride]]] [-F]
[-G gpe_dsc] [-g grp[,…]] [--glb ...] [-h] [--hdf] [--hdr_pad nbr] 
[-L dfl_lvl] [-l path] [-n loop]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] [--nsm_fl|grp] [--nsm_sfx sfx]
[-O] [-o output-file] [-p path] [--ppc ...] [-R] [-r] [--ram_all] [--rth_dbl|flt]
[-t thr_nbr] [--unn] [-v var[,…]] [-X ...] [-x] [-y op_typ]
[input-files] [output-file]


nces performs gridpoint statistics (including, but not limited to, averages) on variables across an arbitrary number (an ensemble) of input-files and/or of input groups within each file. Each file (or group) receives an equal weight. nces was formerly (until NCO version 4.3.9, released December, 2013) known as ncea (netCDF Ensemble Averager)66. For example, nces will average a set of files or groups, weighting each file or group evenly. This is distinct from ncra, which performs statistics only over the record dimension(s) (e.g., time), and weights each record in each record dimension evenly.

The file or group is the logical unit of organization for the results of many scientific studies. Often one wishes to generate a file or group which is the statistical product (e.g., average) of many separate files or groups. This may be to reduce statistical noise by combining the results of a large number of experiments, or it may simply be a step in a procedure whose goal is to compute anomalies from a mean state. In any case, when one desires to generate a file whose statistical properties are equally influenced by all the inputs, then nces is the operator to use.

Variables in the output-file are the same size as the variable hyperslab in each input file or group, and each input file or group must be the same size after hyperslabbing 67 nces does allow files to differ in the record dimension size if the requested record hyperslab (see Hyperslabs) resolves to the same size for all files. nces recomputes the record dimension hyperslab limits for each input file so that coordinate limits may be used to select equal length timeseries from unequal length files. This simplifies analysis of unequal length timeseries from simulation ensembles (e.g., the CMIP3 IPCC AR4 archive).

nces works in one of two modes, file ensembles or group ensembles. File ensembles are the default (equivalent to the old ncea) and may also be explicitly specified by the ‘--nsm_fl’ or ‘--ensemble_file’ switches. To perform statistics on ensembles of groups, a newer feature, use ‘--nsm_grp’ or ‘--ensemble_group’. Members of a group ensemble are groups that share the same structure, parent group, and nesting level. Members must be leaf groups, i.e., not contain any sub-groups. Their contents usually have different values because they are realizations of replicated experiments. In group ensemble mode nces computes the statistics across the ensemble, which may span multiple input files. Files may contain members of multiple, distinct ensembles. However, all ensembles must have at least one member in the first input file. Group ensembles behave as an unlimited dimension of datasets: they may contain an arbitrary and extensible number of realizations in each file, and may be composed from multiple files.

Output statistics in group ensemble mode are stored in the parent group by default. If the ensemble members are /cesm/cesm_01 and /cesm/cesm_02, then the computed statistic will be in /cesm in the output file. The ‘--nsm_sfx’ option instructs nces to instead store output in a new child group of the parent created by attaching the suffix to the parent group’s name, e.g., ‘--nsm_sfx='_avg'’ would store results in the output group /cesm/cesm_avg:

nces --nsm_grp        
nces --nsm_grp --nsm_sfx='_avg'

See Statistics vs. Concatenation, for a description of the distinctions between the statistics tools and concatenators. As a multi-file operator, nces will read the list of input-files from stdin if they are not specified as positional arguments on the command line (see Large Numbers of Files).

Like ncra and ncwa, nces treats coordinate variables as a special case. Coordinate variables are assumed to be the same in all ensemble members, so nces simply copies the coordinate variables that appear in ensemble members directly to the output file. This has the same effect as averaging the coordinate variable across the ensemble, yet does not incur the time- or precision- penalties of actually averaging them. ncra and ncwa allow coordinate variables to be processed only by the linear average operation, regardless of the arithmetic operation type performed on the non-coordinate variables (see Operation Types). Thus it can be said that the three operators (ncra, ncwa, and nces) all average coordinate variables (even though nces simply copies them). All other requested arithmetic operations (e.g., maximization, square-root, RMS) are applied only to non-coordinate variables. In these cases the linear average of the coordinate variable will be returned.


Consider a model experiment which generated five realizations of one year of data, say 1985. Imagine that the experimenter slightly perturbs the initial conditions of the problem before generating each new solution. Assume each file contains all twelve months (a seasonal cycle) of data and we want to produce a single file containing the ensemble average (mean) seasonal cycle. Here the numeric filename suffix denotes the realization number (not the month):

nces 85_0[1-5].nc
nces -n 5,2,1

These three commands produce identical answers. See Specifying Input Files, for an explanation of the distinctions between these methods. The output file,, is the same size as the inputs files. It contains 12 months of data (which might or might not be stored in the record dimension, depending on the input files), but each value in the output file is the average of the five values in the input files.

In the previous example, the user could have obtained the ensemble average values in a particular spatio-temporal region by adding a hyperslab argument to the command, e.g.,

nces -d time,0,2 -d lat,-23.5,23.5 85_??.nc

In this case the output file would contain only three slices of data in the time dimension. These three slices are the average of the first three slices from the input files. Additionally, only data inside the tropics is included.

As of NCO version 4.3.9 (released December, 2013) nces also works with groups (rather than files) as the fundamental unit of the ensemble. Consider two ensembles, /ecmwf and /cesm stored across three input files,, and Ensemble members would be leaf groups with names like /ecmwf/01, /ecmwf/02 etc. and /cesm/01, /cesm/02, etc. These commands average both ensembles:

nces --nsm_grp
nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1
nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1

The first command stores averages in the output groups /cesm and /ecmwf, while the second stores minima in the output groups /cesm/cesm_min and /ecmwf/ecmwf_min: The third command demonstrates that sub-setting and hyperslabbing work as expected. Note that each input file may contain different numbers of members of each ensemble, as long as all distinct ensembles contain at least one member in the first file.

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.7 ncflint netCDF File Interpolator


ncflint [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]
[--cnk_byt sz_byt] [--cnk_csh sz_byt] [--cnk_dmn nm,sz_lmn]
[--cnk_map map] [--cnk_min sz_byt] [--cnk_plc plc] [--cnk_scl sz_lmn]
[-D dbg] [-d dim,[min][,[max][,[stride]]] [--fl_fmt fl_fmt]
[-F] [--fix_rec_crd] [-G gpe_dsc] [-g grp[,…]] [--glb ...] [-h] [--hdr_pad nbr]
[-i var,val3] [-L dfl_lvl] [-l path] [-N]
[--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o file_3] [-p path] [--ppc ...] [-R] [-r] [--ram_all] 
[-t thr_nbr] [--unn] [-v var[,…]] [-w wgt1[,wgt2]] [-X ...] [-x]
file_1 file_2 [file_3]


ncflint creates an output file that is a linear combination of the input files. This linear combination is a weighted average, a normalized weighted average, or an interpolation of the input files. Coordinate variables are not acted upon in any case, they are simply copied from file_1.

There are two conceptually distinct methods of using ncflint. The first method is to specify the weight each input file contributes to the output file. In this method, the value val3 of a variable in the output file file_3 is determined from its values val1 and val2 in the two input files according to val3 = wgt1*val1 + wgt2*val2 . Here at least wgt1, and, optionally, wgt2, are specified on the command line with the ‘-w’ (or ‘--weight’ or ‘--wgt_var’) switch. If only wgt1 is specified then wgt2 is automatically computed as wgt2 = 1 - wgt1. Note that weights larger than 1 are allowed. Thus it is possible to specify wgt1 = 2 and wgt2 = -3. One can use this functionality to multiply all values in a given file by a constant.

As of NCO version 4.6.1 (July, 2016), the ‘-N’ switch (or long-option equivalents ‘--nrm’ or ‘--normalize’) implements a variation of this method. This switch instructs ncflint to internally normalize the two supplied (or one supplied and one inferred) weights so that wgt1 = wgt1/(wgt1 + wgt2 and wgt2 = wgt2/(wgt1 + wgt2 and . This allows the user to input integral weights, say, and to delegate the chore of normalizing them to ncflint. Be careful that ‘-N’ means what you think, since the same switch means something quite different in ncwa.

The second method of using ncflint is to specify the interpolation option with ‘-i (or with the ‘--ntp’ or ‘--interpolate’ long options). This is the inverse of the first method in the following sense: When the user specifies the weights directly, ncflint has no work to do besides multiplying the input values by their respective weights and adding together the results to produce the output values. It makes sense to use this when the weights are known a priori.

Another class of problems has the arrival value (i.e., val3) of a particular variable var known a priori. In this case, the implied weights can always be inferred by examining the values of var in the input files. This results in one equation in two unknowns, wgt1 and wgt2: val3 = wgt1*val1 + wgt2*val2 . Unique determination of the weights requires imposing the additional constraint of normalization on the weights: wgt1 + wgt2 = 1. Thus, to use the interpolation option, the user specifies var and val3 with the ‘-i’ option. ncflint then computes wgt1 and wgt2, and uses these weights on all variables to generate the output file. Although var may have any number of dimensions in the input files, it must represent a single, scalar value. Thus any dimensions associated with var must be degenerate, i.e., of size one.

If neither ‘-i’ nor ‘-w’ is specified on the command line, ncflint defaults to weighting each input file equally in the output file. This is equivalent to specifying ‘-w 0.5’ or ‘-w 0.5,0.5’. Attempting to specify both ‘-i’ and ‘-w’ methods in the same command is an error.

ncflint does not interpolate variables of type NC_CHAR and NC_STRING. This behavior is hardcoded.

By default ncflint interpolates or multiplies record coordinate variables (e.g., time is often stored as a record coordinate) not other coordinate variables (e.g., latitude and longitude). This is because ncflint is often used to time-interpolate between existing files, but is rarely used to spatially interpolate. Sometimes however, users wish to multiply entire files by a constant that does not multiply any coordinate variables. The ‘--fix_rec_crd’ switch was implemented for this purpose in NCO version 4.2.6 (March, 2013). It prevents ncflint from multiplying or interpolating any coordinate variables, including record coordinate variables.

Depending on your intuition, ncflint may treat missing values unexpectedly. Consider a point where the value in one input file, say val1, equals the missing value mss_val_1 and, at the same point, the corresponding value in the other input file val2 is not misssing (i.e., does not equal mss_val_2). There are three plausible answers, and this creates ambiguity.

Option one is to set val3 = mss_val_1. The rationale is that ncflint is, at heart, an interpolator and interpolation involving a missing value is intrinsically undefined. ncflint currently implements this behavior since it is the most conservative and least likely to lead to misinterpretation.

Option two is to output the weighted valid data point, i.e., val3 = wgt2*val2 . The rationale for this behavior is that interpolation is really a weighted average of known points, so ncflint should weight the valid point.

Option three is to return the unweighted valid point, i.e., val3 = val2. This behavior would appeal to those who use ncflint to estimate data using the closest available data. When a point is not bracketed by valid data on both sides, it is better to return the known datum than no datum at all.

The current implementation uses the first approach, Option one. If you have strong opinions on this matter, let us know, since we are willing to implement the other approaches as options if there is enough interest.


Although it has other uses, the interpolation feature was designed to interpolate file_3 to a time between existing files. Consider input files and containing variables describing the state of a physical system at times time = 85 and time = 87. Assume each file contains its timestamp in the scalar variable time. Then, to linearly interpolate to a file which describes the state of the system at time at time = 86, we would use

ncflint -i time,86

Say you have observational data covering January and April 1985 in two files named and, respectively. Then you can estimate the values for February and March by interpolating the existing data as follows. Combine and in a 2:1 ratio to make

ncflint -w 0.667
ncflint -w 0.667,0.333

Multiply by 3 and by -2 and add them together to make

ncflint -w 3,-2

This is an example of a null operation, so should be identical (within machine precision) to

Multiply all the variables except the coordinate variables in the file by by 0.8:

ncflint --fix_rec_crd -w 0.8,0.0

The use of ‘--fix_rec_crd’ ensures, e.g., that the time coordinate, if any, is not scaled (i.e., multiplied).

Add to to obtain, then subtract from to obtain

ncflint -w 1,1
ncflint -w 1,-1

Thus ncflint can be used to mimic some ncbo operations. However this is not a good idea in practice because ncflint does not broadcast (see ncbo netCDF Binary Operator) conforming variables during arithmetic. Thus the final two commands would produce identical results except that ncflint would fail if any variables needed to be broadcast.

Rescale the dimensional units of the surface pressure prs_sfc from Pascals to hectopascals (millibars)

ncflint -C -v prs_sfc -w 0.01,0.0
ncatted -a units,prs_sfc,o,c,millibar

Next: , Previous: , Up: Reference Manual   [Contents][Index]

4.8 ncks netCDF Kitchen Sink


ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [-b fl_bnr] [-C] [-c] [--cdl]
[--cnk_byt sz_byt] [--cnk_csh sz_byt] [--cnk_dmn nm,sz_lmn]
[--cnk_map map] [--cnk_min sz_byt] [--cnk_plc plc] [--cnk_scl sz_lmn]
[-D dbg] [-d dim,[min][,[max][,[stride]]]
[-F] [--fix_rec_dmn dim] [--fl_fmt fl_fmt] [--fmt_val format]
[-G gpe_dsc] [-g grp[,…]] [--glb ...] [--grp_xtr_var_xcl]
[-H] [-h] [--hdn] [--hdr_pad nbr] [--jsn] [--jsn_fmt lvl] 
[-L dfl_lvl] [-l path]
[-M] [-m] [--map map-file] [--md5] [--mk_rec_dmn dim]
[--no_blank] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] 
[-O] [-o output-file] [-P] [-p path] [--ppc ...] [--prn_fl print-file]
[-Q] [-q] [-R] [-r] [--rad] [--ram_all] [--rgr ...] [--rnr=wgt]
[-s format] [-u] [--unn] [-V] [-v var[,…]] [-X ...] [-x] [--xml]
input-file [[output-file]]


The nickname “kitchen sink” is a catch-all because ncks combines most features of ncdump and nccopy with extra features to extract, hyperslab, multi-slab, sub-set, and translate into one versatile utility. ncks extracts (a subset of the) data from input-file, regrids it according to map-file if specified, then writes in netCDF format to output-file, and optionally writes it in flat binary format to fl_bnr, and optionally prints it to screen.

ncks prints netCDF input data in ASCII, CDL, JSON, or NcML/XML text formats to stdout, like (an extended version of) ncdump. By default ncks prints CDL format. Option ‘-s’ (or long options ‘--sng_fmt’ and ‘--string’) permits the user to format data using C-style format strings, while option ‘--cdl’ outputs CDL, option ‘--jsn’ (or ‘json’) outputs JSON, option ‘--trd’ (or ‘traditional’) outputs “traditional” format, and option ‘--xml’ (or ‘ncml’) outputs NcML. The “traditional” tabular format is intended to be easy to search for the data you want, one datum per screen line, with all dimension subscripts and coordinate values (if any) preceding the datum. ncks exposes many flexible controls over printed output, including CDL, JSON, and NcML.

Options ‘-a’, ‘--cdl’, ‘-F’, ‘--fmt_val’, ‘-H’, ‘--hdn’, ‘--jsn’, ‘-M’, ‘-m’, ‘-P’, ‘--prn_fl’, ‘-Q’, ‘-q’, ‘-s’, ‘--trd’, ‘-u’, ‘-V’, and ‘--xml’ (and their long option counterparts) control the presence of data and metadata and their formatted location and appearance when printed.

ncks extracts (and optionally creates a new netCDF file comprised of) only selected variables from the input file (similar to the old ncextr specification). Only variables and coordinates may be specifically included or excluded—all global attributes and any attribute associated with an extracted variable are copied to the screen and/or output netCDF file. Options ‘-c’, ‘-C’, ‘-v’, and ‘-x’ (and their long option synonyms) control which variables are extracted.

ncks extracts hyperslabs from the specified variables (ncks implements the original nccut specification). Option ‘-d’ controls the hyperslab specification. Input dimensions that are not associated with any output variable do not appear in the output netCDF. This feature removes superfluous dimensions from netCDF files.

ncks will append variables and attributes from the input-file to output-file if output-file is a pre-existing netCDF file whose relevant dimensions conform to dimension sizes of input-file. The append features of ncks are intended to provide a rudimentary means of adding data from one netCDF file to another, conforming, netCDF file. If naming conflicts exist between the two files, data in output-file is usually overwritten by the corresponding data from input-file. Thus, when appending, the user should backup output-file in case valuable data are inadvertantly overwritten.

If output-file exists, the user will be queried whether to overwrite, append, or exit the ncks call completely. Choosing overwrite destroys the existing output-file and create an entirely new one from the output of the ncks call. Append has differing effects depending on the uniqueness of the variables and attributes output by ncks: If a variable or attribute extracted from input-file does not have a name conflict with the members of output-file then it will be added to output-file without overwriting any of the existing contents of output-file. In this case the relevant dimensions must agree (conform) between the two files; new dimensions are created in output-file as required. When a name conflict occurs, a global attribute from input-file will overwrite the corresponding global attribute from output-file. If the name conflict occurs for a non-record variable, then the dimensions and type of the variable (and of its coordinate dimensions, if any) must agree (conform) in both files. Then the variable values (and any coordinate dimension values) from input-file will overwrite the corresponding variable values (and coordinate dimension values, if any) in output-file 68.

Since there can only be one record dimension in a file, the record dimension must have the same name (though not necessarily the same size) in both files if a record dimension variable is to be appended. If the record dimensions are of differing sizes, the record dimension of output-file will become the greater of the two record dimension sizes, the record variable from input-file will overwrite any counterpart in output-file and fill values will be written to any gaps left in the rest of the record variables (I think). In all cases variable attributes in output-file are superseded by attributes of the same name from input-file, and left alone if there is no name conflict.

Some users may wish to avoid interactive ncks queries about whether to overwrite existing data. For example, batch scripts will fail if ncks does not receive responses to its queries. Options ‘-O’ and ‘-A’ are available to force overwriting existing files and variables, respectively.

Options specific to ncks

The following summarizes features unique to ncks. Features common to many operators are described in Shared features.


Switches ‘-a’, ‘--abc’, and ‘--alphabetizeturn-off the default alphbetization of extracted fields in ncks only. These switches are misleadingly named and were deprecated in ncks as of NCO version 4.7.1 (December, 2017).

This is the default behavior so these switches are no-ops included only for completeness. By default, NCO extracts, prints, and writes specified output variables to disk in alphabetical order. This tends to make long output lists easier to search for particular variables. Again, no option is necessary to write output in alphabetical order. Until NCO version 4.7.1 (December, 2017), ncks used the -a, --abc, or --alphabetize switches to turn-off the default alphabetization. These names were counter-intuitive and needlessly confusing. As of NCO version 4.7.1, ncks uses the new switches --no_abc, --no-abc, --no_alphabetize, or --no-alphabetize, all of which are equivalent. The --abc and --alphabetize switches are now no-ops, i.e., they write the output in the unsorted order of the input. The -a switch is now completely deprecated in favor of the clearer long option switches.

-b file

Activate native machine binary output writing to binary file file. Also ‘--fl_bnr’ and ‘--binary-file’. Writing packed variables in binary format is not supported. Metadata is never output to the binary file. Examine the netCDF output file to see the variables in the binary file. Use the ‘-C’ switch, if necessary, to avoid wanting unwanted coordinates to the binary file:

% ncks -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data
% ls -l bnr.dat | cut -d ' ' -f 5 # 200 B contains time and one_dmn_rec_var
% ls -l bnr.dat
% ncks -C -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data
% ls -l bnr.dat | cut -d ' ' -f # 40 B contains one_dmn_rec_var only

As of NCO version 4.6.5 (March, 2017), ncks can print human-legible calendar strings corresponding to time values with UDUnits-compatible date units of the form time-since-basetime, e.g., ‘days since 2000-01-01’ and a CF calendar attribute, if any. Enact this with the ‘--calendar’ (also ‘--cln’, ‘prn_lgb’, and ‘datestamp’) option when printing in any mode. Invoking this option when dbg_lvl >= 1 in CDL mode prints both the value and the calendar string (one in comments):

zender@aerosol:~$ ncks -D 1 --cal -v tm_365 ~/nco/data/
    double tm_365 ;
      tm_365:units = "days since 2013-01-01" ; // char
      tm_365:calendar = "365_day" ; // char

    tm_365 = "2013-03-01"; // double value: 59
zender@aerosol:~$ ncks -D 1 -v tm_365 ~/nco/data/
    tm_365 = 59; // calendar format: "2013-03-01"

This option is similar to the ncdump-t’ option. As of NCO version 4.6.8 (August, 2017), ncks CDL printing supports finer-grained control of date formats with the ‘--dt_fmt=dt_fmt’ (or ‘--date_format’) option. The dt_fmt is an enumerated integer from 0–3. Values dt_fmt=0 or 1 correspond to the short format for dates that are the default. The value dt_fmt=2 requests the “regular” format for dates, dt_fmt=3 requests the full ISO-8601 format with the “T” separator:

ncks -H -m -v time_bnds -C --dt_fmt=value ~/nco/data/
# Value:    Output:
# 0,1       1964-03-13 09:08:16        # Default, short format
# 2         1964-03-13 09:08:16.000000 # Regular format
# 3         1964-03-13T09:08:16.000000 # ISO8601 'T' format

Note that ‘--dt_fmt’ automatically implies ‘--cal’ makes that options superfluous.


Change record dimension dim in the input file into a fixed dimension in the output file. Also ‘--no_rec_dmn’. Before NCO version 4.2.5 (January, 2013), the syntax for --fix_rec_dmn did not permit or require the specification of the dimension name dim. This is because the feature only worked on netCDF3 files, which support only one record dimension, so specifying its name was not necessary. netCDF4 files allow an arbitrary number of record dimensions, so the user must specify which record dimension to fix. The decision was made that starting with NCO version 4.2.5 (January, 2013), it is always required to specify the dimension name to fix regardless of the netCDF file type. This keeps the code simple, and is symmetric with the syntax for --mk_rec_dmn, described next.

As of NCO version 4.4.0 (January, 2014), the argument all may be given to ‘--fix_rec_dmn’ to convert all record dimensions to fixed dimensions in the output file. Previously, ‘--fix_rec_dmn’ only allowed one option, the name of a single record dimension to be fixed. Now it is simple to simultaneously fix all record dimensions. This is useful (and nearly mandatory) when flattening netCDF4 files that have multiple record dimensions per group into netCDF3 files (which are limited to at most one record dimension) (see Group Path Editing).


As of NCO version 4.4.0 (January, 2014), the ‘--hdn’ or ‘--hidden’ options print hidden (aka special) attributes. This is equivalent to ‘ncdump -s’. Hidden attributes include: _Format, _DeflateLevel, _Shuffle, _Storage, _ChunkSizes, _Endianness, _Fletcher32, and _NOFILL. Previously ncks ignored all these attributes in CDL/XML modes. Now it prints these attributes as appropriate in all modes. As of NCO version 4.4.6 (September, 2014), ‘--hdn’ also prints the extended file format (i.e., the format of the file or server supplying the data) as _SOURCE_FORMAT. As of NCO version 4.6.1 (August, 2016), ‘--hdn’ also prints the hidden attributes _NCProperties, _IsNetcdf4, and _SuperblockVersion for netCDF4 files so long as NCO is linked against netCDF library version 4.4.1 or later. Users are referred to the Unidata netCDF Documentation, or the man pages for ncgen or ncdump, for detailed descriptions of the meanings of these hidden attributes.


As of NCO version 4.3.3 (July, 2013), ncks can print extracted data and metadata to screen (i.e., stdout) as valid CDL (network Common data form Description Language). CDL is the human-readable “lingua franca” of netCDF ingested by ncgen and excreted by ncdump. As of NCO version 4.6.9 (September, 2017), ncks prints CDL by default, and the “traditional” mode must be explicitly selected with ‘--trd’. Compare ncks “traditional” with CDL printing:

zender@roulee:~$ ncks --trd -v one ~/nco/data/
one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
one attribute 0: long_name, size = 3 NC_CHAR, value = one

one = 1 

zender@roulee:~$ ncks --cdl -v one ~/nco/data/
netcdf in {

    float one ;
    one:long_name = "one" ;

    one = 1 ;

} // group /

Users should note the NCO’s CDL mode outputs successively more verbose additional diagnostic information in CDL comments as the level of debugging increases from zero to two. For example printing the above with ‘-D 2’ yields

zender@roulee:~$ ncks -D 2 --cdl -v one ~/nco/data/
netcdf in {
  // ncgen -k classic -b -o in.cdl

    float one ; // RAM size = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes, ID = 147
      one:long_name = "one" ; // char

    one = 1 ; 

} // group /

ncgen converts CDL-mode output into a netCDF file:

ncks -v one ~/nco/data/ > ~/in.cdl
ncgen -k netCDF-4 -b -o ~/ ~/in.cdl
ncks -v one ~/

The HDF version of ncgen, often named hncgen, h4_ncgen, or ncgen-hdf, converts netCDF3 CDL into an HDF file:

/usr/hdf4/bin/ncgen -b -o ~/in.hdf ~/in.cdl # HDF ncgen (local builds)
/usr/bin/hncgen     -b -o ~/in.hdf ~/in.cdl # Same as HDF ncgen (RPM packages?)
/usr/bin/h4_ncgen   -b -o ~/in.hdf ~/in.cdl # Same as HDF ncgen (Anaconda)
/usr/bin/ncgen-hdf  -b -o ~/in.hdf ~/in.cdl # Same as HDF ncgen (Debian packages?)
hdp dumpsds ~/in.hdf                        # ncdump/h5dump-equivalent for HDF4
h4_ncdump dumpsds ~/in.hdf                  # ncdump/h5dump-equivalent for HDF4

Note tha