NCO News and Announcements from 20030624 (version 2.8.3) and earlier

2003/06/24: A potpurri of old news which is no longer needed on the homepage

• As of version 2.8.3, ncap consumes significantly less memory than before.
• New “binary” operator ncbo does file addition, subtraction, multiplication, and division
• ncap adds user-invokable type conversion functions char(), short(), int(), float(), double().
• libnco_c++ reads into and writes from type long double
• Developer code has fully integrated autoconf/libtool build procedure
• NCO allows ncap to unpack variables stored with scale_factor and add_offset convention
• NCO includes ncap, a scriptable operator capable of performing arbitrary arithmetic operations
• NCO code includes libnco_c++, a new C++ interface to netCDF
• NCO now supports multithreading with OpenMP-compliant compilers
• NCO code contains working packing and unpacking primitives

2002/02/03: NCO version 2.2 is released. Differences with version 2.0

• ncap, the netCDF Arithmetic Processor
• libnco_c++, a C++ API for libnetcdf
• Various minor bugfixes introduced in 2.X releases

2001/05/07: NCO version 2.0 is released. Differences with version 1.2.2 are completely internal and oriented toward enhancing I/O performance and modernizing the NCO API:

• NCO 2.x is now completely based on the netCDF 3.x API. NCO 1.2.x will continue to be maintained for its ability to work with HDF4 files, but no new features will be added.

2001/02/11: NCO version 1.2.2 is released. Differences with version 1.2.1 include three new features and one bugfix:

• Weights are allowed with ncwa in min/max/ttl operations. New "var:att" syntax allows ncrename to rename attributes for specific variables rather than whole file, e.g.,
  

  ncrename -a os_name:eunuchs,unix in.nc out.nc

  Previously, ncrename would rename the same attribute for all variables in a file.
• ncks has new -q option to allow for printing arrays without printing associated dimensions
• The build system has been modified so that make HDF4=Y will build NCO with HDF4 compatibility (previously this had to be done by hand).
• The internal representations of NC_CHAR and NC_BYTE data have been swapped in order to agree with the netCDF standard. The effects of this last change should not be visible to the user.

2000/08/28: NCO version 1.2.1 is released. Differences with version 1.2 include one new feature and one bugfix.

1. Operators now perform automatic type conversion of integers to doubles before arithmetic. This feature allows ncwa, ncra, and ncea to handle averaging of longs, shorts, bytes, and chars in a more satisfactory manner. One immediate benefit is that ncwa no longer complains when user's try to apply weights (e.g., gaussian weights) all less than one to the averaging of an integer (e.g., nlons). Formerly ncwa would issue a warning, a workaround, and then dump core (because weights converted to integers were all zeros). Now that arithmetic is performed in double precision (rather than in the precision of the variable), ncwa handles this situation correctly.

2. A bug introduced in NCO 1.2 which broke the -x switch has been fixed

3. Thanks to Henry Butowsky for implementing the type conversion feature
Thanks to Ethan Davis for improving the build procedure

2000/07/30: nco-1.2 is released. Summary of new features and user-visible changes:

• The names of three of the operators have been made somewhat obsolete because they are no longer just arithmetic averagers. ncra, ncea, and ncwa have all been given brain transplants so that the type of operation they perform may be specified on the command line. The following operations are now supported by the new -y switch: average (default), total, minimum, maximum, root-mean-square, square of the mean, mean of the squares, square root. For example,

  ncra -O -y rms -n 12,4,1 h0001.nc 2000_rms.nc # Root-mean-square
  ncea -O -y min -n 12,4,1 h0001.nc 2000_min.nc # Minimum
  ncwa -O -y max -a lon,lat h0001.nc 200001_max_x.nc # Maxima
  

  Averaging is still the default so no existing usage needs to be modified. It is also possible to compute spatial and temporal variances and standard deviations using a combination of three or four NCO commands.
• All operators may now be DODS clients. DODS enables network transparent data access to any DODS server. Thus DODS-enabled NCO can operate on remote files accessible through any DODS server without transferring the files. Only the required data (e.g., the variable or hyperslab specified) are transferred. See the Users Guide for complete details.
• Fortran compatibility has been dropped. If a volunteer wants to continue to maintain the option to perform arithmetic in Fortran for reasons of speed, contact the project.

• NCO is now an Open Source project licensed under the GPL. These new capabilities are the result of contributions from Henry Butowski, Bill Kocik, and Roland Schweitzer. For more information on how to contribute to NCO, visit the NCO administration homepage.
• New NCO releases and important announcements will be made on the netCDF list and on the new nco-announce mailing list.
• Questions, suggestions, ideas, and pleas for help should be posted to the appropriate discussion forum at the NCO administration site. NCO developers monitor these open forums, and your message will get a quicker response if it is posted on one of these forums:
  • Open Discussion Forum: General discussion, discussion of tips, tricks, future directions for NCO.
  • Help Forum: Installation problems, user's questions (e.g., "How do I average x and y...?")
  • Developers forum: Discussion of patches, technical questions related to modifying NCO, and discussion of specific items on the TODO list and how best to address them

2000/05/15: nco-1.1.48 fixes a bug introduced in nco-1.1.15 (November 1998) with ncdiff version 1.5. That bug could cause incorrect answers when the broadcast feature was used and dimensions were defined in a certain order (that appears to be favored by oceanographers). All NCO users are urged to upgrade NCO in order to avoid the potential ncdiff bug as well as to take advantage of the new features in ncra and ncrcat.

2000/05/10: nco-1.1.47 adds capability of ncra, and ncrcat to handle superfluous initial files. These multifile operators used to require that hyperslabs along the record dimension contain valid points in the first input file, although superfluous trailing files were allowed. Now superfluous files are also allowed anywhere in the file argument list for coordinate value hyperslabs (e.g., -d time,2.0,3.0), and at the beginning and end of the file argument list for dimension index hyperslabs (e.g., -d time,2,3). This can dramatically simplify analysis of file collections when the location of certain record coordinates or indices is not known a priori, or is tedious to compute. Users may now operate on specific segments of a timeseries (e.g., February values) without knowing which files contain February. Simply input the entire timeseries of files to the operators and they will automagically find and operate on only the February values. For example,

ncra -O -d time,2.0,3.0 -n 1000,4,1 h0001.nc feb.nc

ncrcat -O -F -d time,2,,12 -n 1000,4,1 h0037.nc feb.nc

Sylvia Murphy of the NCAR Community Climate System Model Support Network organized a workshop in February, 2000 which devoted some time to NCO. Here are links to the very useful pamphlet and slide presentation she developed to introduce and to summarize the NCO operators. The pamphlet is an excellent primer on NCO's capabilities, though not as definitive as the Users Guide.

2000/03/05: Source and prebuilt binary executable RPMs for NCO are now available for download.

2000/01/26: Fixed longstanding NCO bug which caused infrequent but unpredictable core dumps on machines with PIDs longer than 6 digits, e.g., all recent SGI machines. A core dump could occur on these machines when the PID was seven digits long and the filename length was one less than a multiple of eight. Kudos to Juliana Rew and Brian Eaton for identifying this bug. It is recommended that all users upgrade to the the new version. NCO is now distributed under the terms of the GNU General Public License (GPL).

1999/12/26: Added support for printing character arrays as strings in ncks. Added documentation and examples of stride capability to ncks, ncra, and ncrcat portions of Users Guide.

1999/12/14: Added feature to ncatted so that C-language escape sequences may be used in editing character attributes. Text formatting of attributes now supports \n, \t, and most other printf-style sequences. See the Users Guide for details.

1999/12/06: Fixed bug in ncatted so that character-valued attributes (strings) may contain commas, and be of zero length. Formerly ncatted would drop all text after the first comma.

1999/10/04: The build configuration has been simplified and made more robust. Now typing make test will invoke a self-test where NCO attempts to download a small (~65 kb) file on which to perform a battery of NCO tests. If the equations output by the make test command are numerically accurate, then you can have high confidence that NCO is installed and working correctly.

1999/08/31: A bug involving ncwa, weights, and missing values has been identified and fixed. When ncwa was used to average variables with user-specified weights (i.e., -w wgt was used) and every value in a target hyperslab was the missing value (i.e., missing_value), then ncwa would erroneously set the average value of the target hyperslab to 0.0 rather than to missing_value. Thus some values which should have been missing_value in the output file were, instead, set to 0.0. No operators besides ncwa were affected by this bug. This bug is fixed in nco-1.1.38. A new feature has been implemented in ncwa. ncwa used to exit with an error whenever it was called without any user-specified dimensions to average, i.e., when the -a was not invoked. Now when ncwa is invoked without any user-specified dimensions, it averages over all dimensions. Thus the behavior of -a is now analogous to that of -d and of -v. See the Users Guide for details.

1999/08/29: NCO now builds completely with pure C source files by default. In the past, NCO used Fortran routines to perform floating point arithmetic and pure C builds could be manually enabled by specifying -DC_ONLY in the build environment. The interface between Fortran and C, however, was very difficult to maintain on the wide variety of platforms supported by NCO. Now pure C builds are the default and Fortran arithmetic must be manually enabled by specifying -DUSE_FORTRAN_ARITHMETIC in the build environment. See the Users Guide for details. The Fortran code is no longer supported and will probably be deprecated in future releases.

1999/08/05: NCAR users will be pleased to know that NCO now takes advantage of the local msrcp command, if it exists in a standard location on the local network. This allows all operators to retrieve files from the NCAR mass storage unit when msrcp is available on the local system. This change is transparent to the user, so that commands should still specify the paths to the files on the mass store with, e.g., ncks -O -R -l ./ /ZENDER/nc/in.nc foo.nc. See the Users Guide for more details.

1999/07/03: ncatted now replaces missing values of variables whose missing_value attribute is changed. When the internal floating point representation of a missing value, e.g., 1.0e36, differs between machines then netCDF files produced on those machines will have incompatible missing values. ncatted now changes the missing_value attribute and the missing data self-consistently. This allows the user to change the missing values in files from different machines to a single value so that the files may then be concatenated together, e.g., by ncrcat, without losing any information.

1999/05/09: ncks now alphabetizes output by default. To turn off alphabetization and return to the previous behavior (variables reported in the order they were written to disk) simply specify `-a' on the command line. Yes, that means the behavior of `-a' has been toggled.

ncrcat and ncra now support a stride in the record dimension (and only the record dimension!), e.g., ncra -d time,1,100,12 in1.nc in2.nc ... out.nc should now work correctly. Strides in any dimension remain available with ncks, of course.

ncrcat and ncra now support index-based hyperslabbing in the record dimension across files, e.g., ncra -d time,1,100 in1.nc in2.nc ... out.nc The Users Guide claimed this feature had already been implemented, but, due to an oversight, that was not true. Attempting to use this new feature in older versions of NCO resulted in an "index out of range" error. Thanks to John Truesdale for pointing this out.

1999/04/20: Fixed bug where ncrcat and ncra omitted the last slice of the record dimension when `-F' (Fortran indexing) was user-specified AND user-specified hyperslab information was provided for some dimensions BUT not for the record dimension. This bug only affected ncrcat and ncra and only under these conditions. Thanks to John Sheldon for pointing this out.

1999/04/04: By default, extracted variables appear in the same order in which they were written to the source netCDF file. However, this order is often essentially random, making it hard to locate a particular variable in a dump of many variables. To help address this, ncks will now alphabetize variables. Simply specify `-a' on the command line.

1999/01/21: An optional token has been added to NCO which eliminates all netCDF 3.x calls and thus restores compatibility of NCO with HDF files.

1998/12/02: ncwa has been updated again since a user discovered a bug which occurred when a weight (`-w') and a mask (`-m') were both given to ncwa. Unfortunately, ncwa sometimes produced incorrect answers in this scenario so if you ever used weights and masks at the same time you are strongly urged to upgrade your NCO and rerun the commands. This bug only affected variables averaged with both the weight and mask options enabled at the same time, no other variables would have been affected.

The `-n' and `-W' normalization options to ncwa have been removed, possibly permanently, but at least until their implementation is simplified and redesigned. The `-N' option to ncwa remains the same. All the new behavior is fully documented in the rewritten and expanded ncwa section in the NCO Users Guide.

1998/11/24: Choose whether to weight your coordinates in ncwa A `-I' switch has been added to ncwa. Henforth, by default, ncwa applies weights or masks to coordinate variables just like any other variable. Previously, ncwa did not weight coordinate variables at all, and always produced simple arithmetic averages of coordinate variables. Use `ncwa -I' to recover the old behavior (i.e., arithmetic rather than masked, weighted averages of coordinate variables). A problem where ncwa would refuse to average variables which did not contain any of the averaged dimensions has also been fixed.

1998/07/07: Reduce bloating! The `-h' switch has been enabled on all operators. Using this switch suppresses the automatic history feature in NCO. Now you can decide whether each change to a file warrants an additional line in the history attribute. The functionality of the old `ncks -h' switch is now invoked with `ncks -m' (the mnemonic is `metadata').