$Header$ -*-text-*-

netCDF Operators NCO version 5.4.0 set sail

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco/nco (Source Code, Issues, Releases)

What's new?
Version 5.4.0 contains a raft of features and fixes.
Some notable features:
All operators allow setting the netCDF4 metadata block size.
Many operators now optionally preserve orphaned dimensions.
ncclimo can now climatologize simulation ensembles.
ncremap tolerates absent hybrid p0/P0 variables.

Users interested in these features should upgrade to 5.4.0. 

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncclimo now generates ensemble monthly climatologies.
Simply feed the files from all ensemble members to ncclimo
using any supported input method. Redirecting standard input
is often most concise. Consider three ensemble members:

# Example: Three Member Decadal-mean Ensemble Climatology
drc_top=/big/doe/computer/E3SMv3
caseid1=v3.LR.historical_0051
caseid2=v3.LR.historical_0101
caseid3=v3.LR.historical_0151
fl_nsm1=${drc_top}/${caseid1}/run/${caseid1}.elm.h0.185?-??.nc
fl_nsm2=${drc_top}/${caseid2}/run/${caseid2}.elm.h0.185?-??.nc
fl_nsm3=${drc_top}/${caseid3}/run/${caseid3}.elm.h0.185?-??.nc
ls ${fl_nsm1} ${fl_nsm2} ${fl_nsm3} | ncclimo --fml_nm=ensemble ...

This approach scales to any number of ensemble members.
All the usual climo-mode options, such as regridding, work.
For N-member ensembles, ncclimo averages N-times
as many months as a single member climo to get the mean
January, February, etc. This takes about N-times longer.
Seasonal and annual means are computed from monthly means so
their speed is unchanged, as is regridding speed.
For now, the output statistics are climatological means.
Future work could support outputting standard deviations.
Let us know what you would like to see in future versions.

B. ncclimo --mth_end now defaults to 11 months after --mth_srt.
Formerly, users specifying a non-January start month (--mth_srt)
were required to also explicitly provide --mth_end, even though in
climatology mode the end month must always be 11 months after the
start month.  Now --mth_end is automatically set to mth_srt+11 (mod
12) when it is not specified by the user.  For example, specifying
--mth_srt=5 now automatically sets --mth_end=4 (May through April
annual cycle).  Explicitly-specified --mth_end values are still
accepted and take precedence.  In timeseries mode, --mth_end may
still be explicitly specified when the range of months spans a
non-integral number of years. Alternatively, one can now avoid
the --mth_srt and --mth_end options entirely even with non-January
start months. Do this by specifying the years in YYYYMM format:

ncclimo -s 2013 -e 2014 --mth_srt=5 ... # mth_end automatically set to 4
ncclimo -s 2013 -e 2014 --mth_srt=5 --mth_end=4 ... # same, explicit
ncclimo -s 201305 -e 201404 ... # Omit --mth_srt/end options.

http://nco.sf.net/nco.html#yr_srt
http://nco.sf.net/nco.html#mth_srt
http://nco.sf.net/nco.html#mth_end

C. The --rad (retain_all_dimensions) option is now supported by many
data operators: ncap2, ncecat, nces, ncflint, ncpdq, and ncwa.
Previously only ncks supported --rad. When --rad is specified,
dimensions that are not referenced by any extracted variable (orphaned
dimensions) are retained in the output file rather than silently
dropped. The E3SM ELM model, for example, uses orphan dimensions to
trigger different physics treatments. When ncks is invoked in printing
mode, --rad also causes orphaned dimensions to be printed
(they are not printed by default). Synonyms --retain_all_dimensions,
--orphan_dimensions, and --rph_dmn are also accepted. Thanks to Jon
Wolfe (LANL) for prompting this feature.

ncap2 --rad -s 'one=1' in.nc out.nc # Keep all dimensions
nces --rad in.nc out.nc # Keep all dimensions
ncks --rad -m -v T in.nc # Print all dimensions

http://nco.sf.net/nco.html#rad

D. All data operators now support the newish netCDF ability
to set the metadata block size in netCDF4 files. Option --blk_sz
allows users to specify sizes other than the default 2048 bytes.
Large files chunked files often require much more space because
the metadata block is where HDF5 stores the location of every
chunk. If this list grows too long it must point to another
metadata block and HDF5 must follow the trail, possibly through
thousands of metadata blocks, just to find a single chunk.
CMIP7 submissions will need to take advantage of larger sizes.
One megabyte is a good size for large, chunked, netCDF4 files.
Setting blk_sz=0 will reset the block size to the default value
of the active netCDF4/HDF5 library (which may increase beyond
2048 B in the future). This option only affects HDF5 files.

ncks -4 -O ~/nco/data/in.nc ~/foo.nc # Default 2048 B
ncks -4 -O --blk=0 ~/nco/data/in.nc ~/foo.nc # Default
ncks -4 -O --blk=1048576 ~/nco/data/in.nc ~/foo.nc # 1 MB

http://nco.sf.net/nco.html#blk_sz

E. ncremap now gracefully tolerates the absence of the reference
surface pressure variable in hybrid sigma-pressure coordinate
datasets. Previouly these variables (usually named p0 or P0) had to be
contained in the the vertical grid of the input data. However, the
de facto convention is that p0=P0=100000 Pa. Now when the reference
pressure is missing (as is the case with older versions of EAMxx),
ncremap supplies this value internally and completes without failing.

F. ncremap now supports regridding MODIS MOD10CM datasets.
The dimensions in those dataset have been added to ncremap's
internal database so explicitly specifying which is latitude
and which is longitude is unnecessary.

G. ncclimo and ncremap now use the srun "--exclusive" option
to prevent compute nodes from sharing their resources with
concurrently running others jobs. This been shown to substantially
increase throughput of some NCO-based analysis workflows such as
LIVVkit. Thanks to Mike Kelleher (ORNL) for suggesting this.

H. In ncremap the --alg option is now a synonym for --alg_typ.
Just a small change to reduce typing.

I. ncremap and ncclimo have been significantly refactored.
They now run in "set -eou" mode and use modernized expression
evaluation and test constructs that fail on any Bash-reported errors.
This makes debugging easier. Both codes have now been tested with
the most common use-cases. However, we might have missed some
"unbound variables". Please send us an error report if you notice
this. The workaround is to run with an earlier version of NCO.

BUG FIXES:

A. Previous versions of the BitGroom algorithm could produce
incorrect answers when single-precision variables were
quantized with NSD=7 significant digits. The maximum value
now accepted is NSD=6. Thanks to Ed Hartnett for catching this.

B. Certain combinations of climo-mode and input-mode could
cause the CASEID to be prepended to filenames twice.
The workaround is to use a different input method.
The solution is to upgrade.

Full release statement at http://nco.sf.net/ANNOUNCE
    
KNOWN PROBLEMS DUE TO NCO:

This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems. 
These problems occur with NCO 5.4.0 built/tested under
MacOS 26.5.2 with netCDF 4.10.2-development on HDF5 2.2.0.
and with Linux FC42 with netCDF 4.9.2 on HDF5 1.14.4.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr