Known Problems through 2012 (version 4.2.3)
Older Generic Run-time Problems:
- MM3 slowdown:
A longstanding “feature” of netCDF3 was identified in
March, 2012, and is now known by the tag MM3.
The MM3 issue can lead to unusually slow performance.
The problem is triggered by an aggregate pattern of file access so the
workaround must be implemented in the application software (e.g., NCO)
rather than in the netCDF library itself.
The name MM3 fits because the problem is normally encountered on
Multi-record Multi-variable netCDF3 files.
And we call our “solution” the MM3-workaround.
If you encounter unusually slow NCO performance while using NCO to
analyze MM3 files on a large blocksize filesystem,
chances are you are encountering an MM3-induced slowdown.
NCO release 4.1.0 implements the MM3-workaround for ncks.
It speeds-up common ncks sub-setting on NCAR's GLADE by 10-50x.
MM3-induced slowdowns are present in other NCO operators and we are
prioritizing our MM3-patches to those encountered most often.
Thanks to Gary Strand for reporting this problem, and to Russ Rew for
creating the workaround algorithm, which is also now in nccopy.
- NOFILL bug:
All netCDF versions prior to 4.1.3 may create corrupt netCDF3 files
when linked to any version of NCO except 4.0.8.
The solution is to install netCDF version 4.1.3 or later.
The corruption occurs silently (without warning or error messages).
The problem has been seen "in the wild" only on filesystems with
large block sizes (e.g., Lustre), although it may be more widespread.
It is caused by a netCDF bug that NCO triggers by invoking
NOFILL mode for faster writes. Hence it is called the NOFILL bug.
The bug is hard to trigger, it depends on a rare interaction of
filesystem block-size, hyperslab size, and order-of-variable writing.
The bug exists in all versions of netCDF through 4.1.2.
If you have a large block filesystem and cannot upgrade your netCDF
library, then use NCO version 4.0.8, which disables NOFILL mode (and
thus writes files more slowly).
NCO 4.0.8 and will workaround the NOFILL bug on all versions of netCDF
(i.e., 4.1.2 and earlier).
Hence NCO 4.0.8 will always correctly write netCDF3 files.
Other temporary workarounds include creating only netCDF4 files
(e.g., ncks -4 ...) instead of netCDF3 files.
The NOFILL patch included in NCO 4.0.8 was subsequently removed
in NCO 4.0.9, which assumes that netCDF 4.1.3 or later is installed.
- Degenerate hyperslabbing bug:
Versions ???—4.0.6 could return incorrect hyperslabs when
user-specified hyperslabs did not include at least one point.
In such cases, instead of returning no data, hyperslabs could return all data.
To determine whether your NCO is affected by this bug, run these commands:
ncks -O -v lat -d lat,20.,20.001 ~/nco/data/in.nc ~/foo.nc;ncks -H ~/foo.nc
If the returned hyperslab contains any data, then your NCO is buggy
(because that hyperslab should be empty).
This can lead to incorrect answers for hyperslabs that should be empty.
Analogous problems would occur with empty auxiliary coordinate bounding boxes.
Although most users do not specify empty hyperslabs, we urge all users
to upgrade to NCO 4.0.7+ just to be safe.
- Threading problems with MSA:
NCO version 3.9.5 has a nasty bug that causes threaded arithmetic
operators, e.g., nces to produce incorrect results under some
The problem may occur whenever OpenMP is enabled and the operators
run on a multi-core CPU with more than one thread.
These incorrect answers, if generated, are relatively easy to notice.
The number of threads used to generate a file is, by default, recorded
in the global attribute nco_openmp_thread_number which may
be examined with ncks -M foo.nc | grep nco_openmp_thread_number.
The only action that will correct a file that you think (or know)
contains corrupted data because of this NCO bug is to re-process the
file with a non-buggy NCO version.
Version 3.9.5 is buggy and should be upgraded ASAP.
Be careful with data processed using this NCO version on multi-core CPUs.
The (one-line!) patch to fix this bug in 3.9.5 is
- Index-based hyperslab problems:
NCO versions 2.7.3—2.8.3 have a nasty bug that causes
index-based hyperslabs, e.g., -d lat,1, to
behave like value-based hyperslabs, e.g., -d lat,1.0 under
Unfortunately, the incorrect answers generated may be hard to notice!
This problem was most often enountered by users trying to assemble
monthly averages using the stride feature of ncrcat.
One common symptom is that the time-offset of the output file is
Versions 2.7.3—2.8.3 are buggy and should be upgraded ASAP.
Re-do any data-processing that used index-based hyperslabbing with
these versions of NCO.
Older Operator-specific Run-time Problems:
- ncks bug with auxiliary coordinates:
Versions 4.2.x–4.3.1 of ncks did not correctly
support auxiliary coordinates (specified with -X).
Auxiliary coordinates continued to work with the other hyperslabbing
NCO operators. Auxiliary coordinates once again work in all
hyperslabbing operators, including on netCDF4 group files in operators
that support them.
Fixed in version 4.3.2.
- ncatted bug on implicit attribute names:
Versions 4.2.x–4.3.0 of ncatted could segfault when
processing attributes specified implicitly (i.e., by leaving the
attribute field blank in the -a specification.
Fixed in version 4.3.1.
- ncbo bug handling certain special variables:
Version 4.3.0 of ncbo inadvertently always turns off
to variable list processing.
This may cause some grid-related variables (e.g., ntrm and nbdate)
and some non-grid variables (e.g., ORO and gw) to be
arithmetically processed (e.g., subtracted) even when that makes no
sense in most climate model datasets.
Fixed in version 4.3.1.
- ncks bug copying metadata:
Version 4.2.6 of ncks does not copy variable metadata by default.
Thus output files appear stripped of metadata.
One can work around this problem in 4.2.6 by specifying the -m option.
Otherwise an upgrade is recommended.
Fixed in version 4.3.0.
- ncks bug subsetting variables:
Version 4.2.4 of ncks sometimes dumps core
when subsetting variables with -v var.
Fixed in version 4.2.5.
- ncks bug with altering record dimensions:
Version 4.2.4 of ncks ignored both the
--mk_rec_dmn and the --fix_rec_dmn switches.
It exited successfully without altering the record variable.
Fixed in version 4.2.5.
- nces bug with non-record files:
Versions 4.2.1—4.2.3 of nces incorrectly referenced
the record variable on files which do not contain it.
This caused a segmentation violation and core dump.
- ncra bug when last file(s) is/are superfluous:
Versions 4.2.1—4.2.3 of ncra incorrectly skipped
writing the results of the final normalization when trailing files
were superfluous (not used).
In the most common case, all values are zeros in the output file.
Upgrade if you call ncra with trailing superfluous files.
- ncecat bug when files generated with -n:
Version 4.2.2 of ncecat could incorrectly skip the first
input file in the default mode (RECORD_AGGREGATE) when
the -n NINTAP switch is used to automate filename generation.
Upgrade if you use ncecat -n.
- ncra bug handling CF coordinates attributes
that contain the name of the record coordinate:
Versions 4.0.3—4.0.4 of ncra incorrectly treat the
record variable (usually time) as a fixed variable if it
is specified in the coordinates attribute of any variable in
a file processed with CCM/CCSM/CF metadata conventions.
This bug caused core dumps, and even weirder behavior like
creating imaginary time slices in the ouput.
Upgrade recommended if you work with NCAR CCSM/CESM model output.
One workaround that does not require NCO upgrades is to remove the
record coordinate name (usually time) from
the coordinates attribute of all variables in CF-compliant
files before processing the file with ncra.
- ncra bug averaging YYYYMMDD-format date
variables in CCSM/CF-compliant files:
Versions ???—4.0.5 of ncra contain a bug which
produces an incorrect average (usually zero) of the date
variable which many CCSM/CF-compliant files use to track model dates
in the human-readable YYYYMMDD-format.
Averaging YYYYMMDD-format integers is intrinsically difficult, since
such dates have calendar assumptions built-in.
NCO attempts this in CCSM/CF-compliant files by using the
nbdate (beginning date) and time (days
since nbdate) variables to find the average date,
converting that to YYYYMMDD, and writing that as the average value
- ncks bug hyperslabbinging fixed netCDF4 dimensions:
Versions 4.0.3—4.0.4 of ncks contain a bug which
triggers a core-dump when hyperslabbing (along a non-record
dimension) a netCDF4-format input file into a netCDF4-format output
file, e.g., ncks -d 0,1,lat in4.nc out4.nc.
Three workarounds that do not require NCO upgrades (or downgrades) are
to explicitly specify chunking with, e.g.,
ncks --cnk_plc=all -d 0,1,lat in4.nc out4.nc, or, to use
nces instead of ncks for hyperslabbing, e.g.,
nces -d 0,1,lat in4.nc out4.nc (nces does a no-op
when there is only one input file), or to write to a netCDF3 file,
ncks -3 -d 0,1,lat in4.nc out3.nc.
- Core dump with ncks:
Printing variables to screen with ncks can trigger a segfault
in NCO 3.9.9—4.0.3.
Users may upgrade, downgrade, or apply this one-line patch to 3.9.9 sources:
Remove this line
—near line 751 of nco/src/nco/nco_netcdf.c—
should fix the problem.
The problem in later NCO versions is due to a different bug and this
patch will not work.
- ncrename erroneous error exit:
Versions 4.0.1—4.0.3 of ncrename contain a bug where
commands like ncrename -a .old_nm,new_nm in.nc out.nc
would, if old_nm did not exist, write the correct file and
then exit with an error message although no error had occurred.
The files written were fine, and the error message can be safely
ignored. This was due to not clearing an extraneous return code.
- ncbo segmentation fault:
ncbo versions 4.0.0—4.0.2 incorrectly refreshed
internal metadata, leading to segmentation faults and core dumps with
some exacting compilers, notably xlC on AIX.
- ncra segmentation fault:
ncra versions 4.0.0—4.0.1 mishandled some CF-compliant
dates, leading to segmentation faults and core dumps.
- Arithmetic problems with ncap division, modulo, and exponentiation:
ncap versions < 3.0.1 incorrectly exponentiate
variables to variable powers (V^V).
We recommend that all ncap users upgrade.
ncap versions up to 2.9.1 incorrectly handle division,
modulo, and exponentiation operations of the form S/V,
S%V, and S^V where first operand (S) is
scalar (i.e., either typed directly in the ncap script or
converted from an attribute) and the second operand (V) is
a full variable (i.e., stored in a file or computed by ncap).
Instead of the requested quantity, ncap returned
V/S, V%S, and V^S.
In other words ncap treated some non-commutative operations
as commutative. This is now fixed.
V/V, V%V, V^V,
V/S, V%S, V^S,
S/S, S%S, and S^S operations were never
We recommend that all ncap users upgrade.
- Incorrect ncbo output for packed input:
ncbo versions ???—3.2.0 incorrectly write differences
of packed input. This only affects packed variables.
- Problems with ncflint and missing_values:
The algorithm ncflint used to perform interpolation in
versions up to 2.9.4 was not commutative.
It returned the weighted valid datum when the other datum was
missing_value, or it returned missing_value,
depending on the order the input files were specified.
As of version 2.9.5, ncflint always returns
missing_value when either input datum is
Possible future implementations are discussed
- Problems with ncra and nces when missing_value = 0.0:
The algorithm ncra and nces used to perform
arithmetic in versions up to 2.9.2 breaks if missing_value
Why, you ask?
Running average (or total, etc.) algorithms must initialize the answer
This is done since the sum accumulates in place as ncra and
nces proceeds across records and files.
(Normalizing this accumulation by the total number of records is the
The old algorithm compared both the current running average and the
new record to the missing_value.
If either comparison matched, then nothing accumulated for that
This zero-initialization led to a state where it was impossible
to ever recognize valid data.
As a result nothing accumulated and the answer was always zero.
The record and ensemble averages would also fail (in a non-obvious)
way whenever an intermediate sum equalled missing_value.
The chances of the latter event ever happening are exceedingly
The new algorithm compares only the new record to the
This fixes both problems and is faster, too.
- Packing problems with ncwa:
NCO versions ???—2.9.0 have a bug that causes ncwa
to fail (produce garbage answers) when processing packed
NC_FLOAT data. Version 2.9.1 fixes this problem.
This problem may have been noticed most by
OPeNDAP users since many
netCDF climate datasets served by
OPeNDAP are packed
Upgrade to 2.9.1 if you use ncwa on packed data.
- Packing problems with ncap:
NCO versions 2.8.4—2.8.6 have a bug that causes the ncap
intrinsic packing function pack() to fail.
Version 2.8.7 fixes this problem.
Older Platform-specific Run-time Problems:
- Float-valued intrinsic arithmetic functions in ncap on AIX:
ncap versions through 4.0.4 have a bug that causes all float-valued
intrinsic math functions to fail under AIX.
Float-valued math functions are the ISO C99 functions, e.g.,
cosf(), fabsf(), logf().
The user does not invoke these functions directly—
the user always specifies the generic function name, e.g.,
cos(), abs(), log().
NCO automatically calls the native single precision (i.e.,
float-valued) math functions when the generic function argument
is a native float (e.g., naked constants like 1.0f or
variables stored as NC_FLOAT).
Double precision arguments cause NCO to invoke the standard
(double-valued) form of the generic function, e.g., cos(),