[RFC 004] Operation Review Comment 16

our "operational" experience with OPeNDAP is extensive.

We have been running an on-line "portal" for distributed access to hundreds of variables from distributed sites for approximately 5 years (http://ferret.pmel.noaa.gov/NVODS/servlets/dataset);

We have a similar duration of experience running distributed model inter-comparison sites in support of the international Global Ocean Data Assimilation Experiment (GODAE)

Since 1999 OPeNDAP has been a major capability of the "operational" (not the usual use of the word) support that we supply to hundreds of Ferret users, many of whom depend upon OPeNDAP access.

PMEL has been an operational user of OPeNDAP to provide access to many ocean datasets (http://www.ferret.noaa.gov/cgi-bin/dods/nph-dods/data/);
In case you have not been in touch with them -- NOAA/GFDL has been an operational user of OPeNDAP to provide access to their climate model outputs for several years;

I assume that you are already in touch with data serving institutions such as NOAA/CDC, the Columbia University IRI, U. Hawaii ADPRC, US GODAE Server, etc.
Key operational challenges have included:

managing the client-side "cache" used to enhance OPeNDAP performance (a standard feature of the OPeNDAP client software);

the need for "translation" capabilities to unify the handling of gridded and non-gridded data (available as a BETA capability, only, at this time);

keeping up with new releases of OPeNDAP software has been rendered more difficult through its dependency on the C++ (so-called) standard libs.

Associated with this have been complex versioning dependencies that have made multi-platform support much more challenging. In the latest OPeNDAP ALPHA versions there is a move towards Java and pure-C alternatives, which will simplify these dependencies considerably.

Sorry for being last minute with this response. As you know I am just returning from three weeks out of the office.

--------
OPeNDAP is an indispensable part of our overall software strategy that enables scientific users to access remote data translated from a variety of formats. We incorporate the OPeNDAP netCDF client libraries into our own code and make a number of reference datasets available using the OPeNDAP netCDF server.
NOAA's TMAP group has been using OPeNDAP software (earlier known as DODS) in our own code since the August 1999 release of Ferret.
As software developers, we use both OPeNDAP client libraries, source code and applications to meet the needs of our users. Our two products, Ferret and the Live Access Server (LAS), provide others with access to data and data products that make use of OPeNDAP components internally. Other OPeNDAP enabled products we utilize include
dncdump (OPeNDAP enabled version of ncdump)
asciival (OPeNDAP ASCII data dumper)
Our Ferret software utilizes the netCDF API for data access and OPeNDAP support for this API is very complete. The match for us was very good. Our software also operates in a very heterogeneous Internet environment and OPeNDAP's use of the HTTP protocol has eliminated headaches associated with local firewalls.
At one point we investigated the use of CORBA for data transport but quickly dismissed it as unsuitable for heterogeneous Internet environments.
Converting our Ferret visualization and analysis program into an OPeNDAP enabled program was extremely easy -- we just needed to relink the code with the OPeNDAP netCDF libraries. Installation of the OPeNDAP netCDF server was also very easy for us.
The use of OPeNDAP by our end users has made it much easier for us to find and fix bugs in our own software. When a user has problems working with a particular data file we often have them put that data in an OPeNDAP server so that we can access it remotely ourselves to reproduce the error they are seeing. As some of our users work with very large model datasets, this has been extremely helpful.
We are very happy with data access performance using OPeNDAP. Often, our connection to OPeNDAP data sources is good enough that it is difficult to tell we are using remote instead of local files. OPeNDAP support of netCDF subsetting and 'striding' are very important in this regard.
Working in a NOAA research lab we have always had very high bandwidth.
As developers of software we often find ourselves working with beta release of new OPeNDAP products. Occasionally, we encounter bugs or poorly supported features while testing these products but this is a normal part of software development. It has not added to our general administrative workload. The workhorse products for us -- netCDF server and client libraries -- have been very robust and well supported.
Earlier releases of the netCDF client libraries were limited by the netCDF 2G file size limit. Recent releases, compiled on 64 bit machines have eliminated this restriction. We have not encountered any file size related issues with the OPeNDAP servers on our system.
1.I don't have numbers on data volumes handled by our OPeNDAP servers.


0.Our most recent stats show 68,472 non-robot OPeNDAP accesses on our server in one month from 1557 unique IP addresses.

I'm sorry I don't have more complete statistical information.