[RFC 004] Operation Review Comment 2

Review of DAP2 operational experience:

NASA's Earth Science Data Systems Standards Process Group (SPG) is considering the Data Access Protocol, Version 2, (DAP2) for adoption as a community standard.  This is the second review of DAP2, this one focusing on its operation experience.� The questions below are provided to guide your feedback. You only need to answer questions applicable to you.  Please send comments to ese�rfc�004@spg.gsfc.nasa.gov.�

1. Describe in a sentence or two your overall operational experience related to DAP2 (e.g., scientific analysis; science users, server operation; database management; or data translation, etc).� What kinds of DAP2 systems do you have experience with?� (e.g., OPeNDAP netCDF server, OPeNDAP HDF server, OPeNDAP Matlab Toolkit, Ferret, GrADS, and etc).
1. How long have you been using DAP2 operationally?

I started working with DODS (now called OPeNDAP) in 2001. I installed a DODS netCDF server and wrote some client applications based on the IDL client. Our intention was to collect model output generated by various groups of CTM and CCM modelers involved in the EU funded project TRADEOFF and to provide easy online access to these data. I am still using the server occasionally for other things. Recently we used it to provide online access to ECMWF weather forecast fields (which we first processed into netCDF files and then put on the server) during an aircraft measurement campaign in Brazil.

What types of applications do you use DAP2 systems for?� Are the DAP2 systems applicable to your applications? (e.g., Do they work well with the data types and data manipulations in your application?) ��

2. How many of your applications use DAP2?

                                                               i.      Total number of applications

                                                             ii.      Percentage of applications

I have only two applications using DAP2 which is probably less than 10% of the applications I am working with.

2. Why do you choose to use DAP2 systems over other systems for your applications?
1. What alternative technologies did you consider?

There was no real alternative when I decided to use DAP2. Of course I also considered putting all model output on an ftp server. However, the modelers would typically be interested in only a few model fields, possibly only at specific model layers, and therefore transfer of the entire (and huge) model output files was not a good option.

3. Are the DAP2 systems easy to use?� (e.g., Is it hard to learn how to use DAP2 systems?)
1. How has the use of DAP2 affected your user support workload?

Unfortunately user support workload was very large. The main reason was that the DODS IDL client was (and still is) not in a good shape. Installation of the client on different platforms was very difficult (usually no binaries available and the make files were not working without some adaptations). Some groups never managed to install the client (despite skilled system administrators) which severely limited its usage in the project. I think these problems are specifically related to the IDL client and probably do not apply to others. E.g. the program geturl was easy to install on different computers.

4. Does the performance of the DAP2 systems you have experienced meet your requirements?� (e.g., Does it take a long time to access data in DAP2 systems?)
1. Have your bandwidth issues changed since you have been using DAP2 systems? (e.g., Have you seen increased bandwidth requirements because of increased data access or DAP2 overhead, decreased bandwidth requirements because of reduced data volume from subsetting, no impact on bandwidth because of low usage?)

The best test for this was our recent measurement campaign in Brazil. We had a 1 Mbit internet connection in the field which I used to access ECMWF data (1x1 deg resolution, 60level data; I think about 3 MB per field). Even though the connection was usually much slower than 1 Mbit the bandwidth was sufficient to work with. I do not know whether DAP2 overhead etc. is decreasing the transfer rate significantly. I simply did not checked this.

5. What operational challenges do the DAP2 systems present? (e.g., Does it require advanced processing power, large amounts of memory, complex configuration, etc.? Are the systems easy to deploy and maintain?)
1. How has the use of DAP2 affected your systems administration workload?

The IDL DODS client used up large amounts of memory because of the way it was programmed. When accessing a data DDS the client would return an IDL structure mimicking the structure of the remote data set. The IDL structure was using up as much memory as the entire remote data file which sometimes became impracticable. This behavior of the client should definitely be changed (and I think people like Ethan Davies are aware of this).

6. How well do the DAP2 systems scale to large numbers of simultaneous users, or to large datasets?
1. How much data does your DAP2 system handle in a typical month?

                                                               i.      Total data volume

                                                             ii.      Total number of data files

                                                            iii.      Percentage of data volume

                                                           iv.      Percentage of data files

The load on our DODS server was always low. There was hardly ever more than one user accessing the data simultaneously. Thus, I have no idea of where the limits of the system really are. Total data volume per month may have been of the order of a few 100 MB. Usually less (or none at all).

7. Can you provide information on user statistics of your DAP2 systems?�
1. How many users does your DAP2 system handle in a typical month?

                                                               i.      Total number of DAP2 users

                                                             ii.      Percentage of your overall users

2. Of the feedback you have received from your users on DAP2, what percentage is positive and what is negative?
3. How have the user statistics changed over time?

No, I can�t.