«Medium-Range Weather Prediction Austin Woods Medium-Range Weather Prediction The European Approach The story of the European Centre for Medium-Range ...»
GETDATA was beginning to show its weaknesses as the volume of the ECMWF archives, and demand for access to them, grew. Use of magnetic tapes meant that data were organised sequentially; this was far from ideal.
The limit to the number of tapes that could be mounted for reading was being reached. A more comprehensive system was clearly required. It was Archives and Graphics: towards MARS, MAGICS and Metview 187 “planned to make it a major objective of the Centre over the next few years to increase its on-line storage capacity and its data handling capability”.
Work began in 1982, with Peter Gray and Dick Dixon of the Computer Division starting to make plans.
After some consultation with the Director Lennart Bengtsson, Daniel Söderman decided to set up a project team, with John Hennessy as project leader, to design and implement the application software for the data-handling project. Hennessy later became Section Head of the group in charge of the archives. The team started work in November 1982. Appropriate meteorological scientists at the Centre and Computer Division staff were nominated to liase with the project team.
During its 17th session, the Council, following the recommendations of its Technical Advisory Committee, authorized the Director to conclude the contract for a Data-Handling Subsystem “after completion of a detailed study to be made jointly with IBM UK Ltd. to confirm that the performance of the system will meet the Centre’s requirement”. The study was performed by Dick Dixon and David Dent from ECMWF and Mr N Bartlett and Mr P Goody from IBM UK Ltd. during the period 23 March to 3 June 1983. The 24-page report of this study concluded: the Common File System (CFS) “can be implemented and maintained on the proposed hardware configuration using a general-purpose network. The CFS package (including its user interfaces on the computers connected to the Data Handling Processor) when suitably modified will meet ECMWF’s functional requirements. The manpower required to make these modifications is estimated at approximately 76 man-months.” The projected requirements had been estimated at about 5,000 MB stored and 10,000 MB retrieved daily.
Choosing CFS, the Common File System, was a groundbreaking decision on the Centre’s part. As far as is known, no other meteorological centre was using, or even contemplating, any comparable data management software at that time. The Centre had a unique opportunity to shape the way this product would develop. It grasped the opportunity by providing input on the specification of such new features as magnetic tape support, tape “families” and multiple partial data access. ECMWF staff members were able to collaborate on developing the code for these features: the CFS code was well-structured, it was written in the PL/1 high-level programming language and it had a reasonably comprehensive set of internal documentation, produced by staff at the Kirtland Air Force Weapons Laboratory and at Los Alamos Scientific Laboratory (LASL), the originator of CFS. Several LASL staff members visited the Centre for weeks at a time to help in this joint venture. One in
188 Chapter 15particular, Emily Willbanks, had a long association with the Centre that continued even after CFS was finally retired from service.
The Meteorological Archival and Retrieval System (MARS) introduced in June 1985 ran on an IBM/MVS mainframe: an IBM 4341, and used CFS as the underlying data storage management software.
Originally, MARS was designed to handle field data only. It was extended in the following years to become a complete archive, which would store, and retrieve, vast quantities of data. The meteorological observations that had been used as model input, as well as all analysis and forecast fields, results of research experiments, Member State results from work on the Centre’s computers, data from the Re-analysis Project, and more were eventually all in MARS.
The IBM 4341 system had 8 MB of memory, an online capacity of 12.5 GB of disk space, six IBM 3420 tape drives and an IBM 3851-A01 mass storage system with sufficient cartridges to hold 35 GB; soon extended to 105 GB. This machine was the first robotic tape device that the Centre installed. A key element in the implementation of MARS was the development of the “data highway” to provide the necessary high-speed links between the Centre’s different computers; this would be the Centre’s first Local Area Network (LAN).
In 1985, the Commission for Basic Systems (CBS) of WMO recognised the need for new codes for efficient transfer and storage of meteorological data between and within data processing centres. Different codes would be required for observational data on the one hand, and for forecast and analysis products on the other. The Centre decided that where possible, its archives would store data in internationally agreed forms. Daniel Söderman was the originator of a new efficient code for forecast and analysis products “Grid In Binary” or GRIB. ECMWF staff participated in the earliest stages in the development of GRIB. With Söderman’s strong backing, this code was approved by CBS in October 1985, and the related Binary Universal Form for Representation (BUFR) of meteorological data, was approved in early 1988.
At the beginning of MARS development, neither GRIB nor BUFR had yet been developed, although an experimental version of the GRIB code was available, and this format was used. Partly because of the Centre’s experiences with the code, some changes were made before the code was adopted by WMO. In the absence of GRIB and BUFR, MARS formats were to be machine-independent ECMWF binary format.
In MARS, the underlying data organisation was hidden from the user. A MARS retrieval was expressed in meteorological terms (date, parameter, Archives and Graphics: towards MARS, MAGICS and Metview 189 level) and not in terms of “files”. The data were stored using standard meteorological formats, which are machine independent.
Data storage design was logical. Frequently-used files were held on-line on disks. Less frequently used files were kept on cartridges in the mass storage device, together with large files that would use up too much valuable disk space. Files that were used only infrequently were stored on magnetic tapes. Since no manual intervention was needed to mount the cartridges, data on these were effectively on-line, but took longer to access than the data on disk. The CFS software maintained this hierarchical file system. The remaining data were kept on tapes in racks in the computer hall, requiring an operator to retrieve and mount the tape in response to a request for littleused data or very large files.
Related to MAGICS was Metview, the ECMWF visualization software, developed under a co-operative project between the Centre and the Brazilian Centre for Weather Prediction and Climate Studies (INPE/CPTEC), with assistance from Météo France. Metview was designed to retrieve data from MARS, and tranform it in a form that MAGICS could handle. It matured to become a highly adaptable modular package, with the aim of providing “desktop publishing” capacities to the operational and research meteorologist. The computational capacity of Metview rested on an easy to learn, high-level macro language particularly adapted to weather data. Metview, MARS and MAGICS are used at the time of writing to produce the plots for the ECMWF websites. Metview is the user interface, used to request for example some forecasts. The Metview request goes to MARS to retrieve the fields from the archives. It then uses MAGICS to create the contours, titles, map background and so on. Finally it puts the forecasts on the user’s screen or plotter.
Beginning in 1989, all field data previously retrieved by GETDATA, with the exception of forecasts from the years before 1985, were gradually converted to GRIB format and re-archived under MARS, thus extending the MARS archive back to the beginning of the Centre’s operations.
Observation archiving in MARS started in 1990. Observations from earlier years were systematically converted to BUFR and archived. Not only were the observations stored. The observations went through many quality control checks for accuracy before being used in the analysis; the results of all these checks, including substituted values, flags indicating the accuracy of the data, and bias information were also stored in the archives, never to be discarded.
MARS retrievals were interfaced to MAGICS, providing an impressive plotting and display service with a simple common interface between the
190 Chapter 15two. By 1987, MARS retrievals were being used freely and often by Member State users.
To meet requests for data quickly and efficiently, work began in 1985 to create a series of special compact data sets that were to be provided by the ECMWF Data Services, together with software tools allowing direct extraction, or simple tape copying, of reasonable subsets of data in internationally accepted formats. These included high-resolution global analyses from the FGGE year, data sets from the Tropical Ocean–Global Atmosphere (TOGA) experiment, a large set of data from over Europe collected during the Alpine Experiment (ALPEX), and analyses and forecasts from other global forecasting centres such as Bracknell and Washington.
By now, the volume of tapes was beginning to present a storage problem in the computer hall. Further, in the mid-1980s it was felt that the CFS system would need replacement within a few years time. The Centre embarked on a study of available systems. Few of the systems on the market could meet the end-of-decade requirements, and those that could were inferior to the existing CFS system. The study showed just how good the existing system was! The initial assumption that the market would provide, and clearly indicate, an appropriate successor, was quickly proved over-optimistic.
Plans were made to extend the life of CFS. ECMWF staff visited Los Alamos National Laboratory, and vice versa, working on joint development projects to enhance, and extend the usability of CFS.
In June 1987 an IBM 3090-150E was installed; this replaced the IBM 4341 in October. Difficult technical work followed throughout 1988, implementing a new operating system, required to speed up access to the archives.
A new utility “ECFILE” for storing and retrieving data not suitable for saving as MARS data started to be used from October 1988; by then 11 GB of data were being transferred daily between the CRAY X-MP/48 and the data archiving system.
In December 1990, Council approved the purchase of an automated cartridge library system from Storage Technology Ltd, to improve the Centre’s archival storage. Four of these very large modules or silos had been installed by September 1992. On 4 January 1992, an IBM ES/9000-580 was implemented, and immediately improved the performance of the data handling system. The IBM ES/9000-580 was upgraded to an IBM ES/9000-720 on 29 January 1994. In 1998, an IBM SP2 system replaced the ES 9000 data-handling computer.
MAGICS was being used by thirteen Member States as well as the National Meteorological Services of Australia and India by 1989. MicroMAGICS, a version of MAGICS to be run on IBM PCs, was developed by the Brazil’s INPE/CPTEC in that year. In 1989, GETDATA was finally brought to an end.
Archives and Graphics: towards MARS, MAGICS and Metview 191 The Centre was now dispatching several hundred tapes of archive data to users worldwide each year. The Cyber tapes were copied, over a period of months, to the high-density data handling tape cartridges. About 25,000 1 ⁄2-inch 9-track tapes were disposed of in 1989.
In 1994, MARS software was enhanced to allow ECMWF staff to access the archives from their newly installed workstations. The beginning of serious work on the Re-analysis Project meant many internal changes to the MARS system. Further, the MARS client software had to be ported to a new computer, a Fujitsu VPP300/16 that was installed to port codes and enable the Centre to become familiar with the Fujitsu UXP/V operating system that would be installed on the VPP700 mainframe computer later that year.
CFS was becoming increasingly difficult to support. Los Alamos was moving to a new product, the High Performance Storage System (HPSS), and no new development was being done on CFS. It was becoming obvious that the Centre would have to move to a new system for its data management requirements. In 1995, the Centre concluded a contract with IBM to supply a new Data Handling System (DHS), which would eventually replace the CFS-based system. The new DHS would use the Adstar Data Storage Manager (ADSM) instead of CFS as the underlying management system used by MARS. A new utility was developed called ECFS, the ECMWF File Management System, which was to become the replacement for ECFILE. ADSM ran on AIX, IBM’s Unix operating system and could be distributed over a set of servers, rather than having to rely on a single mainframe as did CFS. This meant that the system could grow incrementally, purchasing server hardware year by year as necessary, rather than having to buy a large mainframe from time to time.
In 1995, Baudouin Raoult and Manuel Fuentes began design of a new MARS system. The system was completely rewritten using Object Oriented design in the C++ programming language running under Unix. By 1997, the new version was ready for trial, and the “back-archiving” — copying to new media — from CFS to ADSM started. At the end of 1998, the CFSbased MARS system was switched off, ending 15 years of exceptional service. A total of 32 TB (32 million million bytes) of data was backarchived in 18 months. The Object Oriented approach allowed rapid development of MARS. A web interface was created, giving users the ability to navigate through the vast archive, and retrieve and plot sample fields.
A new system to index fields by parameter, data source or time was quickly being used by many scientists to find data in the archive.
Moving to the new data archiving system was not without its problems.
The Centre was pushing the capabilities of the hardware and software to the