Despite domain-specific digital archives are growing in number and size, there is a lack of studies describing their architectures and runtime characteristics. The traces described at this site have been used to investigate the storage landscape of the European Centre for Medium-Range Weather Forecasts (ECMWF) whose storage capacity has reached 100 PB and experiences an annual growth rate of about 45%. Out of this storage, the work has been used to examine a 14.8 PB user archive and a 37.9 PB object database for metereological data over a period of 29 and 50 months, respectively.
Furthermore, we used the data to optimize the cartridge management and reduced the load latencies by a factor of 2.1 and the number of load operations by 2.5, compared to standard library settings. We also developped eviction and placement strategies, which can additionally lower the load latency by 20%, achieving the same performance level with considerably fewer drives.
Citation
Please cite the following papers in case you are using our traces for your research
2016
- Markus Masker, Lars Nagel, Tim Suess, Andre Brinkmann, and Lennart Sorth. 2016. Simulation and Performance Analysis of the ECMWF Tape Library System. In SC ’16: PROCEEDINGS OF THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, 252–263. Author/Publisher URL
2015
- Matthias Grawinkel, Lars Nagel, Markus Mäsker, Federico Padua, Andre Brinkmann, and Lennart Sorth. 2015. Analysis of the ECMWF Storage Landscape. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST), 15–27. Author/Publisher URL
Download
The traces are available at the ECMWF web site.
The image of the ECMWF tape library has been provided by A. Brookes/ECMWF Copernicus under CC 2.0.