Advancements in the fields of experimental particle, astroparticle, hadron, and nuclear physics have produced ever-increasing amounts of research data. Higher resolution of modern instruments have led to huge data sets, reaching the order of millions of terabytes and beyond, and require not only extensive computational and storage resources but also their user-friendly access.
FIDIUM aims to solve these challenges by virtually combining several complex infrastructures that offer a number of services. This includes pooling together dedicated CPU and storage resources at large data centers, temporary resources at HPC centers of partners, and commercial cloud systems. Providing an production system that can handle this highly heterogenous environment requires a high degree of abstraction and the development of the necessary software tools.
For processing the corresponding data sets, FIDIUM will combine modern storage systems of several data centers into data lakes. For unified and efficient data lake access, FIDIUM will use temporary caches at the local sites so that HPC resources can be incorporated into scientific workflows, e.g., from high-energy physics. Hierarchical storage management and intelligent data placement will utilize ad hoc file systems, among others.
Funding
FIDIUM is funded by the Germany Federal Ministry of Education and Research (BMBF) from 01.10.2021 — 30.09.2024.
Project Partners
- Rheinisch-Westfälische Technische Hochschule Aachen, Coordinator
- Rheinische Friedrich-Wilhelms-Universität Bonn
- Goethe Universität Frankfurt am Main
- Albert-Ludwigs-Universität Freiburg
- Georg-August-Universität Göttingen
- Universität Hamburg
- Karlsruher Institut für Technologie
- Ludwig-Maximilians-Universität München
- Bergische Universität Wuppertal
Associated Partners
- CERN
- DESY
- GridKa
- GSI Helmholtzzentrum für Schwerionenforschung
Contact
Publications
2024
- Yingjin Qian, Marc-André Vef, Patrick Farrell, Andreas Dilger, Xi Li, Shuichi Ihara, Yinjin Fu, Wei Xue, and André Brinkmann. 2024. Combining Buffered I/O and Direct I/O in Distributed File Systems. In Proceedings of the 22nd USENIX Conference on File and Storage Technologies (FAST), Santa Clara, CA, USA, February 27-29, 17–33. Author/Publisher URL
2023
- Marc-André Vef, Alberto Miranda, Ramon Nou, and André Brinkmann. 2023. From Static to Malleable: Improving Flexibility and Compatibility in Burst Buffer File Systems. In 2nd International Workshop on Malleability Techniques Applications in High-Performance Computing (HPCMall). Author/Publisher URL
- Nafiseh Moti, André Brinkmann, Marc-André Vef, Philippe Deniel, Jesus Carretero, Philip Carns, Jean-Thomas Acquaviva, and Reza Salkhordeh. 2023. The I/O Trace Initiative: Building a Collaborative I/O Archive to Advance HPC. In Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. DOI Author/Publisher URL
- Yingjin Qian, Wen Cheng, Lingfang Zeng, Xi Li, Marc-André Vef, Andreas Dilger, Siyao Lai, Shuichi Ihara, Yong Fan, and André Brinkmann. 2023. Xfast: Extreme File Attribute Stat Acceleration for Lustre. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Denver, CO, USA, November 12-17, 96:1–96:12. DOI Author/Publisher URL