Author:
Sobolev Egor,Schmidt Philipp,Malka Janusz,Hammer David,Boukhelef Djelloul,Möller Johannes,Ahmed Karim,Bean Richard,Bermúdez Macías Ivette Jazmín,Bielecki Johan,Bösenberg Ulrike,Carinan Cammille,Dall’Antonia Fabio,Esenov Sergey,Fangohr Hans,Ferreira de Lima Danilo Enoque,Ferreira Maia Luís Gonçalo,Firoozi Hadi,Flucke Gero,Gessler Patrick,Giovanetti Gabriele,Koliyadu Jayanath,Madsen Anders,Michelat Thomas,Schuh Michael,Sikorski Marcin,Silenzi Alessandro,Sztuk-Dambietz Jolanta,Turcato Monica,Turkot Oleksii,Wrigley James,Aplin Steve,Hauf Steffen,Wrona Krzysztof,Gelisio Luca
Abstract
The European XFEL is a megahertz repetition-rate facility producing extremely bright and coherent pulses of a few tens of femtoseconds duration. The amount of data generated in the context of user experiments can exceed hundreds of gigabits per second, resulting in tens of petabytes stored every year. These rates and volumes pose significant challenges both for facilities and users thereof. In fact, if unaddressed, extraction and interpretation of scientific content will be hindered, and investment and operational costs will quickly become unsustainable. In this article, we outline challenges and solutions in data reduction.