LAURENT, M. Olivier (2021) Towards a Dynamic Pipeline for Clustering Runs of HPC Applications PRE - Research Project, ENSTA.



The calculation power of supercomputers has greatly increased over the years thanks to massively parallel architectures. However, this power is assessed mainly with benchmarks efficiently parallelized which only very lightly interact with the memory, such as LINPACK. In reality, performance is much more mixed and depends on the way these computers perform their Input/Output (IO) operations. Indeed, IO operations are often the limiting factor in real applications which use huge amounts of data. Moreover, these operations might be complex to understand. In this context, and to help users optimize the use of their supercomputers, the Data Management team of Bull has developed tools which monitor the IO operations of application runs. Since these tools produce a data which is not easy to analyze, a part of the team works on a plugin called IOPA. Its capabilities include clustering, i.e. regrouping runs of applications considered as similar. While it only works for fixed runs today, the objective of the internship is to assess the possibility of making the plugin more flexible. That is allowing the user to add and remove runs from a previous experiment. This is especially expected to shorten the waiting time. This report proposes a theoretical study as well as analyses of the results obtained with near-production-quality python implementations. It provides insights on the parts of the clustering pipeline which seem worth being modified to improve the user's experience.

Item Type:Thesis (PRE - Research Project)
Uncontrolled Keywords:HPC, IOs, Pipeline, Time-Series, Clustering, Production
Subjects:Information and Communication Sciences and Technologies
ID Code:8426
Deposited By:M. Olivier LAURENT
Deposited On:08 déc. 2021 09:36
Dernière modification:08 déc. 2021 09:36

Repository Staff Only: item control page