The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing
costs of data movement and IO, and the reduced mean times between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. We address this challenge by proposing a novel hybrid approach CoREC that combines dynamic replication with erasure coding based on data access patterns. CoREC is a scalable resilient in-memory data staging runtime for large-scale in-situ workflows. We also design a new load balancing and conflict avoiding encoding, and a low overhead, lazy data recovery scheme to decrease computation cost further. CoREC is implemented on the top of DataSpaces, an open-source data staging framework and the Jerasure open-source library which performs encode/decode operations. We make an experimental evaluation using synthetic benchmarks as well as the S3D combustion simulation and analysis workflow on the Titan Cray XK7 at Oak Ridge National Laboratory (ORNL). The experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall
storage efficiency at large scales.