CS Events Monthly View

Qualifying Exam

Enabling data-driven adaptations for large-scale in-situ scientific workflows


Download as iCal file

Thursday, October 22, 2020, 04:30pm - 06:00pm


Speaker: Zhe Wang

Location : Remote via Webex


Prof. Manish Parashar (Advisor)

Prof. Uli Kremer

Prof. Sudarsun Kannan

Prof. Mridul Aanjaneya

Event Type: Qualifying Exam

Abstract: As scientific workflows increasingly target extreme-scale resources, the imbalance between higher computational capabilities, generated data volumes, and available I/O bandwidth limits the ability to translate these scales into insights. In-situ workflows (and the in-situ approach) leverage storage levels close to the computation in novel ways to reduce the amount of data movement and I/O required. However, managing the execution of in-situ workflows presents several challenges, especially in cases where computations are dynamic and data-driven, and resources are constrained and shared. In these cases, the placement of tasks and data, the execution of the workflow as well as the management have to be adapted at runtime to ensure that objectives in terms of performance and scalability are satisfied. In this presentation, I will first discuss the state-of-art in data-driven adaptive management of in-situ workflows aimed at reducing overheads and improving execution time. Specifically, I will explore key scenarios that trigger adaptation and summarize mechanisms that have been used to address the trigger. I will use this exploration to formulate an approach for data-driven adaptation for in-situ workflows based on change detection, adaptation goals, adaptation policies, and adaptation mechanism for different types of triggers, and will analyze associated research challenges. I will then present our recent work, “Staging Based Task Execution for Data-driven, In-Situ Scientific Workflows.” This work compares different paradigms for in-situ task placement. In this work, we show that the proper choice for scheduling in-situ tasks can decrease workflow execution overheads significantly. Specifically, I will discuss a model that captures the different factors that influence the mapping, execution, and performance of data-driven in-situ workflows and experimentally study the impact of different mapping decisions and execution patterns. I will finally discuss the design, implementation, and experimental evaluation of a data-driven in-situ workflow execution framework that leverages user-defined task-triggers to enable efficient and scalable in-situ workflow execution.


Meeting link: https://rutgers.webex.com/rutgers/j.php?MTID=ma82e100942829448eab7d00337d92d14

Meeting number: 120 597 4102
Password: HdBJrpMX749