"Granular Data Processing on HPCs Using an Event Service"
EMSL Project ID
48652
Abstract
This is a continuation into FY15 of work begun in July 2014. We will demonstrate a novel fine grained workflow engine suited to the efficient exploitation of distributed and architecturally diverse computing resources. The demonstration will illustrate the approach in an implementation, the ATLAS Event Service, that processes HEP/NP data in a continuous stream at the event level rather than the file level, dispatching single or few event workloads to processing nodes which then stage fine grained outputs in near real time to a data service that can in turn be used as data source for downstream processing. While the demo will involve (at a minimum) ATLAS simulation processing, the approach is generalizable not only to other HEP/NP experiments but to any workflow that can support fine grained partitioning of the processing and its outputs. Other experiments interested and potentially participating on a best effort basis include ALICE and Belle II.
Project Details
Start Date
2014-10-27
End Date
2015-09-30
Status
Closed
Released Data Link
Team
Principal Investigator