"Granular Data Processing on HPCs Using an Event Service"
EMSL Project ID
48545
Abstract
We will demonstrate a novel fine grained workflow engine suited to the efficient exploitation of distributed and architecturally diverse computing resources. The demonstration will illustrate the approach in an implementation, the ATLAS Event Service, that processes HEP/NP data in a continuous stream at the event level rather than the file level, dispatching single or few event workloads to processing nodes which then stage fine grained outputs in near real time to a data service that can in turn be used as data source for downstream processing. While the demo will involve (at a minimum) ATLAS simulation processing, the approach is generalizable not only to other HEP/NP experiments but to any workflow that can support fine grained partitioning of the processing and its outputs. Other experiments interested and potentially participating on a best effort basis include ALICE and Belle II.Project Details
Start Date
2014-07-21
End Date
2014-09-30
Status
Closed
Released Data Link
Team
Principal Investigator