Back to Search
Start Over
Scaling Embedded In-Situ Indexing with DeltaFS
- Source :
- SC
- Publication Year :
- 2018
- Publisher :
- IEEE, 2018.
-
Abstract
- Analysis of large-scale simulation output is a core element of scientific inquiry, but analysis queries may experience significant I/O overhead when the data is not structured for efficient retrieval. While in-situ processing allows for improved time-to-insight for many applications, scaling in-situ frameworks to hundreds of thousands of cores can be difficult in practice. The DeltaFS in-situ indexing is a new approach for in-situ processing of massive amounts of data to achieve efficient point and small-range queries. This paper describes the challenges and lessons learned when scaling this in-situ processing function to hundreds of thousands of cores. We propose techniques for scalable all-to-all communication that is memory and bandwidth efficient, concurrent indexing, and specialized LSM-Tree formats. Combining these techniques allows DeltaFS to control the cost of in-situ processing while maintaining 3 orders of magnitude query speedup when scaling alongside the popular VPIC particle-in-cell code to 131,072 cores.
- Subjects :
- Speedup
Computer science
Search engine indexing
020206 networking & telecommunications
020207 software engineering
02 engineering and technology
Data modeling
Computer engineering
Scalability
0202 electrical engineering, electronic engineering, information engineering
Code (cryptography)
Overhead (computing)
Point (geometry)
Scaling
Subjects
Details
- Database :
- OpenAIRE
- Journal :
- SC18: International Conference for High Performance Computing, Networking, Storage and Analysis
- Accession number :
- edsair.doi...........137ce8cf893b003450c177e130ebfb46