Velox Memory Management

Velox memory system is designed for safely running highly variable query workloads within a fixed memory resource. It provides the query execution with all the required memory allocation functions and optimizes both physical memory allocation and query memory allocation patterns. It provides fair-memory sharing among queries by memory arbitration and disk spilling techniques. It provides the total memory capacity enforcement by managing the physical memory on its own.

Xiaoxuan Meng
Software Engineer at Meta

Transcript

1. Velox Memory Management Xiaoxuan Meng 3/27/24
2. 1. Background 2. System Overview 3. Memory Pool 4.Memory Arbitrator 5. Memory Allocator 6. Memory Allocation 7. Next Steps Agenda 2
3. Background 3 • Velox memory system is designed for safely running highly variable query workloads within a fixed memory resource ◦ Memory Allocation ▪ Optimize large physical memory allocations ▪ Optimize query memory allocation patterns ◦ Memory Sharing ▪ Query level memory usage tracking ▪ Memory arbitration among running queries for fair memory sharing ◦ Memory Capacity ▪ Allocate memory from OS directly and total capacity enforcement ▪ Dynamic memory sharing between query and cache
4. System Overview 4 • Memory Manager: memory pool management • Memory Pool: query memory allocation and usage tracking • Memory Arbitrator: arbitrate query memory capacity for fair sharing • Memory Allocator: physical memory allocation and total capacity enforcement
5. Memory Pool 5 Memory Pool Type • Root Pool: capacity enforcement • Aggr Pool: memory usage aggregation • Leaf Pool: memory allocation and usage tracking • Pool Hierarchy: mapping query execution plan Memory Reservation • Quantized memory reservation • Capacity enforcement through memory arbitration Memory Reclaim • Memory Reclaimer connects memory arbitration with the query object • Reclaim used memory through disk spilling
6. Memory Arbitrator 6 Memory Arbitrator arbitrates memory capacity across queries to achieve fair memory sharing: • Ensures total allocated query memory capacities is within arbitrator capacity limit • On startup, a query gets initial capacity from memory arbitrator • During execution, a query can grow its capacity from the arbitrator ◦ Local memory arbitration: reclaim from the query itself if reach max query capacity ◦ Global memory arbitration: reclaim from the queries using more memory
7. Memory Arbitration Process 7 The memory arbitration process within the arbitrator: 1. The requestor pool enters arbitration state ◦ Put the driver thread in suspended state is not considered as running in task pause 2. Captures the memory usage snapshot of the candidate pools for arbitration selection 3. Reclaims used memory from the requestor itself if reaches max query capacity: local memory arbitration 4. Sort candidates by free capacity to reclaim unused capacity by shrinking their capacity 5. Sort candidates by memory usage to reclaim used memory through disk spilling: global memory arbitration 6. Sort candidates by memory capacity and abort the candidate with the largest capacity 7. Grow the requestor capacity on success 8. The requestor pool leaves arbitration state
8. Memory Reclaim Process 8 The memory reclaimer reclaims the used memory from a query by traversing its memory pool hierarchy: 1. Query memory reclaimer reclaims from each child task pool 1. Task memory reclaimer pauses the task execution 2. Task memory reclaimer reclaims from each child plan node pool 1. Node memory reclaimer reclaims from each child operator pool 1. Operator memory reclaimer triggers disk spilling in operator 2. Operator memory reclaimer shrinks the query pool capacity to reclaim the freed memory 3. Task memory reclaimer resumes task execution
9. Memory Allocator 9 Memory Allocator allocates physical memory and ensures the total usage is within total memory capacity Memory Allocation • Allocate large chunk of memory from OS directly through mapped memory • Use SizeClass handles the non-contiguous memory allocations • Delegate small allocations (<3KB) to std::malloc Capacity Enforcement • Track the physical memory usage of mapped memory space • Ensure physical memory usage is within the total memory capacity limit • Shrinks cache space when new allocation hits capacity limit
10. End-to-End Memory Allocation 10
11. Next Steps 11 Memory Arbitration • Parallelize the local arbitration processing to avoid cross-query execution blocking Memory Efficiency • improve the file writer memory efficiency to avoid early stripe flush • improve memory efficiencies in non-spillable operators Disk Spilling • optimize spill data sorting by leveraging prefix key sort • disk io optimization: write pipelining and read ahead File Cache • improve cache replacement algorithm to prevent cache pollution from one-time data scan
12. Thank you! 12
13. Q/A 13