Windows Kernel Heap: Segment heap in windows kernel Part 1

Slide 1

Slide 1 text

Windows Kernel Heap Part 1: Segment heap in windows kernel [email protected] @scwuaptx Angelboy

Slide 2

Slide 2 text

• Angelboy • Researcher at DEVCORE • CTF Player • HITCON / 217 • Chroot • Co-founder of pwnable.tw • Speaker • HITB GSEC 2018/AVTokyo 2018/VXCON Whoami

Slide 3

Slide 3 text

Kernel Pool • Non Paged Pool • Memory that will not be page out • Paged Pool • Memory that will be page out • Session Paged Pool • Memory that will be page out • Each session is independent

Slide 4

Slide 4 text

Memory Allocator in kernel • ExAllocatePool/ExAllocatePoolWithTag • ExFreePool/ExFreePoolWithTag • Nt Heap • RtlpAllocateHeap • RtlpFreeHeap

Slide 5

Slide 5 text

Memory Allocator in kernel • ExAllocatePoolWithTag/ExFreePoolWithTag • Before 19H1 (1903) • Kernel Pool Allocator • From 19H1 ~ Now • Segment heap • Note • Segment heap in kernel is very similar to segment heap in userland • Some size and conﬁg are diﬀerence • We will focus on kernel in this slide • Windows 10 20H2

Slide 6

Slide 6 text

Memory Allocator in kernel • Segment Heap Overview • Frontend Allocation • Low FragmentationHeap • Variable Size Allocation • Backend Allocation • Segment Allocation • Large Block Allocation

Slide 7

Slide 7 text

Memory Allocator in kernel • Segment Heap Overview ntoskrnl.exe ExAllocatePool WithTag ntoskrnl.exe ExAllocateHeap Pool LFH RtlpHpLfh ContextAllocate VS Allocation RtlpHpVsContext AllocateInternal Segment Allocation RtlpHpSegAlloc Block Allocation RtlpHpLarge Alloc FrontEnd allocator Backend Allocator

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Memory Allocator in kernel • If FrontEnd's memory is  insuﬃcient,  it will be allocated  from BackEnd to  FrontEnd ﬁrst ntoskrnl.exe ExAllocatePool WithTag ntoskrnl.exe ExAllocateHeap Pool LFH RtlpHpLfh ContextAllocate VS Allocation RtlpHpVsContext AllocateInternal Segment Allocation RtlpHpSegAlloc Block Allocation RtlpHpLarge Alloc FrontEnd allocator Backend Allocator

Slide 13

Slide 13 text

Memory Allocator in kernel • Important structure • _EX_POOL_HEAP_MANAGER_STATE • _SEGMENT_HEAP • _RTLP_HP_HEAP_GLOBALS

Slide 14

Slide 14 text

Memory Allocator in kernel • nt!ExPoolState (_EX_POOL_HEAP_MANAGER_STATE) • The core structure in the Kernel pool memory allocator • Whether it is Page Pool or Non page pool, etc. are managed by this structure

Slide 15

Slide 15 text

Memory Allocator in kernel • ExPoolState (_EX_POOL_HEAP_MANAGER_STATE) • HeapManager (_RTLP_HP_HEAP_MANAGER ) • Store some Global variables, some Metadata, etc. • NumberOfPool • Number of Pool Node • Default 1 NumberOfPool (8bytes) HeapManager (0x38d0bytes) PoolNode[64] (0x20c0*64) … 0x0 0x38d0 0x3900 0x86900

Slide 16

Slide 16 text

Memory Allocator in kernel • ExPoolState (_EX_POOL_HEAP_MANAGER_STATE) • PoolNode (_EX_HEAP_POOL_NODE) • Each node will have four Heaps corresponding to diﬀerent segment heaps, such as Paged/ Nonpaged pool, etc. NumberOfPool (8bytes) HeapManager (0x38d0bytes) PoolNode[64] (0x20c0*64) … 0x0 0x38d0 0x3900 0x86900

Slide 17

Slide 17 text

Memory Allocator in kernel • _SEGMENT_HEAP • The core structure in the segment heap • Each Pool will have a _SEGMENT_HEAP structure. When allocating memory, it will decide which HEAP to allocate according to the pool type. • In ExPoolState • These are four segment heap will be created and initialized when the system is turned on • ExPoolState.PoolNode[0].Heap[0-4]

Slide 18

Slide 18 text

Memory Allocator in kernel • _SEGMENT_HEAP • ExPoolState.PoolNode[0].Heap[0] -> NonPagedPool • ExPoolState.PoolNode[0].Heap[1] -> NonPagedPoolNx • ExPoolState.PoolNode[0].Heap[2] -> PagedPool

Slide 19

Slide 19 text

Memory Allocator in kernel • _SEGMENT_HEAP • EnvHandle (RTL_HP_ENV_HANDLE) • environment handle of segment heap • Signature • Signature of segment heap • It always 0xddeeddee • AllocatedBase • Point to the end of the entire structure of _SEGMENT_HEAP, used to allocate the structure required by LFH Allocator, after allocation, it will point to the end of the allocated structure … … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 0x0 EnvHandle (10bytes) … UserContext (8bytes) … … 0x28

Slide 20

Slide 20 text

Memory Allocator in kernel • _SEGMENT_HEAP • SegContexts (_HEAP_SEG_CONTEXT) • The core structure of the back-end manager • Divided into two segcontexts according to size • 0x20000 < Size <= 0x7f000 • 0x7f000 < Size <= 0x7f0000 … … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 0x0 EnvHandle (10bytes) … UserContext (8bytes) … … 0x28

Slide 21

Slide 21 text

Memory Allocator in kernel • _SEGMENT_HEAP • VsContext (_HEAP_VS_CONTEXT) • The core structure of front-end VS Allocator • LfhContext (_HEAP_LFH_CONTEXT) • The core structure of the front-end LowFragmentationHeap Allocator … … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 0x0 EnvHandle (10bytes) … UserContext (8bytes) … … 0x28

Slide 22

Slide 22 text

Memory Allocator in kernel • nt!RtlpHpHeapGlobals (_RTLP_HP_HEAP_GLOBALS) • In Segment Heap, many ﬁelds, values, and functions are mostly encoded • This structure will be used to store related key and other information

Slide 23

Slide 23 text

Memory Allocator in kernel • RtlpHpHeapGlobals (_RTLP_HP_HEAP_GLOBALS) • HeapKey • Used for VS Allocator, Segment Allocator • LfhKey • Used for LowFragmentationHeap • Both are 8 byte random value HeapKey (8bytes) LfhKey (8bytes) _RTLP_HP_HEAP_GLOBALS … 0x0 0x8

Slide 24

Slide 24 text

Memory Allocator in kernel • Segment Heap • Frontend Allocation • Low FragmentationHeap • Variable Size Allocation • Backend Allocation • Segment Allocation • Large Block Allocation

Slide 25

Slide 25 text

Low FragmentationHeap • Size • In the case of default • Size <= 0x200 and LFH is enabled for this size • The activation timing is very similar to that of NtHeap. It will be activated after 18 consecutive allocations of the same size block

Slide 26

Slide 26 text

Low FragmentationHeap • Data Structure • Memory allocation mechanism

Slide 27

Slide 27 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • The core structure of the LFH Allocator • Used to manage the memory allocated by LFH, record all the information and structure of LFH in the heap

Slide 28

Slide 28 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • BackendCtx (_HEAP_SEG_CONTEXT) • Point to the Backend allocator used by the LFH … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[129]

Slide 29

Slide 29 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Callbacks (_HEAP_SUBALLOCATOR_CALLBACKS) • Call back function table (use to allocate/ release subsegments) • Allocate • Free • Commit • Decommit • ExtendContext … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[129]

Slide 30

Slide 30 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Callbacks (_HEAP_SUBALLOCATOR_CALLBACKS) • Function pointer 會被 encode 過 • The value of callbacks will be xored by • RtlpHpHeapGlobals.HeapKey • Address of LfhContext • Function pointer … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[129]

Slide 31

Slide 31 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Conﬁg (_RTL_HP_LFH_CONFIG) • Used to indicate the attributes of the LFH Allocator • It will be used to determine whether the size of allocation is within the scope of LFH allocator … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[129]

Slide 32

Slide 32 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Conﬁg (_RTL_HP_LFH_CONFIG) • MaxBlockSize • The size of the max block in LFH • WitholdPageCrossingBlocks • Are there any cross-page blocks • DisableRandomization • Whether to disable randomization of LFH MaxBlockSize (2bytes) WitholdPageCrossingBlocks (1bit) _RTL_HP_LFH_CONFIG 0x0 0x2 DisableRandomization (1bit) 0x2

Slide 33

Slide 33 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Buckets (_HEAP_LFH_BUCKET) • Bucket array, each buckets corresponds to blocks in a speciﬁc size range • When LFH is enabled, it will point to the _HEAP_LFH_BUCKET structure … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[129]

Slide 34

Slide 34 text

Low FragmentationHeap • LfhContext (_HEAP_LFH_CONTEXT) • Buckets (_HEAP_LFH_BUCKET) • If it is not enabled, it will be used to indicate the number of times the block of the size is allocated. • The last 2 bytes are always 1 • The next 2 byte is the number of allocations • Each allocation will add 0x21 … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[0] … 0x0021 0x0001 Buckets[0]

Slide 35

Slide 35 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • The bucket and its related structure will be allocated only when the LFH is enabled • LFH Allocator is use the structure to manage the block corresponding to the Size

Slide 36

Slide 36 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • State (_HEAP_LFH_SUBSEGMENT_OW NER) • Used to indicate the status of the Buckets • This structure is used to manage the memory pool of LFH … State (0x38bytes) TotalBlockCount (8bytes) _HEAP_LFH_BUCKET 0x0 0x38 0x40 TotalSubsegmentCount (8bytes) ReciprocalBlockSize (8bytes) … AfﬁnitySlots[] 0x48 0x60

Slide 37

Slide 37 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • State (_HEAP_LFH_SUBSEGMENT_OW NER) • BucketIndex • The index of the bucket IsBuckets (1bit) _HEAP_LFH_SUBSEGMENT_OWNER 0x0 0x1 BucketIndex (1byte) … 0x8 0x18 … AvailableSubsegmentCount (8B) AvailableSubsegmentList (8B) FullSubsegmentList (8bytes) 0x20

Slide 38

Slide 38 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • State (_HEAP_LFH_SUBSEGMENT_OW NER) • AvailableSubsegmentCount • Number of available subsegments IsBuckets (1bit) _HEAP_LFH_SUBSEGMENT_OWNER 0x0 0x1 BucketIndex (1byte) … 0x8 0x18 … AvailableSubsegmentCount (8B) AvailableSubsegmentList (8B) FullSubsegmentList (8bytes) 0x20

Slide 39

Slide 39 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • State (_HEAP_LFH_SUBSEGMENT_OWNER) • AvailableSubsegmentList (_LIST_ENTRY) • Point to the next available subsegment in the bucket • FullSubsegmentList (_LIST_ENTRY) • Point to the next used up subsegment in the bucket IsBuckets (1bit) _HEAP_LFH_SUBSEGMENT_OWNER 0x0 0x1 BucketIndex (1byte) … 0x8 0x18 … AvailableSubsegmentCount (8B) AvailableSubsegmentList (8B) FullSubsegmentList (8bytes) 0x20

Slide 40

Slide 40 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • AﬃnitySlots (_HEAP_LFH_AFFINITY_SLOT) • The main structure used to manage the memory pool using by LFH • There is only one by default … State (0x38bytes) TotalBlockCount (8bytes) _HEAP_LFH_BUCKET 0x0 0x38 0x40 TotalSubsegmentCount (8bytes) ReciprocalBlockSize (8bytes) … *AfﬁnitySlots[] 0x48 0x60

Slide 41

Slide 41 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • AﬃnitySlots (_HEAP_LFH_AFFINITY_SLOT) • State (_HEAP_LFH_SUBSEGMENT_O WNER) • Same structure as State in Bucket, but this one is mainly used to manage subsegment State (0x38bytes) ActiveSubsegment (8bytes) _HEAP_LFH_AFFINITY_SLOT 0x0 0x38

Slide 42

Slide 42 text

Low FragmentationHeap • Buckets (_HEAP_LFH_BUCKET) • AﬃnitySlots (_HEAP_LFH_AFFINITY_SLOT) • ActiveSubsegment (_HEAP_LFH_FAST_REF) • Point to the subsegment being used • The lowest 12 bits indicate how many blocks are available in the subsegment State (0x38bytes) ActiveSubsegment (8bytes) _HEAP_LFH_AFFINITY_SLOT 0x0 0x38

Slide 43

Slide 43 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • The memory pool of LFH is very similar in structure to UserBlock in NtHeap, but each block does not have header and other metadata. • Once there is not enough memory, it will take subsegment from Buckets->State.availableSubsegmentList ﬁrst, if it has no available subsegment , it will allocate from backend allocator for a new subsegment • Mainly managed by buckets->AﬃnitySlots

Slide 44

Slide 44 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • ListEntry(_LIST_ENTRY) • Flink • Point to the next full/available LFH subsegment • Blink • Point to the previous full/available LFH subsegment ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 45

Slide 45 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • Owner (_HEAP_LFH_SUBSEGMENT_OWN ER) • Point to the structure that manages the subsegment • Point back to the AﬃnitySlots.State of the bucket to which it belongs ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 46

Slide 46 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • FreeCount • The number of times the block is released • Not the number of freed blocks of the subsegment ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 47

Slide 47 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • BlockCount • the total number of block in the subsegment • FreeHint • The index of block that the last allocated • If the index of block that the last freed larger than freehint. It will be updated to the index • Used in allocated algorithm ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 48

Slide 48 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • Location • Indicates the location of the subsegment • 0 : AvailableSubsegmentList • 1 : FullSubsegmentList • 2 : The entire subsegment will be returned to back-end soon ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 49

Slide 49 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • BlockOffsets (_HEAP_LFH_SUBSEGMENT_ENC ODED_OFFSETS) • Used to indicate the block size of the subsegment and the offset of the first block in the subsegment ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 50

Slide 50 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • BlockSize • The size of block in the subsegment • The size is the original size • FirstBlockOffset • The offset of the first block • FirstBlock = subsegment + FirstBlockOffset BlockSize (2bytes) _HEAP_LFH_SUBSEGMENT_ENCODED_OFFSETS 0x0 FirstBlockOffset (2bytes) 0x2

Slide 51

Slide 51 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • BlockOﬀsets (_HEAP_LFH_SUBSEGMENT_ENC ODED_OFFSETS) • The value will be encoded • EncodedData = RtlpHpHeapGlobals.LfhKey^Blo ckOﬀsets^(subsegment >> 12) ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 52

Slide 52 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • BlockBitmap • Indicates the usage status of the block in the subsegment, and every two bits corresponds to a block • bit 0 : is_busy bit • bit 1 : unused bytes ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 53

Slide 53 text

Low FragmentationHeap • LFH Subsegments (_HEAP_LFH_SUBSEGMENT) • Block • The allocated memory return to user. • If the unused byte in the corresponding BlockBitmap is 1, then the last two bytes of the block are used to represent unused bytes • If there is only 1 byte, it will be recorded as 0x8000 ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) _HEAP_LFH_SUBSEGMENT 0x0 0x10 0x20 Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … 0x22 0x24 0x26 0x28 0x30

Slide 54

Slide 54 text

Low FragmentationHeap … BackendCtx (8bytes) Callbacks (0x28bytes) Conﬁg (4bytes) _HEAP_LFH_CONTEXT 0x0 0x8 0x3c … 0x80 … Buckets[0] Buckets[1] … 0x88 _HEAP_LFH_BUCKET … State (0x38bytes) TotalBlockCount TotalSubsegmentCount ReciprocalBlockSize … AfﬁnitySlots[] State ActiveSubsegment _HEAP_LFH_AFFINITY_SLOT IsBuckets BucketIndex … … AvailableSubsegmentCount AvailableSubsegmentList FullSubsegmentList _HEAP_LFH_SUBSEGMENT_OWNE ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … _HEAP_LFH_SUBSEGMENT

Slide 55

Slide 55 text

Low FragmentationHeap • Data Structure • Memory allocation mechanism

Slide 56

Slide 56 text

Low FragmentationHeap • Allocate • When allocating memory smaller than LfhContext- >Conﬁg.MaxBlockSize, it will ﬁrst check whether the corresponding bucket is LFH enabled • bucket index = RtlpLfhBucketIndexMap[needbytes+0xf] • Check : Buckets[index]->State & 1 == 1 (disable） • If it’s not enabled, it will update(RtlpHpLfhBucketUpdateStats) the usage times corresponding bucket.

Slide 57

Slide 57 text

Low FragmentationHeap • Allocate • When allocating memory smaller than LfhContext->Config.MaxBlockSize, it will first check whether the corresponding bucket is LFH enabled • The buckets[idx] will record the number of times when it is not enabled • When we allocated a chunk : the corresponding buckets[idx] += 0x210000 • Once (buckets[idx] >> 16) & 0x1f> 0x10 or (buckets[idx] >> 16)> 0xff00 it will enable LFH (RtlpHpLfhBucketActivate)

Slide 58

Slide 58 text

Low FragmentationHeap • Allocate • About enable LFH (RtlpHpLfhBucketActivate) • At the beginning, it will allocate the required structure of the bucket (RtlpHpHeapExtendContext) • Bucket, owner, aﬃnityslot … • The allocation method is directly allocated from _SEGMENG_HEAP- >AllocatedBase (usually pointing to the end of the segment heap structure) • After allocation, the bucket related structure will be initialized by RtlpHpLfhBucketInitialize

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Low FragmentationHeap _SEGMENT_HEAP … … Signature (4bytes) … AllocatedBase (8bytes) … LfhContext (0x4c0bytes) 0x10 0xe8 0x340 BackendCtx (8bytes) Callbacks (0x28bytes) Config (4bytes) _HEAP_LFH_CONTEXT … … Buckets[0] Buckets[1] … State (0x38bytes) **AffinitySlots … State ActiveSubsegment 0x800 0xa40 AffinitySlots[0] 0x8e0 0x860 _HEAP_LFH_BUCKET _HEAP_LFH_AFFINITY_SLOT *_HEAP_LFH_AFFINITY_SLOT

Slide 61

Slide 61 text

Low FragmentationHeap • Allocate • After LFH is enabled, LFH will be used as long as the block of the size is allocated • Implementation function is nt!RtlpHpLfhSlotAllocate • Next, it will check if there is an available block in ActiveSubsegment • The check is to take the lowest 12 bits of ActiveSubsegment (representing the number of blocks that can be allocated), and if there are available block, it will allocate from the subsegment • If not, it will conﬁrm whether the subsegment really has no block that can be allocated

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

• Allocate • Which block will be taken? • In the case that the subsegment have available blocks, it will be similar to the LFH block in NtHeap, it would take the value of RtlpLowFragHeapRandomData[x] ﬁrst. • Next time it will retrieve the value from RtlpLowFragHeapRandomData[x+1] • x is 1 byte ，x = rand() % 256 after 256 rounds • RtlpLowFragHeapRandomData is a 256-bytes array ﬁlled with random value • The range of random value is 0x0 - 0x7f Low FragmentationHeap

Slide 65

Slide 65 text

Low FragmentationHeap • Allocate • Next, it will retrieve value of Bitmap[Index] according to the value of subsegment->FreeHint • Index = (2*freehint) >> 6 • After retrieving the value of the Bitmap, it will look for the bit corresponding ﬁrst allocated block • Then adding the random index, it is the block to be allocated

Slide 66

Slide 66 text

Low FragmentationHeap • Allocate • If the block is allocated, it will tack the next nearest block • Finally, after updating the bit map and unused byte, it will return the memory block

Slide 67

Slide 67 text

Low FragmentationHeap • Get an index • Searchwidth = RtlpSearchWidth[Bucketindex]; • randval = RtlpLowFragHeapRandomData[x] • Bitmap = BlockBitmap[(2*freehint) >> 6] _HEAP_LFH_SUBSEGMENT ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[0] Block Block … BlockBitmap[1]

Slide 68

Slide 68 text

Low FragmentationHeap • Get an index • Firstnotfreeidx = ﬁrst not free block in Bitmap • val = (searchwidth *randval >> 7) & 0x1FFFFFE • blockindex = (ﬁrstnotfreeidx + val) & 0x3f _HEAP_LFH_SUBSEGMENT ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[0] Block Block … BlockBitmap[1]

Slide 69

Slide 69 text

Low FragmentationHeap • Check if the BusyBitmap correspond to index is 0 • If it is not zero, it will take the next nearest block • If it is zero, it will set the corresponding BlockBitmap, and conﬁrm whether there is unused byte _HEAP_LFH_SUBSEGMENT ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[0] Block Block … BlockBitmap[1]

Slide 70

Slide 70 text

Low FragmentationHeap • If there are unused byte, set unusedbyte at the end of the block • Return the block _HEAP_LFH_SUBSEGMENT ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[0] Block Block … BlockBitmap[1]

Slide 71

Slide 71 text

Low FragmentationHeap • Allocate • If the lowest 12 bit of the ActiveSubsegment is 0, it would check whether subsegment->FreeCount is greater than 1 • if the value is greater than 1 (indicating that the subsegment still have allocatable blocks), it will update the lowest 12 bit at the ActiveSubsegment • If the subsegment has no block that can be allocated, it will be ﬁlled from Buckets[idx]->State.AvailableSubsegmentList

Slide 72

Slide 72 text

Low FragmentationHeap • Allocate • In no subsegment available, It would allocate a new subsegment from the back-end allocator and initialize the subsegment (RtlpHpLfhSubsegmentCreate) • It will initialize BlockOffsets, BitMap, etc. • After initialization, it will add to AffinitySlots->State.AvailableSubsegmentList, and point subsegment->Owner back to AffinitySlots->State • AffinitySlots->ActiveSubsegment is also set to this subsegment and the lowest 12 bit is initialized

Slide 73

Slide 73 text

Low FragmentationHeap _SEGMENT_HEAP … … AllocatedBase (8bytes) … LfhContext (0x4c0bytes) 0xe8 0x340 BackendCtx (8bytes) Callbacks (0x28bytes) Config (4bytes) _HEAP_LFH_CONTEXT … … Buckets[0] Buckets[1] … State (0x38bytes) **AffinitySlots … State ActiveSubsegment 0x800 0xa40 AffinitySlots[0] 0x8e0 0x860 _HEAP_LFH_BUCKET _HEAP_LFH_AFFINITY_SLOT *_HEAP_LFH_AFFINITY_SLOT ListEntry (10bytes) Owner (8bytes) FreeCount (2bytes) BlockCount (2bytes) FreeHint (2bytes) Location (1byte) … BlockOffsets (4bytes) … … BlockBitmap[] Block … IsBuckets BucketIndex … … AvailableSubsegmentCount AvailableSubsegmentList FullSubsegmentList _HEAP_LFH_SUBSEGMENT_OWNER _HEAP_LFH_SUBSEGMENT SegContexts (0x180bytes) VsContext (0xc0ytes) 0x280 0x100

Slide 74

Slide 74 text

Low FragmentationHeap • Free • Main implementation function is nt!RtlpHpLfhSubsegmentFreeBlock • At the beginning, it will take subsegment.BlockOffsets first and decode it • According to the content of BlockOffsets, it will retrieve block size and firstblockoffset • According to the previous info, the index of block can be calculated • Clear the corresponding bitmap and set FreeCount = FreeCount + 1

Slide 75

Slide 75 text

Low FragmentationHeap • Free • If FreeCount == BlockCount-1 means that all the blocks of the subsegment are released, the subsegment will be removed from the AvaliableSubsegmentList • There will also be double linked list checks when removing here • It will release subsegment to backend Allocator

Slide 76

Slide 76 text

Memory Allocator in kernel • Segment Heap • Frontend Allocation • Low FragmentationHeap • Variable Size Allocation • Backend Allocation • Segment Allocation • Large Block Allocation

Slide 77

Slide 77 text

Variable Size Allocation • Size • Size <= 0x200 and not enable LFH • 0x200 < Size <= 0xfe0 • 0xfe0 < Size <= 0x20000 and (Size & 0xﬀf) != 0

Slide 78

Slide 78 text

Variable Size Allocation • Data Structure • Memory allocation mechanism

Slide 79

Slide 79 text

Variable Size Allocation • Chunk • The basic structure of memory allocator • In front of the chunk in VS allocation, there are some related metadata that record information of the chunk • _HEAP_VS_CHUNK_HEADER • _HEAP_VS_CHUNK_FREE_HEADER • Similar to back-end allocator in NtHeap

Slide 80

Slide 80 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • MemoryCost • Only used when freed User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit)

Slide 81

Slide 81 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • UnsafeSize • The size of chunk • The value is right shift by 4 bits • UnasfePrevSize • The size of previous chunk • The value is right shift by 4 bits User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit)

Slide 82

Slide 82 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • Allocated • Indicates whether the chunk is allocated • The value is 1, if it is allocated User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit)

Slide 83

Slide 83 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • Chunk header will be encoded • Encoded header is xor with • Chunk header • Chunk address • RtlpHpHeapGlobals.HeapKey User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit) Encode

Slide 84

Slide 84 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • EncodedSegmentPageOﬀset • Indicates the index of pages of the chunk in the VS subsegment • It is used to ﬁnd the VS subsegment • It will also be encoded. User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit)

Slide 85

Slide 85 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • EncodedSegmentPageOﬀset is xor with • (int8)Chunk address • (int8)RtlpHpHeapGlobals.HeapKe y • SegmentPageOﬀset User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit) Encode

Slide 86

Slide 86 text

Variable Size Allocation • _HEAP_VS_CHUNK_HEADER • UnusedBytes • Indicates whether the allocated chunk has unused memory User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) EncodedSegmentPageOffset (1byte) Inused UnusedBytes (1bit) SkipDuringWalk (1bit) Spare (22 bit)

Slide 87

Slide 87 text

Variable Size Allocation • _HEAP_VS_CHUNK_FREE_HEADER • ＭemoryCost • Indicates how many pages of memory need to be committed when the chunk is allocated User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) Freed Node (18byte)

Slide 88

Slide 88 text

Variable Size Allocation • _HEAP_VS_CHUNK_FREE_HEADER • 8 byte header is in the same as allocated chunk • Node (_RTL_BALANCED_NODE) • Node of rbtree and Freed chunk will be stored in a rbtree structure User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) Freed Node (18byte)

Slide 89

Slide 89 text

Variable Size Allocation • _HEAP_VS_CHUNK_FREE_HEADER • Node (RTL_BALANCED_NODE) • Children • Left/Right • The left and right subtrees of the Node • ParentValue • Parent node of this node User Data MemoryCost (2byte) UnsafeSize (2byte) UnsafePrevSize (2byte) Allocated (1byte) Padding (1byte) Freed Left (8byte) Right (8byte) ParentValue (8byte)

Slide 90

Slide 90 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • The core structure of the VS Allocator • Used to manage the memory allocated by VS allocator, and record all the information and structure of VS allocator in the heap

Slide 91

Slide 91 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • Lock • Just for lock • LockType (_RTLP_HP_LOCK_TYPE) • The type of Lock can be divided into • Paged/NonPaged/TypeMax LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 92

Slide 92 text

Variable Size Allocation • FreeChunkTree (_RTL_RB_TREE) • In VS Allocation, after free a chunk, the chunk will be placed in the FreeChunkTree of the heap, and the chunk will be inserted in the FreeChunkTree according to the size. • If the size of chunk is larger than the node, it will be placed in right subtree  otherwise, will be placed in left subtree • If there is no larger chunk than the chunk, the right subtree is NULL, and the other side is also • There will be a node check when taken out of the tree

Slide 93

Slide 93 text

Variable Size Allocation • FreeChunkTree (_RTL_RB_TREE) • Root • Point to the root of the rbtree • Encoded • Indicates whether the root has been encoded (default disable) • About encode : • EncodedRoot = Root ^ FreeChunkTree Encoded Root FreeChunkTree 0x0 0x8

Slide 94

Slide 94 text

Variable Size Allocation • FreeChunkTree (_RTL_RB_TREE) Left Right Chunk header (0x420) Parent (NULL) Left Right Chunk header (0x210) Parent Left Right Chunk header (0x510) Parent Encoded Root FreeChunkTree

Slide 95

Slide 95 text

Variable Size Allocation • DelayFreeContext (_HEAP_VS_DELAY_FREE_CONTEXT) • When enable delay free (default in kernel but disable in usermode) and size of chunk < 0x1000, after Free a chunk, it will not be free immediately, but will be added to a singly linked list called DelayFreeContext, until the number of chunks in the linked list exceeds 0x20, the chunks in the linked list will be freed at one time • Next pointer will be put at the beginning of user data • FILO • If you want to check whether delay free is enable, you can check VsContext- >Conﬁg

Slide 96

Slide 96 text

Variable Size Allocation • DelayFreeContext (_HEAP_VS_DELAY_FREE_CONTEXT) • Depth • The number of chunks in the Linked list • NextEntry • Point to next chunk • At this time chunk is still Allocated Depth (2byte) NextEntry (8byte) Sequence (6byte) DelayFreeContext Listhead

Slide 97

Slide 97 text

Variable Size Allocation • DelayFreeContext (_HEAP_VS_DELAY_FREE_CONTEXT) Depth (2byte) NextEntry (8byte) Sequence (6byte) DelayFreeContext Next Chunk header (16 byte) Null Chunk header (16 byte) Listhead

Slide 98

Slide 98 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • The memory pool of VS allocation • Once there is not enough for allocation, it will allocated from the backend allocator for a new subsegment • Each subsegment will be linked in a linked list

Slide 99

Slide 99 text

… Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • ListEntry(_LIST_ENTRY) • Flink • Point to next subsegment • Blink • Point to previous subsegment ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 100

Slide 100 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • ListEntry(_LIST_ENTRY) • The value would be encoded • Flink/Blink will xor with following value • Address of ListEntry • Address of next/prev subsegment … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 101

Slide 101 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • ListEntry(_LIST_ENTRY) • There is also double linked list check here … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 102

Slide 102 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • CommitBitmap • Indicates the status of page commit in the subsegment, and the page starts from the beginning of the subsegment • CommitLock • Lock used when Commit … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 103

Slide 103 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • Size • Size of VS subsegment • The value is right shift by 4 bits • Signature • Signature for veriﬁcation, make sure that the subsegment is found when free … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 104

Slide 104 text

Variable Size Allocation • VS Subsegment (_HEAP_VS_SUBSEGMENT) • Behind the VS subsegment header is the memory pool of VS Allocator. At the beginning, the memory pool of subsegment is a large chunk • Split when allocated • Coalesce to-be-freed block with neighbors … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) 0x0 0x10 0x18 0x20 0x22 Chunk header 0x30 Chunk header Chunk header

Slide 105

Slide 105 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • BackendCtx • Point to the Backend allocator (_HEAP_SEG_CONTEXT) structure used by the VS Allocator LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 106

Slide 106 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • Callbacks (_HEAP_SUBALLOCATOR_CALLBACKS) • Call back function table (Use to allocate/free subsegment) • Allocate • Free • Commit • Decommit • ExtendContext LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 107

Slide 107 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • Callbacks (_HEAP_SUBALLOCATOR_CALLBACKS) • Function pointer will be encoded • Callbacks will xor with following value • RtlpHpHeapGlobals.HeapKey • VsContext address • Function pointer LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 108

Slide 108 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • Conﬁg (_RTL_HP_VS_CONFIG) • Used to represent the attributes of the Vs Allocator • PageAlignLargeAllocs (default 1 in kernel) • FullDecommit • EnableDelayFree (default 1 in kernel) LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 109

Slide 109 text

Variable Size Allocation • VsContext (_HEAP_VS_CONTEXT) • Conﬁg (_RTL_HP_VS_CONFIG) • Used to represent the attributes of the Vs Allocator • PageAlignLargeAllocs (default 0 in user mode) • FullDecommit • EnableDelayFree (default 0 in usermode) LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0

Slide 110

Slide 110 text

Variable Size Allocation LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0 Encoded Root VS Subsegment ListEntry CommitBitmap CommitLock Size (2byte) Signature (15bit) FullCommit (1bit) Free Chunk header (0x300) Node Chunk header Chunk (0x230) Node Chunk (0x430) Node NextEntry

Slide 111

Slide 111 text

Variable Size Allocation LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0 Blink Flink VS Subsegment ListEntry CommitBitmap CommitLock Size (2byte) Signature (15bit) FullCommit (1bit) Free Chunk header (0x300) Node Chunk header ListEntry Subsegm ent NextEntry

Slide 112

Slide 112 text

Variable Size Allocation LockType VsContext Lock FreeChunkTree SubsegmentList … DelayFreeContext … BackendCtx Callbacks Conﬁg 0x0 0x8 0x10 0x20 0x40 0x80 0x88 0xb0 NextEntry Depth VS Subsegment ListEntry CommitBitmap CommitLock Size (2byte) Signature (15bit) FullCommit (1bit) Free Chunk header (0x300) Node Chunk header NextEntry

Slide 113

Slide 113 text

Variable Size Allocation • Data Structure • Memory allocation mechanism

Slide 114

Slide 114 text

Variable Size Allocation • Allocate • Main implementation function is nt!RtlpHpVsContextAllocateInternal • It will calculate the required chunk size at the beginning • Then it will ﬁnd a suitable chunk from FreeChunkTree in VsContext • Start searching from the root, when the required chunk is larger than the node, continue searching from the right subtree until it is found or is NULL

Slide 115

Slide 115 text

Variable Size Allocation • Allocate • If the chunk that can be allocated is not found, a subsegment will be allocated and the subsegment will be added to the VsContext, and then start searching from FreeChunkTree again • It will create a large chunk when it initialize the subsegment. • RtlpHpVsSubsegmentCreate • Request memory (RtlpHpSegVsAllocate) from backend, and create a new subsegment, the minimum size is (0x10000)

Slide 116

Slide 116 text

Variable Size Allocation • Allocate • RtlpHpVsContextAddSubsegment • Add subsegment to VsContext • It will determinate how to split the subsegment according to whether to enable PageAlignLargeAllocs • If PageAlignLargeAllocs is set, the subsegment will be split into two chunks, one is the ﬁrst chunk behind the subsegment structure, and the second is the page alignment, the user data of the chunk is aligned, and both chunks are added to FreeChunkTree • If not, the entire subsegment will be treated as a large chunk and added to FreeChunkTree

Slide 117

Slide 117 text

Variable Size Allocation … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) Chunk header (size:0xffd0) … ListEntry CommitBitmap CommitLock Size (2byte) VS Subsegment Signature (15bit) FullCommit (1bit) Chunk header (size:0xfb0) Chunk header (size:0xf020) 0xffff..2030 0xffff..2030 0xffff..2fe0 Split Page alignment

Slide 118

Slide 118 text

Variable Size Allocation • Allocate • If the found size of chunk size larger then request size, the chunk will be split, and the remaining chunks will be re-added to FreeChunkTree • RtlpHpVsChunkSplit • It will remove the chunk out of FreeChunkTree, and split the chunk, and re-added re-added FreeChunkTree as a new Freed chunk. • This will also be split according to whether the page alignment is or not. If the size of the chunk to be split exceeds 1 page, it split remainder chunk into two pieces according to the page • If request size < chunk size, unused byte will be recorded at the last 2 bytes of chunk

Slide 119

Slide 119 text

Variable Size Allocation … Chunk header (size:0x11020) … Chunk header (size:0x3920) 0xffff..3fe0 0xffff..3fe0 Split allocate 0x3920 Chunk header (size:0x6e0) 0xffff..7900 0xffff..8fe0 Chunk header (size:0xd020) Remainder

Slide 120

Slide 120 text

Variable Size Allocation • Free • Main implementation function is nt!RtlpHpVsContextFree • The signature and Allocated bytes of the subsegment will be verified at first • Subsegment->Size ^ Subsegment->Signature ^ 0x2BED == 0 • BSOD if verification fails

Slide 121

Slide 121 text

Variable Size Allocation • Free • Next, it will check if the number of chunks in VsContext- >DelayFreeContext is greater than 0x20 • VsContext->DelayFreeContext.Depth > 0x20 • If it is less than 0x20, the chunk will be inserted at the beginning of the linked list and then return • VsContext->DelayFreeContext.NextEntry

Slide 122

Slide 122 text

Variable Size Allocation • Free • If DelayFreeContext> 0x20, then the chunks in the linked list will be freed one by one • When it free a chunk, EncodedSegmentPageOﬀset will be used to ﬁnd the VS subsegment of the chunk, and verify the Allocated byte and segment signature

Slide 123

Slide 123 text

Variable Size Allocation • Free • Next, it will check whether the front and next chunks can be merged. If VsContext->Config.flag has enable PageAlignLargeAllocs, then one more chunk will be check.  The merged chunk will be moved out of FreeChunkTree first. There is a tree structure check. After the merge, update prev_size and Size • RtlpHpVsChunkCoalesce • After merging, if address of chunk+0x20 is the beginning of page, the chunk will be page aligned and split into two pieces

Slide 124

Slide 124 text

Variable Size Allocation • FreeChunkTree check Left Right Chunk header (0x420) Parent Left Right Chunk header (0x210) Parent Left Right Chunk header (0x510) Parent ＰＬＲ R->Parent->Right == R L->Parent->Left == L P->Left->Parent == P P->Right->Parent == P

Slide 125

Slide 125 text

Variable Size Allocation • Free • If the size of the merged chunk is exactly equal to the size of the subsegment, the entire subsegment will be moved out of the linked list. • There is also a double linked list check here and it will release the subsegment to the backend allocator • If it is not equal, it will calculate MemoryCost and Segmentpageoﬀset of the chunk and then encode it. • Finally, look for a suitable position in FreeChunkTree to insert

Slide 126

Slide 126 text

Pool Header • When the size of allocation <= 0xfe0 • Both LFH and VS Allocator will allocate additional 0x10 byte (64bit) • It is used to store the Pool Header. Pool Header is used for Pool Allocator before 19H1 . • Now it is basically used in a few cases such as CacheAligned, PoolQuota and PoolTrack • The header contains • Size • PreviousSize • Pool type • PoolIndex

Slide 127

Slide 127 text

Pool Header • When the size of allocation <= 0xfe0 Chunk header (16 byte) _POOL_HEADER (16 byte) _POOL_HEADER (16 byte) LFH VS Return ptr

Slide 128

Slide 128 text

Pool Header • Pool Header (_POOL_HEADER) • PreviousSize • Used in the CacheAligned case, indicating the oﬀset between the previous Pool header and the header • Pool Index • Useless in Segment Heap PoolIndex (1byte) PreviousSize (1byte) PoolType (1byte) BlockSize (1byte) PoolTag (4 bytes) ProcessBilled (8 bytes) 0x0 0x1 0x2 0x3 0x4 0x8 Pool header

Slide 129

Slide 129 text

Pool Header • Pool Header (_POOL_HEADER) • BlockSize • The size of the block • The value is right shift by 4 bits • PoolType • The pooltype of the block PoolIndex (1byte) PreviousSize (1byte) PoolType (1byte) BlockSize (1byte) PoolTag (4 bytes) ProcessBilled (8 bytes) 0x0 0x1 0x2 0x3 0x4 0x8 Pool header

Slide 130

Slide 130 text

Pool Header • Pool Header (_POOL_HEADER) • PoolTag • The tag string is ﬁlled in when the block is allocated • When you use ExAllocatePoolWithTag, you can specify the pooltag • ProcessBilled • Used for PoolQuota and CacheAligned PoolIndex (1byte) PreviousSize (1byte) PoolType (1byte) BlockSize (1byte) PoolTag (4 bytes) ProcessBilled (8 bytes) 0x0 0x1 0x2 0x3 0x4 0x8 Pool header

Slide 131

Slide 131 text

Dynamic Lookaside • If the size of allocation 0x201

Slide 132

Slide 132 text

Dynamic Lookaside • _SEGMENT_HEAP • UserContext (_RTL_DYNAMIC_LOOKASIDE) • The core structure of the dynamic lookaside … … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 0x0 EnvHandle (10bytes) … UserContext (8bytes) … … 0x28

Slide 133

Slide 133 text

Dynamic Lookaside • UserContext (_RTL_DYNAMIC_LOOKASIDE) • EnabledBucketBitmap • A bitmap is used to indicate which buckets have enable lookaside • BucketCount • The total number of buckets in lookaside EnabledBucketBitmap (8bytes) BucketCount (4bytes) ActiveBucketCount (4bytes) Buckets[64] (0x40bytes) 0x0 0x8 0xc 0x40 _RTL_DYNAMIC_LOOKASIDE

Slide 134

Slide 134 text

Dynamic Lookaside • UserContext (_RTL_DYNAMIC_LOOKASIDE) • ActiveBucketCount • The number of buckets with enable lookaside • Buckets (_RTL_LOOKASIDE) • Used to manage the structure of diﬀerent sizes of lookaside, which is where the lookaside are EnabledBucketBitmap (8bytes) BucketCount (4bytes) ActiveBucketCount (4bytes) Buckets[64] (0x40bytes) 0x0 0x8 0xc 0x40 _RTL_DYNAMIC_LOOKASIDE

Slide 135

Slide 135 text

Dynamic Lookaside • Buckets (_RTL_LOOKASIDE) • ListHead (_SLIST_HEADER) • The structure of the head of the Singly linked list contains the length of the linked list, and the linked list itself, which is common in windows kernel • Depth • The number of chunks that can be stored in the bucket ListHead (16bytes) Depth (2bytes) … 0x0 0x10 _RTL_LOOKASIDE

Slide 136

Slide 136 text

Dynamic Lookaside • _SLIST_HEADER • Depth • The number of nodes in the linked list • NextEntry • Point to next node • In the Dynamic Lookaside, it points to the Userdata of the freed chunk, behind the pool header Depth (2bytes) … NextEntry (8 byte) 0x0 0x8 _SLIST_HEADER

Slide 137

Slide 137 text

Dynamic Lookaside … … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 0x0 EnvHandle (10bytes) … UserContext (8bytes) … … 0x28 EnabledBucketBitmap (8bytes) BucketCount (4bytes) ActiveBucketCount (4bytes) Buckets[0] (0x40bytes) _RTL_DYNAMIC_LOOKASIDE Depth (2bytes) … _RTL_LOOKASIDE Depth (2bytes) … NextEntry (8 byte) Chunk header (16 byte) _POOL_HEADER (16 byte) VS chunk … Buckets[1] (0x40bytes)

Slide 138

Slide 138 text

Memory Allocator in kernel • Segment Heap • Frontend Allocation • Low FragmentationHeap • Variable Size Allocation • Backend Allocation • Segment Allocation • Large Block Allocation

Slide 139

Slide 139 text

Segment Allocation • Size (in the kernel) • Size is a multiple of page • Size <= 0x7f0000 • Size is not a multiple of page • 0x20000 < Size <= 0x7f0000 • It’s a bit diﬀerent in userland.

Slide 140

Slide 140 text

Segment Allocation • it can be divided into two categories. The diﬀerence between the two is that the block units are diﬀerent. • 0x20000 < Size <= 0x7f000 • 0x1000 as the unit of block • 0x7f000 < Size <= 0x7f0000 • 0x10000 as the unit of block • Both unit are called page here

Slide 141

Slide 141 text

Segment Allocation • Data Structure • Memory allocation mechanism

Slide 142

Slide 142 text

Segment Allocation • Page • In the case of size <= 0x7f000, the allocation unit of Segment Allocation • For example, if you allocate 0x1450 bytes, it would allocate 2 page • 0x7f000 < Size <= 0x7f0000 • It will use 0x10000 as the allocation unit • In the following slides, I will use 0x1000 as an example Page Page Page Page 0x1000 0x10000 Pages

Slide 143

Slide 143 text

Segment Allocation • Block (chunk) • Consists of single or multiple pages • The memory block allocated by Segment Allocatior • As shown in the ﬁgure, the block is composed of 3 pages Page Page Page Page 0x1000 Block

Slide 144

Slide 144 text

Segment Allocation • Page Segment (_HEAP_PAGE_SEGMENT) • memory pool of segment allocation • Once there is no available page segment, a new page segment will be allocated from the system(MmAllocatePoolMemory) , but only the required structure will be allocated at the beginning • Each page segment will be inserted into a double linked list

Slide 145

Slide 145 text

Segment Allocation • _HEAP_PAGE_SEGMENT • ListEntry (_LIST_ENTRY) • Flink • Point to the next page segment in the Linked list • Blink • Point to the previous page segment in the Linked list … ListEntry Signature _HEAP_PAGE_SEGMENT … … DescArray[256] 0x0 0x10 0x40 Pages 0x2000   0x10000

Slide 146

Slide 146 text

Segment Allocation • _HEAP_PAGE_SEGMENT • Signature • The signature of the page segment is used to verify the page segment • Signature will xor with following value • Address of page segment • Address of Segcontext • RtlpHpHeapGlobals • 0xA2E64EADA2E64EAD … ListEntry Signature _HEAP_PAGE_SEGMENT … … DescArray[256] 0x0 0x10 0x40 Pages 0x2000 0xA2E64EADA2E64EAD

Slide 147

Slide 147 text

• _HEAP_PAGE_SEGMENT • DescArray (_HEAP_PAGE_RANGE_DESCRIPTOR) • Page range descriptor array • Each element corresponds to each page one by one • Pages • Ｍemory pool of segment allocator Segment Allocation … ListEntry Signature _HEAP_PAGE_SEGMENT … … DescArray[256] 0x0 0x10 0x40 Pages 0x2000 0xA2E64EADA2E64EAD

Slide 148

Slide 148 text

Segment Allocation • Page range descriptor (_HEAP_PAGE_RANGE_DESCRIPTOR) • Descriptor for page • Indicates the status (Allocated or Freed) and information of each page in the page segment (whether the page is the beginning of a block, size of block, etc.) • It can be divided into allocated and freed • The page range descriptor in the freed state will be stored in FreePageRanges which is a rbtree structure

Slide 149

Slide 149 text

Segment Allocation … ListEntry Signature _HEAP_PAGE_SEGMENT … … DescArray[0x02] 0x0 0x10 0x20 0x2000 DescArray[0x03] … Page Page … 0x3000 … 0x40

Slide 150

Slide 150 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • TreeSignature • Signature of page range descriptor • The values is always 0xccddccdd • Only at the beginning of Block • UnusedBytes • Unusedbytes in an allocated block Segment Allocation TreeSignature UnusedBytes Allocated … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 Key (4 bytes) 0x1c …

Slide 151

Slide 151 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • RangeFlag • Used to indicate the page status • Bit 1 : allocated bit • Bit 2 : block header bit • Bit 3 : Committed • LFH : RangeFlag & 0xc = 8 • VS : RangeFlag & 0xc = 0xc Segment Allocation TreeSignature UnusedBytes Allocated … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 Key (4 bytes) 0x1c …

Slide 152

Slide 152 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • CommitedPageCount • Indicates the number of pages committed in the corresponding page Segment Allocation TreeSignature UnusedBytes Allocated … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 Key (4 bytes) 0x1c …

Slide 153

Slide 153 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • Key(_HEAP_DESCRIPTOR_KEY) • Store the size of the page corresponding to the page descriptor and the number of committed pages Segment Allocation TreeSignature UnusedBytes Allocated … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 Key (4 bytes) 0x1c …

Slide 154

Slide 154 text

• _HEAP_DESCRIPTOR_KEY • EncodedCommittedPageCount • ~EncodedCommittedPageCount is the number of pages committed in the block • Only used in block header • UniCount • The size of Block • Value is page count Segment Allocation EncodedCommittedPageCount (2bytes) _HEAP_DESCRIPTOR_KEY 0x0 0xA2E64EADA2E64EAD LargePageCost (1bytes) 0x2 UnitCount (1bytes) 0x3

Slide 155

Slide 155 text

Segment Allocation TreeSignature UnusedBytes Allocated … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 Key (4 bytes) 0x1c … EncodedCommittedPageCount (2bytes) _HEAP_DESCRIPTOR_KEY LargePageCost (1bytes) UnitCount (1bytes) 0x1c 0x1f

Slide 156

Slide 156 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • UnitOffset • If page is not block header • The UnitCount field will be called UnitOffset • Indicates the offset of the page in the block Segment Allocation TreeSignature UnusedBytes Allocated and not header … RangeFlag (1byte) 0x0 0x4 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 … 0x1c … … UnitOffset (1byte) 0x1f

Slide 157

Slide 157 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • TreeNode (_RTL_BALANCED_NODE) • The node in the FreePageRanges Segment Allocation TreeNode (0x18 bytes) Freed RangeFlag (1byte) 0x0 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 … 0x1c Key (4 bytes)

Slide 158

Slide 158 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • TreeNode (_RTL_BALANCED_NODE) • Left • Point to a page descriptor that block size is smaller than its. • Right • Point to a page descriptor that block size is greater than its. • ParentValue • Point to parent node • The lowest 1 bit will determine whether to encode Parent Segment Allocation TreeNode Freed RangeFlag (1byte) 0x0 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 … Left Right ParentValue 0x8 0x10 Key (4 bytes)

Slide 159

Slide 159 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • Key(_HEAP_DESCRIPTOR_KEY) • Same as the case of allocated Segment Allocation TreeNode (0x18 bytes) Freed RangeFlag (1byte) 0x0 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 … Key (4 bytes)

Slide 160

Slide 160 text

• _HEAP_PAGE_RANGE_DESCRIPTOR • UnitOffset • If page is not block header • The UnitCount field will be called UnitOffset • Indicates the offset of the page in the block Segment Allocation TreeNode (0x18 bytes) Freed and not header RangeFlag (1byte) 0x0 0xA2E64EADA2E64EAD 0x18 CommittedPageCount (1byte) 0x19 … … 0x1c … UnitOffset (1byte) 0x1f

Slide 161

Slide 161 text

Segment Allocation • SegContexts (_HEAP_SEG_CONTEXT) • The core structure of the Segment allocation • Used to manage the memory allocated by segment allocator, and record all the information and structure of segment allocator in the heap • There are two SegContexts in each Heap • Size <= 0x7f000 • 0x7f000 < Size < 0x7f0000

Slide 162

Slide 162 text

• SegContexts (_HEAP_SEG_CONTEXT) • SegmentMask • A mask used to ﬁnd Page Segment • Page segment = block ptr & SegmentMask • UnitShift • Used to calculate the index of the page descriptor • Index = block ptr >> UnitShit Segment Allocation SegmentMask (8bytes) UnitShift (1bytes) SegContext PagesPerUnitShift (1byte) FirstDescriptorIndex (1byte) 0x0 0x8 0xA2E64EADA2E64EAD 0x9 … 0xa VsContext (8bytes) 0x18 LfhContext (8bytes) … Heap (8bytes) 0x38 0x20 SegmentListHead (10bytes) … 0x48 … FreePageRanges (10bytes) 0x60

Slide 163

Slide 163 text

• SegContexts (_HEAP_SEG_CONTEXT) • PagePerUnitShift • (0x1 << PagePerUnitShift) indicates the size of a page in the SegContext • The size of page unit is (0x1 << PagePerUnitShift)*0x1000 • If the value is zero • Page unit is 0x1000 • FirstDescriptorIndex • The index of the ﬁrst Page Descriptor in the SegContext Segment Allocation SegmentMask (8bytes) UnitShift (1bytes) SegContext PagesPerUnitShift (1byte) FirstDescriptorIndex (1byte) 0x0 0x8 0xA2E64EADA2E64EAD 0x9 … 0xa VsContext (8bytes) 0x18 LfhContext (8bytes) … Heap (8bytes) 0x38 0x20 SegmentListHead (10bytes) … 0x48 … FreePageRanges (10bytes) 0x60

Slide 164

Slide 164 text

• SegContexts (_HEAP_SEG_CONTEXT) • LfhContext (_HEAP_LFH_CONTEXT) • Point to the LFH allocator in the segment heap • VsContext (_HEAP_VS_CONTEXT) • Point to the VS allocator in the segment heap Segment Allocation SegmentMask (8bytes) UnitShift (1bytes) SegContext PagesPerUnitShift (1byte) FirstDescriptorIndex (1byte) 0x0 0x8 0xA2E64EADA2E64EAD 0x9 … 0xa VsContext (8bytes) 0x18 LfhContext (8bytes) … Heap (8bytes) 0x38 0x20 SegmentListHead (10bytes) … 0x48 … FreePageRanges (10bytes) 0x60

Slide 165

Slide 165 text

• SegContexts (_HEAP_SEG_CONTEXT) • Heap(_SEGMENT_HEAP) • Point to the segment heap to which it belongs • SegmentListHead (_LIST_ENTRY) • Point to the page segment in the segment allocator • The linked list is a double linked list with integrity check Segment Allocation SegmentMask (8bytes) UnitShift (1bytes) SegContext PagesPerUnitShift (1byte) FirstDescriptorIndex (1byte) 0x0 0x8 0xA2E64EADA2E64EAD 0x9 … 0xa VsContext (8bytes) 0x18 LfhContext (8bytes) … Heap (8bytes) 0x38 0x20 SegmentListHead (10bytes) … 0x48 … FreePageRanges (10bytes) 0x60

Slide 166

Slide 166 text

Segment Allocation • FreePageRanges (_RTL_RB_TREE) • In Segment Allocation, after releasing a block, the page descriptor of the block will be inserted into the FreePageRanges of the SegContext according to the size • If the block size is greater than the node, the page descriptor will be inserted into the right subtree, otherwise will be will be inserted into left subtree. • If there is no greater than the page descriptor, the right subtree is NULL and the other side is also • There will be a node check when the node is taken out of the tree

Slide 167

Slide 167 text

• SegContexts (_HEAP_SEG_CONTEXT) • FreePageRanges (_RTL_RB_TREE) • Root • Point to the root of the rbtree • Encoded • Indicates whether the root has been encoded (default disable) • About encode • EncodedRoot = Root ^ FreeChunkTree Segment Allocation 0xA2E64EADA2E64EAD Encoded Root FreePageRanges 0x0 0x8

Slide 168

Slide 168 text

Segment Allocation 0x0 0x8 0x9 0xa 0x18 0x38 0x20 0x48 SegmentMask UnitShift (0xc) SegContext PagesPerUnitShift (0x0) FirstDescriptorIndex (0x2) … VsContext LfhContext … Heap SegmentListHead … … FreePageRanges 0x60 Page Segment … ListEntry Signature … … DescArray[0x02] DescArray[0x03] … Page 0x2 Page 0x3 … Freed Block Blink Flink Encoded Root TreeNode RangeFlag (0x2) … UnitCount (0x2) 0x2000 Page Descriptor

Slide 169

Slide 169 text

Segment Allocation … _HEAP_PAGE_SEGMENT 0x0 0x40 Block 0x2000 DescArray[0x02] DescArray[0x03] 0x60 … _HEAP_VS_SUBSEGMENT Chunk header Chunk header Chunk header … _HEAP_PAGE_SEGMENT 0x0 0x40 Block 0x2000 DescArray[0x02] DescArray[0x03] 0x60 … _HEAP_LFH_SUBSEGMENT LFH Block LFH Block

Slide 170

Slide 170 text

Segment Allocation • Data Structure • Memory allocation mechanism

Slide 171

Slide 171 text

Segment Allocation • Allocate • The allocation is based on page as the unit to allocate, and divided into • Size <= 0x7f000 • Page uses 0x1000 bytes as a unit • 0x7f000 < Size < 0x7f0000 • Page uses 0x10000 bytes as a unit

Slide 172

Slide 172 text

Segment Allocation • Allocate • The allocation is based on page as the unit to allocate • For example, if it allocate 0x1337 bytes, segment allocation will allocate 0x2000, which is 2 page units, and the extra memory 0x2000-0x1337 will be recorded in unused byte

Slide 173

Slide 173 text

Segment Allocation • Allocate • Main implementation function is nt!RtlpHpSegAlloc • It will use RtlpHpSegPageRangeAllocate to get freed page descriptor or create a new page descriptor • First, it will search from FreePageRanges. • Start searching from the root, when the required block is larger than the node, continue searching from the right subtree until it is found or is NULL

Slide 174

Slide 174 text

Segment Allocation • Allocate • If no suitable page descriptor is found, a new page segment will be allocated and the first page descriptor of the page segment will be initialized, and then this page descriptor will be used for allocated. The page segment will be inserted into the SegmentListHead. • In fact, it only allocated the memory required for the page segment and page descriptor structure, and the block part is not allocated at first • RtlpHpSegSegmentAllocate : Allocate page segment • RtlpHpSegSegmentInitialize : Initialize the first page descriptor • RtlpHpSegHeapAddSegment : insert into SegmentListHead

Slide 175

Slide 175 text

Segment Allocation • Allocate • If it found a suitable or created a page descriptor, the page descriptor will be removed from FreePageRanges • When the required number of pages is smaller than the block, splitting will be done. • The page descriptor corresponding to the remainder block will be inserted into FreePageRanges

Slide 176

Slide 176 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block TreeNode RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x2) First

Slide 177

Slide 177 text

Slide 178

Slide 178 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block NULL RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x2) First Remove node Allocate 0x1337

Slide 179

Slide 179 text

Slide 180

Slide 180 text

Slide 181

Slide 181 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block NULL RangeFlag (0x2) … UnitSize (0x2) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) Unused RangeFlag (0x2) CommittedPageCount (1) UnitOffset (0x0) First Split Block  and update  page descriptor   of next  block Allocate 0x1337 Block First

Slide 182

Slide 182 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block NULL RangeFlag (0x2) … UnitSize (0x2) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) Unused RangeFlag (0x2) CommittedPageCount (1) UnitOffset (0x0) First Insert   page descriptor  of next block   to   FreePageRange Allocate 0x1337 Block First

Slide 183

Slide 183 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block NULL RangeFlag (0x2) … UnitSize (0x2) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeNode RangeFlag (0x2) CommittedPageCount (1) UnitOffset (0x0) First Update   page   descriptor  of allocated  block Allocate 0x1337 Block First

Slide 184

Slide 184 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Block TreeSignature RangeFlag (0x3) … UnitSize (0x2) Unused RangeFlag (1) CommittedPageCount (1) UnitOffset (0x1) TreeNode RangeFlag (0x2) CommittedPageCount (1) UnitOffset (0x0) First | Allocated Allocate 0x1337 Block First Allocated

Slide 185

Slide 185 text

Segment Allocation • Allocate • After setting the page descriptor, it will check whether all pages in the block are committed • It will add up the CommittedPageCount of all page descriptors in the block • If the block needs committed, it will commit memory to the speciﬁed VA • RtlpHpSegMgrCommit ->RtlpHpAllocVA->MmAllocatePoolMemory • Then update the CommittedPagecount of all page descriptors corresponding to the block

Slide 186

Slide 186 text

Segment Allocation • Allocate • Finally, it will return the block that correspond to the page descriptor. • Block = (Page descriptor & SegmentMask) + ((index of Page descriptor) << SegContext->Unitshift)

Slide 187

Slide 187 text

Segment Allocation • Free • At the beginning, it will verify whether the page segment where the ptr is located is legal • RtlpHpSegDescriptorValidate • Verify: Page segment->signature • 0xA2E64EADA2E64EAD == Page segment^SegContext^ RtlpHpHeapGlobals^signature • Verify that the page descriptor of the page where the ptr is located is Allocated • Double Free check

Slide 188

Slide 188 text

Segment Allocation • Free • Remark • from free pointer to page segment • Free pointer & segment mask • Page descriptor =   page_segment + 0x20 * ((free pointer - page segment) >> segcontext->UnitShift) • _HEAP_SEG_CONTEXT =  (page segment)^(page segment- >Signature)^0xA2E64EADA2E64EAD^RtlpHpHeapGlobals.HeapKey 

Slide 189

Slide 189 text

Segment Allocation • Free • Next, we will see if the free pointer is at the beginning of block • If free pointer is not at the beginning of the block, it will check the RangeFlag of the Page descriptor to determine whether to use VS Allocator or Lfh Allocator to release the memory • If free pointer is not at the beginning of the block, it means that the free pointer is managed by segment allocation, and RtlpHpSegPageRangeShrink will be used

Slide 190

Slide 190 text

Segment Allocation • Free • Then the Allocated bit of page descriptor correspond to block will be cleared, and it will check whether the previous and following blocks are Freed. If it is Freed, it will be merged • RtlpHpSegPageRangeCoalesce • The way to ﬁnd the previous block is to check whether the page descriptor of previous page is at the beginning of the block. If it is not the beginning, it will use the UnitOﬀset of the page descriptor of previous page to calculate the page descriptor of the previous block. • The following is calculated using the UnitCount of the current page descriptor • Determine whether the RangeFlag of the Page descriptor at the beginning of the block is Allocated • It use page descriptor of the block at the beginning to check whether the block is allocated.

Slide 191

Slide 191 text

Segment Allocation • Free • If the previous block is Free • Remove the Page descriptor of the previous block from FreePageRanges • Clear the ﬁrst bit of the RangeFlag of the Page descriptor of the Block which we want to free • Update the UnitCount of the Page descriptor of the previous block • Update UnitOﬀset of the last page descriptor of Block after merge

Slide 192

Slide 192 text

Segment Allocation • Free • If the following block is Free • Remove the page descriptor of following block from FreePageRanges • Clear the ﬁrst bit of the RangeFlag of the Page descriptor of the following block • Update the UnitCount of the Page descriptor of the block which we want to free • Update UnitOﬀset of the last page descriptor of Block after merge

Slide 193

Slide 193 text

Segment Allocation • Free • Finally, inserted it to FreePageRanges according to the block size

Slide 194

Slide 194 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeNode RangeFlag (0x2) … UnitSize (0x2) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x3) CommittedPageCount (1) UnitSize (0x1) First Free 0x4000 Block First | Allocated

Slide 195

Slide 195 text

Slide 196

Slide 196 text

Slide 197

Slide 197 text

Slide 198

Slide 198 text

Slide 199

Slide 199 text

Slide 200

Slide 200 text

Slide 201

Slide 201 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeSignature RangeFlag (0x2) … UnitSize (0x2) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x3) CommittedPageCount (1) UnitSize (0x1) First Free 0x4000 Block First | Allocated Update  size of prev  block

Slide 202

Slide 202 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeSignature RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x3) CommittedPageCount (1) UnitSize (0x1) First Free 0x4000 First | Allocated Clean Allcated  and ﬁrst bit   of page   descriptor

Slide 203

Slide 203 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeSignature RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x0) CommittedPageCount (1) UnitSize (0x1) First Free 0x4000 Set the last  page descriptor  of the free block

Slide 204

Slide 204 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeSignature RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x0) CommittedPageCount (1) UnitOffset (0x2) First Free 0x4000 Insert  the page   descriptor to  FreePageRages

Slide 205

Slide 205 text

Segment Allocation … _HEAP_PAGE_SEGMENT DescArray[0x02] DescArray[0x03] … Page 2 Page 3 Page 4 Page segment DescArray[0x04] 0x0 0x40 0x60 0x80 0x2000 0x3000 0x4000 … Free Block TreeNode RangeFlag (0x2) … UnitSize (0x3) Unused RangeFlag (0) CommittedPageCount (1) UnitOffset (0x1) TreeSignature RangeFlag (0x0) CommittedPageCount (1) UnitOffset (0x2) First Free 0x4000

Slide 206

Slide 206 text

Memory Allocator in kernel • Segment Heap • Frontend Allocation • Low FragmentationHeap • Variable Size Allocation • Backend Allocation • Segment Allocation • Large Block Allocation

Slide 207

Slide 207 text

Large Block Allocation • Size • Size > 0x7f0000

Slide 208

Slide 208 text

Large Block Allocation • Only use rbtree to manage. Compared to other Allocations, it is much simpler. In fact, it almost directly allocate for a large block of memory from the system and stores it with rbtree. • Release is also removed from rbtree and returned to the system directly.

Slide 209

Slide 209 text

Large Block Allocation • Data Structure • Memory allocation mechanism

Slide 210

Slide 210 text

Large Block Allocation • _SEGMENT_HEAP • LargeAllocMetadata (_RTL_RB_TREE) • Used to manage the allocated memory by Large Allocation • It is also a rbree structure, and the address of allocated memory is used as KEY … … Dfa … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP … LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) … VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 EnvHandle (10bytes) 0x0

Slide 211

Slide 211 text

Large Block Allocation • _SEGMENT_HEAP • LargeReservedPages • Number of pages reserved for Large Block Allocation • LargeCommittedPages • Number of committed pages for Large Allocation … … Dfa … Signature (4bytes) LargeAllocMetadata (10bytes) _SEGMENT_HEAP … LargeReservedPages (8bytes) LargeCommittedPages (8bytes) … AllocatedBase (8bytes) SegContexts (0x180bytes) … VsContext (0xc0bytes) LfhContext (0x4c0bytes) 0x10 0x48 0x58 0x60 0xe8 0x100 0x280 0x340 EnvHandle (10bytes) 0x0

Slide 212

Slide 212 text

Large Block Allocation • _HEAP_LARGE_ALLOC_DATA • TreeNode (_RTL_BALANCED_NODE) • Left • Point to a node whose VirtualAddress is smaller than the node • Right • Point to a Node whose VirtualAddress is greater than the Node • ParentValue is pointed to parent node • The lowest 1 bit will determine whether to encode Parent node VirtualAddress (8 bytes) … _HEAP_LARGE_ALLOC_DATA 0x0 0x18 AllocatedPages (52 bit) 0x20:12 Left Right ParentValue

Slide 213

Slide 213 text

Large Block Allocation • _HEAP_LARGE_ALLOC_DATA • VirtualAddress • The lowest 16 bits as Unusedbytes • Address of large block VirtualAddress (8 bytes) … _HEAP_LARGE_ALLOC_DATA 0x0 0x18 AllocatedPages (52 bit) 0x20:12 TreeNode (0x18 bytes)

Slide 214

Slide 214 text

Large Block Allocation • _HEAP_LARGE_ALLOC_DATA • AllocatedPages • The number of allocated pages VirtualAddress (8 bytes) … _HEAP_LARGE_ALLOC_DATA 0x0 0x18 AllocatedPages (52 bit) 0x20:12 TreeNode (0x18 bytes)

Slide 215

Slide 215 text

Large Block Allocation • Data Structure • Memory allocation mechanism

Slide 216

Slide 216 text

Large Block Allocation • Allocate • The allocation is based on page as the unit. • Main implementation function is nt!RtlpHpMetadataAlloc • At the beginning, it will allocate memory for storing Large block Metadata (_HEAP_LARGE_ALLOC_DATA) • RtlpHpMetadataHeapCtxGet & RtlpHpMetadataHeapStart • It will determine which Heap to allocate from according to the value of Segment heap->EnvHandle • ExPoolState->HeapManager.MetadataHeaps[idx]

Slide 217

Slide 217 text

Large Block Allocation • Allocate • The allocation is based on page as the unit. • Main implementation function is nt!RtlpHpMetadataAlloc • Next, it will use RtlpHpAllocVA to allocated memory, and then store the VirtualAddress in Metadata • Finally, insert the Metadtata into Segment heap->LargeAllocMetadata

Slide 218

Slide 218 text

Large Block Allocation • Free • Main implementation function is (RtlpHpLargeFree) • First ﬁnd the node correspond to the free ptr from Segment heap- >LargeAllocMetadata, and remove the node • Use RtlpHpFreeVA to release memory • Finally release the memory storing Metadata (RtlpHpMetadataFree) of the released memory

Slide 219

Slide 219 text

Reference • Segment Heap • https://www.blackhat.com/docs/us-16/materials/us-16-Yason- Windows-10-Segment-Heap-Internals.pdf • https://www.sstic.org/media/SSTIC2020/SSTIC-actes/ pool_overﬂow_exploitation_since_windows_10_19h1/SSTIC2020- Article-pool_overﬂow_exploitation_since_windows_10_19h1- bayet_fariello.pdf

Slide 220

Slide 220 text

About Part 2 • We will talk about window kernel pool exploit • Way to spray and defragment in diﬀerent heap • Metadata corruption in segment heap • …etc