Slide 1

Slide 1 text

Introduction to the Linux Block I/O Layer Johannes Thumshirn – SUSE Labs [email protected]

Slide 2

Slide 2 text

2 Agenda • I/O Architecture • Scalability issues • Single Queue vs. Multi Queue • Queue depth vs. Multiple queues

Slide 3

Slide 3 text

3 I/O Architecture (simplified) Buffer cache VFS File System Disk Block Layer Device Driver SCSI ML Application Userspace Kernel

Slide 4

Slide 4 text

4 Submission Handling (SQ) • Per-process pluglist ‒ Analogy of a plug in a kitchen sink • I/O – Scheduler ‒ NOOP ‒ Deadline ‒ CFQ • Dispatch queue

Slide 5

Slide 5 text

5 Scalability Issues App CPU 1 App CPU 2 App CPU 3 App CPU 4 File System Block Layer Driver

Slide 6

Slide 6 text

6 Single Queue vs. Multiple Queues • Software queues are per-cpu ‒ No more globally shared state → no lock contention • Software queues are mapped to hardware queues ‒ Only very few devices support multiple queues • Completion IRQs on “submission-local” CPU ‒ Reduction of IPIs ‒ Keep cache lines hot • Drawback: No I/O scheduler

Slide 7

Slide 7 text

7 Queue Depth vs. Multiqueue CPU CPU CPU CPU CPU Req “ classic” single queue CPU CPU Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... single queue with TCQ/NCQ ... ... ... ... ... ... ... multi queue

Slide 8

Slide 8 text

8 Request tagging https://upload.wikimedia.org/wikipedia/commons/6/6a/Peg_dhcp_from_what_the_hack_holland_2005.jpg

Slide 9

Slide 9 text

9 Software Queues vs. Hardware Queues APP SW Q SW Q SW Q SW Q HW Q HW Q HW Q HW Q APP SW Q SW Q SW Q SW Q

Slide 10

Slide 10 text

10 References • Jan Kara @ Kernel recipes 2015 ‒ https://youtu.be/2tu__ZHC0mI • Jens Axboe @ Kernel recipes ‒ https://youtu.be/VIdKBD9-Ozg

Slide 11

Slide 11 text

Questions?

Slide 12

Slide 12 text

Thank you. Join the conversation, contribute & have a lot of fun! www.opensuse.org

Slide 13

Slide 13 text

13 Have a Lot of Fun, and Join Us At: www.opensuse.org

Slide 14

Slide 14 text

General Disclaimer This document is not to be construed as a promise by any participating organisation to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. openSUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for openSUSE products remains at the sole discretion of openSUSE. Further, openSUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All openSUSE marks referenced in this presentation are trademarks or registered trademarks of SUSE LLC, in the United States and other countries. All third-party trademarks are the property of their respective owners. License This slide deck is licensed under the Creative Commons Attribution-ShareAlike 4.0 International license. It can be shared and adapted for any purpose (even commercially) as long as Attribution is given and any derivative work is distributed under the same license. Details can be found at https://creativecommons.org/licenses/by-sa/4.0/ Credits Template Richard Brown [email protected] Design & Inspiration openSUSE Design Team http://opensuse.github.io/branding- guidelines/