Introduction to the Linux Block I/O Layer Johannes Thumshirn – SUSE Labs [email protected] 

2 Agenda • I/O Architecture • Scalability issues • Single Queue vs. Multi Queue • Queue depth vs. Multiple queues 

3 I/O Architecture (simplified) Buffer cache VFS File System Disk Block Layer Device Driver SCSI ML Application Userspace Kernel 

4 Submission Handling (SQ) • Per-process pluglist ‒ Analogy of a plug in a kitchen sink • I/O – Scheduler ‒ NOOP ‒ Deadline ‒ CFQ • Dispatch queue 

5 Scalability Issues App CPU 1 App CPU 2 App CPU 3 App CPU 4 File System Block Layer Driver 

6 Single Queue vs. Multiple Queues • Software queues are per-cpu ‒ No more globally shared state → no lock contention • Software queues are mapped to hardware queues ‒ Only very few devices support multiple queues • Completion IRQs on “submission-local” CPU ‒ Reduction of IPIs ‒ Keep cache lines hot • Drawback: No I/O scheduler 

7 Queue Depth vs. Multiqueue CPU CPU CPU CPU CPU Req “ classic” single queue CPU CPU Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... Req 1 Req 2 Req 3 Req 4 Req 5 Req N ... single queue with TCQ/NCQ ... ... ... ... ... ... ... multi queue 

8 Request tagging https://upload.wikimedia.org/wikipedia/commons/6/6a/Peg_dhcp_from_what_the_hack_holland_2005.jpg 

9 Software Queues vs. Hardware Queues APP SW Q SW Q SW Q SW Q HW Q HW Q HW Q HW Q APP SW Q SW Q SW Q SW Q 

10 References • Jan Kara @ Kernel recipes 2015 ‒ https://youtu.be/2tu__ZHC0mI • Jens Axboe @ Kernel recipes ‒ https://youtu.be/VIdKBD9-Ozg 

Questions? 

Thank you. Join the conversation, contribute & have a lot of fun! www.opensuse.org 

13 Have a Lot of Fun, and Join Us At: www.opensuse.org 

General Disclaimer This document is not to be construed as a promise by any participating organisation to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. openSUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for openSUSE products remains at the sole discretion of openSUSE. Further, openSUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All openSUSE marks referenced in this presentation are trademarks or registered trademarks of SUSE LLC, in the United States and other countries. All third-party trademarks are the property of their respective owners. License This slide deck is licensed under the Creative Commons Attribution-ShareAlike 4.0 International license. It can be shared and adapted for any purpose (even commercially) as long as Attribution is given and any derivative work is distributed under the same license. Details can be found at https://creativecommons.org/licenses/by-sa/4.0/ Credits Template Richard Brown [email protected] Design & Inspiration openSUSE Design Team http://opensuse.github.io/branding- guidelines/