Upgrade to Pro — share decks privately, control downloads, hide ads and more …

/proc/irq/<IRQ>/ smp_affinity

Takuya ASADA
September 22, 2012

/proc/irq/<IRQ>/ smp_affinity

Takuya ASADA

September 22, 2012
Tweet

More Decks by Takuya ASADA

Other Decks in Technology

Transcript

  1. /proc/irq//
    smp_affinity
    @syuu1228
    12೥9݄22೔౔༵೔

    View Slide

  2. cat /proc/interrupts
    12೥9݄22೔౔༵೔

    View Slide

  3. $ cat /proc/interrupts
    CPU0 CPU1
    0: 36843 61973 IO-APIC-edge timer
    1: 518 591 IO-APIC-edge i8042
    8: 1 0 IO-APIC-edge rtc0
    9: 387 321 IO-APIC-fasteoi acpi
    12: 11236 79 IO-APIC-edge i8042
    16: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
    17: 12 11 IO-APIC-fasteoi uhci_hcd:usb7
    18: 0 0 IO-APIC-fasteoi uhci_hcd:usb8
    19: 6531 235 IO-APIC-fasteoi ehci_hcd:usb2
    20: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    21: 85 69 IO-APIC-fasteoi uhci_hcd:usb4
    22: 0 0 IO-APIC-fasteoi uhci_hcd:usb5
    23: 20 15 IO-APIC-fasteoi ehci_hcd:usb1
    40: 0 0 PCI-MSI-edge PCIe PME, pciehp
    41: 0 0 PCI-MSI-edge PCIe PME, pciehp
    42: 0 0 PCI-MSI-edge PCIe PME, pciehp
    43: 33199 31756 PCI-MSI-edge ahci
    44: 19 1583 PCI-MSI-edge eth0
    45: 10 11 PCI-MSI-edge mei
    46: 41104 59303 PCI-MSI-edge i915
    47: 170268 708 PCI-MSI-edge iwlwifi
    48: 78 77 PCI-MSI-edge snd_hda_intel
    NMI: 1406 1430 Non-maskable interrupts
    LOC: 342351 327074 Local timer interrupts
    SPU: 0 0 Spurious interrupts
    PMI: 1406 1430 Performance monitoring interrupts
    IWI: 0 0 IRQ work interrupts
    RES: 83855 108903 Rescheduling interrupts
    CAL: 223 195 Function call interrupts
    TLB: 1486 1665 TLB shootdowns
    TRM: 0 6 Thermal event interrupts
    THR: 0 0 Threshold APIC interrupts
    MCE: 0 0 Machine check exceptions
    MCP: 11 11 Machine check polls
    ERR: 0
    MIS: 0
    ஸ౓ྑ͘ෳ਺ίΞ΁෼ࢄ͞ΕͯΔʁ
    12೥9݄22೔౔༵೔

    View Slide

  4. smp_affinity
    ׂΓࠐΈઌCPU܈ΛϏοτͰࢦఆ͢Δ
    ͱɺࢦఆ͞ΕͨCPU܈ʹϥ΢ϯυϩϏ
    ϯͰׂΓࠐ·ΕΔ
    # cat /proc/irq/43/smp_affinity
    3
    # echo 1 > /proc/irq/43/smp_affinity
    ͜ΕΛॻ͖׵͑Ε͹͍͍ͷ͔ʂ
    12೥9݄22೔౔༵೔

    View Slide

  5. ʔ ׬ ʔ
    12೥9݄22೔౔༵೔

    View Slide

  6. ͦΕͰ͸#kernelvmΒ͘͠ͳ͍
    bitͷཱͬͯΔίΞ΁ׂΓࠐΈ෼ࢄ
    bit͕ҰͭͳΒ͍ͭ΋ͦ͜΁ׂΓࠐΈ
    ͜ΕɺͲ͏ͳͬͯΔΜͩΖ͏ʁ
    # cat /proc/irq/43/smp_affinity
    3
    # echo 1 > /proc/irq/43/smp_affinity
    12೥9݄22೔౔༵೔

    View Slide

  7. ΠϚυΩͳx86ͷׂΓࠐΈ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH(South bridge)
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI
    Capable
    Devices
    IPI
    MSI
    Legacy PCI
    8259A Intr
    12೥9݄22೔౔༵೔

    View Slide

  8. Local APIC
    CPUຖͷׂΓࠐΈίϯτϩʔϥɹׂΓࠐΈΛڐՄɾϚεΫͨ͠
    ΓɺEOIͨ͠Γ
    γεςϜશମͰҰҙͳAPIC IDΛ࣋ͭ
    ଞͷLAPIC΁ׂΓࠐΊΔʢIPI: Inter-processor Interrupt)
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH(South bridge)
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI
    Capable
    Devices
    IPI
    MSI
    Legacy PCI
    8259A Intr
    12೥9݄22೔౔༵೔

    View Slide

  9. I/O APIC
    ݶΒΕͨIRQΛෳ਺ͷPCIσόΠεͰڞ༗
    ֤IRQΛͲͷLAPIC΁సૹ͢Δ͔Λఆٛ͢ΔRedirection TableʢI/
    O APIC্ͷϨδελ܈ʣΛ࣋ͭ
    PCIσόΠε͸I/O APICΛ௨ͯ͡LAPIC΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH(South bridge)
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI
    Capable
    Devices
    IPI
    MSI
    Legacy PCI
    8259A Intr
    12೥9݄22೔౔༵೔

    View Slide

  10. MSI capable PCI devices
    ֤σόΠε͕೚ҙͷ਺ͷIRQΛ࣋ͯΔ
    ֤IRQͷׂΓࠐΈઌLAPIC͸PCIσόΠεͷPCI Configuration
    Spaceʹ࣋ͭ
    PCIσόΠε͸௚઀LAPIC΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH(South bridge)
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI
    Capable
    Devices
    IPI
    MSI
    Legacy PCI
    8259A Intr
    12೥9݄22೔౔༵೔

    View Slide

  11. ׂΓࠐΈઌͷࢦఆํ๏
    Physical Destination Mode
    Destination FieldʹAPIC IDΛࢦఆ
    ʮৗʹಉ͡CPU΁ׂΓࠐΈʯΛ࣮ݱ
    Logical Destination Mode(Flat Model)
    Destination FieldͷbitͰѼઌLAPIC܈ͷൣғΛදݱ
    Delivery Mode
    Fixed
    ࢦఆൣғͷશͯͷLAPIC΁ׂΓࠐΈ
    Lowest Priority
    ࢦఆൣғͷதͰɺ࠷΋TPRʢTask Priority Registerʣͷ
    ஋͕௿͍CPU΁ׂΓࠐΈ
    12೥9݄22೔౔༵೔

    View Slide

  12. Lowest Priority Mode
    ࢦఆൣғͷதͰɺ࠷΋TPRʢTask
    Priority Registerʣͷ஋͕௿͍LAPIC
    ΁ׂΓࠐΈ
    ࠷খ஋ͷTPRΛ࣋ͭLAPIC͕ෳ਺͋Δ
    ৔߹͸ϥ΢ϯυϩϏϯͰ̍ͭબ୒
    12೥9݄22೔౔༵೔

    View Slide

  13. echo 1 > /proc/irq//
    smp_affinityͨ࣌͠ʹԿ͕ى͖Δͷ͔
    MSIͳPCIeσόΠεͰ͔֬ΊͯΈΔ
    Intel Pro/1000(e1000e)
    PCI Configuration Space͸rootͳ
    Β/sys/bus/pci͔ΒಡΊΔ
    →Ϣʔβϥϯυ͔ΒಡΊΔ
    →΋͔ͯ͠͠ɿlspci
    12೥9݄22೔౔༵೔

    View Slide

  14. ʊਓਓ ਓਓʊ
    ʼ lspci ʻ
    ʉY^Y^Y^Yʉ
    12೥9݄22೔౔༵೔

    View Slide

  15. # lspci -vvvv -s 00:19.0
    00:19.0 Ethernet controller: Intel Corporation 82567LM Gigabit Network Connection (rev 03)
    Subsystem: Lenovo Device 20ee
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
    Interrupt: pin A routed to IRQ 44
    Region 0: Memory at f2600000 (32-bit, non-prefetchable) [size=128K]
    Region 1: Memory at f2625000 (32-bit, non-prefetchable) [size=4K]
    Region 2: I/O ports at 1840 [size=32]
    Capabilities: [c8] Power Management version 2
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Address: 00000000fee0300c Data: 41b9
    Capabilities: [e0] PCI Advanced Features
    AFCap: TP+ FLR+
    AFCtrl: FLR-
    AFStatus: TP-
    Kernel driver in use: e1000e
    Kernel modules: e1000e
    ੺͘ృͬͨॴ͕MSIͷϑΟʔϧυ͚ͩΕͲɺ͜
    Εͩͱྑ͘෼͔Βͳ͍ͷͰlspciΛforkͯ͠
    MSIϑΟʔϧυΛදࣔ͢ΔπʔϧΛ࡞੒
    https://gist.github.com/1568777
    12೥9݄22೔౔༵೔

    View Slide

  16. # gcc -lpci msireg.c
    # ./a.out 00:19.0
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
    address_hi=0
    address_lo=fee0300c dest_mode=logical redirection=lowpri dest_id=3
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    Logical modeͰLowpriɺdestid=3ɺ
    vector=185ʹͳͬͯΔ
    # echo 1 > /proc/irq/44/smp_affinity
    # ./a.out 00:19.0
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
    address_hi=0
    address_lo=fee0100c dest_mode=logical redirection=lowpri dest_id=1
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    dest_id͕1ʹॻ͖׵Θͬͨʂ
    12೥9݄22೔౔༵೔

    View Slide

  17. ༨ஊ
    ௐࢠʹ৐ͬͯϢʔβϥϯυ͔ΒϑΟʔ
    ϧυॻ͖׵͑ͨΒԿ͔͓͔͘͠ͳΓ·
    ͨ͠
    Χʔωϧ͞Μͱ݌՞ͯ͠Δ͔ɺϑΟʔ
    ϧυͷॻ͖ํؒҧ͔͑ͨ…
    12೥9݄22೔౔༵೔

    View Slide

  18. Intel® 64 and IA-32 Architectures Software Developer
    Manuals
    Intel® I/O Controller Hub 10 (ICH10) Family Datasheet
    linux/Documentation/IRQ-affinity.txt
    Understanding the Linux Kernel, 3rd Edition
    PCI Local Bus Specification Revision 3.0
    ࠷ۙͷPCΞʔΩςΫνϟʹ͓͚ΔׂΓࠐΈϧʔςΟϯάͷ࢓૊Έ
    /proc/irq//smp_affinityͷॻ͖׵͑ͰPCIίϯϑΟάϨ
    ʔγϣϯۭؒ͸ͲͷΑ͏ʹॻ͖׵ΘΔ͔
    Interrupts on xv6
    ࢀߟࢿྉ
    12೥9݄22೔౔༵೔

    View Slide