Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interrupt Affinityについて

Interrupt Affinityについて

Takuya ASADA

June 28, 2014
Tweet

More Decks by Takuya ASADA

Other Decks in Technology

Transcript

  1. Interrupt Affinity

    ʹ͍ͭͯ
    @syuu1228

    View Slide

  2. Interrupt Affinity?
    ࠓ࣌ͷී௨ͷ1$͸ϚϧνίΞͰ͋Δ
    σόΠε̍ɿ*32̍ɿ$16͍ͬͺ͍

    ͱ͍͏ؔ܎ʹͳ͍ͬͯΔ
    σόΠε͸Ͳ͜ʹׂΓࠐΜͰ͍Δͷʁ

    -JOVY͸Ͳͷ$16ʹׂΓࠐ·͍ͤͯΔͷʁ

    ˠ*OUFSSVQU"⒏OJUZ

    View Slide

  3. Interrupt Affinityͷ
    ϢʔβΠϯλϑΣʔε

    View Slide

  4. $ cat /proc/interrupts
    CPU0 CPU1
    0: 36843 61973 IO-APIC-edge timer
    1: 518 591 IO-APIC-edge i8042
    8: 1 0 IO-APIC-edge rtc0
    9: 387 321 IO-APIC-fasteoi acpi
    12: 11236 79 IO-APIC-edge i8042
    16: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
    17: 12 11 IO-APIC-fasteoi uhci_hcd:usb7
    18: 0 0 IO-APIC-fasteoi uhci_hcd:usb8
    19: 6531 235 IO-APIC-fasteoi ehci_hcd:usb2
    20: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    21: 85 69 IO-APIC-fasteoi uhci_hcd:usb4
    22: 0 0 IO-APIC-fasteoi uhci_hcd:usb5
    23: 20 15 IO-APIC-fasteoi ehci_hcd:usb1
    40: 0 0 PCI-MSI-edge PCIe PME, pciehp
    41: 0 0 PCI-MSI-edge PCIe PME, pciehp
    42: 0 0 PCI-MSI-edge PCIe PME, pciehp
    43: 33199 31756 PCI-MSI-edge ahci
    44: 19 1583 PCI-MSI-edge eth0
    45: 10 11 PCI-MSI-edge mei
    46: 41104 59303 PCI-MSI-edge i915
    47: 170268 708 PCI-MSI-edge iwlwifi
    48: 78 77 PCI-MSI-edge snd_hda_intel
    NMI: 1406 1430 Non-maskable interrupts
    $16ຖͷׂΓࠐΈ౷ܭΛදࣔ
    /proc/interrupts

    View Slide

  5. /proc/irq/*/smp_affinity
    ׂΓࠐΈઌ$16܈ΛϏοτͰࢦఆ͢Δ
    ͱɺࢦఆ͞Εͨ$16܈ʹϥ΢ϯυϩϏ
    ϯͰׂΓࠐ·ΕΔʢϏοτ͕୯ҰͳΒ
    ৗʹͦͷ$16΁ׂΓࠐΈʣ
    # cat /proc/irq/43/smp_affinity
    3
    # echo 1 > /proc/irq/43/smp_affinity
    ͜ΕΛॻ͖׵͑ΔͱׂΓࠐΈઌ$16͕มΘΔ

    View Slide

  6. LinuxͰͷσϑΥϧτ஋
    શ$16ͷϏοτ͕༗ޮʹͳ͓ͬͯΓɺ
    ͭͷׂΓࠐΈ͕શ$16΁ϥ΢ϯυϩ
    Ϗϯ͢Δ
    ϥ΢ϯυϩϏϯׂΓࠐΈʹରԠ͠ͳ͍1$
    Ͱ͸͋Δ$16΁ݻఆతʹׂΓࠐΉ
    ʢDQV ʣ

    View Slide

  7. /proc/irq/*/smp_affinity_list
    ώϡʔϚϯϦʔμϒϧͳ[email protected]⒏OJUZ
    # echo ff > /proc/irq/43/smp_affinity
    # cat /proc/irq/43/smp_affinity_list

    0-7

    View Slide

  8. /proc/irq/*/smp_affinity_hint
    σόΠευϥΠό͔ΒϢʔβϥϯυσʔ
    ϞϯʢJSRCBMBODFʣ΁͜ͷ*32ʹద੾
    ͳ$16NBTLΛ௨஌

    View Slide

  9. irqbalance
    ඵ͝ͱʹγεςϜෛՙঢ়گΛνΣο
    Ϋɺ֤*32ͷ[email protected]⒏OJUZΛॻ͖׵͑
    ͯ$16ෛՙΛެฏʹ͢ΔσʔϞϯ
    POFTIPUϞʔυͰ͸ىಈ࣌ʹద੾ͱࢥ
    ΘΕΔ஋Λ[email protected]⒏OJUZʹઃఆͯ͠
    FYJU

    View Slide

  10. MultiQueue NICͷׂΓࠐΈ
    $16ຖʹૹड৴ΩϡʔɺΩϡʔຖʹ
    .4*9ׂΓࠐΈΛ࣋ͭ
    ਖ਼͍͠[email protected]⒏OJUZ͸ݻఆతʹܾ·ͬ
    ͍ͯΔ
    ෳ਺$16΁ࢄΒͯ͠͸ͳΒͳ͍
    47: 7602 0 0 0 0 3 23 0 PCI-MSI-edge p1p1-TxRx-0
    48: 0 7602 0 0 0 0 13 12 PCI-MSI-edge p1p1-TxRx-1
    49: 12 0 7605 0 0 0 10 0 PCI-MSI-edge p1p1-TxRx-2
    50: 0 12 0 7602 3 0 10 0 PCI-MSI-edge p1p1-TxRx-3
    51: 0 0 12 0 7602 3 10 0 PCI-MSI-edge p1p1-TxRx-4
    52: 0 0 0 20 0 7602 13 0 PCI-MSI-edge p1p1-TxRx-5
    53: 0 0 0 0 12 0 7612 3 PCI-MSI-edge p1p1-TxRx-6
    54: 3 0 0 0 0 13 10 7602 PCI-MSI-edge p1p1-TxRx-7
    55: 0 2 0 0 0 0 2 0 PCI-MSI-edge p1p1

    View Slide

  11. ιʔείʔυ
    QSPDJOUFSSVQUT

    GTQSPDJOUFSSVQUTD
    QSPDJSR

    LFSOFMJSRQSPDD
    JSRCBMBODF

    IUUQTHJUIVCDPNJSRCBMBODF
    JSRCBMBODF

    View Slide

  12. PCΞʔΩςΫνϟʹ͓͚Δ

    ׂΓࠐΈϧʔςΟϯά

    View Slide

  13. PC্ͷׂΓࠐΈίϯϙʔωϯτ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View Slide

  14. Local APIC
    $16ຖͷׂΓࠐΈίϯτϩʔϥɹׂΓࠐΈΛڐՄɾϚεΫ͠
    ͨΓɺ&0*ͨ͠Γ
    γεςϜશମͰҰҙͳ"1*$*%Λ࣋ͭ
    ଞͷ-"1*$΁ׂΓࠐΊΔʢ*1**OUFSQSPDFTTPS*OUFSSVQU

    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View Slide

  15. LAPIC - Interrupt
    Command Register(ICR)
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFS4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    %FTUJOBUJPO4IPSUIBOEɿѼઌͷϚεΫํࣜʢࣗ$16ΛϚεΫɾ
    ΞϯϚεΫͳͲʣ
    %FTUJOBUJPO*%ɿѼઌ

    View Slide

  16. ιʔείʔυ
    *$3

    BSDIYJODMVEFBTNJQJI

    View Slide

  17. σʔλγʔτ
    *OUFMšBOE*"
    "SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT

    *OUFSSVQU$PNNBOE
    3FHJTUFS *$3

    View Slide

  18. ิ଍ɿx2APIC
    -"1*$*%͸ैདྷCJU͕ͩͬͨɺίΞ਺ͷ૿ՃʹΑ
    ΓϏοτ਺͕଍Γͳ͘ͳͬͯ͠·ͬͨͷͰCJU΁
    ϏοτΛ௥Ճ
    ͜Εʹ൐ͬͯɺҎ߱Ͱղઆ͢Δ֤ϨδελͰͷׂΓ
    ࠐΈઌ-"1*$ࢦఆϑΟʔϧυͷΞυϨε෯΋Ϗοτ
    ௥Ճ͞Εͨ

    View Slide

  19. σʔλγʔτ
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    (-044"3:

    View Slide

  20. I/O APIC
    ݶΒΕͨຊ਺ͷ෺ཧׂΓࠐΈઢΛ֤1$*σόΠεͰڞ༗

    ʢ1$*Fʹ͸෺ཧׂΓࠐΈઢ͸ແ͘ɺϝοηʔδϯάʹΑͬͯ*0"1*$ܦ༝ׂΓࠐΈΛ
    ΤϛϡϨʔτʣ
    ֤*32ΛͲͷ-"1*$΁సૹ͢Δ͔Λఆٛ͢Δ3FEJSFDUJPO5BCMF

    ʢ*0"1*$্ͷϨδελ܈ʣΛ࣋ͭ
    1$*σόΠε͸*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View Slide

  21. I/O APIC - REDIR_TBL
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFSZ4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    *OUFSSVQU*OQVU1JO1PMBSJUZɿׂΓࠐΈϐϯͷۃੑ
    3FNPUF*33ɿࣗಈతʹ&0*͢Δ͔Ͳ͏͔
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    .BTLɿׂΓࠐΈϚεΫ
    %FTUJOBUJPO*%ɿѼઌ

    View Slide

  22. View Slide

  23. σʔλγʔτ
    *OUFMš*0$POUSPMMFS)VC
    *$)
    'BNJMZ%BUBTIFFU

    3&%*[email protected]#-3FEJSFDUJPO
    5BCMF

    View Slide

  24. ϨΨγσόΠε
    ϨΨγσόΠε͔ΒͷׂΓࠐΈΛ1$*-1$CSJEHFͰ1$*΁ม׵
    1$*σόΠεಉ༷ʹ*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔ
    δΛૹ৴
    ޓ׵ੑͷͨΊ"ʹ΋઀ଓ͞Ε͍ͯΔͱࢥΘΕΔ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View Slide

  25. 8259A PIC
    লུʢ·ͩ࢒ͬͯ·͕͢ɺ΋͏࢖Θͳ
    ͍Ͱ͠ΐ͏ʣ

    View Slide

  26. MSI(-X) capable PCI devices
    ֤σόΠε͕೚ҙͷ਺ͷ*32Λ࣋ͯΔ
    ֤*32ͷׂΓࠐΈઌ-"1*$͸1$*σόΠεͷ1$*
    $POpHVSBUJPO4QBDFʹ࣋ͭ
    1$*σόΠε͸௚઀-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View Slide

  27. MSI Capability on

    PCI configuration space
    Device ID Vendor ID

    Reserved Cap. Pointer

    Capability x CAP ID(x)
    Next Pointer

    Capability y CAP ID(yy)
    Next Pointer

    MSICTL CAP ID(D0h)
    Next Pointer
    MSIAR
    MSIDR

    View Slide

  28. MSI Capability - MSICTL
    লུʢ.4*༗ޮɺCJUɺෳ਺ׂΓࠐ
    Έ༗ޮͳͲͷϑΟʔϧυʣ

    View Slide

  29. MSI Capability - MSIAR
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    3FEJSFDUJPO)JOUɿMPHJDBMNPEFˍ
    -PX1SJͰͷSFEJSFDUJPOΛ༗ޮԽ
    %FTUJOBUJPO*%ɿѼઌ

    View Slide

  30. MSI Capability - MSIDR
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FMJWFSZ4UBUVTɿׂΓࠐΈঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤο
    δτϦΨ

    View Slide

  31. MSI-Xͷ৔߹
    লུʢϑΟʔϧυ͕֦ு͞Εͯ୔ࢁׂ
    ΓࠐΈ͕࣋ͯΔ͕ɺ࢓૊Έ͸ಉ͡ʣ

    View Slide

  32. ιʔείʔυ
    .4* 9
    $BQBCJMJUZ

    [email protected]

    View Slide

  33. σʔλγʔτ
    *OUFMš$IJQTFUBOE*OUFMš
    $IJQTFU

    .4*$5-.4*$POUSPM
    3FHJTUFS

    .4*"3.4*"EESFTT
    3FHJTUFS

    .4*%3.4*%BUB3FHJTUFS

    View Slide

  34. ֤ํࣜͰڞ௨ͳཁૉ
    ҎԼͷࡾͭͷϑΟʔϧυͰѼઌ͕ࢦఆ
    ͞ΕΔ
    %FTUJOBUJPO.PEF
    %FMJWFSZ.PEF
    %FTUJOBUJPO*%

    View Slide

  35. ׂΓࠐΈઌͷࢦఆํ๏
    1IZTJDBM%FTUJOBUJPO.PEF
    %FTUJOBUJPO'JFMEʹ"1*$*%Λࢦఆ

    ʮৗʹಉ͡$16΁ׂΓࠐΈʯΛ࣮ݱ
    -PHJDBM%FTUJOBUJPO.PEF 'MBU.PEFM

    %FTUJOBUJPO'JFMEͷCJUͰѼઌ-"1*$܈ͷൣғΛදݱ
    %FMJWFSZ.PEF
    'JYFE

    ࢦఆൣғͷશͯͷ-"1*$΁ׂΓࠐΈ
    -PXFTU1SJPSJUZ

    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL1SJPSJUZ3FHJTUFSʣͷ஋͕
    ௿͍$16΁ׂΓࠐΈ

    View Slide

  36. Logical Destination Mode

    (Cluster Model)
    Y"1*$ͷ৔߹ɺ

    CJU$MVTUFS*%

    CJU-PHJDBM*%
    'MBU.PEFMʹൺ΂ͯଟ͘ͷ$16਺Λ
    αϙʔτͰ͖Δ

    View Slide

  37. Lowest Priority Mode
    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL
    1SJPSJUZ3FHJTUFSʣͷ஋͕௿͍-"1*$΁
    ׂΓࠐΈ
    ୠ͠ɺ-JOVYͰ͸ಈ࡞தʹ513Λߋ৽͠ͳ
    ͍ˠશ-"1*$ͷ513͸શͯಉ͡஋
    ࠷খ஋ͷ513Λ࣋ͭ-"1*$͕ෳ਺͋Δ৔
    ߹͸ϥ΢ϯυϩϏϯͰ̍ͭબ୒

    View Slide

  38. σʔλγʔτ
    *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF%FWFMPQFS
    .BOVBMT

    *OUFSSVQU$PNNBOE3FHJTUFS *$3

    1IZTJDBM%FTUJOBUJPO.PEF

    -PHJDBM%FTUJOBUJPO.PEF

    -PXFTU1SJPSJUZ%FMJWFSZ.PEF
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    -PHJDBM%FTUJOBUJPO3FHJTUFS
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ%BUBTIFFU

    "VUPNBUJD3PUBUJPO.PEF &RVBM1SJPSJUZ%FWJDFT

    4QFDJpD3PUBUJPO.PEF 4QFDJpD1SJPSJUZ

    View Slide

  39. echo 1 > /proc/irq//
    smp_affinityͨ࣌͠ʹԿ͕ى͖Δͷ͔
    .4*ͳ1$*FσόΠεͰ͔֬ΊͯΈΔ
    *OUFM1SP FF

    1$*$POpHVSBUJPO4QBDF͸SPPUͳΒ
    TZTCVTQDJ͔ΒಡΊΔ

    ˠϢʔβϥϯυ͔ΒಡΊΔ

    ˠ΋͔ͯ͠͠ɿMTQDJ


    View Slide

  40. # lspci -vvvv -s 00:19.0
    00:19.0 Ethernet controller: Intel Corporation 82567LM Gigabit Network Connection (rev 03)
    Subsystem: Lenovo Device 20ee
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
    Interrupt: pin A routed to IRQ 44
    Region 0: Memory at f2600000 (32-bit, non-prefetchable) [size=128K]
    Region 1: Memory at f2625000 (32-bit, non-prefetchable) [size=4K]
    Region 2: I/O ports at 1840 [size=32]
    Capabilities: [c8] Power Management version 2
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Address: 00000000fee0300c Data: 41b9
    Capabilities: [e0] PCI Advanced Features
    AFCap: TP+ FLR+
    AFCtrl: FLR-
    AFStatus: TP-
    Kernel driver in use: e1000e
    Kernel modules: e1000e
    ੺͘ృͬͨॴ͕.4*ͷϑΟʔϧυ͕ͩɺ͜Εͩ
    ͱಡΊͳ͍ͷͰMTQDJΛGPSLͯ͠.4*ϑΟʔϧ
    υΛදࣔ͢ΔπʔϧΛ࡞੒

    IUUQTHJTUHJUIVCDPN

    View Slide

  41. # gcc -lpci msireg.c!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0300c dest_mode=logical redirection=lowpri dest_id=3!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    -PHJDBMNPEFͰ-PXQSJɺEFTUJEɺ
    WFDUPSʹͳ͍ͬͯΔ
    # echo 1 > /proc/irq/44/smp_affinity!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0100c dest_mode=logical redirection=lowpri dest_id=1!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    [email protected]͕ʹॻ͖׵Θͬͨ

    View Slide

  42. Affinityηοτ࣌ͷΧʔωϧͷڍಈ
    γεςϜάϩʔόϧͳ%FTUJOBUJPO.PEFͱ
    %FMJWFSZ.PEFͷઃఆΛอ࣋

    ʢ[email protected]@NPEF 

    [email protected]@NPEFʣ
    "⒏OJUZηοτ࣌ʹ͜ͷ஋Λࢀরɺ%FTUJOBUJPO
    *%ͱซͤͯॻ͖ࠐΉ
    [email protected]@[email protected]
    [email protected]

    View Slide

  43. OSॳظԽ࣌ͷׂΓࠐΈϞʔυઃఆ
    ؀ڥʹΑͬͯҟͳΔυϥΠόΛ࢖༻

    [email protected]@D [email protected]
    [email protected]ʜ
    [email protected]ʢඇYBQJDϞʔυʣ

    %FMJWFSZ.PEF-PX1SJ

    %FTUJOBUJPO.PEF-PHJDBM
    [email protected]ʢඇYBQJDϞʔυ $164ʣ

    %FMJWFSZ.PEF'JYFE

    %FTUJOBUJPO.PEF1IZTJDBM
    [email protected]@QIZTʢYBQJDϞʔυ "$1*Ͱ1IZTJDBMϞʔυࢦఆʣ

    ಉ্
    ϥ΢ϯυϩϏϯׂΓࠐΈͰ͖Δ؀ڥ͸Ұఆ৚݅Λຬͨͨ͠৔߹ͷΈ

    View Slide

  44. *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ
    %BUBTIFFU
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO
    *OUFMš$IJQTFUBOE*OUFMš$IJQTFU
    MJOVY%PDVNFOUBUJPO*32B⒏OJUZUYU
    6OEFSTUBOEJOHUIF-JOVY,FSOFM SE&EJUJPO
    1$*-PDBM#VT4QFDJpDBUJPO3FWJTJPO
    ࢀߟࢿྉ

    View Slide

  45. ࠓ·Ͱʹ·ͱΊͨࢿྉ
    ࠷ۙͷ1$ΞʔΩςΫνϟʹ͓͚ΔׂΓࠐΈϧʔςΟϯάͷ࢓૊Έ
    QSPDJSR*[email protected]⒏OJUZͷॻ͖׵͑Ͱ1$*ίϯϑΟάϨʔ
    γϣϯۭؒ͸ͲͷΑ͏ʹॻ͖׵ΘΔ͔
    QSPDJSR*[email protected]⒏OJUZ
    *OUFSSVQUTPOYW
    ϋΠύʔόΠβͷ࡞ΓํʙͪΌΜͱཧղ͢ΔԾ૝Խٕज़ʙୈ̐ճ*
    0Ծ૝ԽʮׂΓࠐΈฤɾͦͷ̍ʯ

    View Slide