Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interrupt Affinityについて

Interrupt Affinityについて

Takuya ASADA

June 28, 2014
Tweet

More Decks by Takuya ASADA

Other Decks in Technology

Transcript

  1. Interrupt Affinity

    ʹ͍ͭͯ
    @syuu1228

    View full-size slide

  2. Interrupt Affinity?
    ࠓ࣌ͷී௨ͷ1$͸ϚϧνίΞͰ͋Δ
    σόΠε̍ɿ*32̍ɿ$16͍ͬͺ͍

    ͱ͍͏ؔ܎ʹͳ͍ͬͯΔ
    σόΠε͸Ͳ͜ʹׂΓࠐΜͰ͍Δͷʁ

    -JOVY͸Ͳͷ$16ʹׂΓࠐ·͍ͤͯΔͷʁ

    ˠ*OUFSSVQU"⒏OJUZ

    View full-size slide

  3. Interrupt Affinityͷ
    ϢʔβΠϯλϑΣʔε

    View full-size slide

  4. $ cat /proc/interrupts
    CPU0 CPU1
    0: 36843 61973 IO-APIC-edge timer
    1: 518 591 IO-APIC-edge i8042
    8: 1 0 IO-APIC-edge rtc0
    9: 387 321 IO-APIC-fasteoi acpi
    12: 11236 79 IO-APIC-edge i8042
    16: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
    17: 12 11 IO-APIC-fasteoi uhci_hcd:usb7
    18: 0 0 IO-APIC-fasteoi uhci_hcd:usb8
    19: 6531 235 IO-APIC-fasteoi ehci_hcd:usb2
    20: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
    21: 85 69 IO-APIC-fasteoi uhci_hcd:usb4
    22: 0 0 IO-APIC-fasteoi uhci_hcd:usb5
    23: 20 15 IO-APIC-fasteoi ehci_hcd:usb1
    40: 0 0 PCI-MSI-edge PCIe PME, pciehp
    41: 0 0 PCI-MSI-edge PCIe PME, pciehp
    42: 0 0 PCI-MSI-edge PCIe PME, pciehp
    43: 33199 31756 PCI-MSI-edge ahci
    44: 19 1583 PCI-MSI-edge eth0
    45: 10 11 PCI-MSI-edge mei
    46: 41104 59303 PCI-MSI-edge i915
    47: 170268 708 PCI-MSI-edge iwlwifi
    48: 78 77 PCI-MSI-edge snd_hda_intel
    NMI: 1406 1430 Non-maskable interrupts
    $16ຖͷׂΓࠐΈ౷ܭΛදࣔ
    /proc/interrupts

    View full-size slide

  5. /proc/irq/*/smp_affinity
    ׂΓࠐΈઌ$16܈ΛϏοτͰࢦఆ͢Δ
    ͱɺࢦఆ͞Εͨ$16܈ʹϥ΢ϯυϩϏ
    ϯͰׂΓࠐ·ΕΔʢϏοτ͕୯ҰͳΒ
    ৗʹͦͷ$16΁ׂΓࠐΈʣ
    # cat /proc/irq/43/smp_affinity
    3
    # echo 1 > /proc/irq/43/smp_affinity
    ͜ΕΛॻ͖׵͑ΔͱׂΓࠐΈઌ$16͕มΘΔ

    View full-size slide

  6. LinuxͰͷσϑΥϧτ஋
    શ$16ͷϏοτ͕༗ޮʹͳ͓ͬͯΓɺ
    ͭͷׂΓࠐΈ͕શ$16΁ϥ΢ϯυϩ
    Ϗϯ͢Δ
    ϥ΢ϯυϩϏϯׂΓࠐΈʹରԠ͠ͳ͍1$
    Ͱ͸͋Δ$16΁ݻఆతʹׂΓࠐΉ
    ʢDQV ʣ

    View full-size slide

  7. /proc/irq/*/smp_affinity_list
    ώϡʔϚϯϦʔμϒϧͳTNQ@B⒏OJUZ
    # echo ff > /proc/irq/43/smp_affinity
    # cat /proc/irq/43/smp_affinity_list

    0-7

    View full-size slide

  8. /proc/irq/*/smp_affinity_hint
    σόΠευϥΠό͔ΒϢʔβϥϯυσʔ
    ϞϯʢJSRCBMBODFʣ΁͜ͷ*32ʹద੾
    ͳ$16NBTLΛ௨஌

    View full-size slide

  9. irqbalance
    ඵ͝ͱʹγεςϜෛՙঢ়گΛνΣο
    Ϋɺ֤*32ͷTNQ@B⒏OJUZΛॻ͖׵͑
    ͯ$16ෛՙΛެฏʹ͢ΔσʔϞϯ
    POFTIPUϞʔυͰ͸ىಈ࣌ʹద੾ͱࢥ
    ΘΕΔ஋ΛTNQ@B⒏OJUZʹઃఆͯ͠
    FYJU

    View full-size slide

  10. MultiQueue NICͷׂΓࠐΈ
    $16ຖʹૹड৴ΩϡʔɺΩϡʔຖʹ
    .4*9ׂΓࠐΈΛ࣋ͭ
    ਖ਼͍͠TNQ@B⒏OJUZ͸ݻఆతʹܾ·ͬ
    ͍ͯΔ
    ෳ਺$16΁ࢄΒͯ͠͸ͳΒͳ͍
    47: 7602 0 0 0 0 3 23 0 PCI-MSI-edge p1p1-TxRx-0
    48: 0 7602 0 0 0 0 13 12 PCI-MSI-edge p1p1-TxRx-1
    49: 12 0 7605 0 0 0 10 0 PCI-MSI-edge p1p1-TxRx-2
    50: 0 12 0 7602 3 0 10 0 PCI-MSI-edge p1p1-TxRx-3
    51: 0 0 12 0 7602 3 10 0 PCI-MSI-edge p1p1-TxRx-4
    52: 0 0 0 20 0 7602 13 0 PCI-MSI-edge p1p1-TxRx-5
    53: 0 0 0 0 12 0 7612 3 PCI-MSI-edge p1p1-TxRx-6
    54: 3 0 0 0 0 13 10 7602 PCI-MSI-edge p1p1-TxRx-7
    55: 0 2 0 0 0 0 2 0 PCI-MSI-edge p1p1

    View full-size slide

  11. ιʔείʔυ
    QSPDJOUFSSVQUT

    GTQSPDJOUFSSVQUTD
    QSPDJSR

    LFSOFMJSRQSPDD
    JSRCBMBODF

    IUUQTHJUIVCDPNJSRCBMBODF
    JSRCBMBODF

    View full-size slide

  12. PCΞʔΩςΫνϟʹ͓͚Δ

    ׂΓࠐΈϧʔςΟϯά

    View full-size slide

  13. PC্ͷׂΓࠐΈίϯϙʔωϯτ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View full-size slide

  14. Local APIC
    $16ຖͷׂΓࠐΈίϯτϩʔϥɹׂΓࠐΈΛڐՄɾϚεΫ͠
    ͨΓɺ&0*ͨ͠Γ
    γεςϜશମͰҰҙͳ"1*$*%Λ࣋ͭ
    ଞͷ-"1*$΁ׂΓࠐΊΔʢ*1**OUFSQSPDFTTPS*OUFSSVQU

    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View full-size slide

  15. LAPIC - Interrupt
    Command Register(ICR)
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFS4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    %FTUJOBUJPO4IPSUIBOEɿѼઌͷϚεΫํࣜʢࣗ$16ΛϚεΫɾ
    ΞϯϚεΫͳͲʣ
    %FTUJOBUJPO*%ɿѼઌ

    View full-size slide

  16. ιʔείʔυ
    *$3

    BSDIYJODMVEFBTNJQJI

    View full-size slide

  17. σʔλγʔτ
    *OUFMšBOE*"
    "SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT

    *OUFSSVQU$PNNBOE
    3FHJTUFS *$3

    View full-size slide

  18. ิ଍ɿx2APIC
    -"1*$*%͸ैདྷCJU͕ͩͬͨɺίΞ਺ͷ૿ՃʹΑ
    ΓϏοτ਺͕଍Γͳ͘ͳͬͯ͠·ͬͨͷͰCJU΁
    ϏοτΛ௥Ճ
    ͜Εʹ൐ͬͯɺҎ߱Ͱղઆ͢Δ֤ϨδελͰͷׂΓ
    ࠐΈઌ-"1*$ࢦఆϑΟʔϧυͷΞυϨε෯΋Ϗοτ
    ௥Ճ͞Εͨ

    View full-size slide

  19. σʔλγʔτ
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    (-044"3:

    View full-size slide

  20. I/O APIC
    ݶΒΕͨຊ਺ͷ෺ཧׂΓࠐΈઢΛ֤1$*σόΠεͰڞ༗

    ʢ1$*Fʹ͸෺ཧׂΓࠐΈઢ͸ແ͘ɺϝοηʔδϯάʹΑͬͯ*0"1*$ܦ༝ׂΓࠐΈΛ
    ΤϛϡϨʔτʣ
    ֤*32ΛͲͷ-"1*$΁సૹ͢Δ͔Λఆٛ͢Δ3FEJSFDUJPO5BCMF

    ʢ*0"1*$্ͷϨδελ܈ʣΛ࣋ͭ
    1$*σόΠε͸*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View full-size slide

  21. I/O APIC - REDIR_TBL
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFSZ4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    *OUFSSVQU*OQVU1JO1PMBSJUZɿׂΓࠐΈϐϯͷۃੑ
    3FNPUF*33ɿࣗಈతʹ&0*͢Δ͔Ͳ͏͔
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    .BTLɿׂΓࠐΈϚεΫ
    %FTUJOBUJPO*%ɿѼઌ

    View full-size slide

  22. ιʔείʔυ
    3&%*3@5#-

    BSDIYLFSOFMBQJDJP@BQJDD

    View full-size slide

  23. σʔλγʔτ
    *OUFMš*0$POUSPMMFS)VC
    *$)
    'BNJMZ%BUBTIFFU

    3&%*3@5#-3FEJSFDUJPO
    5BCMF

    View full-size slide

  24. ϨΨγσόΠε
    ϨΨγσόΠε͔ΒͷׂΓࠐΈΛ1$*-1$CSJEHFͰ1$*΁ม׵
    1$*σόΠεಉ༷ʹ*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔ
    δΛૹ৴
    ޓ׵ੑͷͨΊ"ʹ΋઀ଓ͞Ε͍ͯΔͱࢥΘΕΔ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View full-size slide

  25. 8259A PIC
    লུʢ·ͩ࢒ͬͯ·͕͢ɺ΋͏࢖Θͳ
    ͍Ͱ͠ΐ͏ʣ

    View full-size slide

  26. MSI(-X) capable PCI devices
    ֤σόΠε͕೚ҙͷ਺ͷ*32Λ࣋ͯΔ
    ֤*32ͷׂΓࠐΈઌ-"1*$͸1$*σόΠεͷ1$*
    $POpHVSBUJPO4QBDFʹ࣋ͭ
    1$*σόΠε͸௚઀-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip

    View full-size slide

  27. MSI Capability on

    PCI configuration space
    Device ID Vendor ID

    Reserved Cap. Pointer

    Capability x CAP ID(x)
    Next Pointer

    Capability y CAP ID(yy)
    Next Pointer

    MSICTL CAP ID(D0h)
    Next Pointer
    MSIAR
    MSIDR

    View full-size slide

  28. MSI Capability - MSICTL
    লུʢ.4*༗ޮɺCJUɺෳ਺ׂΓࠐ
    Έ༗ޮͳͲͷϑΟʔϧυʣ

    View full-size slide

  29. MSI Capability - MSIAR
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    3FEJSFDUJPO)JOUɿMPHJDBMNPEFˍ
    -PX1SJͰͷSFEJSFDUJPOΛ༗ޮԽ
    %FTUJOBUJPO*%ɿѼઌ

    View full-size slide

  30. MSI Capability - MSIDR
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FMJWFSZ4UBUVTɿׂΓࠐΈঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤο
    δτϦΨ

    View full-size slide

  31. MSI-Xͷ৔߹
    লུʢϑΟʔϧυ͕֦ு͞Εͯ୔ࢁׂ
    ΓࠐΈ͕࣋ͯΔ͕ɺ࢓૊Έ͸ಉ͡ʣ

    View full-size slide

  32. ιʔείʔυ
    .4* 9
    $BQBCJMJUZ

    BSDIYLFSOFMBQJDJP@BQJDD

    View full-size slide

  33. σʔλγʔτ
    *OUFMš$IJQTFUBOE*OUFMš
    $IJQTFU

    .4*$5-.4*$POUSPM
    3FHJTUFS

    .4*"3.4*"EESFTT
    3FHJTUFS

    .4*%3.4*%BUB3FHJTUFS

    View full-size slide

  34. ֤ํࣜͰڞ௨ͳཁૉ
    ҎԼͷࡾͭͷϑΟʔϧυͰѼઌ͕ࢦఆ
    ͞ΕΔ
    %FTUJOBUJPO.PEF
    %FMJWFSZ.PEF
    %FTUJOBUJPO*%

    View full-size slide

  35. ׂΓࠐΈઌͷࢦఆํ๏
    1IZTJDBM%FTUJOBUJPO.PEF
    %FTUJOBUJPO'JFMEʹ"1*$*%Λࢦఆ

    ʮৗʹಉ͡$16΁ׂΓࠐΈʯΛ࣮ݱ
    -PHJDBM%FTUJOBUJPO.PEF 'MBU.PEFM

    %FTUJOBUJPO'JFMEͷCJUͰѼઌ-"1*$܈ͷൣғΛදݱ
    %FMJWFSZ.PEF
    'JYFE

    ࢦఆൣғͷશͯͷ-"1*$΁ׂΓࠐΈ
    -PXFTU1SJPSJUZ

    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL1SJPSJUZ3FHJTUFSʣͷ஋͕
    ௿͍$16΁ׂΓࠐΈ

    View full-size slide

  36. Logical Destination Mode

    (Cluster Model)
    Y"1*$ͷ৔߹ɺ

    CJU$MVTUFS*%

    CJU-PHJDBM*%
    'MBU.PEFMʹൺ΂ͯଟ͘ͷ$16਺Λ
    αϙʔτͰ͖Δ

    View full-size slide

  37. Lowest Priority Mode
    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL
    1SJPSJUZ3FHJTUFSʣͷ஋͕௿͍-"1*$΁
    ׂΓࠐΈ
    ୠ͠ɺ-JOVYͰ͸ಈ࡞தʹ513Λߋ৽͠ͳ
    ͍ˠશ-"1*$ͷ513͸શͯಉ͡஋
    ࠷খ஋ͷ513Λ࣋ͭ-"1*$͕ෳ਺͋Δ৔
    ߹͸ϥ΢ϯυϩϏϯͰ̍ͭબ୒

    View full-size slide

  38. σʔλγʔτ
    *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF%FWFMPQFS
    .BOVBMT

    *OUFSSVQU$PNNBOE3FHJTUFS *$3

    1IZTJDBM%FTUJOBUJPO.PEF

    -PHJDBM%FTUJOBUJPO.PEF

    -PXFTU1SJPSJUZ%FMJWFSZ.PEF
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    -PHJDBM%FTUJOBUJPO3FHJTUFS
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ%BUBTIFFU

    "VUPNBUJD3PUBUJPO.PEF &RVBM1SJPSJUZ%FWJDFT

    4QFDJpD3PUBUJPO.PEF 4QFDJpD1SJPSJUZ

    View full-size slide

  39. echo 1 > /proc/irq//
    smp_affinityͨ࣌͠ʹԿ͕ى͖Δͷ͔
    .4*ͳ1$*FσόΠεͰ͔֬ΊͯΈΔ
    *OUFM1SP FF

    1$*$POpHVSBUJPO4QBDF͸SPPUͳΒ
    TZTCVTQDJ͔ΒಡΊΔ

    ˠϢʔβϥϯυ͔ΒಡΊΔ

    ˠ΋͔ͯ͠͠ɿMTQDJ


    View full-size slide

  40. # lspci -vvvv -s 00:19.0
    00:19.0 Ethernet controller: Intel Corporation 82567LM Gigabit Network Connection (rev 03)
    Subsystem: Lenovo Device 20ee
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
    Interrupt: pin A routed to IRQ 44
    Region 0: Memory at f2600000 (32-bit, non-prefetchable) [size=128K]
    Region 1: Memory at f2625000 (32-bit, non-prefetchable) [size=4K]
    Region 2: I/O ports at 1840 [size=32]
    Capabilities: [c8] Power Management version 2
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Address: 00000000fee0300c Data: 41b9
    Capabilities: [e0] PCI Advanced Features
    AFCap: TP+ FLR+
    AFCtrl: FLR-
    AFStatus: TP-
    Kernel driver in use: e1000e
    Kernel modules: e1000e
    ੺͘ృͬͨॴ͕.4*ͷϑΟʔϧυ͕ͩɺ͜Εͩ
    ͱಡΊͳ͍ͷͰMTQDJΛGPSLͯ͠.4*ϑΟʔϧ
    υΛදࣔ͢ΔπʔϧΛ࡞੒

    IUUQTHJTUHJUIVCDPN

    View full-size slide

  41. # gcc -lpci msireg.c!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0300c dest_mode=logical redirection=lowpri dest_id=3!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    -PHJDBMNPEFͰ-PXQSJɺEFTUJEɺ
    WFDUPSʹͳ͍ͬͯΔ
    # echo 1 > /proc/irq/44/smp_affinity!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0100c dest_mode=logical redirection=lowpri dest_id=1!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    EFTU@JE͕ʹॻ͖׵Θͬͨ

    View full-size slide

  42. Affinityηοτ࣌ͷΧʔωϧͷڍಈ
    γεςϜάϩʔόϧͳ%FTUJOBUJPO.PEFͱ
    %FMJWFSZ.PEFͷઃఆΛอ࣋

    ʢBQJDJSR@EFTU@NPEF 

    BQJDJSR@EFMJWFSZ@NPEFʣ
    "⒏OJUZηοτ࣌ʹ͜ͷ஋Λࢀরɺ%FTUJOBUJPO
    *%ͱซͤͯॻ͖ࠐΉ
    OBUJWF@DPNQPTF@NTJ@NTH

    BSDIYLFSOFMBQJDJP@BQJDD

    View full-size slide

  43. OSॳظԽ࣌ͷׂΓࠐΈϞʔυઃఆ
    ؀ڥʹΑͬͯҟͳΔυϥΠόΛ࢖༻

    BSDIYLFSOFMBQJDBQJD@qBU@D YBQJD@DMVTUFSD
    YBQJD@QIZTDʜ
    BQJD@qBUʢඇYBQJDϞʔυʣ

    %FMJWFSZ.PEF-PX1SJ

    %FTUJOBUJPO.PEF-PHJDBM
    BQJD@QIZTqBUʢඇYBQJDϞʔυ $164ʣ

    %FMJWFSZ.PEF'JYFE

    %FTUJOBUJPO.PEF1IZTJDBM
    BQJD@YBQJD@QIZTʢYBQJDϞʔυ "$1*Ͱ1IZTJDBMϞʔυࢦఆʣ

    ಉ্
    ϥ΢ϯυϩϏϯׂΓࠐΈͰ͖Δ؀ڥ͸Ұఆ৚݅Λຬͨͨ͠৔߹ͷΈ

    View full-size slide

  44. *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ
    %BUBTIFFU
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO
    *OUFMš$IJQTFUBOE*OUFMš$IJQTFU
    MJOVY%PDVNFOUBUJPO*32B⒏OJUZUYU
    6OEFSTUBOEJOHUIF-JOVY,FSOFM SE&EJUJPO
    1$*-PDBM#VT4QFDJpDBUJPO3FWJTJPO
    ࢀߟࢿྉ

    View full-size slide

  45. ࠓ·Ͱʹ·ͱΊͨࢿྉ
    ࠷ۙͷ1$ΞʔΩςΫνϟʹ͓͚ΔׂΓࠐΈϧʔςΟϯάͷ࢓૊Έ
    QSPDJSR*32TNQ@B⒏OJUZͷॻ͖׵͑Ͱ1$*ίϯϑΟάϨʔ
    γϣϯۭؒ͸ͲͷΑ͏ʹॻ͖׵ΘΔ͔
    QSPDJSR*32TNQ@B⒏OJUZ
    *OUFSSVQUTPOYW
    ϋΠύʔόΠβͷ࡞ΓํʙͪΌΜͱཧղ͢ΔԾ૝Խٕज़ʙୈ̐ճ*
    0Ծ૝ԽʮׂΓࠐΈฤɾͦͷ̍ʯ

    View full-size slide