Interrupt Affinityについて

Transcript

1. Interrupt Affinity

   ʹ͍ͭͯ
   @syuu1228
   

   View Slide

2. Interrupt Affinity?
   ࠓ࣌ͷී௨ͷ1$͸ϚϧνίΞͰ͋Δ
   σόΠε̍ɿ*32̍ɿ$16͍ͬͺ͍

   ͱ͍͏ؔ܎ʹͳ͍ͬͯΔ
   σόΠε͸Ͳ͜ʹׂΓࠐΜͰ͍Δͷʁ

   -JOVY͸Ͳͷ$16ʹׂΓࠐ·͍ͤͯΔͷʁ
   

   ˠ*OUFSSVQU"⒏OJUZ
   

   View Slide

3. Interrupt Affinityͷ
   ϢʔβΠϯλϑΣʔε
   

   View Slide

4. $ cat /proc/interrupts
   CPU0 CPU1
   0: 36843 61973 IO-APIC-edge timer
   1: 518 591 IO-APIC-edge i8042
   8: 1 0 IO-APIC-edge rtc0
   9: 387 321 IO-APIC-fasteoi acpi
   12: 11236 79 IO-APIC-edge i8042
   16: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
   17: 12 11 IO-APIC-fasteoi uhci_hcd:usb7
   18: 0 0 IO-APIC-fasteoi uhci_hcd:usb8
   19: 6531 235 IO-APIC-fasteoi ehci_hcd:usb2
   20: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
   21: 85 69 IO-APIC-fasteoi uhci_hcd:usb4
   22: 0 0 IO-APIC-fasteoi uhci_hcd:usb5
   23: 20 15 IO-APIC-fasteoi ehci_hcd:usb1
   40: 0 0 PCI-MSI-edge PCIe PME, pciehp
   41: 0 0 PCI-MSI-edge PCIe PME, pciehp
   42: 0 0 PCI-MSI-edge PCIe PME, pciehp
   43: 33199 31756 PCI-MSI-edge ahci
   44: 19 1583 PCI-MSI-edge eth0
   45: 10 11 PCI-MSI-edge mei
   46: 41104 59303 PCI-MSI-edge i915
   47: 170268 708 PCI-MSI-edge iwlwifi
   48: 78 77 PCI-MSI-edge snd_hda_intel
   NMI: 1406 1430 Non-maskable interrupts
   $16ຖͷׂΓࠐΈ౷ܭΛදࣔ
   /proc/interrupts
   

   View Slide

5. /proc/irq/*/smp_affinity
   ׂΓࠐΈઌ$16܈ΛϏοτͰࢦఆ͢Δ
   ͱɺࢦఆ͞Εͨ$16܈ʹϥ΢ϯυϩϏ
   ϯͰׂΓࠐ·ΕΔʢϏοτ͕୯ҰͳΒ
   ৗʹͦͷ$16΁ׂΓࠐΈʣ
   # cat /proc/irq/43/smp_affinity
   3
   # echo 1 > /proc/irq/43/smp_affinity
   ͜ΕΛॻ͖׵͑ΔͱׂΓࠐΈઌ$16͕มΘΔ
   

   View Slide

6. LinuxͰͷσϑΥϧτ஋
   શ$16ͷϏοτ͕༗ޮʹͳ͓ͬͯΓɺ
   ͭͷׂΓࠐΈ͕શ$16΁ϥ΢ϯυϩ
   Ϗϯ͢Δ
   ϥ΢ϯυϩϏϯׂΓࠐΈʹରԠ͠ͳ͍1$
   Ͱ͸͋Δ$16΁ݻఆతʹׂΓࠐΉ
   ʢDQV ʣ
   

   View Slide

7. /proc/irq/*/smp_affinity_list
   ώϡʔϚϯϦʔμϒϧͳTNQ@B⒏OJUZ
   # echo ff > /proc/irq/43/smp_affinity
   # cat /proc/irq/43/smp_affinity_list

   0-7
   

   View Slide

8. /proc/irq/*/smp_affinity_hint
   σόΠευϥΠό͔ΒϢʔβϥϯυσʔ
   ϞϯʢJSRCBMBODFʣ΁͜ͷ*32ʹద੾
   ͳ$16NBTLΛ௨஌
   

   View Slide

9. irqbalance
   ඵ͝ͱʹγεςϜෛՙঢ়گΛνΣο
   Ϋɺ֤*32ͷTNQ@B⒏OJUZΛॻ͖׵͑
   ͯ$16ෛՙΛެฏʹ͢ΔσʔϞϯ
   POFTIPUϞʔυͰ͸ىಈ࣌ʹద੾ͱࢥ
   ΘΕΔ஋ΛTNQ@B⒏OJUZʹઃఆͯ͠
   FYJU
   

   View Slide

10. MultiQueue NICͷׂΓࠐΈ
    $16ຖʹૹड৴ΩϡʔɺΩϡʔຖʹ
    .4*9ׂΓࠐΈΛ࣋ͭ
    ਖ਼͍͠TNQ@B⒏OJUZ͸ݻఆతʹܾ·ͬ
    ͍ͯΔ
    ෳ਺$16΁ࢄΒͯ͠͸ͳΒͳ͍
    47: 7602 0 0 0 0 3 23 0 PCI-MSI-edge p1p1-TxRx-0
    48: 0 7602 0 0 0 0 13 12 PCI-MSI-edge p1p1-TxRx-1
    49: 12 0 7605 0 0 0 10 0 PCI-MSI-edge p1p1-TxRx-2
    50: 0 12 0 7602 3 0 10 0 PCI-MSI-edge p1p1-TxRx-3
    51: 0 0 12 0 7602 3 10 0 PCI-MSI-edge p1p1-TxRx-4
    52: 0 0 0 20 0 7602 13 0 PCI-MSI-edge p1p1-TxRx-5
    53: 0 0 0 0 12 0 7612 3 PCI-MSI-edge p1p1-TxRx-6
    54: 3 0 0 0 0 13 10 7602 PCI-MSI-edge p1p1-TxRx-7
    55: 0 2 0 0 0 0 2 0 PCI-MSI-edge p1p1
    

    View Slide

11. ιʔείʔυ
    QSPDJOUFSSVQUT

    GTQSPDJOUFSSVQUTD
    QSPDJSR

    LFSOFMJSRQSPDD
    JSRCBMBODF

    IUUQTHJUIVCDPNJSRCBMBODF
    JSRCBMBODF
    

    View Slide

12. PCΞʔΩςΫνϟʹ͓͚Δ

    ׂΓࠐΈϧʔςΟϯά
    

    View Slide

13. PC্ͷׂΓࠐΈίϯϙʔωϯτ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip
    

    View Slide

14. Local APIC
    $16ຖͷׂΓࠐΈίϯτϩʔϥɹׂΓࠐΈΛڐՄɾϚεΫ͠
    ͨΓɺ&0*ͨ͠Γ
    γεςϜશମͰҰҙͳ"1*$*%Λ࣋ͭ
    ଞͷ-"1*$΁ׂΓࠐΊΔʢ*1**OUFSQSPDFTTPS*OUFSSVQU
    
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip
    

    View Slide

15. LAPIC - Interrupt
    Command Register(ICR)
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFS4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    %FTUJOBUJPO4IPSUIBOEɿѼઌͷϚεΫํࣜʢࣗ$16ΛϚεΫɾ
    ΞϯϚεΫͳͲʣ
    %FTUJOBUJPO*%ɿѼઌ
    

    View Slide

16. ιʔείʔυ
    *$3

    BSDIYJODMVEFBTNJQJI
    

    View Slide

17. σʔλγʔτ
    *OUFMšBOE*"
    "SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT

    *OUFSSVQU$PNNBOE
    3FHJTUFS *$3
    
    

    View Slide

18. ิ଍ɿx2APIC
    -"1*$*%͸ैདྷCJU͕ͩͬͨɺίΞ਺ͷ૿ՃʹΑ
    ΓϏοτ਺͕଍Γͳ͘ͳͬͯ͠·ͬͨͷͰCJU΁
    ϏοτΛ௥Ճ
    ͜Εʹ൐ͬͯɺҎ߱Ͱղઆ͢Δ֤ϨδελͰͷׂΓ
    ࠐΈઌ-"1*$ࢦఆϑΟʔϧυͷΞυϨε෯΋Ϗοτ
    ௥Ճ͞Εͨ
    

    View Slide

19. σʔλγʔτ
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    (-044"3:
    

    View Slide

20. I/O APIC
    ݶΒΕͨຊ਺ͷ෺ཧׂΓࠐΈઢΛ֤1$*σόΠεͰڞ༗

    ʢ1$*Fʹ͸෺ཧׂΓࠐΈઢ͸ແ͘ɺϝοηʔδϯάʹΑͬͯ*0"1*$ܦ༝ׂΓࠐΈΛ
    ΤϛϡϨʔτʣ
    ֤*32ΛͲͷ-"1*$΁సૹ͢Δ͔Λఆٛ͢Δ3FEJSFDUJPO5BCMF

    ʢ*0"1*$্ͷϨδελ܈ʣΛ࣋ͭ
    1$*σόΠε͸*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip
    

    View Slide

21. I/O APIC - REDIR_TBL
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    %FMJWFSZ4UBUVTɿલճͷׂΓࠐΈͷ഑ૹঢ়گ
    *OUFSSVQU*OQVU1JO1PMBSJUZɿׂΓࠐΈϐϯͷۃੑ
    3FNPUF*33ɿࣗಈతʹ&0*͢Δ͔Ͳ͏͔
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤοδτϦΨ
    .BTLɿׂΓࠐΈϚεΫ
    %FTUJOBUJPO*%ɿѼઌ
    

    View Slide

22. ιʔείʔυ
    3&%*3@5#-

    BSDIYLFSOFMBQJDJP@BQJDD
    

    View Slide

23. σʔλγʔτ
    *OUFMš*0$POUSPMMFS)VC
    *$)
    'BNJMZ%BUBTIFFU

    3&%*3@5#-3FEJSFDUJPO
    5BCMF
    

    View Slide

24. ϨΨγσόΠε
    ϨΨγσόΠε͔ΒͷׂΓࠐΈΛ1$*-1$CSJEHFͰ1$*΁ม׵
    1$*σόΠεಉ༷ʹ*0"1*$Λ௨ͯ͡-"1*$΁ׂΓࠐΈϝοηʔ
    δΛૹ৴
    ޓ׵ੑͷͨΊ"ʹ΋઀ଓ͞Ε͍ͯΔͱࢥΘΕΔ
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip
    

    View Slide

25. 8259A PIC
    লུʢ·ͩ࢒ͬͯ·͕͢ɺ΋͏࢖Θͳ
    ͍Ͱ͠ΐ͏ʣ
    

    View Slide

26. MSI(-X) capable PCI devices
    ֤σόΠε͕೚ҙͷ਺ͷ*32Λ࣋ͯΔ
    ֤*32ͷׂΓࠐΈઌ-"1*$͸1$*σόΠεͷ1$*
    $POpHVSBUJPO4QBDFʹ࣋ͭ
    1$*σόΠε͸௚઀-"1*$΁ׂΓࠐΈϝοηʔδΛૹ৴
    CPU
    Local APIC
    CPU
    Local APIC
    CPU
    Local APIC
    ICH
    8259A PIC
    Timer
    I/O APIC
    Legacy
    PCI
    Devices
    MSI(-X)
    Capable
    Devices
    IPI
    Legacy PCI
    8259A Intr
    MSI(-X)
    PCI-LPC
    bridge
    SuperIO
    Chip
    

    View Slide

27. MSI Capability on

    PCI configuration space
    Device ID Vendor ID
    …
    Reserved Cap. Pointer
    …
    Capability x CAP ID(x)
    Next Pointer
    …
    Capability y CAP ID(yy)
    Next Pointer
    …
    MSICTL CAP ID(D0h)
    Next Pointer
    MSIAR
    MSIDR
    

    View Slide

28. MSI Capability - MSICTL
    লུʢ.4*༗ޮɺCJUɺෳ਺ׂΓࠐ
    Έ༗ޮͳͲͷϑΟʔϧυʣ
    

    View Slide

29. MSI Capability - MSIAR
    %FTUJOBUJPO.PEFɿѼઌϞʔυ
    3FEJSFDUJPO)JOUɿMPHJDBMNPEFˍ
    -PX1SJͰͷSFEJSFDUJPOΛ༗ޮԽ
    %FTUJOBUJPO*%ɿѼઌ
    

    View Slide

30. MSI Capability - MSIDR
    7FDUPSɿѼઌ-"1*$ͷϕΫλ൪߸
    %FMJWFSZ.PEFɿׂΓࠐΈ഑ૹϞʔυ
    %FMJWFSZ4UBUVTɿׂΓࠐΈঢ়گ
    5SJHHFS.PEFɿϨϕϧτϦΨʗΤο
    δτϦΨ
    

    View Slide

31. MSI-Xͷ৔߹
    লུʢϑΟʔϧυ͕֦ு͞Εͯ୔ࢁׂ
    ΓࠐΈ͕࣋ͯΔ͕ɺ࢓૊Έ͸ಉ͡ʣ
    

    View Slide

32. ιʔείʔυ
    .4* 9
    $BQBCJMJUZ

    BSDIYLFSOFMBQJDJP@BQJDD
    

    View Slide

33. σʔλγʔτ
    *OUFMš$IJQTFUBOE*OUFMš
    $IJQTFU

    .4*$5-.4*$POUSPM
    3FHJTUFS

    .4*"3.4*"EESFTT
    3FHJTUFS

    .4*%3.4*%BUB3FHJTUFS
    

    View Slide

34. ֤ํࣜͰڞ௨ͳཁૉ
    ҎԼͷࡾͭͷϑΟʔϧυͰѼઌ͕ࢦఆ
    ͞ΕΔ
    %FTUJOBUJPO.PEF
    %FMJWFSZ.PEF
    %FTUJOBUJPO*%
    

    View Slide

35. ׂΓࠐΈઌͷࢦఆํ๏
    1IZTJDBM%FTUJOBUJPO.PEF
    %FTUJOBUJPO'JFMEʹ"1*$*%Λࢦఆ

    ʮৗʹಉ͡$16΁ׂΓࠐΈʯΛ࣮ݱ
    -PHJDBM%FTUJOBUJPO.PEF 'MBU.PEFM
    
    %FTUJOBUJPO'JFMEͷCJUͰѼઌ-"1*$܈ͷൣғΛදݱ
    %FMJWFSZ.PEF
    'JYFE

    ࢦఆൣғͷશͯͷ-"1*$΁ׂΓࠐΈ
    -PXFTU1SJPSJUZ

    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL1SJPSJUZ3FHJTUFSʣͷ஋͕
    ௿͍$16΁ׂΓࠐΈ
    

    View Slide

36. Logical Destination Mode

    (Cluster Model)
    Y"1*$ͷ৔߹ɺ

    CJU$MVTUFS*%

    CJU-PHJDBM*%
    'MBU.PEFMʹൺ΂ͯଟ͘ͷ$16਺Λ
    αϙʔτͰ͖Δ
    

    View Slide

37. Lowest Priority Mode
    ࢦఆൣғͷதͰɺ࠷΋513ʢ5BTL
    1SJPSJUZ3FHJTUFSʣͷ஋͕௿͍-"1*$΁
    ׂΓࠐΈ
    ୠ͠ɺ-JOVYͰ͸ಈ࡞தʹ513Λߋ৽͠ͳ
    ͍ˠશ-"1*$ͷ513͸શͯಉ͡஋
    ࠷খ஋ͷ513Λ࣋ͭ-"1*$͕ෳ਺͋Δ৔
    ߹͸ϥ΢ϯυϩϏϯͰ̍ͭબ୒
    

    View Slide

38. σʔλγʔτ
    *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF%FWFMPQFS
    .BOVBMT

    *OUFSSVQU$PNNBOE3FHJTUFS *$3
    

    1IZTJDBM%FTUJOBUJPO.PEF

    -PHJDBM%FTUJOBUJPO.PEF

    -PXFTU1SJPSJUZ%FMJWFSZ.PEF
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO

    -PHJDBM%FTUJOBUJPO3FHJTUFS
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ%BUBTIFFU

    "VUPNBUJD3PUBUJPO.PEF &RVBM1SJPSJUZ%FWJDFT
    

    4QFDJpD3PUBUJPO.PEF 4QFDJpD1SJPSJUZ
    
    

    View Slide

39. echo 1 > /proc/irq//
    smp_affinityͨ࣌͠ʹԿ͕ى͖Δͷ͔
    .4*ͳ1$*FσόΠεͰ͔֬ΊͯΈΔ
    *OUFM1SP FF
    
    1$*$POpHVSBUJPO4QBDF͸SPPUͳΒ
    TZTCVTQDJ͔ΒಡΊΔ

    ˠϢʔβϥϯυ͔ΒಡΊΔ

    ˠ΋͔ͯ͠͠ɿMTQDJ

    

    View Slide

40. # lspci -vvvv -s 00:19.0
    00:19.0 Ethernet controller: Intel Corporation 82567LM Gigabit Network Connection (rev 03)
    Subsystem: Lenovo Device 20ee
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
    Interrupt: pin A routed to IRQ 44
    Region 0: Memory at f2600000 (32-bit, non-prefetchable) [size=128K]
    Region 1: Memory at f2625000 (32-bit, non-prefetchable) [size=4K]
    Region 2: I/O ports at 1840 [size=32]
    Capabilities: [c8] Power Management version 2
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
    Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
    Address: 00000000fee0300c Data: 41b9
    Capabilities: [e0] PCI Advanced Features
    AFCap: TP+ FLR+
    AFCtrl: FLR-
    AFStatus: TP-
    Kernel driver in use: e1000e
    Kernel modules: e1000e
    ੺͘ృͬͨॴ͕.4*ͷϑΟʔϧυ͕ͩɺ͜Εͩ
    ͱಡΊͳ͍ͷͰMTQDJΛGPSLͯ͠.4*ϑΟʔϧ
    υΛදࣔ͢ΔπʔϧΛ࡞੒

    IUUQTHJTUHJUIVCDPN
    

    View Slide

41. # gcc -lpci msireg.c!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0300c dest_mode=logical redirection=lowpri dest_id=3!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    -PHJDBMNPEFͰ-PXQSJɺEFTUJEɺ
    WFDUPSʹͳ͍ͬͯΔ
    # echo 1 > /proc/irq/44/smp_affinity!
    # ./a.out 00:19.0!
    Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+!
    address_hi=0!
    address_lo=fee0100c dest_mode=logical redirection=lowpri dest_id=1!
    data=41b9 trigger=edge level=assert delivery_mode=lowpri vector=185
    EFTU@JE͕ʹॻ͖׵Θͬͨ
    

    View Slide

42. Affinityηοτ࣌ͷΧʔωϧͷڍಈ
    γεςϜάϩʔόϧͳ%FTUJOBUJPO.PEFͱ
    %FMJWFSZ.PEFͷઃఆΛอ࣋

    ʢBQJDJSR@EFTU@NPEF 

    BQJDJSR@EFMJWFSZ@NPEFʣ
    "⒏OJUZηοτ࣌ʹ͜ͷ஋Λࢀরɺ%FTUJOBUJPO
    *%ͱซͤͯॻ͖ࠐΉ
    OBUJWF@DPNQPTF@NTJ@NTH

    BSDIYLFSOFMBQJDJP@BQJDD
    

    View Slide

43. OSॳظԽ࣌ͷׂΓࠐΈϞʔυઃఆ
    ؀ڥʹΑͬͯҟͳΔυϥΠόΛ࢖༻

    BSDIYLFSOFMBQJDBQJD@qBU@D YBQJD@DMVTUFSD
    YBQJD@QIZTDʜ
    BQJD@qBUʢඇYBQJDϞʔυʣ

    %FMJWFSZ.PEF-PX1SJ

    %FTUJOBUJPO.PEF-PHJDBM
    BQJD@QIZTqBUʢඇYBQJDϞʔυ $164ʣ

    %FMJWFSZ.PEF'JYFE

    %FTUJOBUJPO.PEF1IZTJDBM
    BQJD@YBQJD@QIZTʢYBQJDϞʔυ "$1*Ͱ1IZTJDBMϞʔυࢦఆʣ

    ಉ্
    ϥ΢ϯυϩϏϯׂΓࠐΈͰ͖Δ؀ڥ͸Ұఆ৚݅Λຬͨͨ͠৔߹ͷΈ
    

    View Slide

44. *OUFMšBOE*""SDIJUFDUVSFT4PGUXBSF
    %FWFMPQFS.BOVBMT
    *OUFMš*0$POUSPMMFS)VC *$)
    'BNJMZ
    %BUBTIFFU
    *OUFMš"SDIJUFDUVSFY"1*$4QFDJpDBUJPO
    *OUFMš$IJQTFUBOE*OUFMš$IJQTFU
    MJOVY%PDVNFOUBUJPO*32B⒏OJUZUYU
    6OEFSTUBOEJOHUIF-JOVY,FSOFM SE&EJUJPO
    1$*-PDBM#VT4QFDJpDBUJPO3FWJTJPO
    ࢀߟࢿྉ
    

    View Slide

45. ࠓ·Ͱʹ·ͱΊͨࢿྉ
    ࠷ۙͷ1$ΞʔΩςΫνϟʹ͓͚ΔׂΓࠐΈϧʔςΟϯάͷ࢓૊Έ
    QSPDJSR*32TNQ@B⒏OJUZͷॻ͖׵͑Ͱ1$*ίϯϑΟάϨʔ
    γϣϯۭؒ͸ͲͷΑ͏ʹॻ͖׵ΘΔ͔
    QSPDJSR*32TNQ@B⒏OJUZ
    *OUFSSVQUTPOYW
    ϋΠύʔόΠβͷ࡞ΓํʙͪΌΜͱཧղ͢ΔԾ૝Խٕज़ʙୈ̐ճ*
    0Ծ૝ԽʮׂΓࠐΈฤɾͦͷ̍ʯ
    

    View Slide