Async, Persistent, Fast, and Stable "Enought" Queue/Worker Using Go and PostgreSQL

Transcript

1. 1
   "TZOD 1FSTJTUFOU 'BTU
   BOE4UBCMF&OPVHI˒
   2VFVF8PSLFS
   6TJOH(PBOE1PTUHSF42-
   (P$PO +1/PW
   "LJSB$IJLV
   

   View Slide

2. NF
   2
   /BNF"LJSB$IJLV
   5XJUUFS!@BDIJLV
   (JU)VC!BDIJLV
   'JSF+VHHMFS
   (PPHMF4FBSDI"LJSB$IJLV'JSF
   

   View Slide

3. BHFOEB
   3
   Ø CBDLHSPVOE QSPEVDUTJ[FUFBN
   
   Ø XPSLFSTFSWFSQBDLBHFTUSVDUVSF
   Ø XIZ1PTUHSF42-GPSRVFVFXPSLFSTZTUFN
   Ø RVFHPMJCSBSZJOUFSOBM
   Ø BQQMJDBUJPODPEF
   Ø PVUTJEFPGUSBOTBDUJPO FYUFSOBM"1*T
   Ø QSPEVDUJPOQFSGPSNBODFTUBUT
   

   View Slide

4. 4
   CBDLHSPVOE
   QSPEVDUTJ[FUFBN
   
   

   View Slide

5. CBDLHSPVOE QSPEVDU
   
   5
   Ø QSPEVDU
   Ø "QQCBTFEJOTUBOU BOESFMPBEBCMFWJSUVBMQMBTUJD
   7*4"QSFQBJEDBSEGPSFWFSZPOF
   Ø J04 "OESPJE 8FC
   Ø IUUQTWBOEMFKQ
   

   View Slide

6. CBDLHSPVOE TJ[F
   
   6
   Ø TJ[F BTPG/PW
   
   Ø PGFOEQPJOUT JOUFSOBM"1*
   
   Ø PGFYUFSOBMTFSWJDFT
   Ø OPUJODMVEJOH4FOUSZ/FX3FMJD4UBUVT1BHFJP
   
   Ø PGUBCMFT 1PTUHSF42-
   
   Ø PGMJOFT (PDPEFXJUIPVUWFOEPS
   
   Ø "MM
   Ø OPUFTUT
   Ø OPHFOFSBUFEDPEF OPUFTUT
   

   View Slide

7. CBDLHSPVOE UFBN
   
   7
   NFNCFS EFTJHO GSPOUFOE CBDLFOE
   *OGSB
   "84
   
   LPOPD
   NPRBEB
   ZPTVLF
   ZPTIJEB
   BDIJLV
   "TPG/PW
   IJSPBLJT
   

   View Slide

8. 8
   XPSLFSTFSWFS
   QBDLBHFTUSVDUVSF
   

   View Slide

9. QSPKFDUTUSVDUVSF
   9
   WBOEMFTFSWFS
   WBOEMFXPSLFS
   WBOEMFDMJ
   WBOEMF TFSWJDF NPEFM
   TFSWJDF
   TFSWJDF
   &YUFSOBM
   "1*
   DMJFOU
   

   View Slide

10. XPSLFSTFSWFSQBDLBHFTUSVDUVSF
    10
    WBOEMFBQJ
    NPEFM
    JOUFSBDUJPOXJUISECNT
    EBUBNPEFMJOUIFTFSWJDF
    OPUKVTUSECNT UBCMFSFQSFTFOUBUJPOT
    TFSWJDF
    EBUBNPEFMTJOUFSBDUXJUIFBDIPUIFS
    BMTP FYUFSOBM"1*TBSFDBMMFEJOTFSWJDFT
    JBQJ
    SFRVFTUSFTQPOTFTUSVDUT
    SFRVFTUWBMJEBUJPOGVODUJPOT
    IBOEMFST KPCT TFSWFS DPOG FUD
    
    FYTSWDMJFOU
    FYTSWDMJFOU
    DNE
    WBOEMFTFSWFS
    WBOEMFXPSLFS
    WBOEMFDMJ
    FYTSWNPDLTFSWFS
    FYTSWNPDLTFSWFS
    
    ANPEFMAQLH GPS3%#.4
    SFMBUFEJOUFSBDUJPOT
    ATFSWJDFAQLH GPSNPEFMT
    BOEFYU"1*JOUFSBDUJPOT
    AWBOEMFAQLH GPSIUUQ
    IBOEMFSTIUUQ
    TFSWFSXPSLFSTBTZODKPCT
    6TFADNEAUPDSFBUF
    EJGGFSFOUUZQFTPGCJOBSJFT
    

    View Slide

11. 11
    1PTUHSF42-
    GPSRVFVFXPSLFS
    

    View Slide

12. 1PTUHSF42-GPSRVFVFXPSLFS
    12
    Ø CBTFMJOFJT DPOTJTUFODZPWFSQFSGPSNBODF
    Ø XFBSFOPUCVJMEJOH35# OPS)'5TZTUFN
    Ø USBOTBDUJPOJTBXFTPNF EJTDVTTFEMBUFS
    
    Ø XFEJEOUXBOUUPBEEBOPUIFSNPWJOHQBSU
    Ø 3BCJUU.2;FSP.2"84424˘
    Ø POMZPOFGVMMUJNFCBDLFOEEFW BDIJLV
    BUUIBUNPNFOU
    Ø XFBSFVTJOH1PTUHSF42-BTPVSQSJNBSZEBUBTUPSF BOE(P
    BTNBJOMBOHVBHF
    Ø RVFVFXPSLFSMJCSBSZDBMMFERVFHP XIJDIXJMMCFEJTDVTTFE
    MBUFS JTXSJUUFOJO(P42- 1PTUHSF42-
    
    

    View Slide

13. 13
    RVFHPMJCSBSZJOUFSOBM
    

    View Slide

14. 2VFBOERVFHP
    14
    Ø IUUQTHJUIVCDPNDIBOLTRVF
    Ø 2VF LFɪ PSLBZ
    
    Ø ˑ2VFJTBIJHIQFSGPSNBODFBMUFSOBUJWFUP%FMBZFE+PC PS
    2VFVF$MBTTJD UIBUJNQSPWFTUIFSFMJBCJMJUZPGZPVS
    BQQMJDBUJPOCZQSPUFDUJOHZPVSKPCTXJUIUIFTBNF "$*%
    HVBSBOUFFT BTUIFSFTUPGZPVSEBUB˒
    Ø IUUQTHJUIVCDPNCHFOUSZRVFHP
    Ø ˑRVFHPJTBGVMMZJOUFSPQFSBCMF(PMBOH QPSUPG $ISJT
    )BOLT 3VCZ2VFRVFVJOHMJCSBSZ GPS1PTUHSF42-2VFVTFT
    1PTUHSF42-TBEWJTPSZMPDLTGPSTQFFEBOESFMJBCJMJUZ˒
    

    View Slide

15. RVFHPPWFSWJFX
    15
    "1*
    4FSWFS
    8PSLFS
    4FSWFS
    RVF@KPCT
    
    
    "1*TFSWFSNPEJGJFTDVSSFOU
    EBUBXIFOBVTFSSFRVFTUT
    • .PEJGZSFRVFTUFEEBUB
    • &ORVFVFBKPCGPSGVSUIFSXPSL
    • %PBCPWFUXPTUFQTJOPOF
    USBOTBDUJPO
    "XPSLFSEFRVFVFTBKPC
    BOEQSPDFTTFTJU
    • -PDLBKPCUPQSFWFOUJUGSPN
    CFJOHFYFDVUFENVMUJQMFUJNFTCZ
    PUIFSXPSLFST
    • 8PSLPOUIFUBTL
    • $PNNJUJGTVDDFFEFE SPMMCBDL
    NPEJGJDBUJPOTJGGBJMFE
    8PSLFS
    1PPM
    8PSLFS
    1PPM
    

    View Slide

16. RVFHPFORVFVF
    16
    Ø *UJTKVTUJOTFSUJOHBSPXUPBUBCMFOBNFE
    RVF@KPCT
    Ø 2VFVF
    Ø RVF@KPCTUBCMFIBTARVFVFADPMVNO
    Ø EJGGFSFOUXPSLFSQPPMTGPSEJGGFSFOURVFVFT
    Ø 1SJPSJUZ
    Ø TNBMMJOU SFQSFTFOUTQSJPSJUZXJUIJOUIF
    TBNFRVFVF
    Ø "SHT
    Ø +40/UZQFDPMVNO
    Ø 5IJTBMMPXT"1*TFSWFSUPFORVFVFBKPCJOB
    USBOTBDUJPOBMPOHXJUIPUIFSEBUBCBTF
    DIBOHFT
    Ø "KPCDBOˏUCFTFFOGSPNXPSLFSTCFGPSF
    UIFGJOBMDPNNJU
    "1*
    4FSWFS
    RVF@KPCT
    

    View Slide

17. RVFHPEFRVFVF
    17
    Ø &BDIXPSLFSTJOBXPSLFSQPPMBSFRVFSZJOHUP
    GJOEBKPCUPXPSLPO
    Ø 'JSTUJO'JSTUPVU MPPLJOHBUSVO@BU
    DPMVNO
    
    Ø -PDLBKPCUPQSFWFOUGSPNPUIFSXPSLFST
    HSBCJOH UIFTBNFKPC
    Ø "EWJTPSZMPDLXJUISFDVSTJWF$5&JTVTFEUP
    MPDLBKPCXJUIPVUCMPDLJOHPUIFSXPSLFST
    Ø 8PSLPOUIFKPC
    Ø *GJUTVDDFFEFE EFMFUFBKPCGSPNRVF@KPCT
    UBCMF
    Ø *GJUGBJMFE JODSFNFOUFSSPS@DPVOU VQEBUF
    SVO@BU GPSUIFOFYUSFUSZ BOESFDPSEFSSPS
    NFTTBHFJORVF@KPCTUBCMF
    8PSLFS
    4FSWFS
    8PSLFS
    1PPM
    8PSLFS
    1PPM
    RVF@KPCT
    

    View Slide

18. RVFHPBEWJTPSZMPDL
    
    18
    Ø 2VFVFTZTUFNOFFETTPNFTPSUPGMPDLJOHNFDIBOJTNUPSVO
    XPSLFSTDPODVSSFOUMZ
    Ø "EWJTPSZ-PDL
    Ø 4JODF1PTUHSF42- SFMFBTFEJO
    
    Ø *UˏTBQSFUUZNBUVBSFGVODUJPO
    Ø ˑ1PTUHSF42- QSPWJEFTBNFBOTGPSDSFBUJOHMPDLTUIBUIBWF
    BQQMJDBUJPOEFGJOFENFBOJOHT5IFTFBSFDBMMFE BEWJTPSZ
    MPDLT CFDBVTFUIFTZTUFNEPFTOPUFOGPSDFUIFJSVTFˋ JUJT
    VQUPUIFBQQMJDBUJPOUPVTFUIFNDPSSFDUMZ˒
    

    View Slide

19. RVFHPBEWJTPSZMPDL
    
    19
    Ø *UEPFTOˏUXSJUFUPUIFEJTL
    Ø -PDLTBSFTUPSFEJOBTIBSFENFNPSZQPPM
    Ø #FUUFSQFSGPSNBODFUIBONBOBHJOHKPCTUBUVTUBCMF
    Ø #FODINBSL
    Ø IUUQKPIUPQHCMPHTQPUKQRVFVFTRVFVFT
    UIFZBMMGBMMEPXOIUNM
    Ø *UEPFTOˏUCMPDLPUIFSRVFSJFT
    Ø 6OMJLFSPXMPDLJOHˑTFMFDU˘GPSVQEBUF˒
    Ø 4FTTJPO"4&-&$5QH@BEWJTPSZ@MPDL
    SFUVSOUSVF
    Ø 4FTTJPO#4&-&$5QH@BEWJTPSZ@MPDL
    SFUVSOGBMTF
    Ø "QQMJDBUJPOOFFETUPIBOEMFUIJT#PPMFBOWBMVF
    

    View Slide

20. RVFHPXPSLFSXPSLFSQPPM
    
    20
    // WorkerPool is a pool of Workers,
    // each working jobs from the queue Queue
    // at the specified Interval using the WorkMap.
    type WorkerPool struct {
    WorkMap WorkMap
    Interval time.Duration
    Queue string
    c *Client
    workers []*Worker
    mu sync.Mutex
    done bool
    }
    

    View Slide

21. RVFHPXPSLFSXPSLFSQPPM
    
    21
    // WorkFunc is a function that performs a Job.
    // If an error is returned, the job
    // is reenqueued with exponential backoff.
    type WorkFunc func(j *Job) error
    // WorkMap is a map of Job names to WorkFuncs
    // that are used to perform Jobs of a
    // given type.
    type WorkMap map[string]WorkFunc
    

    View Slide

22. RVFHPXPSLFSXPSLFSQPPM
    
    22
    // Start starts all of the Workers in the WorkerPool.
    func (w *WorkerPool) Start() {
    w.mu.Lock()
    defer w.mu.Unlock()
    for i := range w.workers {
    w.workers[i] = NewWorker(w.c, w.WorkMap)
    w.workers[i].Interval = w.Interval
    w.workers[i].Queue = w.Queue
    go w.workers[i].Work()
    }
    }
    

    View Slide

23. RVFHPXPSLFSXPSLFSQPPM
    
    23
    // Worker is a single worker that pulls jobs off the specified Queue. If no Job
    // is found, the Worker will sleep for Interval seconds.
    type Worker struct {
    // Interval is the amount of time that this Worker should sleep before trying
    // to find another Job.
    Interval time.Duration
    // Queue is the name of the queue to pull Jobs off of. The default value, "",
    // is usable and is the default for both que-go and the ruby que library.
    Queue string
    c *Client
    m WorkMap
    mu sync.Mutex
    done bool
    ch chan struct{}
    }
    

    View Slide

24. RVFHPXPSLFSXPSLFSQPPM
    
    24
    // Work pulls jobs off the Worker's Queue at its Interval. This function only
    // returns after Shutdown() is called, so it should be run in its own goroutine.
    func (w *Worker) Work() {
    for {
    select {
    case <-w.ch:
    log.Println("worker done")
    return
    case <-time.After(w.Interval):
    for {
    if didWork := w.WorkOne(); !didWork {
    break // didn't do any work, go back to sleep
    }
    }
    }
    }
    }
    

    View Slide

25. RVFHPESBXCBDLT
    
    25
    Ø 4FWFSFQFSGPSNBODFEFHSBEBUJPOVOEFSNBTTJWFFORVFVF BOE
    MPOHSVOOJOHKPCT
    Ø IUUQTCSBOEVSPSHQPTUHSFTRVFVFT
    Ø 4UBUFEQMBJOMZ PVSSPPUQSPCMFNJTUIBUUIFKPCUBCMFT
    JOEFYIBTCFDPNFMFTTVTFGVMUPUIFQPJOUXIFSFVTJOHJU
    JTOUNVDIGBTUFSUIBOBGVMMTFRVFOUJBMTDBO
    Ø IUUQTHJUIVCDPNCSBOEVSRVFEFHSBEBUJPOUFTU
    Ø MPOHSVOOJOHKPC TMFFQTFDBCPVUNJO
    
    Ø BCPVUKPCFORVFVFE QFSTFD
    Ø BGUFSNJO KPCTJOUIFRVFUBCMF
    Ø TJHOJGJDBOUMPDLJOHQFSGPSNBODFEFHSBEBUJPO
    

    View Slide

26. RVFHPESBXCBDLT
    
    26
    Ø BEWJTPSZMPDLJTEBUBCBTFXJEF BOEUIFLFZIBTUPCF
    JOUJOU
    Ø SFMBUJWFMZDPNQMFY8*5)3&$634*7&RVFSZUPMPDLJE
    Ø IUUQTCSBOEVSPSHQPTUHSFTRVFVFTMPDLJOHBMHPSJUINT
    Ø TLJQMPDLFENJHIUQFSGPSNCFUUFSXJUITJNQMFSJNQMFNFOUBUJPO
    Ø GSPN1PTUHSF42-
    Ø IUUQTCMPHOERVBESBOUDPNXIBUJTTFMFDUTLJQMPDLFE
    GPSJOQPTUHSFTRM
    Ø 5IJTNJHIUDPNQMJDBUFFSSPSIBOEMJOH BOEFYQ CBDLPGG SFUSZ
    BCJU CVUXPSUIBUSZ
    Ø 5IPTFBCPWFBSFXIZ*TUBUFEˑFOPVHI˒JOUIFUJUMFPGUIJTTMJEF
    

    View Slide

27. 27
    BQQMJDBUJPODPEF
    

    View Slide

28. BQQMJDBUJPODPEF
    28
    Ø GVOD RVF+PC
    FSSPS
    Ø 5IJTJTUIFTJHOBUVSFXFIBWFUPXSJUF
    Ø 'BJSMZTJNQMF
    Ø RVF+PC DPOUBJOTEBUBCBTFQPPM
    Ø 5IJTJTVTFEBMMEBUBCBTFPQFSBUJPOTJOKPCQSPDFTT
    Ø RVF+PC BMTPIBT"SHT XIJDIDPSSFTQPOETUPBSHT DPMVNOPG
    RVF@KPCTUBCMF
    Ø +40/UZQFDPMVNO
    Ø 6ONBSTIBMM "SHT <>CZUFJOUPQSFEFGJOFETUSVDU
    Ø *UJTWFSZVTFGVMUPIBWFKPCHMPCBMWBMVFT FH"1*DMJFOU
    BQQMJDBUJPODPOGJH
    JOBTUSVDU
    Ø -FUUIBUTUSVDU IBWFKPCNFUIPETTBUJTGJFTˑGVOD RVF+PC
    
    FSSPS˒TJHOBUVSF
    

    View Slide

29. BQQMJDBUJPODPEF
    
    29
    // BaseApp base app global
    values
    type BaseApp struct {
    Config *Config
    EstcClient *estc.Client
    }
    // JobApp job global values
    type JobApp struct {
    BaseApp
    Logger *log.Logger
    }
    IUUQTHJUIVCDPNBDIJLVHPUPEPJUCMPCNBTUFSKPCHP
    

    View Slide

30. BQQMJDBUJPODPEF
    
    30
    IUUQTHJUIVCDPNBDIJLVHPUPEPJUCMPCNBTUFSKPCHP
    // UpdateUserInfo update user info job
    func (app *JobApp) UpdateUserInfo(j *qg.Job) error {
    var args UpdateUserInfoArgs
    if err := json.Unmarshal(j.Args, &args); err != nil {
    return errors.Wrap(err, "failed to unmarshal UpdateTodoETCArgs")
    }
    dbReq := &model.TodoUser{
    UUID: args.UserID,
    Username: args.Username,
    Email: args.Email,
    Status: args.Status,
    }
    if err := service.UpdateUser(j.Tx(), dbReq); err != nil {
    return errors.Wrap(err, "service.UpdateUser failed")
    }
    ctx := app.Context()
    req := &estc.UpdateUserRequest{
    ID: args.UserID,
    UserName: args.Username,
    Email: args.Email,
    }
    u, err := app.EstcClient.UpdateUser(ctx, req)
    if err != nil {
    return errors.Wrap(err, "EstcClient.UpdateUser failed")
    }
    app.Logger.Printf("userID=%s,email=%s", u.ID, u.Email)
    return nil
    }
    

    View Slide

31. BQQMJDBUJPODPEF
    
    31
    IUUQTHJUIVCDPNBDIJLVHPUPEPJUCMPCNBTUFSXPSLFSHP
    jobs := JobApp{
    BaseApp: BaseApp{
    Config: appCfg,
    },
    }
    wm := qg.WorkMap{
    "updateUserJob": jobs.UpdateUserInfo,
    }
    wPoolLow := qg.NewWorkerPool(qc, wm, appCfg.NumWorkers)
    wPoolLow.Queue = QueNameLowPriority
    wPoolHigh := qg.NewWorkerPool(qc, wm, appCfg.NumWorkers)
    wPoolHigh.Queue = QueNameHighPriority
    

    View Slide

32. BQQMJDBUJPODPEFUFTU
    
    32
    IUUQTHJUIVCDPNBDIJLVHPUPEPJUCMPCNBTUFSKPC@UFTUHP
    func TestUpdateUserInfo(t *testing.T) {
    app, tx, _, cleanup := testSetupJobApp(t)
    defer cleanup()
    u := model.TestCreateUserData(t, tx, &model.TodoUser{})
    args := UpdateUserInfoArgs{
    UserID: u.UUID,
    Email: fmt.Sprintf("%s@example.com", u.UUID),
    Status: "inactive",
    Username: "updated-username",
    }
    jsonArgs, err := json.Marshal(args)
    if err != nil {
    t.Fatal(err)
    }
    ts := httptest.NewServer(estc.TestNewMux(estc.DefaultHandlerMap))
    defer ts.Close()
    app.Config.EstcConfig.BaseEndpoint = ts.URL
    j := qg.TestInjectJobTx(&qg.Job{Args: jsonArgs}, tx)
    if err := app.UpdateUserInfo(j); err != nil {
    t.Fatal(err)
    }
    }
    

    View Slide

33. 33
    PVUTJEFPGUSBOTBDUJPO
    FYUFSOBM"1*TFSWJDFT
    

    View Slide

34. 34
    "1*
    4FSWFS
    8PSLFS
    4FSWFS
    RVF@KPCT
    8PSLFS
    1PPM
    8PSLFS
    1PPM
    PVUTJEFPGUSBOTBDUJPO
    &YUFSOBM
    "1*
    PVUTJEFPG
    USBOTBDUJPO
    

    View Slide

35. 35
    &YUFSOBM"1*TFSWJDFT
    

    View Slide

36. 36
    'BJM
    

    View Slide

37. CFQSFQBSFEGPS"1*GBJMVSFT
    37
    Ø ,FFQJUˑKPCUSBOTBDUJPO˒
    Ø :PVEPOˏUIBWFUPQBZFYUSBNFOUBMDPTUGPSEFTJHOJOHUIF
    CFTUSFUSZQSPDFTT
    Ø *GZPVVTFNVMUJQMFUSBOTBDUJPOTJOPOFKPC ZPVIBWFUP
    UIJOLUISPVHIIPXZPVDBOHVBSEUIFEBUBJOUIFGJSTUUY
    XIFOUIFTFDPOEUY GBJMT
    Ø 8IFOJUJTSFBMMZOFFEFEUPIBWFNVMUJQMFUSBOTBDUJPOT UIFO
    DPOTJEFSUPEJWJEFBKPCJOUPUXPKPCT
    Ø 5IFGJSTUKPCFORVFVFUIFTFDPOEKPC
    Ø 8IFOFYUFSOBM"1*BDDFTTJTSFRVJSFE LFFQBTLJOH˒XIBUJGUIJT
    GBJMT DPVMEXFQSPUFDUPVSEBUBJOUFHSJUZ ˒
    

    View Slide

38. +PCDBUFHPSJFT
    38
    Ø $6% ˘$SFBUF6QEBUF%FMFUF
    Ø 3 ˘3FBE
    3%#.4
    &YUFSOBM"1*
    OB 3 $6%
    OB
    3
    $6%
    Ø $BUFHPSZ
    Ø 3%#.4$36% BOESFBE
    FYUFSOBM"1*
    Ø $BUFHPSZ
    Ø 3%#.4$36% BOE$6%
    FYUFSOBM"1*
    

    View Slide

39. $BUFHPSZ
    39
    3%#.4
    &YUFSOBM"1*
    OB 3 $6%
    OB
    3
    $6%
    Ø $BUFHPSZKPCDBOCFDPNQMFUFEXJUIJO3%#.4
    Ø +PCJTQSPUFDUFECZUSBOTBDUJPO
    Ø 4VDDFTTPSGBJMVSFJTBUPNJD
    Ø *GKPCGBJMTCZ3%#.4PQFSBUJPOPSSFBEFYUFSOBM"1*T KVTU
    SPMMCBDLBMMUIFEBUBDIBOHFT BOESFUSZ
    

    View Slide

40. $BUFHPSZ
    40
    3%#.4
    &YUFSOBM"1*
    OB 3 $6%
    OB
    3
    $6%
    Ø $BMMFYUFSOBM"1*TUPDIBOHFUIFJSEBUB BOEBMTPNPEJGZEBUBJO
    3%#.4
    Ø %BUBDIBOHFTJO3%#.4DBOCFSPMMFCBDLFE JGUIFQSPDFTT
    GBJMFE CVUEBUBNPEJGJDBUJPOTUISPVHIFYUFSOBM"1*TDBOˏU
    CFDBODFMFE
    Ø *UNJHIUCFQPTTJCMFUPSFTUPSFFYUFSOBMEBUBCBDLUPQSFWJPVT
    TUBUFJOFYDFQUJPOIBOEMJOH CVUJUXJMMNBLFTJNQMFUIJOHT
    DPNQMJDBUFE
    Ø 5SZOPUUPDBMMUIFTBNF"1*NVMUJQMFUJNFTFWFOXIFOJUIBTUP
    SFUSZ
    Ø 5SZUPDBMMFYUFSOBM"1*BUUIFMBTUQBSUPGUIFQSPDFTT
    Ø 5SZOPUUPVTFNVMUJQMFFYUFSOBM"1*TJOKPC
    Ø USBOTBDUJPO KPC BUNPTUFYUFSOBM$6%"1*
    

    View Slide

41. IPXNVDIJTˑFOPVHI˒
    41
    Ø "TZPVDBOTFF UIFSFJTOPQFSGFDUCFBVUJGVMTJNQMFTPMVUJPOGPS
    FYUFSOBM"1*TFSWJDFFSSPSIBOEMJOH LFFQJOHBMMUIFEBUB
    JOUFHSJUZBOEQFSGPSNBODF˘
    Ø *O+BQBOFTF XFDBMMUIJTTJUVBUJPOˑװ׏גְֻ׃ַזְ˒
    

    View Slide

42. 42
    װ׏גְֹתׅ
    

    View Slide