Goで作る大量プロセス管理機構

Go Conference'19 Summer in Fukuoka

Transcript

1. (PͰ࡞Δ
   େྔϓϩηε؅ཧػߏ  ໟར
   ܒଠ
2. 
    ‣ 'VTJD
   $P
   -UE
   ‣
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   ൃҊऀϓϩμΫτΦʔφʔ
   ‣ *P5ܥͷडୗҊ݅΋
   ‣ (P
   3VCZ
   "84
   403"$0. ໟར
   ܒଠʢ΋͏Γ
   ͚͍ͨʣ
3. 
   
4. 
    ࣗࣾαʔϏε
5. 
    ࣗࣾαʔϏε डୗ։ൃ 8FCαΠτ &$αΠτ FUD ۀ຿ΞϓϦ ίϯαϧ "3 Ϋϥ΢υ *P5 'JO5FDI "* ΠϯλϥΫςΟϒίϯςϯπ εϚϗΞϓϦ
6. ࣗࣾαʔϏε डୗ։ൃ 8FCαΠτ &$αΠτ FUD ۀ຿ΞϓϦ ίϯαϧ "3 Ϋϥ΢υ *P5 'JO5FDI "* ΠϯλϥΫςΟϒίϯςϯπ εϚϗΞϓϦ ύʔτφʔ
7. ࠓ೔͓࿩͍ͨ͜͠ͱ
8. 
    ͷલʹʜ
9. ͬͯͳʹʁ
10. 
     *P5։ൃऀ޲͚
    ٙࣅσʔλੜ੒αʔϏε
11. 
     *P5ͷ࿩
12. 
     Ϟϊ͕ Thing
13. 
     Πϯλʔωοτܦ༝Ͱ Thing
14. 
     αʔόʹσʔλΛૹΓ Backend Thing D a t a
15. 
     ղੳͨ͠Γͯ͠ Backend Thing D a t a
16. 
     Կ͔͢Δ Backend Thing D a t a
17. 
     *P5ͷ࿩ ͓ΘΓ
18. 
     *P5
    ։ൃ͠·͢
19. 
     ςετ͍ͨ͠ Backend Thing D a t a
20. 
     σʔλૹͬͯ΄͍͠ Backend Thing D a t a
21. 
     Ͱ΋
22. 
     Ϟϊʹͩͬͯ
    ౎߹͸͋Δ
23. 
     ͑͐ͬɺ
    ։ൃʹ൒೥΋͔͔Δͷʂʁ
24. 
     ͑͐ͬɺ
    ϗϯτʹյ͞ͳ͍ͱ
    Τϥʔσʔλग़ͤͳͷʂʁ
25. 
     ෛՙࢼݧ༻ʹ୆
    ཉ͍͚͠Ͳʜ
    ݸສԁʂʁ
26. 
     ෛՙࢼݧ༻ʹ୆
    ༻ҙ͚ͨ͠Ͳʜ
    ୭͕ૢ࡞͢Δͷ͜Εʂʁ
27. 
     ςετʹࠔΔ
28. 
     ͦ͏ͩʂ
29. 
     γϛϡϨʔλʔΛ࡞Ζ͏ʂ Backend Simulator D a t a
30. 
     ΍͹͍ɺ
    ϦϦʔεʹؒʹ߹Θͳ͍ʂ
31. 
     ೉͗ͯ͢͠
    ʓʓ͞Μ ࡞ͬͨਓ ͔͠
    ಈ͔ͤͳ͍ʜ
32. 
     ٱ͠ͿΓʹಈ͔ͨ͠Β
    શવಈ͔ͳ͍ʜ
33. 
     ͦ͜Ͱ
34. 
     ࢥ͍Ͳ͓ΓʹͳΔϞϊ Backend D a t a
35. 
     ͍ͭͰ΋
    ཉ͍͠σʔλΛ
    ཉ͍͠ྔ͚ͩ
    ૹͬͯ͘ΕΔʂ
36. 
     ͏Ε͍͠
37. ը໘Πϝʔδ
     ࢥ͍௨ΓͷϑΥʔϚοτͰ+40/σʔλΛੜ੒ Ծ૝σόΠεͷεςʔλεͱͦͷมԽΛ೚ҙʹઃఆ ͞·͟·ͳ࣌ܥྻσʔλੜ੒ػೳ
38. ͷ
    ͬ͘͟Γͨ͠࢓૊Έ
39. 
     PublicSubnet PrivateSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
40. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ PrivateSubnet Mock Mock Mock Mock Mock
41. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
42. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
43. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
44. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
45. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker ίϯιʔϧͰ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock mockىಈ Mock Mock [ mock ] Go੡όΠφϦ͔Βཱͯͨ Ծ૝σόΠε໾ϓϩηε [ tower ] Go੡όΠφϦ͔Βཱͯͨ mock؅ཧϓϩηε
46. ࠓ೔͓࿩͍ͨ͜͠ͱ վΊͯʂ
47. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
48. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock ͳΜͰϓϩηεʁ ͳΜͰGoʁ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
49. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock ͳΜͰϓϩηεʁ ͳΜͰGoʁ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ Ͳ͏΍ͬͯ େྔͷϓϩηεΛૢ࡞ʁ
50. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ Քಇঢ়گɾϩάૹ৴ PrivateSubnet Mock Mock Mock Mock Mock Mock ͳΜͰϓϩηεʁ ͳΜͰGoʁ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ Ͳ͏΍ͬͯ େྔͷϓϩηεΛૢ࡞ʁ mockͷࢮ׆؂ࢹ͸ Ͳ͏ͯ͠Δͷʁ ઃఆϑΝΠϧ όΠφϦΛDL
51. 
     PublicSubnet PrivateSubnet DynamoDB Lambda SNS SQS S3 Bucket (Lot) S3 Bucket (Binary) Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ Քಇঢ়گɾϩάૹ৴ PrivateSubnet Mock Mock Mock Mock Mock Mock ͳΜͰϓϩηεʁ ͳΜͰGoʁ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ Ͳ͏΍ͬͯ େྔͷϓϩηεΛૢ࡞ʁ mockͷࢮ׆؂ࢹ͸ Ͳ͏ͯ͠Δͷʁ όΠφϦͷ ߋ৽͸ Ͳ͏͢Δͷʁ
52. ͳΜͰϓϩηεʁ
    ͳΜͰ(P
53. ‣ NPDL
    JT
    
     αʔόʔ
    
    ߽՚͗͢Δʜ
     ίϯςφ
    
    एׯ໘౗ͩ͠ɺίϯςφԽ͢Δඞཁ΋ͳ͍
     ϓϩηε
    
    ͪΐ͏Ͳ͍͍ʂ
     εϨου
    
    σόοάେมͦ͏ɺڞ౗Εͦ͠͏ ͳΜͰϓϩηεʁ
    
54. ‣ αʔόʔʹԿ΋Πϯετʔϧͨ͘͠ͳ͍
     ͨͱ͑͹3VCZͩͱ3VCZΛΠϯετʔϧ͢Δඞཁ͕ʜ
    ‣ γϯάϧόΠφϦʹ͢Ε͹ஔ͚ͩ͘Ͱىಈ͕Ͱ͖Δ
    ‣ γϯάϧόΠφϦ͕Ͱ͖Δ΍ͭͰָͦ͠͏ͳͷ
    ‣ ͱ͍͑͹(Pʂ ओ؍ ͳΜͰ(Pʁ
    
55. ‣ ίʔυͷن໛ ςετআ͘
     NPDL
    
    ໿ߦ
     UPXFS
    
    ໿ߦ ͪͳΈʹ
    
56. େྔNPDLͷૢ࡞
57. ‣ ϑΝΠϧϕʔεͰͷૢ࡞
     ಛఆͷσΟϨΫτϦʹϑΝΠϧΛஔ͔ΕͨͷΛݕ஌ͯ͠ॲཧΛߦ͏
     ૢ࡞͕ඞཁͳ෼͚ͩϑΝΠϧΛஔ͚͹͍͍ NPDLͷૢ࡞
     NPDLىಈ NPDLఀࢭ ىಈτϦΨʔ
 ϑΝΠϧઃஔ tower͕ݕ஌͠ mockΛىಈ ఀࢭτϦΨʔ
 ϑΝΠϧઃஔ ֤mock༻ ఀࢭτϦΨʔ
 ϑΝΠϧઃஔ mock͕ݕ஌͠ ࣗݾফ໓
58. ‣ ݪ࢝త͚ͩͲγϯϓϧͰ৴པੑ΋ߴ͘΍Γ΍͍͢
     Կ΋Πϯετʔϧ͠ͳ͍͍ͯ͘
     ॲཧ͕׬ྃͨ͠ޙʹϑΝΠϧΛফ͢Α͏ʹ͓͚ͯ͠͹ɺ
 ΋͠UPXFS్͕தͰ࠶ىಈͯ͠΋ϑΝΠϧΛݟΕ͹ϨδϡʔϜͰ͖Δ
    ‣ ͦͷଞͷ͔ͬ͜Α͛͞ͳ΍Γํ͸ΊΜͲ͍͘͞ؾ͕͢Δ
     )551αʔόʔ
    
     1VC4VC
    
    2VFVF NPDLͷૢ࡞
     
    ૹ৴ࣦഊͨ͠ͱ͖ͷ࠶ૹʜ
    
    ૢ࡞׬ྃલʹUPXFS͕ࢮΜͩ৔߹ʜ
59. ‣ https://github.com/fsnotify/fsnotify ‣ ϑΝΠϧมߋݕ஌ϥΠϒϥϦ GTOPUJGZ
    
60. GTOPUJGZ
     watcher, err := fsnotify.NewWatcher() defer watcher.Close() err = watcher.Add("path/to/trigger/directory") // ϝΠϯϧʔϓ for { select { case ev := <-watcher.Events: // ϑΝΠϧ࡞੒Ҏ֎͸ແࢹ if (ev.Op & fsnotify.Create) == 0 { continue } // τϦΨʔϑΝΠϧͷॲཧ runTriggerProcessor(ev.Name) // Ұ੪ʹmock͕ىಈ͢Δͱड͚ଆ͕େมͳͷͰద੾ʹsleep // (ུ) case err := <-watcher.Errors: // Τϥʔॲཧ case //ͦͷଞॾʑͷॲཧ } }
61. watcher, err := fsnotify.NewWatcher() defer watcher.Close() err = watcher.Add("path/to/trigger/directory") // ϝΠϯϧʔϓ for { select { case ev := <-watcher.Events: // ϑΝΠϧ࡞੒Ҏ֎͸ແࢹ if (ev.Op & fsnotify.Create) == 0 { continue } // τϦΨʔϑΝΠϧͷॲཧ runTriggerProcessor(ev.Name) // Ұ੪ʹmock͕ىಈ͢Δͱड͚ଆ͕େมͳͷͰద੾ʹsleep // (ུ) case err := <-watcher.Errors: // Τϥʔॲཧ case //ͦͷଞॾʑͷॲཧ } } GTOPUJGZ
     ؂ࢹର৅ͷσΟϨΫτϦΛઃఆ func (w *Watcher) Add(name string) error
62. watcher, err := fsnotify.NewWatcher() defer watcher.Close() err = watcher.Add("path/to/trigger/directory") // ϝΠϯϧʔϓ for { select { case ev := <-watcher.Events: // ϑΝΠϧ࡞੒Ҏ֎͸ແࢹ if (ev.Op & fsnotify.Create) == 0 { continue } // τϦΨʔϑΝΠϧͷॲཧ runTriggerProcessor(ev.Name) // Ұ੪ʹmock͕ىಈ͢Δͱड͚ଆ͕େมͳͷͰద੾ʹsleep // (ུ) case err := <-watcher.Errors: // Τϥʔॲཧ case //ͦͷଞॾʑͷॲཧ } } GTOPUJGZ
     ؂ࢹର৅ͷσΟϨΫτϦΛઃఆ func (w *Watcher) Add(name string) error type Event struct { Name string // Πϕϯτ͕͋ͬͨ file/directory Op Op // ૢ࡞छผΛද͢Ϗοτྻ }
63. GTOPUJGZ
     watcher, err := fsnotify.NewWatcher() defer watcher.Close() err = watcher.Add("path/to/trigger/directory") // ϝΠϯϧʔϓ for { select { case ev := <-watcher.Events: // ϑΝΠϧ࡞੒Ҏ֎͸ແࢹ if (ev.Op & fsnotify.Create) == 0 { continue } // τϦΨʔϑΝΠϧͷॲཧ runTriggerProcessor(ev.Name) // Ұ੪ʹmock͕ىಈ͢Δͱड͚ଆ͕େมͳͷͰద੾ʹsleep // (ུ) case err := <-watcher.Errors: // Τϥʔॲཧ case //ͦͷଞॾʑͷॲཧ } } ؂ࢹର৅ͷσΟϨΫτϦΛઃఆ func (w *Watcher) Add(name string) error ev.OpʹϏοτϚεΫΛ͔͚Δ͜ͱͰૢ࡞छผΛ֬ೝ type Op uint32 const ( Create Op = 1 << iota Write Remove Rename Chmod ) type Event struct { Name string // Πϕϯτ͕͋ͬͨ file/directory Op Op // ૢ࡞छผΛද͢Ϗοτྻ }
64. watcher, err := fsnotify.NewWatcher() defer watcher.Close() err = watcher.Add("path/to/trigger/directory") // ϝΠϯϧʔϓ for { select { case ev := <-watcher.Events: // ϑΝΠϧ࡞੒Ҏ֎͸ແࢹ if (ev.Op & fsnotify.Create) == 0 { continue } // τϦΨʔϑΝΠϧͷॲཧ runTriggerProcessor(ev.Name) // Ұ੪ʹmock͕ىಈ͢Δͱड͚ଆ͕େมͳͷͰద੾ʹsleep // (ུ) case err := <-watcher.Errors: // Τϥʔॲཧ case //ͦͷଞॾʑͷॲཧ } } GTOPUJGZ
     ؂ࢹର৅ͷσΟϨΫτϦΛઃఆ func (w *Watcher) Add(name string) error ev.OpʹϏοτϚεΫΛ͔͚Δ͜ͱͰૢ࡞छผΛ֬ೝ type Op uint32 const ( Create Op = 1 << iota Write Remove Rename Chmod ) type Event struct { Name string // Πϕϯτ͕͋ͬͨ file/directory Op Op // ૢ࡞छผΛද͢Ϗοτྻ } sleepೖΕͯ΋ͪΌΜͱશ෦௨஌ͯ͘͠ΕΔͷͰ QueueͬΆ͍ײ͡ʹͳΔ
65. NPDLͷࢮ׆؂ࢹ
66. ‣ NPDLఀࢭॲཧͰਖ਼ৗʹఀࢭͰ͖͔ͨ
     NPDL༻ͷఀࢭτϦΨʔϑΝΠϧΛઃஔ
     ࣗવఀࢭ
    w ࠷େՔಇ࣌ؒܦա
    w ఀࢭεςʔλε΁ͷભҠ
    ‣ NPDL͕ҟৗऴ͍ྃͯ͠ͳ͍͔
     NPDL͕ಥવͷࢮΛܴ͑ΔՄೳੑ΋θϩͰ͸ͳ͍
     ద੾ʹݕ஌ɾ௨஌͠ͳ͚Ε͹ NPDLͷࢮ׆؂ࢹ
    
67. ‣ NPDLఀࢭॲཧͰਖ਼ৗʹఀࢭͰ͖͔ͨ
     NPDL༻ͷఀࢭτϦΨʔϑΝΠϧΛઃஔ
     ࣗવఀࢭ
    w ࠷େՔಇ࣌ؒܦա
    w ఀࢭεςʔλε΁ͷભҠ
    ‣ NPDL͕ҟৗऴ͍ྃͯ͠ͳ͍͔
     NPDL͕ಥવͷࢮΛܴ͑ΔՄೳੑ΋θϩͰ͸ͳ͍
     ద੾ʹݕ஌ɾ௨஌͠ͳ͚Ε͹ NPDLͷࢮ׆؂ࢹ
     ϓϩηεͷ
    ঢ়ଶมԽΛ؂ࢹ
68. NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     processExitChannel chan int func (t *Tower) mockExec(mock MockCode, version string, ...) { // ىಈઃఆϑΝΠϧͷऔಘͱ͔(ུ) // ϓϩηεىಈ var proc *os.Process proc, err = os.StartProcess(binPath, args, attr) // PidΛอଘ mockProcessTable[proc.Pid] = mock // goroutineͰऴྃ؂ࢹ go func(t *Tower, mock MockCode, pid int) { for { var status syscall.WaitStatus var rusage syscall.Rusage wpid, err := syscall.Wait4(pid, &status, syscall.WNOHANG, &rusage) if err != nil || (wpid != 0 && status.Exited()) { processExitChannel <- pid return } time.Sleep(intervalMsec) } }(t, mock, proc.Pid)
69. processExitChannel chan int func (t *Tower) mockExec(mock MockCode, version string, ...) { // ىಈઃఆϑΝΠϧͷऔಘͱ͔(ུ) // ϓϩηεىಈ var proc *os.Process proc, err = os.StartProcess(binPath, args, attr) // PidΛอଘ mockProcessTable[proc.Pid] = mock // goroutineͰऴྃ؂ࢹ go func(t *Tower, mock MockCode, pid int) { for { var status syscall.WaitStatus var rusage syscall.Rusage wpid, err := syscall.Wait4(pid, &status, syscall.WNOHANG, &rusage) if err != nil || (wpid != 0 && status.Exited()) { processExitChannel <- pid return } time.Sleep(intervalMsec) } }(t, mock, proc.Pid) NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     ϓϩηεىಈޙɺgoroutineΛ૸Βͤͯ؂ࢹ͢Δ (ىಈ͍ͯ͠Δmockͷ਺͚ͩgoroutineΛ૸Β͍ͤͯΔ)
70. NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     processExitChannel chan int func (t *Tower) mockExec(mock MockCode, version string, ...) { // ىಈઃఆϑΝΠϧͷऔಘͱ͔(ུ) // ϓϩηεىಈ var proc *os.Process proc, err = os.StartProcess(binPath, args, attr) // PidΛอଘ mockProcessTable[proc.Pid] = mock // goroutineͰऴྃ؂ࢹ go func(t *Tower, mock MockCode, pid int) { for { var status syscall.WaitStatus var rusage syscall.Rusage wpid, err := syscall.Wait4(pid, &status, syscall.WNOHANG, &rusage) if err != nil || (wpid != 0 && status.Exited()) { processExitChannel <- pid return } time.Sleep(intervalMsec) } }(t, mock, proc.Pid) ϓϩηεىಈޙɺgoroutineΛ૸Βͤͯ؂ࢹ͢Δ (ىಈ͍ͯ͠Δmockͷ਺͚ͩgoroutineΛ૸Β͍ͤͯΔ) WNOHANGΦϓγϣϯ: ঢ়ଶมԽ͕ىͬͨ͜ࢠϓϩηε͕ͳ͍৔߹ʹ͙͢ʹ෮ؼ
71. processExitChannel chan int func (t *Tower) mockExec(mock MockCode, version string, ...) { // ىಈઃఆϑΝΠϧͷऔಘͱ͔(ུ) // ϓϩηεىಈ var proc *os.Process proc, err = os.StartProcess(binPath, args, attr) // PidΛอଘ mockProcessTable[proc.Pid] = mock // goroutineͰऴྃ؂ࢹ go func(t *Tower, mock MockCode, pid int) { for { var status syscall.WaitStatus var rusage syscall.Rusage wpid, err := syscall.Wait4(pid, &status, syscall.WNOHANG, &rusage) if err != nil || (wpid != 0 && status.Exited()) { processExitChannel <- pid return } time.Sleep(intervalMsec) } }(t, mock, proc.Pid) NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     ϓϩηεىಈޙɺgoroutineΛ૸Βͤͯ؂ࢹ͢Δ (ىಈ͍ͯ͠Δmockͷ਺͚ͩgoroutineΛ૸Β͍ͤͯΔ) WNOHANGΦϓγϣϯ: ঢ়ଶมԽ͕ىͬͨ͜ࢠϓϩηε͕ͳ͍৔߹ʹ͙͢ʹ෮ؼ status.ExitStatus()Ͱҟৗऴྃ֬ೝ΋ͨ͠Γ͢Δ
72. processExitChannel chan int func (t *Tower) mockExec(mock MockCode, version string, ...) { // ىಈઃఆϑΝΠϧͷऔಘͱ͔(ུ) // ϓϩηεىಈ var proc *os.Process proc, err = os.StartProcess(binPath, args, attr) // PIDΛอଘ mockProcessTable[proc.Pid] = mock // goroutineͰऴྃ؂ࢹ go func(t *Tower, mock MockCode, pid int) { for { var status syscall.WaitStatus var rusage syscall.Rusage wpid, err := syscall.Wait4(pid, &status, syscall.WNOHANG, &rusage) if err != nil || (wpid != 0 && status.Exited()) { processExitChannel <- pid return } time.Sleep(intervalMsec) } }(t, mock, proc.Pid) NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     ϓϩηεىಈޙɺgoroutineΛ૸Βͤͯ؂ࢹ͢Δ (ىಈ͍ͯ͠Δmockͷ਺͚ͩgoroutineΛ૸Β͍ͤͯΔ) ऴྃͨ͠ΒprocessExitChannelʹ௨஌ WNOHANGΦϓγϣϯ: ঢ়ଶมԽ͕ىͬͨ͜ࢠϓϩηε͕ͳ͍৔߹ʹ͙͢ʹ෮ؼ status.ExitStatus()Ͱҟৗऴྃ֬ೝ΋ͨ͠Γ͢Δ
73. NPDLϓϩηεͷঢ়ଶมԽΛ؂ࢹ
     // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ case pid := <-processExitChannel: mock := mockProcessTable[pid] // ऴྃޙͷ͓ย෇͚΋Ζ΋Ζ delete(mockProcessTable, pid) } }
74. ‣ NPDLఀࢭॲཧͰਖ਼ৗʹఀࢭͰ͖͔ͨ
     NPDL༻ͷఀࢭτϦΨʔϑΝΠϧΛઃஔ
     ࣗવఀࢭ
    w ࠷େՔಇ࣌ؒܦա
    w ఀࢭεςʔλε΁ͷભҠ
    ‣ NPDL͕ҟৗऴ͍ྃͯ͠ͳ͍͔
     NPDL͕ಥવͷࢮΛܴ͑ΔՄೳੑ΋θϩͰ͸ͳ͍
     ద੾ʹݕ஌ɾ௨஌͠ͳ͚Ε͹ NPDLͷࢮ׆؂ࢹ
     ϓϩηεͷ
    ঢ়ଶมԽΛ؂ࢹ ఀࢭτϦΨʔϑΝΠϧͱ
    ಈ࡞தͷϓϩηεΛൺֱ
75. ఀࢭτϦΨʔϑΝΠϧͷফԽ֬ೝ
     ticker := time.NewTicker(tickerIntervalMsec) defer ticker.Stop() // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ case tickTime := <-ticker.C: // τϦΨʔϑΝΠϧ͕͋Δͷʹ࢒͍ͬͯΔϓϩηεΛ֬ೝ t.processLeftMock(tickTime) } } func (t *Tower) processLeftMock(cur time.Time) { for mock, term := range t.terminateTable { if cur.Before(term.triggeredAt.Add(ForceTerminateTime)) { continue } pid, processExist := t.mockTable[mock] proc, err := os.FindProcess(pid) err = proc.Signal(os.Kill) delete(t.terminateTable, mock) } }
76. ఀࢭτϦΨʔϑΝΠϧͷফԽ֬ೝ
     ticker := time.NewTicker(tickerIntervalMsec) defer ticker.Stop() // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ case tickTime := <-ticker.C: // τϦΨʔϑΝΠϧ͕͋Δͷʹ࢒͍ͬͯΔϓϩηεΛ֬ೝ t.processLeftMock(tickTime) } } func (t *Tower) processLeftMock(cur time.Time) { for mock, term := range t.terminateTable { if cur.Before(term.triggeredAt.Add(ForceTerminateTime)) { continue } pid, processExist := t.mockTable[mock] proc, err := os.FindProcess(pid) err = proc.Signal(os.Kill) delete(t.terminateTable, mock) } } tickerΛ࢖ͬͯఆظతʹະফԽτϦΨʔϑΝΠϧΛ֬ೝ
77. ఀࢭτϦΨʔϑΝΠϧͷফԽ֬ೝ
     ticker := time.NewTicker(tickerIntervalMsec) defer ticker.Stop() // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ case tickTime := <-ticker.C: // τϦΨʔϑΝΠϧ͕͋Δͷʹ࢒͍ͬͯΔϓϩηεΛ֬ೝ t.processLeftMock(tickTime) } } func (t *Tower) processLeftMock(cur time.Time) { for mock, term := range t.terminateTable { if cur.Before(term.triggeredAt.Add(ForceTerminateTime)) { continue } pid, processExist := t.mockTable[mock] proc, err := os.FindProcess(pid) err = proc.Signal(os.Kill) delete(t.terminateTable, mock) } } tickerΛ࢖ͬͯఆظతʹະফԽτϦΨʔϑΝΠϧΛ֬ೝ ઃஔޙҰఆ͕࣌ؒܦաͨ͠΋ͷͷΈ͕֬ೝର৅
78. ఀࢭτϦΨʔϑΝΠϧͷফԽ֬ೝ
     ticker := time.NewTicker(tickerIntervalMsec) defer ticker.Stop() // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ case tickTime := <-ticker.C: // τϦΨʔϑΝΠϧ͕͋Δͷʹ࢒͍ͬͯΔϓϩηεΛ֬ೝ t.processLeftMock(tickTime) } } func (t *Tower) processLeftMock(cur time.Time) { for mock, term := range t.terminateTable { if cur.Before(term.triggeredAt.Add(ForceTerminateTime)) { continue } pid, processExist := t.mockTable[mock] proc, err := os.FindProcess(pid) err = proc.Signal(os.Kill) delete(t.terminateTable, mock) } } tickerΛ࢖ͬͯఆظతʹະফԽτϦΨʔϑΝΠϧΛ֬ೝ ઃஔޙҰఆ͕࣌ؒܦաͨ͠΋ͷͷΈ͕֬ೝର৅ ϓϩηεڧ੍ऴྃ
79. όΠφϦͷߋ৽
80. ‣ NPDL
     Քಇத͸ߋ৽ෆཁ ಉ͡ઃఆͰಈ͖ଓ͚ΔͷͰ
     ىಈ࣌ʹ࠷৽൛ͷόΠφϦΛ࣋ͬͯ͘Ε͹0,
    ‣ UPXFS
     NPDL͕ՔಇதͰ΋ߋ৽͍ͨ͠
     ࠶ىಈ چUPXFSΛఀࢭ
    
    ৽UPXFSΛىಈ Ͱ͖ΔΑ͏ʹ͓ͯ͘͠
     NPDL؂ࢹঢ়گͷϨδϡʔϜΛ͕Μ͹Δʂ όΠφϦͷߋ৽
    
81. ‣ ࠶ىಈखॱ
     <چ>UPXFSͷ࠶ىಈτϦΨʔϑΝΠϧͷݕ஌
     <چ>τϦΨʔϑΝΠϧʹॻ͔Ε͍ͯΔόʔδϣϯΛ4͔Β%-ͯ͠ىಈ
     <چ>ऴྃ
     <৽>NPDLͷ؂ࢹঢ়گΛϨδϡʔϜ
    ‣ ϨδϡʔϜ͠ͳ͍ͱ͍͚ͳ͍΋ͷ
     NPDL$PEFͱ1*%Λؔ࿈෇͚ΔNBQ
     NPDLϓϩηεΛ؂ࢹ͢ΔHPSPVUJOF UPXFSͷ࠶ىಈ
    
82. ࠶ىಈτϦΨʔϑΝΠϧݕ஌dऴྃ
     // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ఆظॲཧͱ͔ case ev := <-t.towerReboot.Events: if (ev.Op & fsnotify.Create) == 0 || (base != towerRebootName) { continue } rebootStrs, err := ioutil.ReadFile(ev.Name) reboot := strings.Split(strings.TrimSpace(string(rebootStrs)), " ") // ࠶ىಈ࣌ʹόΠφϦ͕ଘࡏ͠ͳ͍৔߹͸μ΢ϯϩʔυͯ͘͠Δ(ུ) // ৽towerىಈ _, err = os.StartProcess(binPath, args, attr) // ϝΠϯϧʔϓ͔Βൈ͚Δ return } }
83. ࠶ىಈτϦΨʔϑΝΠϧݕ஌dऴྃ
     // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ఆظॲཧͱ͔ case ev := <-t.towerReboot.Events: if (ev.Op & fsnotify.Create) == 0 || (base != towerRebootName) { continue } rebootStrs, err := ioutil.ReadFile(ev.Name) reboot := strings.Split(strings.TrimSpace(string(rebootStrs)), " ") // ࠶ىಈ࣌ʹόΠφϦ͕ଘࡏ͠ͳ͍৔߹͸μ΢ϯϩʔυͯ͘͠Δ(ུ) // ৽towerىಈ _, err = os.StartProcess(binPath, args, attr) // ϝΠϯϧʔϓ͔Βൈ͚Δ return } } ࠶ىಈτϦΨʔϑΝΠϧͷ࡞੒ΠϕϯτҎ֎͸ແࢹ
84. ࠶ىಈτϦΨʔϑΝΠϧݕ஌dऴྃ
     // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ఆظॲཧͱ͔ case ev := <-t.towerReboot.Events: if (ev.Op & fsnotify.Create) == 0 || (base != towerRebootName) { continue } rebootStrs, err := ioutil.ReadFile(ev.Name) reboot := strings.Split(strings.TrimSpace(string(rebootStrs)), " ") // ࠶ىಈ࣌ʹόΠφϦ͕ଘࡏ͠ͳ͍৔߹͸μ΢ϯϩʔυͯ͘͠Δ(ུ) // ৽towerىಈ _, err = os.StartProcess(binPath, args, attr) // ϝΠϯϧʔϓ͔Βൈ͚Δ return } } ࠶ىಈτϦΨʔϑΝΠϧͷ࡞੒ΠϕϯτҎ֎͸ແࢹ τϦΨʔϑΝΠϧͷத਎ΛݺΜͰόʔδϣϯΛऔಘ
85. ࠶ىಈτϦΨʔϑΝΠϧݕ஌dऴྃ
     // ϝΠϯϧʔϓ for { select { case // τϦΨʔϑΝΠϧͷݕ஌ॲཧͱ͔ϓϩηεऴྃ࣌ͷॲཧͱ͔ఆظॲཧͱ͔ case ev := <-t.towerReboot.Events: if (ev.Op & fsnotify.Create) == 0 || (base != towerRebootName) { continue } rebootStrs, err := ioutil.ReadFile(ev.Name) reboot := strings.Split(strings.TrimSpace(string(rebootStrs)), " ") // ࠶ىಈ࣌ʹόΠφϦ͕ଘࡏ͠ͳ͍৔߹͸μ΢ϯϩʔυͯ͘͠Δ(ུ) // ৽towerىಈ _, err = os.StartProcess(binPath, args, attr) // ϝΠϯϧʔϓ͔Βൈ͚Δ return } } ࠶ىಈτϦΨʔϑΝΠϧͷ࡞੒ΠϕϯτҎ֎͸ແࢹ τϦΨʔϑΝΠϧͷத਎ΛݺΜͰόʔδϣϯΛऔಘ ϝΠϯϧʔϓ͔Βൈ͚Δͱͦͷ··ऴྃ
86. NPDL؂ࢹঢ়گͷϨδϡʔϜ
     func NewTower() *Tower { // ͍Ζ͍ΖॳظԽ(ུ) // PIDϑΝΠϧΛऔಘ pidFiles, err := filepath.Glob(filepath.Join(t.PidsDir(), "mk-*")) for _, pidFile := range pidFiles { mock := MockCode(filepath.Base(pidFile)) // PIDϑΝΠϧ͔ΒPIDΛಡΈऔΓ pidStr, err := ioutil.ReadFile(pidFile) pid, err := strconv.Atoi(strings.TrimSpace(string(pidStr))) // PID͔Β࣮ࡍͷϓϩηεΛݟ͚ͭͯอଘ proc, err := os.FindProcess(pid) t.mockProcessTable[pid] = mock // ϓϩηεऴྃ௨஌goroutineΛ௥Ճ go func(t *Tower, mock MockCode, pid int) { // (ུ) }(&t, mock, pid) } }
87. NPDL؂ࢹঢ়گͷϨδϡʔϜ
     func NewTower() *Tower { // ͍Ζ͍ΖॳظԽ(ུ) // PIDϑΝΠϧΛऔಘ pidFiles, err := filepath.Glob(filepath.Join(t.PidsDir(), "mk-*")) for _, pidFile := range pidFiles { mock := MockCode(filepath.Base(pidFile)) // PIDϑΝΠϧ͔ΒPIDΛಡΈऔΓ pidStr, err := ioutil.ReadFile(pidFile) pid, err := strconv.Atoi(strings.TrimSpace(string(pidStr))) // PID͔Β࣮ࡍͷϓϩηεΛݟ͚ͭͯอଘ proc, err := os.FindProcess(pid) t.mockProcessTable[pid] = mock // ϓϩηεऴྃ௨஌goroutineΛ௥Ճ go func(t *Tower, mock MockCode, pid int) { // (ུ) }(&t, mock, pid) } } ֤mock͸ىಈ࣌ʹPIDϑΝΠϧΛ࡞Δ
88. NPDL؂ࢹঢ়گͷϨδϡʔϜ
     func NewTower() *Tower { // ͍Ζ͍ΖॳظԽ(ུ) // PIDϑΝΠϧΛऔಘ pidFiles, err := filepath.Glob(filepath.Join(t.PidsDir(), "mk-*")) for _, pidFile := range pidFiles { mock := MockCode(filepath.Base(pidFile)) // PIDϑΝΠϧ͔ΒPIDΛಡΈऔΓ pidStr, err := ioutil.ReadFile(pidFile) pid, err := strconv.Atoi(strings.TrimSpace(string(pidStr))) // PID͔Β࣮ࡍͷϓϩηεΛݟ͚ͭͯอଘ proc, err := os.FindProcess(pid) t.mockProcessTable[pid] = mock // ϓϩηεऴྃ௨஌goroutineΛ௥Ճ go func(t *Tower, mock MockCode, pid int) { // (ུ) }(&t, mock, pid) } } ֤mock͸ىಈ࣌ʹPIDϑΝΠϧΛ࡞Δ লུͯ͠Δ͚Ͳɺ ϓϩηε͕ݟ͔ͭΒͳ͔ͬͨΒPIDϑΝΠϧΛ࡟আ
89. ·ͱΊ
90. ‣
    
    
    
    
    
    
    
    
    
    
    
    
    ɹɹʹ͓͍ͯɺ(PΛ࢖ͬͯ
 େྔͷϓϩηεΛͲͷΑ͏ʹ؅ཧ͍ͯ͠Δ͔Λ঺հ
     ͦ΋ͦ΋ͳΜͰϓϩηεʁͳΜͰ(P
     ϑΝΠϧϕʔεͷNPDLૢ࡞
     NPDLͷࢮ׆؂ࢹ
     όΠφϦͷߋ৽
    ‣ ʮ໌೔͔Β࢖͑ΔʯΑ͏ͳ಺༰Ͱ͸ͳ͍͸͚ͣͩͲɺ
 ָ͠ΜͰ΋Β͑ͨͳΒ޾͍Ͱ͢
    ‣ (Pͨͷ͍͠ʂ ·ͱΊ
    
91. 
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    ΛҰॹʹ
    ։ൃͯ͘͠ΕΔ஥ؒΛืूதʂ ͝ڵຯ͋Δํ͸ໟར·Ͱʂ Go Ruby AWS IoT ΋ͬͱ͏·͘΍ΕΔʂ ͓΋͠Ζͦ͏ʂ @Fukuoka