Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Kubernetes CronJob Implementation in Detail #k8sjp
Shimpei Otsubo
September 28, 2018
Programming
5
2.5k
Kubernetes CronJob Implementation in Detail #k8sjp
Shimpei Otsubo
September 28, 2018
Tweet
Share
More Decks by Shimpei Otsubo
See All by Shimpei Otsubo
Copy Kubernetes Clusters Really Fast
potsbo
3
1.8k
Go と Wantedly の関係 / How Wantedly uses Go
potsbo
1
540
Deploy Flow at Wantedly
potsbo
2
510
Wrap every method with just one line
potsbo
1
2.2k
Zero yen Keyboard
potsbo
6
2.4k
Kube - The core tool at Wantedly
potsbo
1
4.4k
k8s - Kubernetes 8 Factors
potsbo
12
8.6k
コンテンツ作成に集中するためのプレゼンテーション Tips / Presentation with Confidence
potsbo
6
21k
ConfigMap vs Secret #k8sjp
potsbo
1
950
Other Decks in Programming
See All in Programming
クラウド KMS の活用 / TOKYO BLOCKCHAIN TECH MEETUP 2022
odanado
PRO
0
180
段階的な技術的負債の解消方法.pdf
ko2ic
2
880
パスワードに関する最近の動向
kenchan0130
1
320
実践的!FPGA開発セミナー vol.11
fixstars
0
110
読みやすいコード クラスメソッド 2022 年度新卒研修
januswel
0
2.9k
ZOZOTOWNにおけるDatadogの活用と、それを支える全社管理者の取り組み / 2022-07-27
tippy
1
3k
Pythonで鉄道指向プログラミング
usabarashi
0
130
Rust on Lambda 大きめCSV生成
atsuyokota
1
390
2022年のモダンCSS改
tonkotsuboy_com
24
16k
atama plusの開発チームはどのように「不確実性」に向き合ってきたか〜2022夏版〜
atamaplus
3
610
動画合成アーキテクチャを実装してみて
satorunooshie
0
540
Rust、何もわからない...#3
estie
0
150
Featured
See All Featured
4 Signs Your Business is Dying
shpigford
169
20k
The World Runs on Bad Software
bkeepers
PRO
57
5.4k
Navigating Team Friction
lara
175
11k
GraphQLの誤解/rethinking-graphql
sonatard
31
6.8k
Scaling GitHub
holman
451
140k
For a Future-Friendly Web
brad_frost
166
7.5k
Keith and Marios Guide to Fast Websites
keithpitt
404
21k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
980
Done Done
chrislema
174
14k
Infographics Made Easy
chrislema
233
17k
Adopting Sorbet at Scale
ufuk
63
7.6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
6
560
Transcript
©2018 Wantedly, Inc. CronJob Implementation in Detail ίʔυ͔ΒಡΈղ͘ CronJob ͷৄࡉ༷
Kubernetes Meetup Tokyo #13 28.Sep.2018 - Shimpei Otsubo - @potsbo
CronJob ʹ·͞Ε͖ͯͨʜ ࣮ߦ࿙Ε ࣮ߦΕ ҙਤ͠ͳ͍࣮ߦ ਂ༻ͷॏ͍+PC͕னؒʹಈ͍ͯ*ODJEFOU &SSPSʹ$PNQMFUFʹ͍ͳ͍ʜ ఔฏؾͰΕΔ BQQMZͨ͠ॠؒʹಈ͍ͯ͠·͏ EFMFUFDSFBUFͩͱൃੜ͠ͳ͍
©2018 Wantedly, Inc.
ຊ൪Ϋϥελͷ$SPO+PCͷԆΛܭଌͯ͠Έͨ ඵ༨༟ඵฏؾͰΕΔ n = 475 $SPO+PCԆώετάϥϜ 0 30 60 Ԇ
ඵ <5 <10 <15 <20 <25 <30 <35 <40 <45 <50 <55 <60 <65 <70 <75 <80 ݸ ˞4UBSUJOH%FBEMJOF4FDPOET͕ͷͷ ©2018 Wantedly, Inc.
// Run the main goroutine responsible for watching and syncing
jobs. func (jm *CronJobController) Run(stopCh <-chan struct{}) { defer utilruntime.HandleCrash() glog.Infof("Starting CronJob Manager") // Check things every 10 second. go wait.Until(jm.syncAll, 10*time.Second, stopCh) <-stopCh glog.Infof("Shutting down CronJob Manager") } $SPO+PC$POUSPMMFS͕ಈ͖࢝ΊΔGVOD ©2018 Wantedly, Inc.
// Run the main goroutine responsible for watching and syncing
jobs. func (jm *CronJobController) Run(stopCh <-chan struct{}) { defer utilruntime.HandleCrash() glog.Infof("Starting CronJob Manager") // Check things every 10 second. go wait.Until(jm.syncAll, 10*time.Second, stopCh) <-stopCh glog.Infof("Shutting down CronJob Manager") } FWFSZTFDPOEͱॻ͍ͯ͋Δ͕࣮શ෦ॲཧඵεϦʔϓ $SPO+PC͕ଟ͍ͱ୯ҐͰΕΔ͜ͱ͋Γͦ͏ ˞ҰճͷTZOD"MMʹຊʹͦΜͳʹ͕͔͔͍࣌ؒͬͯΔͷ͔ະܭଌ ©2018 Wantedly, Inc.
func syncOne(sj *batchv1beta1.CronJob, js []batchv1.Job, now time.Time, jc jobControlInterface, sjc
sjControlInterface, pc podControlInterface, recorder record.EventRecorder) { // 50+ lines omitted times, err := getRecentUnmetScheduleTimes(*sj, now) if err != nil { return } // 10+ lines omitted scheduledTime := times[len(times)-1] tooLate := false if sj.Spec.StartingDeadlineSeconds != nil { tooLate = scheduledTime.Add(time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)).Before(now) } if tooLate { return } // 30 lines omitted jobReq, err := getJobFromTemplate(sj, scheduledTime) if err != nil { return } jobResp, err := jc.CreateJob(sj.Namespace, jobReq) if err != nil { return } // 20+ lines omitted return } ˞దٓ؆ུԽ
func syncOne(sj *batchv1beta1.CronJob, js []batchv1.Job, now time.Time, jc jobControlInterface, sjc
sjControlInterface, pc podControlInterface, recorder record.EventRecorder) { // 50+ lines omitted times, err := getRecentUnmetScheduleTimes(*sj, now) if err != nil { return } // 10+ lines omitted scheduledTime := times[len(times)-1] tooLate := false if sj.Spec.StartingDeadlineSeconds != nil { tooLate = scheduledTime.Add(time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)).Before(now) } if tooLate { return } // 30 lines omitted jobReq, err := getJobFromTemplate(sj, scheduledTime) if err != nil { return } jobResp, err := jc.CreateJob(sj.Namespace, jobReq) if err != nil { return } // 20+ lines omitted return } ະ࣮ߦ࣌ࠁҰཡΛऔಘ ˞దٓ؆ུԽ
func syncOne(sj *batchv1beta1.CronJob, js []batchv1.Job, now time.Time, jc jobControlInterface, sjc
sjControlInterface, pc podControlInterface, recorder record.EventRecorder) { // 50+ lines omitted times, err := getRecentUnmetScheduleTimes(*sj, now) if err != nil { return } // 10+ lines omitted scheduledTime := times[len(times)-1] tooLate := false if sj.Spec.StartingDeadlineSeconds != nil { tooLate = scheduledTime.Add(time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)).Before(now) } if tooLate { return } // 30 lines omitted jobReq, err := getJobFromTemplate(sj, scheduledTime) if err != nil { return } jobResp, err := jc.CreateJob(sj.Namespace, jobReq) if err != nil { return } // 20+ lines omitted return } ൃݟ͕͗ͨ͢ʁఆ ະ࣮ߦ࣌ࠁҰཡΛऔಘ ˞దٓ؆ུԽ
func syncOne(sj *batchv1beta1.CronJob, js []batchv1.Job, now time.Time, jc jobControlInterface, sjc
sjControlInterface, pc podControlInterface, recorder record.EventRecorder) { // 50+ lines omitted times, err := getRecentUnmetScheduleTimes(*sj, now) if err != nil { return } // 10+ lines omitted scheduledTime := times[len(times)-1] tooLate := false if sj.Spec.StartingDeadlineSeconds != nil { tooLate = scheduledTime.Add(time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)).Before(now) } if tooLate { return } // 30 lines omitted jobReq, err := getJobFromTemplate(sj, scheduledTime) if err != nil { return } jobResp, err := jc.CreateJob(sj.Namespace, jobReq) if err != nil { return } // 20+ lines omitted return } +PCൃߦ ൃݟ͕͗ͨ͢ʁఆ ະ࣮ߦ࣌ࠁҰཡΛऔಘ ˞దٓ؆ུԽ
func getRecentUnmetScheduleTimes(sj batchv1beta1.CronJob, now time.Time) ([]time.Time, error) { starts :=
[]time.Time{} sched, err := cron.ParseStandard(sj.Spec.Schedule) if err != nil { return starts, fmt.Errorf("Unparseable schedule: %s : %s", sj.Spec.Schedule, err) } var earliestTime time.Time if sj.Status.LastScheduleTime != nil { earliestTime = sj.Status.LastScheduleTime.Time } else { earliestTime = sj.ObjectMeta.CreationTimestamp.Time } if sj.Spec.StartingDeadlineSeconds != nil { // Controller is not going to schedule anything below this point schedulingDeadline := now.Add(-time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)) if schedulingDeadline.After(earliestTime) { earliestTime = schedulingDeadline } } if earliestTime.After(now) { return []time.Time{}, nil } for t := sched.Next(earliestTime); !t.After(now); t = sched.Next(t) { starts = append(starts, t) } return starts, nil } ˞దٓ؆ུԽ
func getRecentUnmetScheduleTimes(sj batchv1beta1.CronJob, now time.Time) ([]time.Time, error) { starts :=
[]time.Time{} sched, err := cron.ParseStandard(sj.Spec.Schedule) if err != nil { return starts, fmt.Errorf("Unparseable schedule: %s : %s", sj.Spec.Schedule, err) } var earliestTime time.Time if sj.Status.LastScheduleTime != nil { earliestTime = sj.Status.LastScheduleTime.Time } else { earliestTime = sj.ObjectMeta.CreationTimestamp.Time } if sj.Spec.StartingDeadlineSeconds != nil { // Controller is not going to schedule anything below this point schedulingDeadline := now.Add(-time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)) if schedulingDeadline.After(earliestTime) { earliestTime = schedulingDeadline } } if earliestTime.After(now) { return []time.Time{}, nil } for t := sched.Next(earliestTime); !t.After(now); t = sched.Next(t) { starts = append(starts, t) } return starts, nil } Ͳͷఔաڈ͔Β୳͔͢ʁ ˞దٓ؆ུԽ
func getRecentUnmetScheduleTimes(sj batchv1beta1.CronJob, now time.Time) ([]time.Time, error) { starts :=
[]time.Time{} sched, err := cron.ParseStandard(sj.Spec.Schedule) if err != nil { return starts, fmt.Errorf("Unparseable schedule: %s : %s", sj.Spec.Schedule, err) } var earliestTime time.Time if sj.Status.LastScheduleTime != nil { earliestTime = sj.Status.LastScheduleTime.Time } else { earliestTime = sj.ObjectMeta.CreationTimestamp.Time } if sj.Spec.StartingDeadlineSeconds != nil { // Controller is not going to schedule anything below this point schedulingDeadline := now.Add(-time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)) if schedulingDeadline.After(earliestTime) { earliestTime = schedulingDeadline } } if earliestTime.After(now) { return []time.Time{}, nil } for t := sched.Next(earliestTime); !t.After(now); t = sched.Next(t) { starts = append(starts, t) } return starts, nil } Ͳͷఔաڈ͔Β୳͔͢ʁ ͦͷաڈ͔Βࠓ·ͰϚον͢Δ࣌ࠁΛϦετ ˞దٓ؆ུԽ
$SPO+PC͔Β+PCൃߦ·ͰͷྲྀΕ ʮ࠷ޙͷ࣮ߦ࣌ࠁcc࡞࣌ࠁʯ͔Βࠓ·Ͱͷ࣌ࠁީิͷ࠷ޙ͕ StartingDeadlineSeconds ҎͳΒ࣮ߦ ©2018 Wantedly, Inc.
$SPO+PCͷBQQMZ࣌ʹ͍͖ͳΓKPC͕Γग़͢ ໌͔ΒAM 3:00 ʹಈ͍ͯ΄͍͠ 1:00 AM 3:00 AM ୳͢ൣғ 3:00
AM ະ࣮ߦఆ ?:?? ?? ҙਤ͠ͳ͍Job࣮ߦ ࠓͷAM 1:00 ͔Β apply ͷॠؒ·ͰʹࠓͷAM 3:00 ͕ೖͬͯ͠·͏ apply ͷޙʮະ࣮ߦͷ+PCͩʯͱೝࣝ͞Εͯͦͷॠؒಈ͖ग़͢ apply StartingDeadlineSeconds͕খ͚͞Ε͜͜ͰTLJQͯ͠Β͑Δ ࠓ ໌ ࡢ ࠷ޙ 1:00 AM ©2018 Wantedly, Inc.
$SPO+PCͷBQQMZ࣌ʹ͍͖ͳΓKPC͕Γग़͢ ໌͔ΒAM 3:00 ʹಈ͍ͯ΄͍͠ delete/createͩͱ࡞ΒΕͨ࡞ΒΕͨॠ͔ؒΒ୳࢝͠ΊΔͷͰ࣍ͷ".·Ͱ࣮ߦ͞Εͳ͍ ͨͩ͠Ұॠͱ͍͑ CronJob ͕ଘࡏ͠ͳ͍Մೳੑ͕͋ΔͷͰ·ͱΊͯΔͱ࣮ߦ࿙ΕϦεΫ 1:00 AM
3:00 AM delete/create ࠓ ໌ ࡢ 1:00 AM ©2018 Wantedly, Inc.
·ͱΊ ίʔυҙ֎ͱಡΊΔ CronJob ؆୯ʹΕΔ 4UBSUJOH%FBEMJOF4FDPOET৻ॏʹ ©2018 Wantedly, Inc.