Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
データ基盤品質向上のための一年
Search
Toshiki Tsuchikawa
December 13, 2021
0
7.8k
データ基盤品質向上のための一年
DataEngineeringStudy #11
Toshiki Tsuchikawa
December 13, 2021
Tweet
Share
More Decks by Toshiki Tsuchikawa
See All by Toshiki Tsuchikawa
急成長する組織を支えるデータ基盤のこれまで、これから
ttccddtoki
6
670
アジリティの高いデータ基盤を目指して
ttccddtoki
4
1.5k
DMBOKを参考にしたデータマネジメントの取り組み
ttccddtoki
6
2.4k
dbt_Cloudとdbt_Core併用の試み
ttccddtoki
3
1.2k
データ品質を重視したデータ基盤プロダクト開発
ttccddtoki
8
2.3k
タイミーの未来を支えるデータ基盤プロダクト
ttccddtoki
1
720
datatech-jp Casual Talks #3
ttccddtoki
0
1k
Lookerとdbtの共存
ttccddtoki
1
1.4k
[輪読会]実践的データ基盤への処方箋
ttccddtoki
0
260
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
2
170
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
2
290
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
28
900
Optimizing for Happiness
mojombo
376
70k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Mobile First: as difficult as doing things right
swwweet
222
9k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.3k
Scaling GitHub
holman
458
140k
How GitHub (no longer) Works
holman
311
140k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
810
Visualization
eitanlees
146
15k
What's in a price? How to price your products and services
michaelherold
243
12k
Transcript
䚷ᅵᕝ⛱⏕ 䝕䞊䝍ᇶ┙ရ㉁ྥୖ䛾䛯䜑䛾୍ᖺ $XZXKC
┠ḟ Ɣ ⮬ᕫ⤂ Ɣ 㐺ᛶ┘ど䝅䝇䝔䝮䛾ᑟධ Ɣ /RRNHU䛾ᑟධ Ɣ GEW&ORXG䛾ᑟධ Ɣ
䛥䛔䛤䛻
ᅵᕝ⛱⏕ 7VXFKLNDZD7RVKLNL Ɣ ឡ▱┴ฟ㌟ Ɣ ᖺᮾᕤሗ⌮ᕤᏛ㝔༞ Ɣ ᰴᘧ♫䝍䜲䝭䞊 ż
'5( 'DWD5HOLDELOLW\(QJLQHHULQJ 䝏䞊䝮 ż 䝕䞊䝍ᇶ┙䛾㛤Ⓨ䞉ಖᏲ䞉㐠⏝ ż ศᯒᇶ┙䛾㛤Ⓨ䞉ಖᏲ䞉㐠⏝ Ɣ 7ZLWWHU#WYWJB ⮬ᕫ⤂
⣙ᖺ๓䛾䝕䞊䝍ᇶ┙
̐᭦᪂䝍䜲䝭䞁䜾䛾㐜ᘏ %LJ4XHU\䝴䞊䝄䞊䛛䜙᭦᪂䝍䜲䝭䞁䜾䛜㐜䜜䛶䛔䜛 䛸䛾ධಙ䛜ቑຍ 䛭䛾㒔ᗘ'5(䝏䞊䝮䛜༶ᑐᛂ ┘ど䛿䛔䛟䛴䛛ධ䜜䛶䛔䛯䛜䚸䜲䞁䝣䝷䜔䚸䝕䞊䝍䝟 䜲䝥䝷䜲䞁䛺䛹ከᒱ䛻䜟䛯䜛䝞䜾䛾Ⓨ⏕※䛜䛒䜚ᑐᛂ 䛻㛫䛜ྲྀ䜙䜜䜛䜘䛖䛻 䝕䞊䝍ရ㉁䛾ၥ㢟Ⅼ ʍ᭦᪂㢖ᗘ┘ど6/26/$䛾ᑟධ
㐺ᛶ䜢6/,䛸䛧䛶ྛ䝕䞊䝍䝋䞊䝇䛤䛸䛻 6/$䜢ศᯒ ⪅䛸ᐃ⩏ 䜶䝷䞊䝞䝆䜵䝑䝖䜢タ䛡䜛䛣䛸䛷᪂つ㛤Ⓨ䛻㛫䛜 䛡䜛䜘䛖䛻䛺䜛 䝫䝇䝖䝰䞊䝔䝮䛷ᨵၿ䜒ᅇ䜛 %LJ4XHU\䛾᭱⤊᭦᪂้䜢┘ど䛩䜛䛣䛸䛷 '5(䝏䞊 䝮䛜䜶䝷䞊䛻᪩䛟Ẽ䛵䛡䜛䜘䛖䛻䛺䜛 ̐ ̐ ̐ ̐
6/,6/$6/2䛾ᐃ⩏ 6HUYLFH/HYHO,QGLFDWRU 䝃䞊䝡䝇䛾ရ㉁䜢Ᏺ䜛䛯䜑䛾ᣦᶆ 6/, 6/$ 6/2 6HUYLFH/HYHO$JUHHPHQW 6/,䛷ᐃ⩏䛧䛯ᣦᶆ䛻㛵䛩䜛䝃䞊䝡䝇ᥦ౪⪅䛸 䛾ዎ⣙
◚䛳䛯䛻䛹䛖䛩䜛䛛䛺䛹 6HUYLFH/HYHO2EMHFWLYH 6/,䛷ᐃ⩏䛧䛯ᣦᶆ䛾ලయⓗ䛺┠Ᏻ
'5(䝏䞊䝮䛻䛚䛡䜛6/,6/26/$䛾ᐃ⩏ 6HUYLFH/HYHO,QGLFDWRU 䝕䞊䝍䝟䜲䝥䝷䜲䞁䛾㐺ᛶ 䝕䞊䝍䝋䞊䝇䛾᭦᪂䛛䜙䛹 䛾䛟䜙䛔㐜䜜䛶㌿㏦ඛ䛷ᐇ⏝ྍ⬟䛻䛺䜛䛛 6/, 6/$ 6/2 6HUYLFH/HYHO$JUHHPHQW
䝕䞊䝍䝋䞊䝇䛤䛸䛻%LJ4XHU\⏝⪅䛸⤖䜀䜜䛯㐺 ᛶ䛻㛵䛩䜛ዎ⣙ ◚䛳䛯ሙྜ䛿䝫䝇䝖䝰䞊䝔䝮䜢ᐇ 䝕䞊䝍䝋䞊䝇$䛿᪥䛾㐺ᛶ䛷䛾㌿㏦ 6HUYLFH/HYHO2EMHFWLYH '5(䝏䞊䝮ෆ䛷Ỵᐃ䛥䜜䛯䝕䞊䝍䝋䞊䝇䛤䛸䛾㐺 ᛶ䛾┠ᶆ 䝕䞊䝍䝋䞊䝇$䛿KRXU䛾㐺ᛶ䛷䛾㌿㏦
㐺ᛶ䛾┘ど 㐜ᘏ㛫┘ど䝅䝇䝔䝮䛾ᐇ ྛ✀䝕䞊䝍䝟䜲䝥䝷䜲䞁䛾㐺ᛶ䛻㛵䛩䜛 6/2䛸%LJ4XHU\䛾䝯䝍䝕䞊䝍䛷䛒䜛᭱⤊᭦᪂ 㛫䜢ẚ㍑ 6/26/$䜘䜚䜒᭦᪂㛫䛜㐜ᘏ䛧䛶䛔䛯䜙 VODFN䛻㏻▱䛧䚸㞀ᐖᑐᛂ䞉䝫䝇䝖䝰䞊䝔䝮䜢ᐇ 䝁䞊䝗䛿S\WKRQ䛷సᡂ䛧䚸*LWKXE$FWLRQV䛷
ᐃᮇᐇ⾜
䜚㏉䜚 㐺ᛶ䛾6/26/$䜢ᐃ⩏䛩䜛䛣䛸䛻䜘䛳䛶䚸ศᯒ⪅䛿䝕䞊䝍䛾㌿㏦㛫䛻䛚䛡䜛㐜ᘏ䜢ᢕ ᥱ䛧䛺䛜䜙ศᯒ䛜ྍ⬟䛻䛺䛳䛯 '5(䝏䞊䝮䜒㞀ᐖᑐᛂ䜢ᐇ䛩䜛㝿䛾ᇶ‽್䛻䜘䜚䚸㞀ᐖᑐᛂ䛸ᶵ⬟㛤Ⓨ䜢䝞䝷䞁䝇䜘䛟ᐇ ⾜䛩䜛䛣䛸䛜䛷䛝䜛䜘䛖䛻䛺䛳䛯
䜽䜶䝸䛾ಟṇ䛚 㢪䛔䛧䜎䛩 䛣䛾䝎䝑䝅䝳䝪䞊䝗䛾 సᡂ䛚㢪䛔䛧䜎䛩 䝕䞊䝍ရ㉁䛾ၥ㢟Ⅼ ̐ศᯒ䝏䞊䝮䜈䛾౫㢗㐣ከ ศᯒ䝏䞊䝮䛻䜽䜶䝸䛾సᡂ䚸ಟṇ䛺䛹䛾౫㢗䛜ቑຍ ㄡ䛛䛜సᡂ䛧䛯䜽䜶䝸䜢⏝䛩䜛䛣䛸䛷ᑡ䛧䛪䛴䝈 䝺䛜Ⓨ⏕
ʍ/RRNHU䛾ᑟධ /RRNHU䛷๓䛻64/䜢ᐃ⩏䛧䛶䛒䛢䜛䛣䛸䛷䚸䜽䜶䝸 䛾᭩䛝᪉䛻䜘䜛䝈䝺䜢ῶ䜙䛩䚹 ㄡ䛷䜒䝎䝑䝅䝳䝪䞊䝗䜢⡆༢䛻సᡂ䛷䛝䜛䜘䛖䛻䛩䜛 䛣䛸䛷䚸䝕䞊䝍ά⏝䛾ಁ㐍䜢ᮇᚅ
/RRNHU䛾ᑟධ /RRNHU䛻䜘䜛䝕䞊䝍ရ㉁䛾ྥୖ 㢖⦾䛻ศᯒ䛷⏝䛔䜙䜜䜛ᣦᶆ䜢 ':+䛸䛧䛶 /RRNHU䛛䜙ᐃ⩏ /RRN0/䛻䜘䜛64/ᐃ⩏䛾୍ᮏ ([SORUH䛻䜘䜛䝎䝑䝅䝳䝪䞊䝗సᡂ䛾⡆᫆ 5HGDVK䛷⏝䛔䜙䜜䜛䝎䝑䝅䝳䝪䞊䝗䛾⨨
䜚㏉䜚 /RRNHU䛾ᑟධ䛻䜘䜚䚸/RRN0/䛻グ㏙䛧䛯ᣦᶆ䛻㛵䛧䛶䛿ㄡ䛷䜒⡆༢䛻䝎䝑䝅䝳䝪䞊䝗䜢సᡂ 䛷䛝䚸䛭䛾䝕䞊䝍䛾ရ㉁䜒ୖ䛜䛳䛯 /RRNHU䛷ᐃ⩏䛧䛯ᣦᶆ䜢%LJ4XHU\䛛䜙䜒ศᯒྍ⬟䛺≧ែ䛻䛩䜛䛣䛸䛷䚸䝕䞊䝍䝫䞊䝍䝹䜔 5HGDVK䛷䛾䝕䞊䝍䛾ရ㉁䜒㧗䛟䛺䛳䛯 /RRNHU䛿ゝㄒ䛭䛾䜒䛾䛿64/䜢ᇶᮏ䛸䛧䛶䛔䜛䛾䛷䜻䝱䝑䝏䜰䝑䝥䛿⡆༢䛰䛳䛯䛜䚸䝁䞊䝗ᵓ ᡂ䛾タィ䛿㞴䛧䛟䚸ఱᗘ䛛య䛾タィ䜢䜔䜚┤䛧䛯 5HGDVK䛷⏝䛔䜙䜜䛶䛔䜛䝎䝑䝅䝳䝪䞊䝗䜢 /RRNHU䛻⛣⾜䛧䜘䛖䛸䛧䛶䛔䜛䛜
/RRN0/䛻ᐃ⩏ 䛩䜉䛝ᣦᶆ䜒ከ䛟䚸䜎䛰⛣⾜㏵୰䛷䛒䜛 /RRNHU䛾䝁䝭䝳䝙䝔䜱䛜ᗈ䛔䛾䛷䚸♫䛥䜣䛸 /RRNHU䛾ᵓᡂ䛻䛴䛔䛶┦ㄯ䛧䛺䛜䜙㛤Ⓨ䛷䛝 䛯䛾䛷䛩䛤䛟ຓ䛛䛳䛯
̐(7/䛾7ฎ⌮䛾䝒䞊䝹౫Ꮡᛶ䚸⫧䚸㏱᫂ᛶ 䝕䞊䝍㔞䛾ቑຍ䛻క䛔䝬䝇䜻䞁䜾䛺䛹䛾ຍᕤฎ⌮䜢 ⾜䛳䛶䛔䜛HPEXONෆ䛾7ฎ⌮䝁䞊䝗䛾ᐇ⾜㛫䛜⫧ 䛧䛶䛝䛯 WURFFR䛺䛹䛾HPEXON௨እ䛾䝕䞊䝍䝟䜲䝥䝷䜲䞁䛾ᑟ ධ䛻䜘䜚7ฎ⌮䛜HPEXON䛻౫Ꮡ䛧䛶䛔䜛 HPEXON䛾䝁䞊䝗䜢ㄞ䜎䛺䛔䛸 7ฎ⌮䛾ෆᐜ䛜ᢕᥱ䛷 䛝䛪䚸ศᯒ⪅䛛䜙䛩䜛䛸㏱᫂䛺ฎ⌮䛷䛒䜛
䝕䞊䝍ရ㉁䛾ၥ㢟Ⅼ ʍGEW&ORXG䛾ᑟධ䛻䜘䜛(/7䝟䜲䝥䝷䜲䞁䛾ᵓ⠏ GEW&ORXG䜢⏝䛔䛶HPEXON䛷⾜䛳䛶䛔䛯7ฎ⌮䜢௦᭰ 䝕䞊䝍䝟䜲䝥䝷䜲䞁䛻⏝䛔䛶䛔䜛䝒䞊䝹䛻౫Ꮡ䛫䛪 䛻䚸%LJ4XHU\䛻㞟䛧䛯䝕䞊䝍䛻ᑐ䛧䛶ᵝ䚻䛺ຍᕤ䛜 ྍ⬟䛻 ຍᕤ䜢୍⟠ᡤ䛻㞟䜑䜛䛣䛸䛷䚸ศᯒ⪅䛻ຍᕤሗ䜢 㐺ษ䛻ఏ䛘䜙䜜䜛䜘䛖䛻ᮇᚅ 䝬䝇䜻䞁䜾䛺䛹䛾ຍᕤฎ⌮ ¡
䜚㏉䜚 GEW&ORXG䛾ᑟධ䛻䜘䜚䚸䛹䛾䝕䞊䝍䝟䜲䝥䝷䜲䞁䛛䜙䝕䞊䝍䜢㞟䛧䛶䜒ඹ㏻䛾ຍᕤ䛜䛷䛝 䜛䜘䛖䛻䛺䛳䛯 GEW&ORXG䛿䜲䞁䝣䝷䛾ᩚഛ䛜ᚲせ䛺䛯䜑䚸䝏䞊䝮䝯䞁䝞䞊䛜ᑡ䛺䛔 '5(䝏䞊䝮䛻䛿㐺ษ 䛺䝒䞊䝹䛰䛳䛯䚹ゝㄒ䜒 64/䜢ᇶᮏ䛸䛧䛶䛔䜛䛾䛷ᑟධ䜎䛷䛾䝁䝇䝖䛜ప䛛䛳䛯 (7/ᇶ┙䜢(/7ᇶ┙䛻ኚ᭦䛩䜛㝿䛻䚸ຍᕤ䛧䛶䛺䛔䝕䞊䝍䜢≉ᐃ䛾 *&3䝥䝻䝆䜵䜽䝖䛻ಖᏑ
䛩䜛ᚲせ䛜䛒䜛䛜䚸 *&3䛾,$0ᵓᡂ䛜」㞧䛷䛭䛣䛛䜙ᡭ䜢䛴䛡䜛䛾䛜ኚ䛰䛳䛯
᭱㏆䛾䝕䞊䝍ᇶ┙
䛣䜜䛛䜙䛾ရ㉁䛾䛚䛡䜛ㄢ㢟 䝕䞊䝍ရ㉁䛻䛚䛡䜛 6/,䛸䛧䛶㐺ᛶ䜢ᐃ⩏䛧䛯䛜䚸䛻䜒୍ពᛶ䚸ᛶ䛺䛹䛾ᣦᶆ䛜䛒 䜚䚸┘ど䛺䛹䜢㐺ᛂ䛩䜛䛣䛸䛷ရ㉁䛾ྥୖ䜢┠ᣦ䛧䛯䛔 GEW&ORXG䛿'5(䝏䞊䝮䜢୰ᚰ䛸䛧䛯䝕䞊䝍ຍᕤ䚸 /RRNHU䛿ศᯒ⪅䛜୰ᚰ䛸䛺䜛ຍᕤ䜢ᢸᙜ 䛩䜛䛣䛸䛷䝕䞊䝍ศᯒ䛻䛚䛡䜛䝁䝇䝖䚸ရ㉁䛾ྥୖ䜢┠ᣦ䛧䛯䛔 ྛ✀%,䝒䞊䝹䛾ᙺ䜢ศᢸ䛧䚸㐺ษ䛺 %,䝒䞊䝹䛾⏝䜢♫ⓗ䛻㐺ᛂ䛧䚸ရ㉁䛾㧗䛔䝕䞊
䝍ά⏝䜢ಁ㐍䛧䛯䛔
䛥䛔䛤䛻 㼔㼠㼠㼜㼟㻦㻛㻛㼙㼑㼑㼠㼥㻚㼚㼑㼠㻛㼙㼍㼠㼏㼔㼑㼟㻛㼙㻱㻶㼜㻵㼚㼤㻳㻺㼒㼁㼅䈊 㼔㼠㼠㼜㼟㻦㻛㻛㼣㼣㼣㻚㼣㼍㼚㼠㼑㼐㼘㼥㻚㼏㼛㼙㻛㼜㼞㼛㼖㼑㼏㼠㼟㻛㻡㻣㻥㻤㻝㻜䈊