hotdog - a TD tool for DD

TD Tech Talk #1

Transcript

1. Copyright ©2015 Treasure Data. All Rights Reserved. hotdog a TD tool for DD Yamashita, Yuu (@yyuu) 
 Treasure Data, Inc. yuu@treasure-data.com March 11, 2015
2. Copyright ©2015 Treasure Data. All Rights Reserved. about:yyuu • 2014/10 - Software Engineer at Treasure Data Inc. • Past: Infrastructure Engineer at Geisha Tokyo Entertainment Inc. • OSS • pyenv (github: yyuu/pyenv) • xtc (bitbucket: yyuu/xtc) • jetty-nosql-memcached (github: yyuu/jetty-nosql-memcached) • botornado (github: yyuu/botornado) • sbt-extras (github: paulp/sbt-extras) 2
3. Copyright ©2015 Treasure Data. All Rights Reserved. about:yyuu 3
4. Copyright ©2015 Treasure Data. All Rights Reserved. Today’s topic • hotdog - yet another command line tool for DataDog 4
5. Copyright ©2015 Treasure Data. All Rights Reserved. What’s DD? 5
6. Copyright ©2015 Treasure Data. All Rights Reserved. What’s DD? 6 • System metrics • dd-agent • Application metrics • fluent-logger • td-agent • fluent-plugin-metricsense
7. Copyright ©2015 Treasure Data. All Rights Reserved. TD on DD 7
8. Copyright ©2015 Treasure Data. All Rights Reserved. TD on DD • We have *MANY* servers on DD • BIG DATA NEEDS PROCESSORS • We have many DataDog monitors to detect failures • Send notification from DD to Slack and/or PagerDuty 8
9. Copyright ©2015 Treasure Data. All Rights Reserved. TD ❤️ DD 9
10. Copyright ©2015 Treasure Data. All Rights Reserved. Today’s topic • hotdog - yet another command line tool for DataDog 10
11. Copyright ©2015 Treasure Data. All Rights Reserved. TD PD 11 So, what is i-XXXXXXXX?
12. Copyright ©2015 Treasure Data. All Rights Reserved. How to get node info of i-XXXXXXXX? • AWS Management Console • DataDog Infrastructure List • Chef / Knife • … 12
13. Copyright ©2015 Treasure Data. All Rights Reserved. My solution; hotdog 13
14. Copyright ©2015 Treasure Data. All Rights Reserved. My solution; hotdog • Search hosts from DD tags • Search hosts from instance-id • Support expressions on DD tags • AND, OR, NOT • wildcard characters (‘*’, ‘?’, ‘[‘, ‘]’) • Supports cooperation with pssh/GNU Parallel 14
15. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • Parse given expression by PEG parser 15
16. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • Parse given expression by PEG parser • Get tag->hosts map from “GET /api/v1/tags/hosts” 16
17. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • Parse given expression by PEG parser • Get tag->hosts map from “GET /api/v1/tags/hosts” • Build host<->tag index on on-memory SQLite 17
18. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • Parse given expression by PEG parser • Get tag->hosts map from “GET /api/v1/tags/hosts” • Build host<->tag index on on-memory SQLite • Evaluate the expression 18
19. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • Parse given expression by PEG parser • Get tag->hosts map from “GET /api/v1/tags/hosts” • Build host<->tag index on on-memory SQLite • Evaluate the expression • Print the results in specified format (text, json, yam, etc.) 19 small expression interpreter based on parslet
20. Copyright ©2015 Treasure Data. All Rights Reserved. How hotdog works • https://github.com/yyuu/hotdog 20
21. Copyright ©2015 Treasure Data. All Rights Reserved. WE ARE HIRING! 21 Check: www.treasuredata.com