Demisto @Go meetup

a bit on Demisto stack + vendoring in Golang

Transcript

1. Demisto Confidential Golang @Demisto Meir Wahnon | June 2016
2. Demisto Confidential Summary Meir Wahnon, Dev team @Demisto Started GOLANG at Demisto(~0.5 year ago) Big golang fan ever since! In the Past HP open source project (CloudSlang) 2
3. Demisto Confidential Agenda § Why GO? § Our tech stack @Demisto § Vendoring − What is it? − Why you need it? § Q&A
4. Demisto Confidential Demisto § Cyber Security -> Incident Response § Company started ~0.5 year ago § 12 dev § https://www.demisto.com/
5. Demisto Confidential Incident Response? § Incident response is an organized approach to addressing and managing the aftermath of a security breach or attack § The 1990s was the era of protection § Then came detection decade § This decade is one of response
6. Demisto Confidential Why GO? § Because our CEO said so!! § But also because : − Concurrency at the language level − Single binary − Great open source community − Easy to contribute back to the community
7. Demisto Confidential Our Stack § BoltDB § Bleve - for search § Docker - for python sandboxing § Otto – for Javascript sandboxing § React – for UI stuff… CI: § CircleCI § Coveralls – for code coverage
8. Demisto Confidential Vendoring – the why? § A while back… when we just started… § We said that we should use vendoring, but no rush.. We can wait with it… § Then.. § First version release…
9. Demisto Confidential Some Go vendoring history § Early GO versions did not support vendoring § Solution – third party libs: § Godep : https://github.com/tools/godep § basically exploit the implementation details of $GOPATH
10. Demisto Confidential Vendoring beta § Vendoring was first introduced as EXPERIMENTAL on go 1.5 version § Environment variable `GO15VENDOREXPERIMENT`
11. Demisto Confidential Vendoring official § Vendoring was officially introduced in 1.6 as part of GO
12. Demisto Confidential Comparison of Go Vendoring Tools § Godep - https://github.com/tools/godep − ~3,600 github stars − save/restore § Glide - https://github.com/Masterminds/glide − ~2,100 github stars − Yml format − Glide init+install/up § Govendor - https://github.com/kardianos/govendor − ~700 github stars § The full comparison article: − https://medium.com/@onuryilmaz/comparison-of-go-vendoring-tools-acf019ea476f#.7pgyj6bsx
13. Demisto Confidential Govendor § We choose Govendor (same as article) § Json file called vendor.json to describe all project dependencies §
14. Demisto Confidential Govendor cmds § Init : Create the "vendor" folder and the "vendor.json" file. § List : List and filter existing dependencies and packages. § Add : Add packages from $GOPATH. § Update : Update packages from $GOPATH. § Fetch : Add new or update vendor folder packages from remote repository. § Sync : Pull packages into vendor folder from remote repository with revisions from vendor.json file.
15. Demisto Confidential Vendoring issues (for us at least…) § Go tooling does not give “special” treatment to vendor folder. − If you run “./…” it will use the vendor folder too !! − Issue in testing with coverage , or running “go fmt” commands… − https://github.com/golang/go/issues/11659 − WDYT? § Impact on us: − Doing nasty bash stuff to exclude vendor folder from coverage § FOLDER_FOR_COVERAGE=$(go list ./... | grep -v /vendor/ | xargs echo -n | tr ' ' ',') § go list -f '{{if gt (len .TestGoFiles) 0}}"go test -tags integration -timeout 30m -covermode set -coverprofile {{.Name}}.coverprofile - coverpkg xxx {{.ImportPath}}"{{end}}' ./web/... ./services/... | awk -v targets=$FOLDER_FOR_COVERAGE '{ gsub("xxx", targets ); print;}' | sed 's/\"//g' − Also for golint: § golint ./... 2>&1 | grep -v ^vendor | wc -l
16. Demisto Confidential Supporting Vendoring in CircleCI § Vendor.json is our Single source of truth § Each build we are doing: − Govendor sync − Caching vendor folder (except the vendor.json file!!) § Watch out: − In CircleCI $GOPATH is complex (an array of folders by default) − Causes some weird issues when using with govendor − Solution (more nasty bash scripts …) : − IMPORT_PATH: "github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME"
17. Demisto Confidential 17
18. Thanks!