Building an IoT platform with Go and Python

Go and Python are very different languages, but do they complement each other? At Energomonitor, we think so, and we used both to build an IoT platform that processes and analyzes data from our devices in real-time. Where did we use Go? Where did we use Python? Why? What are the strong points of each language? And what are the weak ones? How do the ecosystems compare? And what about the daily operational experience? Finally, what would we do differently knowing what we know today? These are all questions I will attempt to answer, based on our multi-year experience.

Presented at Golang Prague #3.

Details: https://www.meetup.com/Prague-Golang-Meetup/events/251802529/

Transcript

1. Building an IoT platform with Go and Python David Majda @dmajda Golang Prague #3 · Prague · 13 September, 2018
2. Outline What do we do? Data pipeline Python vs. Go What did we learn?
3. What do we do?
7. Data pipeline
11. Typical Go service Service Data Data Config
12. Python vs. Go
13. Comparison Language Ecosystem Performance Concurrency Deployment
14. Language
16. Simple
17. Simple Easy to learn
18. Simple Easy to learn Fast development
19. Simple Easy to learn Fast development Reliable
20. Simple Easy to learn Fast development Reliable No complaints
21. Simple Easy to learn Fast development Reliable No complaints Simple
22. Simple Easy to learn Fast development Reliable No complaints Simple Easy to learn
23. Simple Easy to learn Fast development Reliable No complaints Simple Easy to learn Fast development
24. Simple Easy to learn Fast development Reliable No complaints Simple Easy to learn Fast development Reliable
25. Simple Easy to learn Fast development Reliable No complaints Simple Easy to learn Fast development Reliable Tons of complaints
26. Complaints
27. Complaints Too simple
28. Complaints Too simple Unnecessarily different
29. Complaints Too simple Unnecessarily different Inflexible
30. Complaints Too simple Unnecessarily different Inflexible Likes to be left alone
31. Complaints Too simple Unnecessarily different Inflexible Likes to be left alone Many other small issues
32. { "type": "gateway", "sn": "3FCF54AADCB6E7DC" } { "type": "sensor", "gateway_sn": "3FCF54AADCB6E7DC", "channel": 5 }
33. type Device struct { Type string `json:"type"` SN *string `json:"sn,omitempty"` // pointer! GatewaySN *string `json:"gateway_sn,omitempty"` // pointer! Channel *uint `json:"channel,omitempty"` // pointer! } input := `{"type": "Gateway", "sn": "3FCF54AADCB6E7DC"}` device := Device{} json.Unmarshal([]byte(input), &device) fmt.Println(device) // {Gateway 0x1050c160 <nil> <nil>} output, _ := json.Marshal(device) fmt.Printf("%s\n", output) // {"type":"Gateway","sn":"3FCF54AADCB6E7DC"}
34. @JsonTypeInfo( use = JsonTypeInfo.Id.NAME, include = As.PROPERTY, property = "type" ) @JsonSubTypes({ @JsonSubTypes.Type(value = Gateway.class, name = "gateway"), @JsonSubTypes.Type(value = Sensor.class, name = "sensor") }) public class Device { } @JsonTypeName("gateway") public class Gateway extends Device { public String sn; } @JsonTypeName("sensor") public class Sensor extends Device { public String gatewaySN; public int channel; }
35. { "type": "gateway", "gateway": { "sn": "3FCF54AADCB6E7DC" } } { "type": "sensor", "sensor": { "gateway_sn": "3FCF54AADCB6E7DC", "channel": 5 } }
36. Ecosystem
38. Excellent Except asyncio (later)
39. Excellent Except asyncio (later) Excellent
40. Performance
42. Good :-)
43. Good :-) Better :-)
44. Concurrency
46. Threads 90’s technology
47. Threads 90’s technology asyncio Parallel world Poor usability Bad libraries
48. Threads 90’s technology asyncio Parallel world Poor usability Bad libraries Goroutines & channels Model the real world really nicely
49. Deployment
51. Python 2/3
52. Python 2/3 Virtualenvs
53. Python 2/3 Virtualenvs Packaging
54. Python 2/3 Virtualenvs Packaging Tons of files
55. Python 2/3 Virtualenvs Packaging Tons of files Single binary
56. All invalidated by…
57. What did we learn?
59. Simple services One well-defined task Performance needed Concurrency useful
60. Apps Simple services One well-defined task Performance needed Concurrency useful
61. Apps Complex services Simple services One well-defined task Performance needed Concurrency useful
62. Apps Complex services Scripts Simple services One well-defined task Performance needed Concurrency useful
63. Apps Complex services Scripts Prototyping Simple services One well-defined task Performance needed Concurrency useful
64. What would we do differently?
65. Not much
66. The End David Majda @dmajda Interested in working with us? Let’s talk!