Pro Yearly is on sale from $80 to $50! »

Building an IoT platform with Go and Python

Ebe96461709771a430da9c7c58f9ae5f?s=47 David Majda
September 13, 2018

Building an IoT platform with Go and Python

Go and Python are very different languages, but do they complement each other? At Energomonitor, we think so, and we used both to build an IoT platform that processes and analyzes data from our devices in real-time. Where did we use Go? Where did we use Python? Why? What are the strong points of each language? And what are the weak ones? How do the ecosystems compare? And what about the daily operational experience? Finally, what would we do differently knowing what we know today? These are all questions I will attempt to answer, based on our multi-year experience.

Presented at Golang Prague #3.

Details: https://www.meetup.com/Prague-Golang-Meetup/events/251802529/

Ebe96461709771a430da9c7c58f9ae5f?s=128

David Majda

September 13, 2018
Tweet

Transcript

  1. Building an IoT platform with Go and Python David Majda

    @dmajda Golang Prague #3 · Prague · 13 September, 2018
  2. Outline What do we do? Data pipeline Python vs. Go

    What did we learn?
  3. What do we do?

  4. None
  5. None
  6. None
  7. Data pipeline

  8. None
  9. None
  10. None
  11. Typical Go service Service Data Data Config

  12. Python vs. Go

  13. Comparison Language Ecosystem Performance Concurrency Deployment

  14. Language

  15. None
  16. Simple

  17. Simple Easy to learn

  18. Simple Easy to learn Fast development

  19. Simple Easy to learn Fast development Reliable

  20. Simple Easy to learn Fast development Reliable No complaints

  21. Simple Easy to learn Fast development Reliable No complaints Simple

  22. Simple Easy to learn Fast development Reliable No complaints Simple

    Easy to learn
  23. Simple Easy to learn Fast development Reliable No complaints Simple

    Easy to learn Fast development
  24. Simple Easy to learn Fast development Reliable No complaints Simple

    Easy to learn Fast development Reliable
  25. Simple Easy to learn Fast development Reliable No complaints Simple

    Easy to learn Fast development Reliable Tons of complaints
  26. Complaints

  27. Complaints Too simple

  28. Complaints Too simple Unnecessarily different

  29. Complaints Too simple Unnecessarily different Inflexible

  30. Complaints Too simple Unnecessarily different Inflexible Likes to be left

    alone
  31. Complaints Too simple Unnecessarily different Inflexible Likes to be left

    alone Many other small issues
  32. { "type": "gateway", "sn": "3FCF54AADCB6E7DC" } { "type": "sensor", "gateway_sn":

    "3FCF54AADCB6E7DC", "channel": 5 }
  33. type Device struct { Type string `json:"type"` SN *string `json:"sn,omitempty"`

    // pointer! GatewaySN *string `json:"gateway_sn,omitempty"` // pointer! Channel *uint `json:"channel,omitempty"` // pointer! } input := `{"type": "Gateway", "sn": "3FCF54AADCB6E7DC"}` device := Device{} json.Unmarshal([]byte(input), &device) fmt.Println(device) // {Gateway 0x1050c160 <nil> <nil>} output, _ := json.Marshal(device) fmt.Printf("%s\n", output) // {"type":"Gateway","sn":"3FCF54AADCB6E7DC"}
  34. @JsonTypeInfo( use = JsonTypeInfo.Id.NAME, include = As.PROPERTY, property = "type"

    ) @JsonSubTypes({ @JsonSubTypes.Type(value = Gateway.class, name = "gateway"), @JsonSubTypes.Type(value = Sensor.class, name = "sensor") }) public class Device { } @JsonTypeName("gateway") public class Gateway extends Device { public String sn; } @JsonTypeName("sensor") public class Sensor extends Device { public String gatewaySN; public int channel; }
  35. { "type": "gateway", "gateway": { "sn": "3FCF54AADCB6E7DC" } } {

    "type": "sensor", "sensor": { "gateway_sn": "3FCF54AADCB6E7DC", "channel": 5 } }
  36. Ecosystem

  37. None
  38. Excellent Except asyncio (later)

  39. Excellent Except asyncio (later) Excellent

  40. Performance

  41. None
  42. Good :-)

  43. Good :-) Better :-)

  44. Concurrency

  45. None
  46. Threads 90’s technology

  47. Threads 90’s technology asyncio Parallel world Poor usability Bad libraries

  48. Threads 90’s technology asyncio Parallel world Poor usability Bad libraries

    Goroutines & channels Model the real world really nicely
  49. Deployment

  50. None
  51. Python 2/3

  52. Python 2/3 Virtualenvs

  53. Python 2/3 Virtualenvs Packaging

  54. Python 2/3 Virtualenvs Packaging Tons of files

  55. Python 2/3 Virtualenvs Packaging Tons of files Single binary

  56. All invalidated by…

  57. What did we learn?

  58. None
  59. Simple services One well-defined task Performance needed Concurrency useful

  60. Apps Simple services One well-defined task Performance needed Concurrency useful

  61. Apps Complex services Simple services One well-defined task Performance needed

    Concurrency useful
  62. Apps Complex services Scripts Simple services One well-defined task Performance

    needed Concurrency useful
  63. Apps Complex services Scripts Prototyping Simple services One well-defined task

    Performance needed Concurrency useful
  64. What would we do differently?

  65. Not much

  66. The End David Majda @dmajda Interested in working with us?

    Let’s talk!