Building an IoT platform with Go and Python

Go and Python are very different languages, but do they complement each other? At Energomonitor, we think so, and we used both to build an IoT platform that processes and analyzes data from our devices in real-time. Where did we use Go? Where did we use Python? Why? What are the strong points of each language? And what are the weak ones? How do the ecosystems compare? And what about the daily operational experience? Finally, what would we do differently knowing what we know today? These are all questions I will attempt to answer, based on our multi-year experience.

Presented at Golang Prague #3.

Details: https://www.meetup.com/Prague-Golang-Meetup/events/251802529/

Transcript

1. Building an IoT platform
   with Go and Python
   David Majda
   @dmajda
   Golang Prague #3 · Prague · 13 September, 2018
   

   View Slide

2. Outline
   What do we do?
   Data pipeline
   Python vs. Go
   What did we learn?
   

   View Slide

3. What do we do?
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. Data pipeline
   

   View Slide

8. View Slide

9. View Slide

10. View Slide

11. Typical Go service
    Service
    Data Data
    Config
    

    View Slide

12. Python vs. Go
    

    View Slide

13. Comparison
    Language
    Ecosystem
    Performance
    Concurrency
    Deployment
    

    View Slide

14. Language
    

    View Slide

15. View Slide

16. Simple
    

    View Slide

17. Simple
    Easy to learn
    

    View Slide

18. Simple
    Easy to learn
    Fast development
    

    View Slide

19. Simple
    Easy to learn
    Fast development
    Reliable
    

    View Slide

20. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    

    View Slide

21. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    

    View Slide

22. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    

    View Slide

23. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development
    

    View Slide

24. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development
    Reliable
    

    View Slide

25. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development
    Reliable
    Tons of complaints
    

    View Slide

26. Complaints
    

    View Slide

27. Complaints
    Too simple
    

    View Slide

28. Complaints
    Too simple
    Unnecessarily different
    

    View Slide

29. Complaints
    Too simple
    Unnecessarily different
    Inflexible
    

    View Slide

30. Complaints
    Too simple
    Unnecessarily different
    Inflexible
    Likes to be left alone
    

    View Slide

31. Complaints
    Too simple
    Unnecessarily different
    Inflexible
    Likes to be left alone
    Many other small issues
    

    View Slide

32. {
    "type": "gateway",
    "sn": "3FCF54AADCB6E7DC"
    }
    {
    "type": "sensor",
    "gateway_sn": "3FCF54AADCB6E7DC",
    "channel": 5
    }
    

    View Slide

33. type Device struct {
    Type string `json:"type"`
    SN *string `json:"sn,omitempty"` // pointer!
    GatewaySN *string `json:"gateway_sn,omitempty"` // pointer!
    Channel *uint `json:"channel,omitempty"` // pointer!
    }
    input := `{"type": "Gateway", "sn": "3FCF54AADCB6E7DC"}`
    device := Device{}
    json.Unmarshal([]byte(input), &device)
    fmt.Println(device) // {Gateway 0x1050c160 }
    output, _ := json.Marshal(device)
    fmt.Printf("%s\n", output) // {"type":"Gateway","sn":"3FCF54AADCB6E7DC"}
    

    View Slide

34. @JsonTypeInfo(
    use = JsonTypeInfo.Id.NAME,
    include = As.PROPERTY,
    property = "type"
    )
    @JsonSubTypes({
    @JsonSubTypes.Type(value = Gateway.class, name = "gateway"),
    @JsonSubTypes.Type(value = Sensor.class, name = "sensor")
    })
    public class Device {
    }
    @JsonTypeName("gateway")
    public class Gateway extends Device {
    public String sn;
    }
    @JsonTypeName("sensor")
    public class Sensor extends Device {
    public String gatewaySN;
    public int channel;
    }
    

    View Slide

35. {
    "type": "gateway",
    "gateway": {
    "sn": "3FCF54AADCB6E7DC"
    }
    }
    {
    "type": "sensor",
    "sensor": {
    "gateway_sn": "3FCF54AADCB6E7DC",
    "channel": 5
    }
    }
    

    View Slide

36. Ecosystem
    

    View Slide

37. View Slide

38. Excellent
    Except asyncio (later)
    

    View Slide

39. Excellent
    Except asyncio (later)
    Excellent
    

    View Slide

40. Performance
    

    View Slide

41. View Slide

42. Good :-)
    

    View Slide

43. Good :-) Better :-)
    

    View Slide

44. Concurrency
    

    View Slide

45. View Slide

46. Threads
    90’s technology
    

    View Slide

47. Threads
    90’s technology
    asyncio
    Parallel world
    Poor usability
    Bad libraries
    

    View Slide

48. Threads
    90’s technology
    asyncio
    Parallel world
    Poor usability
    Bad libraries
    Goroutines &
    channels
    Model the real world
    really nicely
    

    View Slide

49. Deployment
    

    View Slide

50. View Slide

51. Python 2/3
    

    View Slide

52. Python 2/3
    Virtualenvs
    

    View Slide

53. Python 2/3
    Virtualenvs
    Packaging
    

    View Slide

54. Python 2/3
    Virtualenvs
    Packaging
    Tons of files
    

    View Slide

55. Python 2/3
    Virtualenvs
    Packaging
    Tons of files
    Single binary
    

    View Slide

56. All invalidated by…
    

    View Slide

57. What did we learn?
    

    View Slide

58. View Slide

59. Simple services
    One well-defined task
    Performance needed
    Concurrency useful
    

    View Slide

60. Apps Simple services
    One well-defined task
    Performance needed
    Concurrency useful
    

    View Slide

61. Apps
    Complex services
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful
    

    View Slide

62. Apps
    Complex services
    Scripts
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful
    

    View Slide

63. Apps
    Complex services
    Scripts
    Prototyping
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful
    

    View Slide

64. What would we do
    differently?
    

    View Slide

65. Not much
    

    View Slide

66. The End
    David Majda
    @dmajda
    Interested in working with us?
    Let’s talk!
    

    View Slide