$30 off During Our Annual Pro Sale. View Details »

Building an IoT platform with Go and Python

David Majda
September 13, 2018

Building an IoT platform with Go and Python

Go and Python are very different languages, but do they complement each other? At Energomonitor, we think so, and we used both to build an IoT platform that processes and analyzes data from our devices in real-time. Where did we use Go? Where did we use Python? Why? What are the strong points of each language? And what are the weak ones? How do the ecosystems compare? And what about the daily operational experience? Finally, what would we do differently knowing what we know today? These are all questions I will attempt to answer, based on our multi-year experience.

Presented at Golang Prague #3.

Details: https://www.meetup.com/Prague-Golang-Meetup/events/251802529/

David Majda

September 13, 2018
Tweet

More Decks by David Majda

Other Decks in Programming

Transcript

  1. Building an IoT platform
    with Go and Python
    David Majda
    @dmajda
    Golang Prague #3 · Prague · 13 September, 2018

    View Slide

  2. Outline
    What do we do?
    Data pipeline
    Python vs. Go
    What did we learn?

    View Slide

  3. What do we do?

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. Data pipeline

    View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. Typical Go service
    Service
    Data Data
    Config

    View Slide

  12. Python vs. Go

    View Slide

  13. Comparison
    Language
    Ecosystem
    Performance
    Concurrency
    Deployment

    View Slide

  14. Language

    View Slide

  15. View Slide

  16. Simple

    View Slide

  17. Simple
    Easy to learn

    View Slide

  18. Simple
    Easy to learn
    Fast development

    View Slide

  19. Simple
    Easy to learn
    Fast development
    Reliable

    View Slide

  20. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints

    View Slide

  21. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple

    View Slide

  22. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn

    View Slide

  23. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development

    View Slide

  24. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development
    Reliable

    View Slide

  25. Simple
    Easy to learn
    Fast development
    Reliable
    No complaints
    Simple
    Easy to learn
    Fast development
    Reliable
    Tons of complaints

    View Slide

  26. Complaints

    View Slide

  27. Complaints
    Too simple

    View Slide

  28. Complaints
    Too simple
    Unnecessarily different

    View Slide

  29. Complaints
    Too simple
    Unnecessarily different
    Inflexible

    View Slide

  30. Complaints
    Too simple
    Unnecessarily different
    Inflexible
    Likes to be left alone

    View Slide

  31. Complaints
    Too simple
    Unnecessarily different
    Inflexible
    Likes to be left alone
    Many other small issues

    View Slide

  32. {
    "type": "gateway",
    "sn": "3FCF54AADCB6E7DC"
    }
    {
    "type": "sensor",
    "gateway_sn": "3FCF54AADCB6E7DC",
    "channel": 5
    }

    View Slide

  33. type Device struct {
    Type string `json:"type"`
    SN *string `json:"sn,omitempty"` // pointer!
    GatewaySN *string `json:"gateway_sn,omitempty"` // pointer!
    Channel *uint `json:"channel,omitempty"` // pointer!
    }
    input := `{"type": "Gateway", "sn": "3FCF54AADCB6E7DC"}`
    device := Device{}
    json.Unmarshal([]byte(input), &device)
    fmt.Println(device) // {Gateway 0x1050c160 }
    output, _ := json.Marshal(device)
    fmt.Printf("%s\n", output) // {"type":"Gateway","sn":"3FCF54AADCB6E7DC"}

    View Slide

  34. @JsonTypeInfo(
    use = JsonTypeInfo.Id.NAME,
    include = As.PROPERTY,
    property = "type"
    )
    @JsonSubTypes({
    @JsonSubTypes.Type(value = Gateway.class, name = "gateway"),
    @JsonSubTypes.Type(value = Sensor.class, name = "sensor")
    })
    public class Device {
    }
    @JsonTypeName("gateway")
    public class Gateway extends Device {
    public String sn;
    }
    @JsonTypeName("sensor")
    public class Sensor extends Device {
    public String gatewaySN;
    public int channel;
    }

    View Slide

  35. {
    "type": "gateway",
    "gateway": {
    "sn": "3FCF54AADCB6E7DC"
    }
    }
    {
    "type": "sensor",
    "sensor": {
    "gateway_sn": "3FCF54AADCB6E7DC",
    "channel": 5
    }
    }

    View Slide

  36. Ecosystem

    View Slide

  37. View Slide

  38. Excellent
    Except asyncio (later)

    View Slide

  39. Excellent
    Except asyncio (later)
    Excellent

    View Slide

  40. Performance

    View Slide

  41. View Slide

  42. Good :-)

    View Slide

  43. Good :-) Better :-)

    View Slide

  44. Concurrency

    View Slide

  45. View Slide

  46. Threads
    90’s technology

    View Slide

  47. Threads
    90’s technology
    asyncio
    Parallel world
    Poor usability
    Bad libraries

    View Slide

  48. Threads
    90’s technology
    asyncio
    Parallel world
    Poor usability
    Bad libraries
    Goroutines &
    channels
    Model the real world
    really nicely

    View Slide

  49. Deployment

    View Slide

  50. View Slide

  51. Python 2/3

    View Slide

  52. Python 2/3
    Virtualenvs

    View Slide

  53. Python 2/3
    Virtualenvs
    Packaging

    View Slide

  54. Python 2/3
    Virtualenvs
    Packaging
    Tons of files

    View Slide

  55. Python 2/3
    Virtualenvs
    Packaging
    Tons of files
    Single binary

    View Slide

  56. All invalidated by…

    View Slide

  57. What did we learn?

    View Slide

  58. View Slide

  59. Simple services
    One well-defined task
    Performance needed
    Concurrency useful

    View Slide

  60. Apps Simple services
    One well-defined task
    Performance needed
    Concurrency useful

    View Slide

  61. Apps
    Complex services
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful

    View Slide

  62. Apps
    Complex services
    Scripts
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful

    View Slide

  63. Apps
    Complex services
    Scripts
    Prototyping
    Simple services
    One well-defined task
    Performance needed
    Concurrency useful

    View Slide

  64. What would we do
    differently?

    View Slide

  65. Not much

    View Slide

  66. The End
    David Majda
    @dmajda
    Interested in working with us?
    Let’s talk!

    View Slide