Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Integration Test in Event Driven Architecture

Integration Test in Event Driven Architecture

LINE Developers Thailand

October 16, 2021
Tweet

More Decks by LINE Developers Thailand

Other Decks in Technology

Transcript

  1. INTEGRATION TEST IN EVENT
    DRIVEN ARCHITECTURE

    View full-size slide

  2. EVENT DRIVEN ARCHITECTURE
    Event
    Event
    Event Producer
    Event
    Event
    Event
    Consumer
    Consumer
    Consumer
    Event Broker
    Topic 1
    Topic 2
    Topic 3

    View full-size slide

  3. Server Event Broker Event Broker
    Frontend
    Consumer
    Consumer
    Consumer
    Database
    Unit Test Unit Test
    Unit Test
    Unit Test
    Unit Test

    View full-size slide

  4. Server Event Broker Event Broker
    Frontend
    Consumer
    Consumer
    Consumer
    Database
    E2E Test

    View full-size slide

  5. Event Broker
    Consumer
    Server Event Broker Event Broker
    Frontend
    Consumer
    Consumer
    Consumer
    Database
    Event Broker
    Server
    Consumer
    Consumer
    Database
    Event Broker
    Consumer
    Event Broker
    Consumer
    Event Broker
    Consumer
    Database
    E2E test is getting more difficult
    Event Broker
    Consumer

    View full-size slide

  6. • Admin can chat with followers
    • Admin can manage tags
    • Admin can search conversation thread
    • Admin can do many things on this page
    OA Plus

    View full-size slide

  7. If a new user sent a message to the channel
    should be able to search conversation thread
    conversion thread should be created
    OA Plus

    View full-size slide

  8. Thread
    Server
    Insert messages
    Webhook
    Server
    Elasticsearch
    HBASE
    Consumer
    Consumer
    Consumer
    Insert threads
    Elasticsearch
    mongoDB
    Consumer
    Consumer
    Consumer
    Consumer
    Web Socket
    Consumer
    Consumer
    We only want to test this
    New message
    event
    Kafka Kafka Kafka
    BEHIND THE SCENE

    View full-size slide

  9. E2E Test
    Integration Test
    Unit Test

    View full-size slide

  10. Thread
    Server
    Insert messages
    Webhook
    Server
    Elasticsearch
    HBASE
    Consumer
    Consumer
    Consumer
    Insert threads
    Elasticsearch
    mongoDB
    Consumer
    Consumer
    Consumer
    Consumer
    Web Socket
    Consumer
    Consumer
    We only want to test this
    New message
    event
    Kafka Kafka Kafka

    View full-size slide

  11. 1. Put test event to Kafka
    2. Wait services to finish processing
    3. Check data in mongoDB and Elasticsearch
    4. Search thread via Thread server
    Thread
    Server
    Insert threads
    Elasticsearch
    mongoDB
    Consumer
    Consumer
    Consumer
    Kafka Kafka
    Put test event
    Check data
    Test API

    View full-size slide

  12. Events
    Events
    Event
    Broker
    Servers
    Consumers
    Databases
    Servers

    View full-size slide

  13. docker-compose.yaml
    Makefile
    mongoDB
    Elasticsearch
    Compose
    Docker
    $ make test.integration
    Thread
    Server
    Kafka
    Test Suite
    Consumer Consumer Consumer
    Consumer Consumer
    mongoDB as
    Migrator
    Run migration scripts
    Delete data
    Delete data Recreate Topics
    Put Event

    View full-size slide

  14. TEST SUITE
    BeforeSuite(func() {
    util = InitUtil(”./integration-test.env", ”./alpha.env")
    util.ResetData()
    util.StartThreadConsumerMessage()
    util.StartThreadServerSave()
    util.StartThreadConsumerIndexSearch()
    util.StartThreadServerSearch()
    util.StartThreadConsumerUpdateManualTags()
    util.StartThreadServer()
    util.StartConsumerUserEvent()
    })

    View full-size slide

  15. TEST SUITE
    // StartThreadServerSave start `thread-server-save`
    func (i *Util) StartThreadServerSave() {
    c := NewKafkaConsumer(TopicThreadSaveBuffer)
    threadDB := InitMongoDB(MongoDBNameThread)
    config := &serverSaveProcessor.Config{
    AccountClient: i.accountCli,
    TopicProducer: TopicThreadEvent,
    LineAPI: serverSaveAPI.NewFetchProfile(&http.Client{}),
    }
    p := serverSaveProcessor.NewService(config)
    c.Use(p)
    go func() {
    if err := c.Consume(); err != nil {
    logger.Panic(errors.Wrap(err, "[StartThreadServerSave]: unable to start consume message"))
    }
    }()
    }

    View full-size slide

  16. TEST SUITE
    When("Message and tag events produced", func() {
    BeforeEach(func() {
    util.ProduceKafkaMsgs("./testdata/mock-event.json", TopicMessageEvent)
    time.Sleep(10 * time.Second) // make sure consumer finished all messages
    })
    It("should create threads and tags", func() {
    By("getting mongodb document")
    // CODE HERE
    By("getting data from elasticsearch")
    // CODE HERE
    By(“get search result from thread server”)
    // CODE HERE
    })
    })

    View full-size slide

  17. INTEGRATE WITH CI/CD

    View full-size slide

  18. BENEFITS 😇
    • Can test that unit test and E2E can’t
    • Flexible
    • Can test each PR
    • More confidence

    View full-size slide

  19. LIMITATIONS 😓
    • External API/Services dependencies
    • Cannot test with large data set
    • Many services/test cases can take time
    • Test flakiness
    • Not easy investigate sometime
    • No framework

    View full-size slide

  20. 3 KEY TAKEAWAYS 🤓
    1. Test cases should be independent
    2. Don’t integrate too many services
    3. Test should not take much time

    View full-size slide