Save 37% off PRO during our Black Friday Sale! »

Integration Test in Event Driven Architecture

Integration Test in Event Driven Architecture

B29f42636a5f1249b640473d49aa4514?s=128

LINE Developers Thailand

October 16, 2021
Tweet

Transcript

  1. None
  2. INTEGRATION TEST IN EVENT DRIVEN ARCHITECTURE

  3. EVENT DRIVEN ARCHITECTURE Event Event Event Producer Event Event Event

    Consumer Consumer Consumer Event Broker Topic 1 Topic 2 Topic 3
  4. Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database

    Unit Test Unit Test Unit Test Unit Test Unit Test
  5. Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database

    E2E Test
  6. Event Broker Consumer Server Event Broker Event Broker Frontend Consumer

    Consumer Consumer Database Event Broker Server Consumer Consumer Database Event Broker Consumer Event Broker Consumer Event Broker Consumer Database E2E test is getting more difficult Event Broker Consumer
  7. OA Plus

  8. • Admin can chat with followers • Admin can manage

    tags • Admin can search conversation thread • Admin can do many things on this page OA Plus
  9. If a new user sent a message to the channel

    should be able to search conversation thread conversion thread should be created OA Plus
  10. Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer

    Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka BEHIND THE SCENE
  11. E2E Test Integration Test Unit Test

  12. Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer

    Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka
  13. 1. Put test event to Kafka 2. Wait services to

    finish processing 3. Check data in mongoDB and Elasticsearch 4. Search thread via Thread server Thread Server Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Kafka Kafka Put test event Check data Test API
  14. Events Events Event Broker Servers Consumers Databases Servers

  15. docker-compose.yaml Makefile mongoDB Elasticsearch Compose Docker $ make test.integration Thread

    Server Kafka Test Suite Consumer Consumer Consumer Consumer Consumer mongoDB as Migrator Run migration scripts Delete data Delete data Recreate Topics Put Event
  16. TEST SUITE BeforeSuite(func() { util = InitUtil(”./integration-test.env", ”./alpha.env") util.ResetData() util.StartThreadConsumerMessage()

    util.StartThreadServerSave() util.StartThreadConsumerIndexSearch() util.StartThreadServerSearch() util.StartThreadConsumerUpdateManualTags() util.StartThreadServer() util.StartConsumerUserEvent() })
  17. TEST SUITE // StartThreadServerSave start `thread-server-save` func (i *Util) StartThreadServerSave()

    { c := NewKafkaConsumer(TopicThreadSaveBuffer) threadDB := InitMongoDB(MongoDBNameThread) config := &serverSaveProcessor.Config{ AccountClient: i.accountCli, TopicProducer: TopicThreadEvent, LineAPI: serverSaveAPI.NewFetchProfile(&http.Client{}), } p := serverSaveProcessor.NewService(config) c.Use(p) go func() { if err := c.Consume(); err != nil { logger.Panic(errors.Wrap(err, "[StartThreadServerSave]: unable to start consume message")) } }() }
  18. TEST SUITE When("Message and tag events produced", func() { BeforeEach(func()

    { util.ProduceKafkaMsgs("./testdata/mock-event.json", TopicMessageEvent) time.Sleep(10 * time.Second) // make sure consumer finished all messages }) It("should create threads and tags", func() { By("getting mongodb document") // CODE HERE By("getting data from elasticsearch") // CODE HERE By(“get search result from thread server”) // CODE HERE }) })
  19. INTEGRATE WITH CI/CD

  20. BENEFITS 😇 • Can test that unit test and E2E

    can’t • Flexible • Can test each PR • More confidence
  21. LIMITATIONS 😓 • External API/Services dependencies • Cannot test with

    large data set • Many services/test cases can take time • Test flakiness • Not easy investigate sometime • No framework
  22. 3 KEY TAKEAWAYS 🤓 1. Test cases should be independent

    2. Don’t integrate too many services 3. Test should not take much time