Integration Test in Event Driven Architecture

Transcript

2. INTEGRATION TEST IN EVENT DRIVEN ARCHITECTURE
3. EVENT DRIVEN ARCHITECTURE Event Event Event Producer Event Event Event Consumer Consumer Consumer Event Broker Topic 1 Topic 2 Topic 3
4. Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database Unit Test Unit Test Unit Test Unit Test Unit Test
5. Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database E2E Test
6. Event Broker Consumer Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database Event Broker Server Consumer Consumer Database Event Broker Consumer Event Broker Consumer Event Broker Consumer Database E2E test is getting more difficult Event Broker Consumer
7. OA Plus
8. • Admin can chat with followers • Admin can manage tags • Admin can search conversation thread • Admin can do many things on this page OA Plus
9. If a new user sent a message to the channel should be able to search conversation thread conversion thread should be created OA Plus
10. Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka BEHIND THE SCENE
11. E2E Test Integration Test Unit Test
12. Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka
13. 1. Put test event to Kafka 2. Wait services to finish processing 3. Check data in mongoDB and Elasticsearch 4. Search thread via Thread server Thread Server Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Kafka Kafka Put test event Check data Test API
14. Events Events Event Broker Servers Consumers Databases Servers
15. docker-compose.yaml Makefile mongoDB Elasticsearch Compose Docker $ make test.integration Thread Server Kafka Test Suite Consumer Consumer Consumer Consumer Consumer mongoDB as Migrator Run migration scripts Delete data Delete data Recreate Topics Put Event
16. TEST SUITE BeforeSuite(func() { util = InitUtil(”./integration-test.env", ”./alpha.env") util.ResetData() util.StartThreadConsumerMessage() util.StartThreadServerSave() util.StartThreadConsumerIndexSearch() util.StartThreadServerSearch() util.StartThreadConsumerUpdateManualTags() util.StartThreadServer() util.StartConsumerUserEvent() })
17. TEST SUITE // StartThreadServerSave start `thread-server-save` func (i *Util) StartThreadServerSave() { c := NewKafkaConsumer(TopicThreadSaveBuffer) threadDB := InitMongoDB(MongoDBNameThread) config := &serverSaveProcessor.Config{ AccountClient: i.accountCli, TopicProducer: TopicThreadEvent, LineAPI: serverSaveAPI.NewFetchProfile(&http.Client{}), } p := serverSaveProcessor.NewService(config) c.Use(p) go func() { if err := c.Consume(); err != nil { logger.Panic(errors.Wrap(err, "[StartThreadServerSave]: unable to start consume message")) } }() }
18. TEST SUITE When("Message and tag events produced", func() { BeforeEach(func() { util.ProduceKafkaMsgs("./testdata/mock-event.json", TopicMessageEvent) time.Sleep(10 * time.Second) // make sure consumer finished all messages }) It("should create threads and tags", func() { By("getting mongodb document") // CODE HERE By("getting data from elasticsearch") // CODE HERE By(“get search result from thread server”) // CODE HERE }) })
19. INTEGRATE WITH CI/CD
20. BENEFITS 😇 • Can test that unit test and E2E can’t • Flexible • Can test each PR • More confidence
21. LIMITATIONS 😓 • External API/Services dependencies • Cannot test with large data set • Many services/test cases can take time • Test flakiness • Not easy investigate sometime • No framework
22. 3 KEY TAKEAWAYS 🤓 1. Test cases should be independent 2. Don’t integrate too many services 3. Test should not take much time