Share
Embed
Integration Test in Event Driven Architecture
LINE Developers Thailand
October 16, 2021
Transcript
22 slides
-
EVENT DRIVEN ARCHITECTURE Event Event Event Producer Event Event Event
Consumer Consumer Consumer Event Broker Topic 1 Topic 2 Topic 3 -
Server Event Broker Event Broker Frontend Consumer Consumer Consumer Database
Unit Test Unit Test Unit Test Unit Test Unit Test -
Event Broker Consumer Server Event Broker Event Broker Frontend Consumer
Consumer Consumer Database Event Broker Server Consumer Consumer Database Event Broker Consumer Event Broker Consumer Event Broker Consumer Database E2E test is getting more difficult Event Broker Consumer -
• Admin can chat with followers • Admin can manage
tags • Admin can search conversation thread • Admin can do many things on this page OA Plus -
If a new user sent a message to the channel
should be able to search conversation thread conversion thread should be created OA Plus -
Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer
Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka BEHIND THE SCENE -
Thread Server Insert messages Webhook Server Elasticsearch HBASE Consumer Consumer
Consumer Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Consumer Web Socket Consumer Consumer We only want to test this New message event Kafka Kafka Kafka -
1. Put test event to Kafka 2. Wait services to
finish processing 3. Check data in mongoDB and Elasticsearch 4. Search thread via Thread server Thread Server Insert threads Elasticsearch mongoDB Consumer Consumer Consumer Kafka Kafka Put test event Check data Test API -
docker-compose.yaml Makefile mongoDB Elasticsearch Compose Docker $ make test.integration Thread
Server Kafka Test Suite Consumer Consumer Consumer Consumer Consumer mongoDB as Migrator Run migration scripts Delete data Delete data Recreate Topics Put Event -
TEST SUITE BeforeSuite(func() { util = InitUtil(”./integration-test.env", ”./alpha.env") util.ResetData() util.StartThreadConsumerMessage()
util.StartThreadServerSave() util.StartThreadConsumerIndexSearch() util.StartThreadServerSearch() util.StartThreadConsumerUpdateManualTags() util.StartThreadServer() util.StartConsumerUserEvent() }) -
TEST SUITE // StartThreadServerSave start `thread-server-save` func (i *Util) StartThreadServerSave()
{ c := NewKafkaConsumer(TopicThreadSaveBuffer) threadDB := InitMongoDB(MongoDBNameThread) config := &serverSaveProcessor.Config{ AccountClient: i.accountCli, TopicProducer: TopicThreadEvent, LineAPI: serverSaveAPI.NewFetchProfile(&http.Client{}), } p := serverSaveProcessor.NewService(config) c.Use(p) go func() { if err := c.Consume(); err != nil { logger.Panic(errors.Wrap(err, "[StartThreadServerSave]: unable to start consume message")) } }() } -
TEST SUITE When("Message and tag events produced", func() { BeforeEach(func()
{ util.ProduceKafkaMsgs("./testdata/mock-event.json", TopicMessageEvent) time.Sleep(10 * time.Second) // make sure consumer finished all messages }) It("should create threads and tags", func() { By("getting mongodb document") // CODE HERE By("getting data from elasticsearch") // CODE HERE By(“get search result from thread server”) // CODE HERE }) }) -
BENEFITS 😇 • Can test that unit test and E2E
can’t • Flexible • Can test each PR • More confidence -
LIMITATIONS 😓 • External API/Services dependencies • Cannot test with
large data set • Many services/test cases can take time • Test flakiness • Not easy investigate sometime • No framework -
3 KEY TAKEAWAYS 🤓 1. Test cases should be independent
2. Don’t integrate too many services 3. Test should not take much time