events within the software. • Supports analysis of processing procedures in past • Helps administrators to find errors in software • Log Management • The total volume of logs increases as the number of logs increases. • Millions of logs need to be retrieved in a short time. 2 Software March 27, 2022 10:24:13 Started app March 27, 2022 10:25:40 Communicated node1 March 27, 2022 10:25:40 Stored file1 March 27, 2022 10:35:00 Stopped app Logs Administrator Search
for large scale data processing • Scatter: The root node splits a task into several sub-tasks, and scatters sub-tasks to leaf nodes. • Gather: Leaf nodes return a result of the sub-task to the root node. • Prerequisite • Applies Scatter-Gather Pattern to log search for distributed tracing 4 Leaf nodes Root node Scatter Gather Admin
all logs in parallel. 5 Search response time The volume of accessed logs corresponding As the volume of accessed logs on search increases, Search response time increases. As the total volume of logs increases, Search response time increases. ◆ Reduction of search response time is useful for trouble-shoot. Short response time reduces the total time for repairing troubles. Needs: The method for reduction the volume of logs on search
distributed tracing • Reduces the number of accessed log data on Search. • Focuses on time-series access patterns of log data and service relations 6 B Microservices A C B Service Relations A C Logs Service Discovery Blocks Placement Rule Leaf nodes Root node Moving blocks by placement rule Istio Admin Search Query Store Phase Search Phase Clustering by datetime & microservice Block List
correspond to Chronological order among logs. 7 Search target period Accessed Blocks in search targets Unaccessed Blocks in search targets Datetime Blocks: Microservice A Blocks: Microservice B Service A sends a request to Service B. =Time-series access patterns Service B writes a log message after Servce A writes a log message. A B Microservices Example) (1)Request (2)Response A B Service Relations Clustering by datetime & microservice Log: sent Reduces the number of accessed blocks on Search Phase Log: received
size. • The number of log messages per block is homogeneous. • The file size which can be read and written simultaneously depends on Disk I/O performance per leaf node. • Block size has to be calculated from Disk I/O performance. • One of the methods is using iostat command which returns I/O performance. 10