Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Collecting and transforming data in WalkMe

Collecting and transforming data in WalkMe

In this talk we examine how WalkMe, the guidance and engagement platform empowers product teams, partners and clients by utilizing and maintaining scalable and cost effective data architecture in order to build innovative products and services delivered based on big amounts of online user behavior data.

Avatar for David Ilievsky

David Ilievsky

December 08, 2014
Tweet

Other Decks in Technology

Transcript

  1. Enterprise Class Guidance & Engagement Platform WalkMe - The Enterprise

    Class Guidance and Engagement Platform © Copyright 2014 WalkMe Inc. Confidential
  2. About myself  My name is David Ilievsky  I’m

    a Full Stack Developer  I work in WalkMe since its early stages  I’m a Software architecture and design freak  Currently I’m running the WalkMe Labs and Devops teams © Copyright 2014 WalkMe Inc. Confidential [email protected] https://www.facebook.com/david.ilievsky https://www.linkedin.com/in/davidilievsky
  3. The Company  Founded 2011, Launched in April 2012. 

    Offices in San Francisco and Tel Aviv, 110+ employees  Hundreds of paying customers from various industries & verticals. © Copyright 2014 WalkMe Inc. Confidential
  4. The Platform © Copyright 2014 WalkMe Inc. Confidential  One

    of a kind Platform to guide and engage prospects, customers, employees or partners through any Web experience.  WalkMe Reduces Complexity to Empower Advanced Selling, Support, Training and improved user experience  Using WalkMe increases conversion rates, reduces support costs, accelerates training and improves customer experience  No integration or changes to the underlying website required.
  5. The Basics #2 WalkMe customer adds the WalkMe JavaScript code

    to his website. The customer publishes the WalkThrus to his users.
  6.  Maximum availability for client side experience (100%)  Low

    latency for fetching the WalkMe files  Very high traffic volume from our customers users (over 1B requests a month)  Analyzing billions of records for WalkMe analytics The Challenges
  7. More Challenges #1  Analyzing WalkMe usage by counting customer’s

    unique visitors  Need to audit a request per end user (even if not using WalkMe)  Need to handle the traffic of all our customers (billions of requests per month)  Startup company, basic DBA knowledge
  8. Data Collection  Tracking unique users by sending requests to

    Cloudfront  Logs files are being analyzed once a day by xplenty  Aggregated data is sent to our RDS
  9. More Challenges #2  Analytics dashboard queries too slow 

    Joins between huge tables (hundreds GBs of data)  Over different time ranges  Real-time analytics data is crucial  Daily Data aggregation?
  10. Data aggregation  Xplenty daily job aggregates the data and

    sends it to the RDS  Very easy setup  Real-time data comes from the regular DB while historic data comes from the aggregated one