Customer Journey

Transcript

1. Turkcell Customer Journey Burak IŞIKLI - Turkcell @burakisikli
2. Hello… Burak IŞIKLI MSc (Computer Science, 2014), Gebze Technical University Developer, BIS, Turkcell Technology since 2009 burak.isikli at {gmail.com}, {turkcell.com.tr} http://tr.linkedin.com/in/burakkk @burakisikli burakisikli.wordpress.com
3. Architecture RDBMS Logs Sqoop Rsync Internal tools HADOOP Oracle Dashboards Archive Ad-Hoc Analysis Mining
4. Customer Journey • Use of products, services, and mobile applications • Receiving advertisements • Contacting call centers • Visiting retail stores and Web pages
5. Moving
6. Requirements-1  Automatizing the extraction and loading layer  Depedency and impact analysis  Create an external Hive table from extracted table  Parallel extraction via SQL query
7. Speed May 2014 July 2014 Nov. 2014 1 TB 1.5 TB 2 TB The amount of data to be processed is rapidly increasing. Data Time
8. Adapt to speed
9. Requirements-1  Automatizing the extraction and loading layer  Depedency and impact analysis  Create an external Hive table from extracted table  Parallel extraction via SQL query X No Hcatalog X No external table support on Sqoop X No parallel extraction via SQL query
10. Tuna  Automatizing the extraction and loading layer  Depedency and impact analysis  Create an external Hive table  Parallel extraction via SQL query TUNA
11. Real World Use Case: Dependency • Which file is the table X loading from?  SVN  Sqoop
13. Real World Use Case: Dependency • Which file is the table X loading from?  SVN  Sqoop What about you?
14. Real World Use Case: Dependency • Which file is the table X loading from?  Unix commands(locate, find)  No hcatalog..etc
15. Real World Use Case: Dependency • Which file is the table X loading from?  Solution: Tuna
16. No development
17. How? • Automatizing the extraction and loading layer  Answer is simple: Smart rule based engine
18. Tuna Flow 1. Get the metadata 2. Validate the data and if necessary change it 3. Create and execute the sqoop query 4. Create and execute the hcatalog query
19. Tuna Features • Independent of database type (Oracle, Teradata...etc)  Store source on a different table with properties
20. Tuna Features • Depedency and impact analysis  Using RDBMS tables for metadata management  Using GUI
21. Tuna Features • Create an external Hive table  Getting metadata from source RDBMS table and map to Hive table  Create a view when source query is given.  No Hcatalog support for external table
22. Tuna Features • Parallel extraction via SQL query  Create a temporary table using the source query  If it’s not allowed, extract serially
23. Tuna Features • Hadoop to hadoop transfer  Resolve the hadoop version  Use distcp
24. Tuna Features • Delta extraction  Look up the maximum delta value  Extract using that value  Find out and store the new maximum delta value
25. Requirements-2  Validate the hive query  Execute the hive query
26. Dolphin  Validate a hive query  Execute a hive query DOLPHIN
27. Dolphin Features • Validate a hive query Possible solutions:  SQL parser such as ANTLR..etc.  Hive command line like EXPLAIN
28. Dolphin Features Validate a hive query Possible solutions: • SQL parser such as ANTLR... etc.  Short execution time X Long development time • Hive command line like EXPLAIN  Easy to understand&short development time X Longer execution time
29. Dolphin Features • Execute a hive query Possible solutions:  Using Hive client  Hive command line
30. Open source • To be released soon on Github repository • Check out our github repository github.com/Turkcell github.com/Turkcell/Shrimp -> Our pig library • If you like this presentation, vote on at Hadoop Summit 2015 Europe
31. Thanks