Why Nielsen Company's Global Buy Platform Relies on Couchbase

WHY NIELSEN COMPANY'S GLOBAL BUY PLATFORM RELIES ON
COUCHBASE DARRELL PRATT ARCHITECTURE LEADER

ABOUT NIELSEN

Help our clients have the most complete understanding of
consumers worldwide. OUR MISSION

Copyright ©2012 The Nielsen Company. ConﬁdenQal and proprietary. 4
OUR ECOSYSTEM COLLABORATIVE DATA

NIELSEN ANSWERS ON DEMAND Flexibility • Dynamic “on-the-fly” processing engine • On-Demand products, markets, periods, buyer groups • User role-based reporting • Custom product definitions, hierarchies and characteristics Speed to Insights • Expedited reporting • Roadmaps and guided analysis • Dynamic reporting Integration • Consistent access channel • Internal and external data sources • Support for client business processes

Consumer foresight for faster, smarter, more
conﬁdent decisions OUR PROMISE to drive growth

CONSUMER DATA

SCAN, PANEL AND LOYALTY •  Scan •  Point of Sales data from 1000’s of retailers •  Weekly data •  Disaggregated, Anonymous data •  Billions of records per week •  Panel •  Data from more than 250,000 households across 25 countries •  Similar to Nielsen Families from View •  Trip and Demographic data •  Millions of records a week •  Loyalty •  Loyalty card data from retailers •  Basket level transacQon data – received daily from thousands of stores •  Some demographic data •  100’s of Millions of items weekly

WHO IS OUR CONSUMER

OUR CLIENTS •  Manufacturers -‐ Kra\, Procter & Gamble, others •  Measure product success •  Understand consumer behaviors •  Target new products or promoQons •  IdenQfy new product opportuniQes •  Product pricing •  Retailers – Safeway, Tesco, Walmart •  Understand consumer buying behavior •  Store performance in market •  Comparison to compeQtors •  Product pricing

OUR REPORTING APPLICATION

REPORT BUILDER

REPORT PLAYER

DATA SELECTION

FEATURES •  Single front-‐end web applicaQon which interfaces with the disparate back-‐end data sources •  Advanced BI capabiliQes •  User expressions •  CondiQonal formabng •  Smart text •  Smart linking of report objects •  Very few limits to what user can request with regards to data •  Most reports to run under 2 minutes maximum •  Loading of applicaQon with most data under 5 seconds

APPLICATION SPECIFICS

HIGH LEVEL VIEW Presenta(on Logic Storage Service Report2Builder Report2Player Data2Selector Spring2MVC Spring2IOC Couchbase Oracle Job2 Management Data22Access2 Service Caching PorColio2 Manager

ARCHITECTURAL VIEW Web Front End Portal ReportApp Workspace Admin Java Middle Tier Spring IOC + Custom Couchbase Cluster ElasticSearch Oracle RAC Netezza Tibco ESB

TECHNOLOGY STACK •  UI built on Sencha Ext JS •  AM Charts used for charQng •  D3 used in some edge case chart types •  Middle Qer composed of Spring MVC and Spring IOC •  JSON REST endpoints through conﬁguraQon •  XML to JSON conversion where needed •  SOA Tier using Tibco AMX 3.2 •  Couchbase 2.1 – Storage, Caching and Search •  Hudson, ArQfactory, Gradle, Jasmine, JS Duck

WHY COUCHBASE

WE STARTED WITH A SEARCH FOR A CACHING SOLUTION

SOLUTION REQUIREMENTS •  Our needs •  Scalability •  Shared cache/storage for separate applicaQons •  Speed •  Out of JVM process •  Support •  As an enterprise, we need 24x7 support •  Our wants •  Document storage •  Map/reduce views •  Full text search

WHY NO-‐SQL •  RelaQon Data Model Overload •  Complexity of objects in system causes churn in DB models •  Poor performance due to complexity •  Need to get out of business of data transformaQons •  Flexibility of data model is near number one requirement •  Scalability with modest hardware and ease •  Data Sharding and replicaQon for reliability •  JSON encoding •  Used throughout UI, important to store as such

COUCHBASE – USAGE •  Storage of naQve JSON data from applicaQon •  User customizaQons of reports •  Report deﬁniQons •  Request instances – Data selecQons •  BI Responses •  Metadata change management •  CharacterisQcs can and do change weekly •  Views created to track user usage and items aﬀected by these changes

DEALING WITH REPORTING DATA

MAKE IT RESPONSIVE •  Asynchronous UI •  ReporQng data gets BIG •  Breaking up a report into objects •  Asynchronously store chunked data in Couchbase •  UI only requests chunk needed

MOVING DATA EFFICIENTLY •  Life of a request for data moves through several systems •  Web, Tibco AMX, Tibco EMS, Composite, Database •  Use Couchbase as a document storage system •  Enables a pass by reference methodology •  Storage of data in format closest to what is displayed to user •  True persistent storage and in-‐memory performance

DEALING WITH CHANGE

DIMENSIONAL DATA CHANGES •  Our data is made up dimensions (Product, Market, Period, Fact…) with each dimension described by CharacterisQcs •  CharacterisQc data is large and changing •  New products introduced •  Human error on ingest •  Manufacturers change their minds •  Changes occur weekly if not daily •  Changes here create waves throughout the system

COPING WITH CHANGE •  Metadata change management process •  Catch the differenQal changes at incepQon •  Send those changes through system on ESB •  User saved data needs to be updated or invalidated •  Saved selecQons •  Saved reports •  Segment definiQons •  All of these items contain this characterisQc data •  Before Couchbase -‐> Stored as CLOBS in Oracle •  Full table scans and programs to read all data and change where it was found •  With Couchbase, MapReduce Views created to easily find items with reference to characterisQc data with changes •  Easy to find, easy to fix. Huge Qme savings

LOOKING FORWARD

WHAT’S NEXT •  Couchbase as a ﬁrst class data storage applicaQon from mainframe acquisiQon •  Full storage of metadata in Couchbase •  Map/reduce views to capture staQsQcs on data usage by clients •  PredicQve analyQcs using collaboraQve data from Couchbase views

Why Nielsen Company's Global Buy Platform Relie...

Why Nielsen Company's Global Buy Platform Relies on Couchbase

Darrell Pratt

Transcript