LIGHTWEIGHT BUSINESS INTELLIGENCE – WITH – RUBY, RAILS, & MONGODB COREY EHMKE @BANTIK

WHO AM I? A developer with a long memory and a longer career Sr. developer at Apartments.com Co-founder of LGBTech A lifelong learner and autodidact

WHAT IS BUSINESS INTELLIGENCE? Collection & organization of mission-critical knowledge Historical view of business operations Tools to support decision making

PRELUDE: THE MARRIAGE OF BUSINESS & TECHNOLOGY

The foundation on which the fun stuff is built A science unto itself Not of great interest to most developers INFRASTRUCTURE

Enable the delivery of products and services Optimize business processes Provide leverage for scaling the business APPLICATIONS

DATA Exists to support critical business decisions Better data leads to better decisions ∴ Data must be as timely and useful as possible

WHEN DEVELOPERS MAKE FRIENDS WITH DATA...

BUSINESS INTELLIGENCE: A FIT IN THREE ACTS

ACT 1: THE NAÏVE APPROACH

THE NAÏVE APPROACH Reporting out of the transactional database Raw SQL embedded in your code for “performance” Granting direct database access to stakeholders

THE NAÏVE APPROACH: SHORTCOMINGS Fighting the schema with complex joins Poor performance Impact on production resources Chance of catastrophic data loss due to...

DON’T TRUST THIS GUY WITH YOUR DATABASE.

ACT 2: THE ENTERPRISE APPROACH

THE ENTERPRISE APPROACH Distinct infrastructure/stack Nightly ETL (extract, transform, load) processes Schema designed for reporting Combination of static and dynamic reports

THE ENTERPRISE APPROACH: SHORTCOMINGS 24 hour delay in data availability Expensive to configure and maintain Requires highly specialized resources Hard to change your mind or adapt Enterprise-y

VERY, VERY ENTERPRISE-Y

DON’T TRUST THESE GUYS WITH YOUR DATABASE, EITHER.

TRADITIONAL BI: THE WRONG HAMMER

TRADITIONAL BI: THE WRONG HAMMER Waterfall approach Painful data migrations Brittle ETL processes No automated testing Logic embedded in the data store “Retro” user interfaces

IN SHORT... TRADITIONAL BI DOESN’T FEEL AGILE.

ACT 3: LIGHTWEIGHT BI

THE FOUR NOBLE GOALS OF THE LIGHTWEIGHT BI APPROACH Provide real-time data to support decisions Leverage familiar technology & infrastructure Use existing development staff Support an iterative, agile approach

TRUST YOUR DEVELOPERS.

GETTING STARTED Collaborate! Determine the KPIs that actually matter Figure out what questions the business is asking on a daily basis

TURN INFERENCES INTO FACTS Find a way to tell a story with your data Design your schema based on facts De-normalize like a boss

PRESENT ANSWERS Provide a central, single source of truth Present the data wherever it’s needed Dashboard design is harder than you think Plan for iterations & ongoing collaboration

PLEASE GODS NO.

DON’T MAKE THESE.

BRING RUBY TO THE PARTY Makes agile, test-driven development easy Quickly deploy new apps Plenty of visualization libraries Powerful SQL & NoSQL ORMs Great data munging capabilities

LEVERAGE MONGODB Flexible and dynamic schemas Support for native datatypes Powerful querying, map/ reduce, and aggregation Fast and performant Easy to scale up

GO AHEAD... TRY THIS AT HOME LIGHTWEIGHT BI TECHNIQUES

PARALLEL DB DEPLOYMENT Modern frameworks support multiple ORMs Use SQL for transactions Use NoSQL for reporting Business logic in your apps, not in your database

STATISTICAL MODELS Collections of facts, not attributes Data spanning deep and wide object graphs De-normalized and optimized for reporting

module  MemberDataProfiles    class  Performance        include  Mongoid::Document        include  Mongoid::Timestamps        field  :member_id,                                  :type  =>  Integer        field  :annual_trunk_frequency,        :type  =>  Integer,    :default  =>  0        field  :average_trunk_value,              :type  =>  Float,        :default  =>  0.0        field  :is_customer,                              :type  =>  Boolean,    :default  =>  false        field  :keep_rate,                                  :type  =>  Float,        :default  =>  0.0        field  :last_transaction_date,          :type  =>  Date        field  :member_creation_date,            :type  =>  Date        field  :total_value,                              :type  =>  Float,        :default  =>  0.0        field  :total_number_of_trunks,        :type  =>  Integer,    :default  =>  0    end end

STREAMING ETL Event-triggered, continuous data extraction Calculations are defined in code rather than SQL or ETL scripts Allow resource-intensive data munging to happen in the background Provide near-real-time data

APIS EVERYWHERE Provide easy access to your data Allow reuse of data in novel ways Quickly build dashboards and data explorers Use the Faceted gem to make building APIs easy

EPILOGUE: ENTER THE LEVIATHAN

A TYPICAL SMALL APP ECOSYSTEM Persistence Service Postgres Ops Members Sales iOS App Inventory Outfitting Messaging Dashboard Reporting Service MongoDB Business Objects MySQL

LEVIATHAN Records events from all applications Subscribes to all message queues Collects and displays real-time data Browse, search, & drill-down interface Longitudinal analysis with dynamic cohorts

CRITICAL DATA POINTS Marketing campaign performance Member on-boarding funnels Order lifecycles Customer service interactions Product performance

class  Event    include  Mongoid::Document    include  Mongoid::Timestamps    field  :label    field  :application    field  :details,  :type  =>  Hash,  :default  =>  {}    def  self.record!(label,  application,  params={})        Event.create(            :label  =>  label,            :application  =>  application,            :details  =>  params        )    end    def  self.search_details(criteria={})        where("details.#{criteria.keys.first}"  =>  /#{criteria.values.first}/i)    end end

No content

No content

No content

No content

No content

No content

No content

THANKS! Corey Ehmke @bantik bantik.github.com