Data & Analytics Framework - Raffaele Lillo, Chief Data Officer of Digital Transformation Team

DAF: Data & Analytics Framework. How public sector can profit from its immense asset, data. Keynote at Traffic Police 2017 (TRAP-2017)

Transcript

1. Data & Analytics Framework: how public sector can profit from its immense asset, data RAFFAELE LILLO Chief Data Officer @ Digital Transformation Team —
2. Who we are and what we do • Digital Transformation Team • Projects Data & Analytics Framework (DAF) • Goals • Ok, but what is DAF? • Projects • Some Architectural Highlights Q&A — DAF
3. Digital Transformation Team
4. The “operating system” of the country: a series of fundamental blocks upon which services for citizens, the Public Administration, and enterprises are built with modern digital products. Vision
5. Make public services for citizens and businesses accessible in an easy manner, via a mobile first approach, with reliable, scalable and fault tolerant architectures, based on clearly defined APIs; support the different central and local government departments in making the best and most data driven decisions, thanks to the adoption of big data and machine learning techniques. Mission
6. —
7. • DAF: Data & Analytics F. • ANPR • Security - Resp. Disclosure • SPID • PagoPA • API Ecosystem • E-Procurement • Developer Community • Digital Citizenship • LexDatafication • ... Projects • 7 "Comuni": Bari, Firenze, Milano, Palermo, Roma, Torino, Venezia • Software House in-house: Sogei, InfoCamere, ACI Informatica, IPZS • PAC: Agenzia Entrate, ISTAT, MISE, MIUR, ANAC, CdC… • PAL: Roma, Milano, Torino, Firenze... Partners
8. DAF: Data & Analytics Framework
9. Data & Analytics Framework (DAF) is a combination of: • A Big Data Platform to centralize and store (data lake), manipulate and standardize (data engine), re-distribute (API & Data Applications) data and insights. • A Data Team (data scientists + data engineers) which uses and evolve the Big Data Platform to analyze data, create ML models and build data applications and data viz. Give us data and a platform...
10. Interoperability (aka Get out of the Silos!) Public data is… public and all PP.AA. should have access to it Democratizing Data (aka Open Data, API & Data Viz) Data should be open (when legally possible), accessible by anyone (and anything) and insightful Data Products (aka deliver value & insights) Machine Learning in interconnected software applications Crowdsourcing (aka data is everywhere, let’s help us out) Citizens (esp. civic hackers) contribute to the surfacing of knowledge … and we shall move the PA
11. Organizational and Managerial Challenge Central Data Office and federated analytics teams Human Resources Data Scientists & Data Engineers to get knowledge from data Technology This is the least complicated one, but still fundamental. Legislative Challenge Balancing Privacy and Public Interest Data Driven Policy needs… Data (and Data Scientists)
12. Introduction of DAF in Piano Triennale 2017-2019 DAF is one of the building blocks of the official document setting the strategy for digitalization of the PA, and signed by the Prime Minister DAF prototype development TD started the development of the platform from scratch around March ‘17, and released an Alpha version the first week of October ‘17 Experimental phase We started working with a selected number of PA to showcase DAF, test it and listen to PA’s needs so to fine-tune the platform before final release Institutionalization of DAF Introduce by law the role of a central data office for the entire PA Our Strategy
13. Mission: Data driven decision making in efficient ways Support PA at all levels to implement informed policies, both ex ante (policy formulation) and ex post (policy monitoring and fine tuning). Centralize common & non-domain specific tasks Provide general purpose data platform once and for all, efficiency in standard data processes, let PA focus on domain specific tasks / analysis Economy of scope towards a center of excellence Reach proper dimension to develop and acquire expensive and idiosyncratic capabilities, and share them with all PA Design and coordinate implementation of Data Policies Help interoperability and usage of state-of-the-art standards and processes in data management and analysis. Stimulates research and collaboration. End Goal: Chief Data Office for the PA
14. High-level Architectural Design Data Ingestion Persistence & Offline analysis (analysis, data processing, model training) Real-Time data processing Operational DB for API and Data applications Communication layer - API
15. (Less) High-Level Architectural Design Hadoop cluster for distributed persistence and processing Kubernetes cluster manages dockerized microservices and external applications. Core Managers: microservices managing core functionalities of DAF External applications natively integrated in DAF Unique identity management system, integrated with HDFS
16. Dataportal - Public Version https://dataportal.daf.teamdigitale.it
17. Dataportal - Private Version https://dataportal-private.daf.teamdigitale.it
18. Data Sources People Firms Smart Cities Fiscality Pension Education Healthcare Cultural events Public Transportatio n Corporate events Energy Traffic Corporate info Balance Sheet Crime Real estate registry
19. Machine Learning Based Applications (aka Data Products) Lex Datafication & Citizen Assistant, Fraud Detection, Citizen Recommendation Engine, Spending Check, Leading Indicators, etc. Data Visualization Thematic dashboards and infographics for citizens and firms API for Interoperability and Open Data Easy and standard access to data within PP.AA. and citizens And much more… The limit is imagination Smart city, analysis for data driven policy making, etc. Use Cases (examples)
20. It… Could… Work!
21. Raffaele Lillo Chief Data Officer raffaele@teamdigitale.governo.it Twitter, Medium: @lilloraffa — Grazie! Cooperate with us, please :) Website http://teamdigitale.governo.it Forum https://forum.italia.it/c/daf Twitter #DatiPubblici #DAF Google Group - Open Data & Analytics data@teamdigitale.governo.it