Alan Dolhaz - Birmingham in Real Time

Transcript

1. Alan Dolhasz // Project Lead Email // alan.dolhasz@bcu.ac.uk Slack Link // https://goo.gl/WSYEHd
2. Overview 1. What is Birmingham in Real Time? 2. Work to Date 3. Current & Future focus
3. Birmingham in Real Time • Started as a loose collaboration between members of Birmingham City University, West Midlands Combined Authority, Birmingham City Council and SMEs/data enthusiasts • Goals: a. Developing a community of stakeholders in Birmingham-centric data b. Developing techniques, tools and systems to assist in collection, storage, analysis and interoperability of data from various sources c. Collaborating with and providing data-analytical expertise to local organisations to help with tackling key smart city issues.
4. Background • BiRT originated from analysis & visualisation of traffic data made available as a result of the OPTICITIES project. • SCOOT loops, car park occupancy sensors and other traffic data made publicly available via UTMC API. • Issues: ◦ Poor data quality ◦ Lack of diagnostic information ◦ Little insight from raw data ◦ Limited capacity within public orgs to develop solutions ◦ Procurement of data/insight from third-parties often expensive
5. Work to Date • Extensive testing of the original API and evaluation of data quality • Development of data validation heuristics • Aggregation & Visualisation ◦ Statistical anomaly detection ◦ Diagnostic measures ◦ Behaviour-based clustering ◦ Cycling trend and seasonality analysis • Development of API Birt.ddns.net
6. Lessons Learned • Many fast-changing data sources ◦ Must provide an easier way for integration of disparate datasets • Range of different tools, standards and formats ◦ Must develop intermediate formats and strive to design tools to be easily reusable and extensible • Data quality issues ◦ Diagnostics, monitoring and bespoke processing pipelines required for different data sources • Open Source! ◦ Councils and public orgs often locked into outdated commercial tools ◦ This stifles innovation and evaluation of new approaches
7. Current Focus & Future Work Phase II of work with WMCA: • Open source tools for automatic data ingest, processing, analysis and visualisation • Provision of common analytical tools • Modular design allowing for bespoke processing pipelines for different data sources • Real-time architecture • Scalable and extensible API SOURCE API SOURCE API SOURCE Data Ingest Scheduler HOT/COLD STORAGE PRE-PROCESSING PIPELINES API Web User Interfaces POST- PROCESSING PIPELINES
8. Alan Dolhasz // Project Lead Email // alan.dolhasz@bcu.ac.uk Slack Link // https://goo.gl/WSYEHd THANKS FOR LISTENING! ANY QUESTIONS?