Alan Dolhaz - Birmingham in Real Time

Alan Dolhaz - Birmingham in Real Time

B62d2fb1b594adea59af817a85c2757b?s=128

Iain Mansell

May 01, 2019
Tweet

Transcript

  1. 2.

    Overview 1. What is Birmingham in Real Time? 2. Work

    to Date 3. Current & Future focus
  2. 3.

    Birmingham in Real Time • Started as a loose collaboration

    between members of Birmingham City University, West Midlands Combined Authority, Birmingham City Council and SMEs/data enthusiasts • Goals: a. Developing a community of stakeholders in Birmingham-centric data b. Developing techniques, tools and systems to assist in collection, storage, analysis and interoperability of data from various sources c. Collaborating with and providing data-analytical expertise to local organisations to help with tackling key smart city issues.
  3. 4.

    Background • BiRT originated from analysis & visualisation of traffic

    data made available as a result of the OPTICITIES project. • SCOOT loops, car park occupancy sensors and other traffic data made publicly available via UTMC API. • Issues: ◦ Poor data quality ◦ Lack of diagnostic information ◦ Little insight from raw data ◦ Limited capacity within public orgs to develop solutions ◦ Procurement of data/insight from third-parties often expensive
  4. 5.

    Work to Date • Extensive testing of the original API

    and evaluation of data quality • Development of data validation heuristics • Aggregation & Visualisation ◦ Statistical anomaly detection ◦ Diagnostic measures ◦ Behaviour-based clustering ◦ Cycling trend and seasonality analysis • Development of API Birt.ddns.net
  5. 6.

    Lessons Learned • Many fast-changing data sources ◦ Must provide

    an easier way for integration of disparate datasets • Range of different tools, standards and formats ◦ Must develop intermediate formats and strive to design tools to be easily reusable and extensible • Data quality issues ◦ Diagnostics, monitoring and bespoke processing pipelines required for different data sources • Open Source! ◦ Councils and public orgs often locked into outdated commercial tools ◦ This stifles innovation and evaluation of new approaches
  6. 7.

    Current Focus & Future Work Phase II of work with

    WMCA: • Open source tools for automatic data ingest, processing, analysis and visualisation • Provision of common analytical tools • Modular design allowing for bespoke processing pipelines for different data sources • Real-time architecture • Scalable and extensible API SOURCE API SOURCE API SOURCE Data Ingest Scheduler HOT/COLD STORAGE PRE-PROCESSING PIPELINES API Web User Interfaces POST- PROCESSING PIPELINES
  7. 8.

    Alan Dolhasz // Project Lead Email // alan.dolhasz@bcu.ac.uk Slack Link

    // https://goo.gl/WSYEHd THANKS FOR LISTENING! ANY QUESTIONS?