Share
Embed
Transcript
8 slides
-
Alan Dolhasz // Project Lead Email // [email protected] Slack Link
// https://goo.gl/WSYEHd -
Overview 1. What is Birmingham in Real Time? 2. Work
to Date 3. Current & Future focus -
Birmingham in Real Time • Started as a loose collaboration
between members of Birmingham City University, West Midlands Combined Authority, Birmingham City Council and SMEs/data enthusiasts • Goals: a. Developing a community of stakeholders in Birmingham-centric data b. Developing techniques, tools and systems to assist in collection, storage, analysis and interoperability of data from various sources c. Collaborating with and providing data-analytical expertise to local organisations to help with tackling key smart city issues. -
Background • BiRT originated from analysis & visualisation of traffic
data made available as a result of the OPTICITIES project. • SCOOT loops, car park occupancy sensors and other traffic data made publicly available via UTMC API. • Issues: ◦ Poor data quality ◦ Lack of diagnostic information ◦ Little insight from raw data ◦ Limited capacity within public orgs to develop solutions ◦ Procurement of data/insight from third-parties often expensive -
Work to Date • Extensive testing of the original API
and evaluation of data quality • Development of data validation heuristics • Aggregation & Visualisation ◦ Statistical anomaly detection ◦ Diagnostic measures ◦ Behaviour-based clustering ◦ Cycling trend and seasonality analysis • Development of API Birt.ddns.net -
Lessons Learned • Many fast-changing data sources ◦ Must provide
an easier way for integration of disparate datasets • Range of different tools, standards and formats ◦ Must develop intermediate formats and strive to design tools to be easily reusable and extensible • Data quality issues ◦ Diagnostics, monitoring and bespoke processing pipelines required for different data sources • Open Source! ◦ Councils and public orgs often locked into outdated commercial tools ◦ This stifles innovation and evaluation of new approaches -
Current Focus & Future Work Phase II of work with
WMCA: • Open source tools for automatic data ingest, processing, analysis and visualisation • Provision of common analytical tools • Modular design allowing for bespoke processing pipelines for different data sources • Real-time architecture • Scalable and extensible API SOURCE API SOURCE API SOURCE Data Ingest Scheduler HOT/COLD STORAGE PRE-PROCESSING PIPELINES API Web User Interfaces POST- PROCESSING PIPELINES -
Alan Dolhasz // Project Lead Email // [email protected] Slack Link
// https://goo.gl/WSYEHd THANKS FOR LISTENING! ANY QUESTIONS?