Augmented Reality

Making an API from scratch is one of the easiest task, but the information gathering and data organisation is a much more complex challenge. Data are dirty and need to be clean and we have to determine the relationships and hierarchies between them. We have created an API that uses many different sources and I will show you how to use it.

Transcript

1. Augmented Reality FALUDI, Bence Database Manager
2. Positions: -  Database Manager @ Mito -  Head Of Development @ Ozmo -  Organizer @ Database Meetup Tasks: -  Data warehouse design -  Data Cleansing & Analytics -  ETL & Python Development -  IT Project Management Introduction FALUDI, Bence b.faludi@mito.hu
3. Dirty raw records we convert into analytically meaningful data. •  Structure the data. •  Interpret data. •  Augment it on different sources and determine the correlations between them. US? Abundant. CEE? Catch up with us. Augmented Reality
4. Data warehouse design Design logical data model and database, create physical database, extraction and transformation modules, etc…
5. Data warehouse design Just primary connections in tables.
6. Application programming interface (API) is a specification of how some software components could interact with each other. In procedural language: In most procedural languages an API is a specification that documents a set of functions that accomplish a specific task. In OOP language: How objects work in that language, usually expressed as set of class methods. In libraries: An API describes and prescribes the expected behavior while the library is actual implementation of this set of rules. API
7. A web API is a method of communication between two devices over the world wide web. Web service where emphasis has been moving to simpler representational state and support their light- weight interfaces. Server side: Programmatic interface with request- response system. Client side: Used within web browser and mostly targeting JavaScript functionality. Web API
8. Programmatic interface to a defined request- response message system, typically expressed in JSON or XML, which is exposed via the web—most commonly by means of an HTTP-based web server. It is moved away from a service-oriented architecture (SOA) with SOAP-based web services towards more cohesive collections of RESTful web resources. These RESTful web APIs are accessible via standard HTTP methods by a variety of HTTP clients including browsers and mobile devices. Web API – Server side
9. Representational State Transfer (REST) is a style of software architecture for distributed systems. REST is leading web API design model. Key goals: 1.  Scalability of component interactions. 2.  Generality of interfaces. 3.  Independent deployment of components. 4.  Intermediary components to reduce latency REST
10. It is a web API implemented using HTTP and REST principles. 1.  Has a base URL for web API http://api.mito.hu/api 2.  Internet media type of the data supported. eg.: JSON, XML 3.  HTTP methods supported. eg.: GET, POST RESTful web API
11. We created a RESTful Web API based on Pyramid web framework and PostgreSQL database. Mito API
12. 1.  We don’t like XLS and CSV, but we had too many of them. Want to rid of them. 2.  Data were collected from previous works and make analytics via founded correlations between them. 3.  Centralization of data validation and cleaning. 4.  Easy to maintain and develop and update the given information. 5.  Collect data which is not known currently. Goals
13. 1.  Collect all data which is necessary. 2.  Determine the hierarchies between them. 3.  Design an architect which easy to expand. 4.  Use an ETL tool to load data into selected database. 5.  Try to fix and normalize the data which is not correct until the data quality will be acceptable. 6.  Use a web framework to create a documentation and API request routes. 7.  Write database jobs to create snapshots (materialized views) for our API request. Response time will be quick. How it started?
14. Programming language: Python Minimal memory usage with C and Python package handling. Good code optimizer, etc… ETL tool: mETL Open source product by Mito for CEU with many transformation for data cleaning and load. Framework: Pyramid Open source lightweight framework with ORM support (SQLAlchemy) and good request route configuration. Database: PostgreSQL + pgAgent Response format: JSON What are we chose?
15. Data organization Exchange rate information for countries. It was the first item. * This data structure diagram is not database schema.
16. Data organization Basic phone validation rules and languages.
17. Data organization Location service details. Hungarian settlements and street list for biggest cities, etc…
18. Data organization Population data for countries, counties, cities, districts or regions.
19. Data organization Crime accidents for every street. Used for location service crime information for only* FHB Otthontérkép. * API will not give you criminal information.
20. Data organization Tax, Company Form, Company Headquarter, Court Registry data sources for ID, Tax number, Firm number validation…
21. Data organization Current sources and hierarchies between them. Just added: •  First name list •  Last name list •  Name popularity
22. Data organization Future: CRM support.
23. Global: •  Phone number validity over 243 country and type classification (e.g.: mobile, landline) •  Email address with DNS check •  Date format Currently only for Hungary: ID number, Tax number, Firm number, Bank account number, IBAN number, EAN, BIC, … Features - Validation
24. Localized list: Countries, Territories, Languages, Currencies Currenty only for Hungary: Counties, Cities, Regions, Districts, Streets, … Lists are made taking into consideration UX best practices. E.g. cities list: The sorting criteria is placed in descending order based on the population of cities, and the towns are stored in alphabetical order after the list of cities. It is working fine in Autocomplete/suggestion field. Features – List with filter
25. Currently only working in Hungary, but our short-term plan to extend to CEE region as soon as possible. •  Country: local name, name, currency, language, population, capital, etc… •  County: local name, area, population, capital, cities, etc… •  City: local name, population, area, region, districts, postal code range, crime statistics, coordinates, etc… •  District: crimes, street numbers, etc… •  Region: local name, population, coordinates, estate prices, districts, etc… •  Street: local name, pcode, coordinates, region, house number range, etc… •  Geolocation Features – Location
26. It has many other feature, but the coverage is very different. •  Name frequency: predicts the number of people having given name based on population and most common name list. •  Currency: change currency with current or historical exchange rate. •  Phone number search: Search phone number in public T-Mobile and Telenor database. * * Landline phone number support soon. Features - Others
27. We have many plans, but let’s see few of them: •  Extend Location API to all cities of Hungary and expand to all CEE region. Location service will be contain venue information as well. •  Reverse geolocation and road route with interpolated coordinates. •  Address and VIN validation. •  Weather API for all supported cities. •  Landline phone search support. If you want to see something, please drop me a line. Future plans
28. If you want to try Mito API for limited time, just use the following key: DTSCIMU More information: http://api.mito.hu Let’s try
29. FALUDI, Bence b.faludi@mito.hu   Thank you for your attention!