Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development of an Artwork Recognition System

Development of an Artwork Recognition System

This talk describes the method and results of a study into the use of SURF-128 for building a system of recognising artworks from photographs.

Presented at Farnborough College of Technology.

Ryan Davies

June 15, 2015
Tweet

Other Decks in Research

Transcript

  1. – Hans Moravec, Mind Children: The Future of Robot and

    Human Intelligence, 1988 “It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”
  2. End User Takes photographs at an art gallery. Wants to

    identify the paintings in them at a later date.
  3. Why not Google Goggles? No way to input images taken

    at an earlier date – only possible to use as augmented reality. Designed for performance, not accuracy. Source: https://play.google.com/store/apps/ details?id=com.google.android.apps.unveil
  4. Why not Google Reverse Image Search? Designed to find similar

    images on webpages based on colour, size, and basic content analysis – not object recognition. Does not provide contextual information – relies on image sources. Because the database contains more than just artwork, presents a lot of irrelevant images for a query image.
  5. Objectives Recognise paintings from photographs with minimum probability of 70%

    For each recognised painting, use an SQL database to provide contextual information: title of the work, artist name, painting description, year of publication, and artist year of birth Provide a user interface for running on Windows 7 and up, built in Visual C#
  6. Objectives Evaluate the robustness of the SURF feature extraction algorithm

    by analysing the results of testing with images that demonstrate changes in scale, orientation, perspective, and obscurities.
  7. Why Visual C#? Type safety Garbage collection Cross platform Integrates

    well with SQL Good user interface tools for Windows platform Source: http://cmap.ihmc.us/xml/ClientNet.html
  8. Why Microsoft SQL Server? Microsoft SQL Server works well with

    Visual C# (.NET) Based on SQL, widely accepted industry standard Good performance, high scalability Source: http://blogs.msdn.com/b/jerrynixon/archive/ 2012/02/26/sql-express-v-localdb-v-sql-compact- edition.aspx
  9. Why Windows? Most used operating system in the world Focus

    on desktop over mobile for increased computing power and availability of tools for higher recall accuracy
  10. Obstacle Avoidance and Navigation in the Real World by a

    Seeing Robot Rover (1980) by Hans Moravec Robot discovers its own path through a maze First attempts failed until using stereo vision Stereo vision requires identifying the same objects across multiple images, to get a sense of depth based on distance between cameras Moravec found that the most useful points to identify across images were corners
  11. Source: Frolova, Simakov (2004) Matching with Invariant Features. Weizmann Institute

    of Science http://alumni.media.mit.edu/~maov/classes/comp_photo_vision08f/lect/18_feature_detectors.pdf
  12. Source: UCF CRCF Video Lectures 2012, Lecture 04 - Interest

    Point Detection http://crcv.ucf.edu/courses/CAP5415/Fall2012/Lecture-4-Harris.pdf
  13. Distinctive Image Features from Scale-Invariant Keypoints (2004) by David G.

    Lowe Algorithm named SIFT (Scale Invariant Feature Transform) Based on properties of human vision Two-stage process: detector and descriptor Detector finds the most interesting areas of an image Descriptor creates a multidimensional vector based on the interest point and its surroundings Results show good invariance to scale, orientation, illumination, and changes to perspective
  14. SURF - Speeded Up Robust Features (2006) by Bay, Gool,

    Tuytelaars Based on SIFT Faster than SIFT Shows good invariance to to scale, rotation, illumination, and changes to perspective Provides ability to improve accuracy at cost to performance: SURF-128
  15. Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration (2009) by

    Lowe and Muja FLANN: Fast Library for Approximate Nearest Neighbors Used for comparing features of database images to the query image SURF features are represented as multidimensional vectors The more dimensions, the more values in each that the search algorithm needs to consider FLANN performs preliminary analysis on a data set to apply the most effective search algorithm
  16. OpenCV Open Computer Vision Library of computer vision algorithms Initially

    developed by Intel Written in C/C++ Provides API’s for accessing SURF and FLANN algorithms
  17. Emgu CV Recursive acronym for ‘Emgu is the Most General

    Unifier’ Provides C# API for OpenCV Integrates with .NET framework
  18. Analysis Most important properties of the system: Accuracy: regularity with

    which paintings are correctly identified Efficiency: processing time to identification
  19. Test Set Test set consists of 26 photographs taken at

    the Tate Modern gallery Photographs deliberately demonstrate changes to perspective For testing, each is transformed to show invariance to scale and rotation Database contains images and details of the 11 artworks photographed
  20. Efficiency Performance is proportionate to image size and complexity The

    large increase in processing time for rotated images was not expected Processing Time in Milliseconds 0 1750 3500 5250 7000 Transformation 25% 50% 75% 100% BW C CC
  21. Accuracy Lowering image size increases accuracy Changes to orientation greatly

    reduce accuracy Indicates SURF-128 is not as rotation invariant as stated Probability (%) 0 20 40 60 80 Transformation 25% 50% 75% 100% BW C CC
  22. Accuracy Changes to perspective significantly reduce accuracy Similarity 0 0.06

    0.12 0.18 0.24 Transformation 25% 50% 75% 100% B&W C CC 24 25 26
  23. The Simplicity Problem Simple artworks do not have enough features

    in order to distinguish them from others in the database This indicates that feature extraction is not the most appropriate matching method for these kinds of images Josef Albers’ Homage to the Square set is an example of simple images that do not match well with feature extraction
  24. Evaluation For 25%, 50%, and 75% scale image sets, met

    70% target. For 100%, black and white, and rotated image sets, accuracy fell below 70% target. Simple artworks brought down average accuracy. Uses SQL database to provide contextual information for matched paintings: title, artist name, description, publication year, and artist year of birth. Visual C# GUI for Windows.
  25. Evaluation Evaluates SURF-128’s applicability to the task of painting identification,

    with the findings: SURF-128 not as invariant to orientation as claimed. More accurate with images closer in size to those in the database. Not the most appropriate solution for paintings with few identifying features.