$30 off During Our Annual Pro Sale. View Details »

Breaking the Machine: Finding Art-Historical Value in the Failure of Big Data and Computing

Breaking the Machine: Finding Art-Historical Value in the Failure of Big Data and Computing

Presented at The College Art Association 2017 annual conference.

In his 1962 book "The Shape of Time", George Kubler wrote, "it is disturbing to those who value the individuality of a thing to have that individuality diminished by classifications and generalizations." (36) This suspicion is as relevant as ever, as specialists in machine learning make ever-more boisterous claims about their ability to mine insights from artistic databases, such as a recent project that claimed to identify "the most creative paintings of all time." (Elgammal and Saleh 2015, http://arxiv.org/abs/1506.00711) Art historians have rightly challenged the implicit impartiality of these researchers' data sources and methods, and contested the intellectual coherence of their stated goals.

However, what might a digital art history look like that deploys machine learning precisely in order to see where the machine breaks? This paper will explore the conditions in which computational models fail to predict or classify, demonstrating how these instances can provoke new questions or analytical approaches. I will discuss a case study in which supervised machine classification was used to analyze a database of seventeenth-century Haarlem still life paintings. The computer's comparative ability, or inability, to discern the shape of different artists' oeuvres based on limited information about composition or subject matter suggested how these painters, as well as those who imitated them, defined their artistic personae. Where the machine breaks, and where it succeeds, also help to surface how decisions as fundamental as one's chosen framework for visual description can fundamentally alter the synthetic historical narratives that one ultimately composes.

Matthew Lincoln

February 18, 2017
Tweet

More Decks by Matthew Lincoln

Other Decks in Research

Transcript

  1. @matthewdlincoln
    Breaking the Machine
    Finding Art-Historical Value in the Failure
    of Big Data and Computing
    Matthew Lincoln, Ph.D
    Data Research Specialist
    College Art Association, New York
    February 18, 2017

    View Slide

  2. The Daisy Linda Ward Collection,
    Ashmolean Museum, Oxford
    Image source: BBC Four
    @matthewdlincoln

    View Slide

  3. River Clegg, “An Honest Museum Audio Tour,” The New Yorker, December 5, 2016
    @matthewdlincoln

    View Slide

  4. Jan van Huysum, Amsterdams
    Historisch Museum
    @matthewdlincoln

    View Slide

  5. Jan van Huysum, Amsterdams
    Historisch Museum
    @matthewdlincoln

    View Slide

  6. Sam Segal, A Flowery Past: A Survey of
    Dutch and Flemish Flower Painting from
    1600 Until the Present (1982).
    @matthewdlincoln

    View Slide

  7. The Daisy Linda Ward Collection,
    Ashmolean Museum, Oxford
    Image source: BBC Four
    @matthewdlincoln

    View Slide

  8. Henry Duval Gregory, “Tabletop Still Lifes in Haarlem, c. 1610-1660: A Study of the
    Relationships between Form and Meaning” (Ph.D. diss., University of Maryland, 2003).
    @matthewdlincoln

    View Slide

  9. Low viewpoint
    @matthewdlincoln
    Willem Claes Heda, Banquet Piece
    with Mince Pie, 1638. National
    Gallery of Art, Washington

    View Slide

  10. High viewpoint
    @matthewdlincoln
    Pieter Claesz, Still life with Turkey Pie,
    1627. Rijksmuseum, Amsterdam.

    View Slide

  11. 1.  Lemon (peeled)
    2.  Candle (extinguished)
    3.  Oyster
    4.  Beer
    5.  Tazza (overturned)
    6.  …
    Willem Claes Heda, Banquet Piece
    with Mince Pie, 1638. National
    Gallery of Art, Washington
    @matthewdlincoln

    View Slide

  12. 1.  Meat pie (Turkey)
    2.  Mince Pie
    3.  Oyster
    4.  Wine (white)
    Pieter Claesz, Still life with Turkey Pie,
    1627. Rijksmuseum, Amsterdam.
    @matthewdlincoln

    View Slide

  13. Compositional
    •  Orientation (por./land.)
    •  Disposition (wedge? pyramid?)
    •  Viewpoint (high/low)
    •  Cropping (tight/expansive)
    •  Height
    •  Width
    Symbolic
    •  Significant Motifs
    •  Illusionistic Signature?
    @matthewdlincoln

    View Slide

  14. @matthewdlincoln
    (image credit: Shih 2013)
    Decision Trees and “Random Forest” learning

    View Slide

  15. @matthewdlincoln
    Willem Claes Heda, Banquet Piece
    with Mince Pie, 1638. National
    Gallery of Art, Washington

    View Slide

  16. @matthewdlincoln
    above-median
    proximity

    View Slide

  17. @matthewdlincoln
    below-median
    proximity

    View Slide

  18. @matthewdlincoln
    Willem Claes Heda, Banquet Piece
    with Mince Pie, 1638. National
    Gallery of Art, Washington

    View Slide

  19. @matthewdlincoln
    More
    Predictable
    More
    Unpredictable
    OOB error
    estimates

    View Slide

  20. @matthewdlincoln
    Some painters had motifs more
    predictable than compositions
    •  2 out of 5 misattributed by
    motif
    •  1 out of 6 misattributed by
    composition
    N = 12 (great performance for
    such a small sample)

    View Slide

  21. @matthewdlincoln
    Others had more
    predictable motifs
    •  1 out of 4 ptgs misattributed by
    composition
    •  1 out of 6 misattributed by
    motif

    View Slide

  22. @matthewdlincoln
    Misattributes almost 100% of Gerrit
    Heda’s compositions to his father

    View Slide

  23. @matthewdlincoln
    Also misattributes 100% of Hedas by
    motif, but splits these mistakes between
    PC and WCH

    View Slide

  24. @matthewdlincoln
    Systematic misattributions
    point to strong affinities
    •  100% of Mahu’s works
    mistakenly given to PC
    judging by motifs alone

    View Slide

  25. Data are subjective
    This is a good thing
    Where models fail can be even more
    informative than where they succeed
    At its best, computing accentuates
    uncertainty and contingency
    @matthewdlincoln

    View Slide

  26. Matthew Lincoln
    matthewlincoln.net
    @matthewdlincoln

    View Slide