Breaking the Machine: Finding Art-Historical Value in the Failure of Big Data and Computing

Presented at The College Art Association 2017 annual conference.

In his 1962 book "The Shape of Time", George Kubler wrote, "it is disturbing to those who value the individuality of a thing to have that individuality diminished by classifications and generalizations." (36) This suspicion is as relevant as ever, as specialists in machine learning make ever-more boisterous claims about their ability to mine insights from artistic databases, such as a recent project that claimed to identify "the most creative paintings of all time." (Elgammal and Saleh 2015, http://arxiv.org/abs/1506.00711) Art historians have rightly challenged the implicit impartiality of these researchers' data sources and methods, and contested the intellectual coherence of their stated goals.

However, what might a digital art history look like that deploys machine learning precisely in order to see where the machine breaks? This paper will explore the conditions in which computational models fail to predict or classify, demonstrating how these instances can provoke new questions or analytical approaches. I will discuss a case study in which supervised machine classification was used to analyze a database of seventeenth-century Haarlem still life paintings. The computer's comparative ability, or inability, to discern the shape of different artists' oeuvres based on limited information about composition or subject matter suggested how these painters, as well as those who imitated them, defined their artistic personae. Where the machine breaks, and where it succeeds, also help to surface how decisions as fundamental as one's chosen framework for visual description can fundamentally alter the synthetic historical narratives that one ultimately composes.

Transcript

1. @matthewdlincoln Breaking the Machine Finding Art-Historical Value in the Failure of Big Data and Computing Matthew Lincoln, Ph.D Data Research Specialist College Art Association, New York February 18, 2017
2. The Daisy Linda Ward Collection, Ashmolean Museum, Oxford Image source: BBC Four @matthewdlincoln
3. River Clegg, “An Honest Museum Audio Tour,” The New Yorker, December 5, 2016 @matthewdlincoln
4. Jan van Huysum, Amsterdams Historisch Museum @matthewdlincoln
5. Jan van Huysum, Amsterdams Historisch Museum @matthewdlincoln
6. Sam Segal, A Flowery Past: A Survey of Dutch and Flemish Flower Painting from 1600 Until the Present (1982). @matthewdlincoln
7. The Daisy Linda Ward Collection, Ashmolean Museum, Oxford Image source: BBC Four @matthewdlincoln
8. Henry Duval Gregory, “Tabletop Still Lifes in Haarlem, c. 1610-1660: A Study of the Relationships between Form and Meaning” (Ph.D. diss., University of Maryland, 2003). @matthewdlincoln
9. Low viewpoint @matthewdlincoln Willem Claes Heda, Banquet Piece with Mince Pie, 1638. National Gallery of Art, Washington
10. High viewpoint @matthewdlincoln Pieter Claesz, Still life with Turkey Pie, 1627. Rijksmuseum, Amsterdam.
11. 1.  Lemon (peeled) 2.  Candle (extinguished) 3.  Oyster 4.  Beer 5.  Tazza (overturned) 6.  … Willem Claes Heda, Banquet Piece with Mince Pie, 1638. National Gallery of Art, Washington @matthewdlincoln
12. 1.  Meat pie (Turkey) 2.  Mince Pie 3.  Oyster 4.  Wine (white) Pieter Claesz, Still life with Turkey Pie, 1627. Rijksmuseum, Amsterdam. @matthewdlincoln
13. Compositional •  Orientation (por./land.) •  Disposition (wedge? pyramid?) •  Viewpoint (high/low) •  Cropping (tight/expansive) •  Height •  Width Symbolic •  Significant Motifs •  Illusionistic Signature? @matthewdlincoln
14. @matthewdlincoln (image credit: Shih 2013) Decision Trees and “Random Forest” learning
15. @matthewdlincoln Willem Claes Heda, Banquet Piece with Mince Pie, 1638. National Gallery of Art, Washington
16. @matthewdlincoln above-median proximity
17. @matthewdlincoln below-median proximity
18. @matthewdlincoln Willem Claes Heda, Banquet Piece with Mince Pie, 1638. National Gallery of Art, Washington
19. @matthewdlincoln More Predictable More Unpredictable OOB error estimates
20. @matthewdlincoln Some painters had motifs more predictable than compositions •  2 out of 5 misattributed by motif •  1 out of 6 misattributed by composition N = 12 (great performance for such a small sample)
21. @matthewdlincoln Others had more predictable motifs •  1 out of 4 ptgs misattributed by composition •  1 out of 6 misattributed by motif
22. @matthewdlincoln Misattributes almost 100% of Gerrit Heda’s compositions to his father
23. @matthewdlincoln Also misattributes 100% of Hedas by motif, but splits these mistakes between PC and WCH
24. @matthewdlincoln Systematic misattributions point to strong affinities •  100% of Mahu’s works mistakenly given to PC judging by motifs alone
25. Data are subjective This is a good thing Where models fail can be even more informative than where they succeed At its best, computing accentuates uncertainty and contingency @matthewdlincoln
26. Matthew Lincoln matthewlincoln.net @matthewdlincoln