CS Events

Abstract:

We design a computational framework that interprets fine art paintings by addressing three principles of art: visual form (how it made), content (subject matter of art), and context (in what circumstances). Specifying the problem as discovering the principles from a large set of digitized paintings, we propose two machine learning algorithms to find visual and content information from paintings.

First, after framing a semantic domain from selected visual concepts, we construct an automatic system that quantifies the relatedness of the formal elements to an input painting. In our system, we introduce a new conceptual space, called `styles’. We show in the joint space, paintings and each of the visual concepts can be properly encoded, so we are able to measure associations between the two modalities in an unsupervised way.

Second, we present another system that learns content from fine art paintings. For content analysis, we adopt a deep convolutional neural network and train it to quantify relevancies of content to an input painting. Additionally, we remark the word embeddings, parameters collected from the last fully connected layer of the neural network, that encode relatedness among content by taking paintings as major contextual resources for the relationship. We demonstrate how computational findings are meaningful especially aligned with iconographic studies in art history, and practically helpful to improve general accessibility on current art retrieval systems.