A Deep Look Inside Paintings

3 minute read

Light and space is the origin of everything. The two gametes which engendered the Universe. Likewise, in the painting, a window to the Universe, the space becomes canvas and the light emerges in the form of colorful pigments. Somebody said once that the painting is just that, light and space. And it is still mysterious how a canvas and a bunch of pigments can condense the deepest corners of the human soul.

In particular, the light confined in the Meninas’ room can summarize itself the history of painting. Everything can be explained therein, any emotion can be found impregnated in those Velazquez’s strokes. The atmosphere in that room has inspired painters, fascinated artists, and hypnotize audiences for centuries until our days. The mere idea of having the possibility to get inside, not the canvas, but the room itself, is so attractive that triggers innumerable imaginary visions. To stand by Velazquez, walk between the Meninas, face the Kings’ mirror, or look through that mysterious door at the end of the room, conforms a spatial-temporal transportation through the folded wrinkles of the Universe.

I don’t remember when I started to be deeply fascinated by the painting and recently, several months ago, I discovered that a neural network could estimate depth from a single image. I instantly dreamed about the idea of having a deep look inside that Meninas’ room.

Neural Models

One of the evolutionary strategies to understand depth from monocular seeing, which means looking through a single eye, has to do with the ability of (visual) remembering that size decreases with distance. In other words, if a person in a picture has the same size than a building, probably the former is near than the latter. Actually, size-scaling was one of the first strategies which painters used to give perspective to their paintings. And the other way around, many people use it to create disparate funny visual illusions like the well-known Ames room.

Nowadays, deep neural networks for computer vision have been trained with so many objects in so many perspectives in so many locations with so many illuminations, that in the end they can learn what should be near and far in the scene. One of the first neural networks to estimate depth from a single image was Monodepth. However, such a net was trained with street images and does not generalize properly with other types of scenes. That took me later to Megadepth, which was trained with thousands of images of city landscapes and impressively generalized better for indoor and human bodies, delivering much cooler results.

Projection and lights

In order to project depth into space, pixels were focal projected instead of using parallel projection. However most of the paintings do not present a clear optical perspective. Thus, as a matter of compromise, all the 3D models were projected with a similar and quite long focal distance. Furthermore, no spotlight was used to illuminate the models. The light contained inside the paintings is so marvelous that generated by itself such chiaroscuro atmospheres.

The deep look

Having a careful look to the generated models, specially when they were 3D projected, one can realize that buildings and elements like roofs, windows, or bridges were prettily reconstructed. Actually, Megadepth was trained specifically with those objects. During its training, it seems that human silhouettes were also somehow considered and so people are gratefully reconstructed as well. However one can notice that several heads, and curiously heads of women, failed to be correctly estimated in their depth across all the processed paintings. On the contrary, the Millet’s pitchfork for instance is nicely molded, even though the net probably never saw a tool like that before. From an artistic point of view, although some of the volumes are visibly wrapped and twisted, new pictorial landscapes emerged so beautifully. And that made me get so excited.

The painters’ paintings

  • Diego Velázquez, Las meninas (1656).
  • Francisco de Goya, Los fusilamientos (1814).
  • Jean-François Millet, L’Angélus (1859).
  • Edgar Degas, Répétition de ballet (1873).
  • Georges Seurat, Un dimanche après-midi à l’Île de la Grande Jatte (1886).
  • Pablo Picasso, Mujer en azul (1901).
  • Ramon Pujolà, Salamanca (2018).

To render these 3D models check DepthPainter out.

The original assets used in this project.