Skip to Main content Skip to Navigation

Deep Learning for Near-duplicated Patterns Discovery and Alignment in Artworks

Xi Shen 1 
1 imagine [Marne-la-Vallée]
LIGM - Laboratoire d'Informatique Gaspard-Monge, ENPC - École des Ponts ParisTech
Abstract : The goal of this thesis is to develop self-supervised learning approaches to artwork analysis. Precisely, we focus on two particular tasks: object discovery and fine alignment in a collection of artworks. Both tasks are extremely challenging in computer vision, the main difficulties include: i) no annotations are available for both tasks; ii) there are differences in the artistic media (oil, pastel, drawing, etc), and imperfections inherent in the copying process.Object discovery aims at identifying repeated visual patterns across a collection of artworks. This is an important application for art historians, as visual links built via the repeated details may indicate authorship and provenance. Apart from artwork analysis, the task is also interesting for applications on natural images, one typical example is that it enable automatically collect training data.Fine alignment takes a pair of images as inputs and predicts pixel-level alignment. Our goal is to design a generic image alignment approach, which allows aligning images with different appearances, viewpoints and styles, such as two frames in a video, two Internet images on the same landmark or even two paintings depicting the same content but with different styles. The precise flow leads to many interesting applications on artworks, such as texture transfer, aligning a group of images/patterns and analysis of copy process, etc. Moreover, the precise optical flow is also beneficial to several important 3D tasks including two-view geometry estimation and 3D reconstruction.The first technical contribution of this thesis is that we introduce a self-supervised approach to adapt a standard deep feature by fine-tuning it on the specific art collection. More specifically, spatial consistency between neighboring feature matches is used as supervision. The adapted feature leads to more accurate style-invariant matching, and we further propose a discovery pipeline, based on multi-resolution feature matching and geometric verification, to identify duplicate patterns in the dataset. Along with the approach, we also propose a dataset Brueghel which allows evaluating one-shot cross-domain art detail detection.Our second contribution is that we show that it is possible to learn co-segmentation for a pair of images on a synthetic dataset. We generate the training pairs by blending objects into a background image such that we have access into ground-truth masks and correspondences. We empirically study two architectures: Sparse Nc-Net and a transformer-based architecture, and show that employing Poisson Blending and style transfer is crucial for generalization. In terms of results, the trained transformer on the proposed dataset achieves surprisingly good performance on various tasks including one-shot cross-domain art detail detection, place recognition and object discovery.The last contribution is a two-stage method for generic image alignment. In the coarse stage, we estimate a homography transformation between a pair of images with standard feature matches and RANSAC; In the fine stage, we design and learn a small Convolutional Neural Network (CNN) to predict pixel-level alignment relying on the reconstruction loss SSIM. Yet simple, the proposed show competitive and better performance across different tasks: optical flow estimation, sparse correspondences evaluation, two-view geometry estimation, and 3D reconstruction, etc. We also show it is possible to align discovered duplicated patterns and images from Internet search.
Document type :
Complete list of metadata
Contributor : ABES STAR :  Contact
Submitted on : Monday, May 2, 2022 - 12:25:14 PM
Last modification on : Friday, June 10, 2022 - 2:58:02 PM


Version validated by the jury (STAR)


  • HAL Id : tel-03583653, version 2



Xi Shen. Deep Learning for Near-duplicated Patterns Discovery and Alignment in Artworks. Artificial Intelligence [cs.AI]. École des Ponts ParisTech, 2021. English. ⟨NNT : 2021ENPC0040⟩. ⟨tel-03583653v2⟩



Record views


Files downloads