Regularization schemes for transfer learning with convolutional networks

Abstract : Transfer learning with deep convolutional neural networks significantly reduces the computation and data overhead of the training process and boosts the performance on the target task, compared to training from scratch. However, transfer learning with a deep network may cause the model to forget the knowledge acquired when learning the source task, leading to the so-called catastrophic forgetting. Since the efficiency of transfer learning derives from the knowledge acquired on the source task, this knowledge should be preserved during transfer. This thesis solves this problem of forgetting by proposing two regularization schemes that preserve the knowledge during transfer. First we investigate several forms of parameter regularization, all of which explicitly promote the similarity of the final solution with the initial model, based on the L1, L2, and Group-Lasso penalties. We also propose the variants that use Fisher information as a metric for measuring the importance of parameters. We validate these parameter regularization approaches on various tasks. The second regularization scheme is based on the theory of optimal transport, which enables to estimate the dissimilarity between two distributions. We benefit from optimal transport to penalize the deviations of high-level representations between the source and target task, with the same objective of preserving knowledge during transfer learning. With a mild increase in computation time during training, this novel regularization approach improves the performance of the target tasks, and yields higher accuracy on image classification tasks compared to parameter regularization approaches.
Complete list of metadatas

Cited literature [162 references]  Display  Hide  Download
Contributor : Abes Star <>
Submitted on : Tuesday, January 14, 2020 - 3:02:13 PM
Last modification on : Wednesday, January 15, 2020 - 9:45:07 AM


Version validated by the jury (STAR)


  • HAL Id : tel-02439193, version 1



Xuhong Li. Regularization schemes for transfer learning with convolutional networks. Technology for Human Learning. Université de Technologie de Compiègne, 2019. English. ⟨NNT : 2019COMP2497⟩. ⟨tel-02439193⟩



Record views


Files downloads