Contributions to global optimization
Quelques contribtuions à l'optimisation globale
Résumé
This work addresses the sequential optimization of an unknown and potentially nonconvex function over a continuous and bounded set. The setup we consider assumes that the function evaluations are noiseless and that the derivative information of the function is unavailable. These problems are of particular interest when the function values are typically expensive to compute, and its derivatives are not available either because the evaluations result from some physical measures, or, more commonly, because it is the result of a possible heavy computer simulation. This includes in particular the problem of tuning the hyperparameters of large neural nets. In the first part, we consider the problem of designing sequential strategies which lead to efficient optimization of an unknown function under the only assumption that it has finite Lipschitz constant. We introduce and analyze a first algorithm, called LIPO, which assumes the Lipschitz constant to be known. Consistency and minimax rates for LIPO are proved, as well as fast rates under an additional Hölder like condition. An adaptive version of LIPO is also introduced for the more realistic setup where the Lipschitz constant is unknown and has to be estimated along with the optimization, and similar theoretical guarantees are shown to hold for the adaptive version of the algorithm. In the second part, we propose to explore concepts from ranking theory based on overlaying level sets in order to develop optimization methods that do not rely on the smoothness of the function. We start by observing that the optimization of the function essentially relies on learning the bipartite rule it induces. Based on this idea, we relate global optimization to bipartite ranking which allows to address the case of functions with weak regularity properties. Novel meta-algorithms for global optimization which rely on the choice of any bipartite ranking method are introduced and theoretical properties are provided in terms of statistical consistency and finite-time convergence toward the optimum. Eventually, the algorithms developed in the thesis are compared to existing state-of-the-art methods over typical benchmark problems for global optimization.
Ce travail de thèse s’intéresse au problème d’optimisation séquentielle d’une fonction inconnue définie sur un ensemble continu et borné. Ce type de problème apparaît notamment dans la conception de systèmes complexes, lorsque l’on cherche à optimiser le résultat de simulations numériques ou plus simplement lorsque la fonction que l’on souhaite optimiser ne présente aucune forme de régularité évidente comme la linéarité ou la convexité. Dans un premier temps, nous nous focalisons sur le cas particulier des fonctions lipschitziennes. Nous introduisons deux nouvelles stratégies ayant pour but d’optimiser n’importe quelle fonction de coefficient de Lipschitz connu puis inconnu. Ensuite, en introduisant différentes mesures de régularité, nous formulons et obtenons des résultats de consistance pour ces méthodes ainsi que des vitesses de convergence sur leurs erreurs d’approximation.
Domaines
Intelligence artificielle [cs.AI]
Origine : Fichiers produits par l'(les) auteur(s)