Abstract : Autonomous robotic systems evolve in unpredictable environments, and have to deal with sensor uncertainties. They are usually built with robustness in mind and not to give a model of their behaviour. These models are necessary for high-level decision making like planning or execution control. In nowadays applications, their are often very simplified with respect to a real application. We propose to talk about automated building of intermediate stochastic models for real-world robotics. First, we are going to explain how to learn hidden Markov models from raw sensor data to hidden internal states. Then we are going to talk about larger models and explain why exact inference in such models is not tractable. We will show an algorithm for learning such models. We then show how to use these models to optimize a robotic behaviour and for the system to decide to learn.