Abstract : The research led in this thesis represents a methodology to contribute to the study of nonlinear dynamic behavior of the system M - O - P in milling. This methodology is directed according to the main objectives relative to this thesis. Indeed, the literature allows giving an overview of the existing work in this context and identifies the vibration phenomena generated by the dynamics of cutting mechanisms and the instability of the involved set cut. In fact, the cutting process generates, as applicable, two types of vibration: forced vibration and self - excited vibration. In a first approach, we developed a model mass - spring (with two degrees of freedom) of the system O - P to study and analyze the dynamic behavior of the elementary machining cell in milling. This model was used to examine the influence of the depth of cut on the stability of the cut. However, this model appears to be insufficient to accurately quantify the vibrations in milling because the sources vibration generated by the dynamics of the machine cannot be neglected. In a second approach, we modeled the overall system Machine - Tool - Part; indeed, a numerical model based on the substructure method by finite element of the M - O - P was developed. This modeling takes into account the effect of inertia of the various rotating elements of the milling machine, the machine structure and shape of the tool. We have conducted simulations of the dynamic behavior of the overall operation of milling. The influence of different parameters on the cutting force such as advance, the depth of cut, and the dynamic responses along the three axes was investigated. This study concluded that these parameters are strongly affected by machining. To complement this numerical study, an experimental study was conducted to study the influence of different cutting parameters. Indeed, the experimental methodology developed has highlighted the most important parameters that influence the overall dynamic behavior of the system M - O - P. This experimental study required the development of an experimental complete to define the parameters of Input / Output and arrange to be tested. The analysis of the results allowed detecting the influence of feed and depth of cut on the level of vibration generated. The outcome of these results put the item on the extreme levels of variables causing the vibration level is higher. Over a confrontation between the experimental and numerical results was performed and showed a good agreement.