In the process of vibration compaction, the "jumping" of vibration wheel may lead to the reduction of compaction quality. Therefore, it is necessary to study the dynamic characteristics of vibration compaction system. In this paper, a coupling nonlinear dynamic model of vibratory roller and subgrade is established. The effect of equivalent stiffness of the subgrade on the nonlinear response of vibration is investigated by using the direct integral method. The results show that the vibration wheel vibrates periodically when the value of the equivalent stiffness is small. With the increase of the stiffness，the vibration wheel jumps and the motion of the system shows period-doubling bifurcation. As the equivalent stiffness increases further, chaotic motion can be observed. Investigation in this paper reveal the evolution of the nonlinear dynamic characteristics of the vibration wheel in practical engineering, which can provide a reference for the design of vibratory roller system.