Nonlinear dynamic model of a coaxial rotor system is established by combining the finite element method and the fixed interface modal synthesis method with nonlinear forces of squeeze film damper and intermediate bearing considered. Nonlinear dynamic characteristics of the co-and counter-rotating coaxial rotor system with different rotational speed ratios are studied and compared in the work. Results show that the unbalance excitation frequencies are dominant in the responses of the rotor system. Due to coupling effect of the intermediate bearing, some combinations of the unbalance excitation frequencies are also observed in the spectrogram while the combinations are different for co-and counter-rotating cases. Also, it is found that the critical speeds of the co-rotating system are equal or slightly higher than those of the counter-rotating case. The orbit and periodicity of the rotor system can be quite different for different speed ratios. Finally, the nonlinear model is validated by comparing the simulation results with the experimental data.