In order to deal with the fluid simulation of the rotor, an algorithm for overset mesh system based on unstructured grids is built. It is suitable for the CFD solver of cell-centred scheme. To capture the blade vortex, the Cartesian grid is adapted all the time during the fluid simulation of the rotor. Alternating digital tree (ADT) algorithm and make, delete, move (MDM) algorithm are used to delete amounts of unused and repeated points generated in every adaptive process. An interpolation method based on gradient is adopted, which greatly simplifies the complexity of the numerical transmission between grids and can satisfy the requirement of accuracy. For Caradonna&Tung and HLISHAPE 7A rotors, calculation in hovering state is conducted and the results are corresponding with the experimental data. It indicates that the algorithm is robust and efficient. After that, the results about capturing blade tip vortex are compared with the simulation without utilizing adaptive gird. It shows that the algorithm can significantly improve the resolution of capturing the blade tip vortex.