Vibration signal contains a large amount of valuable information of machinery working conditions, and faulty vibration signal is generally nonlinear and non-stationary under complex conditions. It is a big challenge to extract effective fault feature and identify faults from vibration signal. A novel machinery fault diagnosis approach via the ensemble local means decomposition (ELMD) and the improved sparse multiscale support vector machine (SMSVM) is proposed in this work. ELMD, adaptive nonlinear and nonstationary signal processing approach, is performed to decompose the multiple modulated faulty components into demodulated mono-components, thus effectively enhancing the faulty features. Improved SMSVM coupled with multiscale analysis and compressive sensing is developed for machinery fault pattern recognition, thus enhancing the performance of multicalss incipient fault identification. The proposed algorithm inherits the merits of sparse representation, multiscale analysis and SVM, and can be generalized to multiclass problem with moderate computation complexity, with better robustness and generalization. The efficiency and effectiveness of the proposed method is validated by synthesis data and experimental data.