A new method for image recognition of train brake shoe based on edge moment invariants matching algorithm had been researched. The dynamic brake shoe images were taken by a vision sensor while the vehicle was moving. Then the statistic of brake shoe image invariant characteristics of rotation, scaling, and shift transformation were applied as the main features to automatically recognize the brake shoes with complicated background. With the presence of the characteristic similarity between the brake shoe surface and the wheel of vehicle, asymmetry of gray distribution of brake shoe images, object variation and occlusion of vehicle brake shoes, an improved moment invariants based on the edge-matching algorithm was proposed to recognize the brake shoe shape and analyze the brake shoe characteristics. The existing moment invariants algorithm was modified using the origin moment instead of central moments, and selected four lower order moments as the feature variable to reduce the calculation error. The experimental results show that the proposed method is effective for solving the above problems. The proposed algorithm can recognize the vehicle brake shoe automatically and can provide an effective way to solve the technical problems in long-standing recognition of the moving brake shoes.