Reliable and accurate detection of fruits during the whole growth period has always been one difficult and important bottleneck for achieving precise, intelligent and efficient orchard management. In order to deal with the insufficiency of sample scale and diversity in actual production scenes, this dataset is constructed focusing on the application of fruit detection in typical orchard operation stages, such as fruit thinning, bagging and picking operations. Specifically, through in-field shooting and data post-processing, the multi-modal image dataset for peach detection is released, which covers the acquisition, classification, labeling, storage and use of multi-modal peach images during fruit thinning, bagging and picking stages under the different natural circumstances, including complex weather, illumination and occlusion. What’s more, the modalities of the dataset get involved in the types of visible light, depth and infrared with the total storage amount of 8.27GB, providing fundamental and valuable image resources for the following research areas, e.g., multi-modal image data fusion and object detection. In addition, the dataset can also be used as a standard library for deep learning modeling in big data environment with the important practical application value for promoting the research of fruit object detection.