As part of industrial quality control in the leather industry, it is important to identify the abnormal features in wet-blue leather samples. Manual inspection of leather samples is the current norm in industrial settings. To comply with the current industrial standards that advocate large-scale automation, visual inspection based leather processing is imperative. Visual inspection of irregular surfaces is a challenging problem as the characteristics of the abnormalities can take a variety of shape and color variations. The aim of this work is to automatically categorize leather images into normal or abnormal by visual analysis of the surfaces. To achieve this aim, a deep learning based approach is devised that learns to recognize regular and irregular leather surfaces and categorize leather images on its basis. To this end, we propose an ensemble of multiple convolutional neural networks for classifying leather images. The proposed ensemble network exhibited competitive performance obtaining 92.68% test accuracy on our own curated leather images dataset.