DEVELOPMENT OF COMPUTER VISION APPROACH FOR RAILROAD TRACK HEALTH MONITORING WITH DRONE DATA

Abstract

As there has been a movement towards the 21st century the quality of pictures have changed with the growth of the Internet and the digital cameras having better and better data to study computer vision. Vision is nowadays used in a gamut of real-world scenarios such as machine inspection, automotive safety, surveillance, fingerprint recognition and biometrics. Some of the industrial vision applications include face detection, visual authentication, stitching and structural health monitoring. These application areas employ a number of computer vision tasks which can be solved using a variety and combination of system methods. The system methods include image acquisition, image pre-processing, image processing, image analysis, feature extraction and decision making. Image acquisition comprises of capturing drone and satellite images, and image pre-processing methods comprise of reduction of image noise, straightening of image by rotation to name a few. The image processing method includes filtering, image analysis method comprises of segmentation to obtain Region of Interest (ROI), feature extraction comprises of computing feature detectors and descriptors while decision making methods include classification, detection and segmentation results. The computer vision tasks include image classification and localization, object detection, semantic segmentation, and instance segmentation. As structural health monitoring for railroad track health monitoring is an essential and prominent industrial application therefore, the computer vision tasks of object detection and segmentation are employed in this domain through implementation of the appropriate feature extraction, machine learning and deep learning (DL) methods. The main emphasis of this thesis is to study development of computer vision approach for railroad track health and assets monitoring in drone data. The general objective is supported by performing the following tasks:  Development of a novel adaptive framework for railroad track extraction in DI.  Development of feature-based template matching method for fishplate detection in railroad track DI.  Development of deep learning-based approach for detection of fishplates in railroad track DI.  Development of deep learning-based methods for asset detection in railroad track drone images. The thesis is organized into seven chapters, which are briefly summarized below: Chapter 1 presents the introduction, theoretical background, motivation, scope and objectives of the current research work. It also describes the study area and data sets used to carry out the thesis objectives. A brief literature review related to the tasks performed in the thesis is given in Chapter 2. Chapter 3 emphasizes the development of an adaptive railroad track extraction method for drone images. Railroad track extraction presents an immediate advantage during railroad inspections, rail surface defects detection, train driver assistance and obstacle identification and provisioning autonomous train systems in an efficient and cost-effective manner. At present, human inspectors, inspection trains, and rail-mounted vehicles equipped with cameras are prevalent image acquisition systems (IAS) in the track extraction module. However, these IAS face various challenges such as high operability cost, railroad closed for normal traffic, and inaccessibility to certain geographical locations. In such scenarios, drones act as effective IAS. Therefore, this chapter presents a novel and adaptive railroad track extraction framework for drone images (DI) which are captured under uneven illumination, at different drone fight heights, with varying rail line orientations, and in complex railroad environments. This framework is termed as DroneRTEF. This work primarily focuses on two aspects of drone-based railroad track images: image enhancement and image analysis. The image enhancement method is used to pre-process the track drone image and then segment it in order to highlight the rail line feature information. This is advantageous as it helps to eliminate or weaken the interference of background information and identify rail lines. The image analysis step is performed by extracting the features (shape features/geometric features) of the identified rail lines. It has been observed that colour spaces in conjunction with masks are quite helpful for rail line identification in DI while working in complex railroad environments. Through this critical analysis it is observed that colour space-based masking seems an effective image enhancement method for the segmentation and identification of rail lines in railroad track DI captured in uneven illumination. Therefore, with regards to the first aspect, a global image enhancement algorithm named adaptive colour space-based masking (ACSM) is developed to enhance railroad track images and identify rail lines. The rail lines and background can be highlighted and homogenized, respectively, in DI captured under various sunlight intensity using ACSM due to its illuminance independence. Hough Transform (HT) (an under constraint line fitting algorithm) serves advantageous over other line detection algorithms such as RANSAC (constraint bound) and Least Squares Fit (over constraint). Also, the Hough transform model can handle a higher percentage of outlier points and is best suited for highly noisy data. Therefore, inspired by these successes, HT is a suitable solution for geometrical representation of hough parameter space of rail lines oriented at varying angles. Nonetheless, the detection of rail lines in drone images captured at different altitudes is another essential step for track extraction at different flight heights. In order to fulfill this aim, Ground Sample Distance (GSD) method is useful as changing the altitude of the drone alters the GSD of the image which in turn helps to detect valid rail lines (rail line pairs) at different flight heights and subsequently extract the railroad track. Hence, with regards to the second aspect, the Hough parameter space analysis-based novel Hough transform-ground sample distance (HT-GSD) method is presented in this chapter. The proposed HT-GSD method emphasizes on rail line detections at varying line orientations and different fight heights. The track extraction is then performed by a coordinate transformation technique. The proposed approach has been successfully tested and validated on test images from four different railroad scenarios comprising of DI. The efficacy of our framework for rail line detection is identified by comparing it with other line detection models such as Inception V4, K-NN mean Euclidean distances, ACSMHT( computed). Performances of these methods are tested using metrics such as precision, recall and accuracies of the detections. Results obtained show that the proposed method is superior to other models as it has achieved a precision and recall of 75% and 90% respectively while the recall value is 84% (comparably lower) to other methods . Therefore, DroneRTEF is an efficient and feasible method for railroad track extraction in DI. Maintenance of railroad track safety is of utmost importance as derailment accidents cause significant loss to life and property. Periodic inspection of railroad tracks and their components is necessary in order to ensure security and well-being of goods as well as humans. Fishplate is an essential component in the railroad track environment hence, periodic maintenance of fishplates is an imperative goal. Maintenance through manual operations involve railroad personnel. These operations are subjective and timeconsuming. Consequently, replacement of manual inspections with unmanned vehicles such as drones is well suited for fast and efficient maintenance operations. Drone images comprising of fishplate instances face challenges such as rail lines having varying orientation and consequent variances in the position and direction of fishplates, different flight heights capturing different sized fishplates, and different environmental factors such as shaking of drones, varied illumination scenarios, partial occlusion affecting the appearance of the fishplates. While the drone images are captured at different altitudes and in aforementioned different railroad scenarios, the fishplate component is very small as compared to the entire drone image (almost invisible to the naked eye) which makes the detection of fishplates in railroad track drone images additionally, a very difficult task. Therefore, automated fishplate detection and monitoring is desired in railroad track drone images. This object detection task is conducted through handcrafted feature engineering in conjunction with template matching methods (Chapter 4) and non-handcrafted feature methods using DL techniques (Chapter 5). The objective of chapter 4 is development of feature-based template matching method for fishplate detection in railroad track drone images. The monitoring of this fishplate component is essential in order to maintain steadiness of the train. For this purpose, the task is undertaken as in the first step normalized correlation coefficient–based template matching method is computed for fishplate detection in drone images. The drone images are captured at a height of 25 metres, preprocessed using median filtering and subsequently, annotated, using the annotation tool, which is then used to subset the fishplate images from the ground truth data. These images are then used for template image creation. The normalized correlation coefficient-based template matching method is then computed in which the uneven template image is slid over the entire source image and the normalized correlation coefficients are computed at every slid between both the areas. The threshold is applied on the coefficient values in order to obtain the accurately detected fishplate bounding boxes. Then image-based statistics and non-maximum suppression are further performed on the obtained detections for reduction of false positives. Image–based statistics is a post processing method utilized for reduction of false positives through mean and standard deviation of the areas inside the bounding boxes. The proposed method has obtained false detections. Therefore, in order to overcome the limitation of false positives and lower sensitivity values in the previous work, we move to the next task where the proposed method encompasses a critical analysis of feature descriptors for development of a novel feature-based template matching technique for efficient fishplate detection in drone data. The images captured using drone images are annotated for both fishplate and no fishplate objects classes. The images for both the classes are then preprocessed. Various features such as entropy, contrast, lacunarity, Zernike moments, Moran’s I, Fractal dimension D, discrete wavelet transform are calculated on annotated images of both the classes. Subsequently, the appropriate feature is chosen, upon computation of the Separability Index (SI) on each of them. The feature with SI value> 1.0 is chosen i.e. horizontal discrete wavelet transform in this work. The preprocessing of template and test image area is performed before feature-based template matching and then sliding window-based Euclidean distance computation is obtained. The Non-maximum suppression method has been used as a post processing method for reduction of false detections. In this work the false reductions are observed to have reduced .