DEVELOPMENT OF A DISTRIBUTED MULTI-MODAL SURVEILLANCE SYSTEM

Abstract

With the increasing availability of inexpensive multimodal sensors and growth in computing and networking technology, Video Surveillance (VS) systems have advanced from traditional analog-based CCTV to multimodal and distributed systems. The new generation of video surveillance is posing novel scientific chal lenges. The objective of this thesis is to address the emerging issues like ro bustness, efficiency, scalability and real time processing, in the development of distributed multimodal surveillance systems. The thesis deals with the following tasks: fusion of information from different sensor modalities, efficient techniques for multimedia data management, scalable video transmission for Distributed Video Surveillance (DVS) and lastly, real-time processing using parallel multicore architecture. Chapter one gives the introduction to automated VS systems and discusses the motivation behind the specific problems that were taken up for present research work. In chapter two of the thesis, a brief review of the current state of the art in the development of multimodal and distributed framework for VS systems is pre sented. In the past, several researchers have focused on VS using single modality, i.e., visible video and in addition, they are mostly targeted for specific applica tions working in a controlled environment. Traditional approach in dealing with the problems of object tracking in difficult conditions have focused on image eniii hancement, developing more powerful algorithms for background updating and extracting additional content data (color, edges, texture etc.) from the visible spectrum video. However, not much work has been reported to utilize multiple different modalities beyond visible spectrum. In addition, there is a research gap in addressing the issues of efficient processing, data management and scalability in the context of large scale deployment of DVS systems. In chapter three, the problem of robust object detection and tracking in dynamic conditions is taken up with the approach of fusing multiple modality information in a generic framework. Recently some attempts have been made to combine information from different modalities like visible spectrum and ther mal infrared. But they perform fusion by assuming that the sensors are reliable and thus, tend to fail when the information from the sensors is contradictory in nature. The proposed method contributes a multi-modal fusion framework and algorithm that overcomes these limitations by a mechanism of belief assign ment to different sensor output, dynamic assessment of sensor reliability and weighted fusion for detection of objects and tracking their position. For this purpose, a belief model is developed, using evidence theory, to combine uncer tain or imprecise information from different sources for determining the validity of a foreground region for tracking. Fuzzy logic is used to model belief mass generation by calculating two measurement features from sensor output. The context of the environment and the past information is also taken into account for giving appropriate weights during the belief fusion. An algorithm for dynamic assessment of sensor reliability is presented and the belief mass is appropriately discounted. An implementation of tracker, with an example of Kalman filter, is done for fusing the sensors measurements of objects position and size, accord ing to their reliability. The approach was evaluated in three extreme conditions

of very rapidly changing illumination, night scene with high sensor noise and a daylight scene with shadows and insulated clothing's. The performance was analyzed using appropriate performance metrics from literature, showing excel lent improvement in the results as compared to individual modalities and other fusion approaches. Chapter four of the thesis deals with an important problem of managing the enormous amount of multimedia stream data, resulting from large scale de ployment of DVS systems. In the past, researchers have attempted to design intelligent systems that performs intelligent recognition of very specialized ab normal events. But quite apart from that, there is a need for efficient techniques for multimedia data management. The thesis contributes a novel framework that attempts to collect informative data and filter out non-informative data in an online setting by learning normal events. The motivation is that in many surveil lance scenarios, most of the time there are normal events which are easier to learn and can be discarded, while anything that deviates from the normal pattern can be collected. The implementation framework demonstrates a possible synergy between multimedia and data mining by showing an effective use of association rule mining in feature selection and a dynamic reservoir sampling algorithm that selects data samples to construct a binary signature of a database. A classi fier is learned using the constructed signature and classification performance is preserved even at low sampling ratio. Chapter five addresses another interesting area of research in DVS system involving efficient processing and scalable transmission of video streams from the cameras to the end users. Pioneer projects for DVS systems do not con sider high processing requirement, network and scalability issues to handle large scale multimedia stream data. This chapter contributes DVS system framework exploiting content based scalable compression scheme and bandwidth adaptive allocation to intelligently transmit important segments of the video sequence over the network. Discrete Wavelet Transform (DWT) based Color Embedded Zerotree Wavelet (CEZW) coding is used to obtain a scalable bitstream that can be transmitted at different rates to different users depending on their network conditions. Object based coding is done to achieve high level of compression and a fast object segmentation method in compressed video is investigated for efficient processing. The architecture includes a novel module called Dynamic Decision Maker (DDM) for dynamic allocation of network bandwidth to differ ent frame constituents based on their relative importance, perceptual quality and available estimate of network bandwidth. The experimental results over different video sequences demonstrate interesting performance in its adaptive behavior to bandwidth availability and preservation of image quality even at very low trans mission rate. Chapter six of the thesis addresses the challenge of real time processing of the VS algorithms, which are becoming more and more data and computational intensive in nature. This requires a high performance computational solution either through distributed computing or parallel processing. Use of grid and cluster based distributed architecture is not desirable due to large communication overhead. Emerging Cell Broadband Engine based on multi-core parallel chip architecture offers high computing power for real time processing of large scale multimedia streams. The chapter presents a generic solution to the problem of real-time processing by parallel implementation on Cell. In our approach we analyze the different modules of a typical VS workload and explore the scope of parallelization based on the data and control dependencies between them. Ageneric hybrid model for restructuring and scheduling the different tasks for vi concurrent execution on the different processing cores is proposed. The proposed implementation demonstrates significant speed up and thus the achievement of desired frame processing rate. Finally in chapter seven, the contributions made in the thesis are summarized and scope of future work is outlined. First, a novel and generic multi-modal fusion approach was contributed to improve the robustness of object detection. Then the problem of data management was addressed by proposing a novel framework for on-line surveillance data management. An end to end DVS system framework was presented along with efficient algorithm design for real time processing and scalable transmission over network.