Home

Research

Automated Real Time Visual Surveillance

Visual Sensor Networks

Aerial Video Exploitation

Multi-frame Data Association

Video Coding

Radiometric Analysis

Alliances in Graphs

Publications

Resume

Photography

 



Research

Automated Real Time Visual Surveillance

A large part of my research focuses on solving issues related to real-time automated surveillance and monitoring systems and spans most of the stages of the video processing pipeline. These include target detection, intra-sensor and inter-sensor target tracking, classification, scene modeling, and event detection.

I was one of the principal architects of a real time automated surveillance and monitoring system called KNIGHT. KNIGHT is a 'smart', fully automated, real-time surveillance and monitoring system that detects important changes, events, and activities using computer vision techniques, flags significant events, and presents a summary in terms of key frames and textual description of activities to a monitoring officer for final analysis and response decision. The system is robust to illumination changes and weather conditions. Originally developed for the Orlando Police Department, KNIGHT has been used for projects funded by Florida Department of Transportation, DARPA Small Business Technology Transfer (STTR) program, and Lockheed Martin Corporation.

I have also developed key technologies to support a multi-sensor maritime surveillance system has been operational at a port in Florida for more than a year where it is being used as experimental test-bed and technology demonstrations. It consists of a large number of high and low resolution heterogeneous sensors including: static color sensors (1MB-11 MB resolution), PTZ units, thermal and low-light sensors, static and mobile omni sensors, and radar.

Related Publications

  1. "Probabilistic Modeling of Scene Dynamics for Applications in Visual Surveillance," with Saleemi I. and Shah M., IEEE Transactions on Pattern Analysis and Machine Intelligence, to appear.
  2. "Automated Visual Analysis in Large Scale Sensor Networks," with Rasheed Z., Cao X., Liu H., Yu L., Lee M., Ramnath K., Choe T., Javed O., and Haering N., ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC), California, September 2008.
  3. " Automated Surveillance in Realistic Scenarios," with Shah M. and Javed O., IEEE Multimedia, vol. 14, issue 1, Jan-March, 2007.
  4. "Automatic Visual Analysis for Transportation Security," with Haering N, IEEE Conference on Technologies for Homeland Security, 2007.
  5. "A Hierarchical Approach to Robust Background Subtraction using Color and Gradient Information," with Javed O. and Shah M., IEEE Workshop on Motion and Video Computing, Orlando, Dec 2002.
  6. "Automatic Visual Analysis for Transportation Security," with Haering N., IEEE Conference on Technologies for Homeland Security.
  7. "Multiple Vehicle Tracking in Surveillance Videos," with Zhai Y., Berkowitz P., Miller A., Varket A., White B., and Shah M., CLEAR Evaluation Campaign and Workshop, 2006.
  8. "Visual Monitoring of Railroad Grade Crossing," with Sheikh Y., Zhai Y., and Shah M., SPIE Defense and Security Symposium, April 2004.

Return to top of the page

Visual Sensor Networks

I am interested in developing efficient algorithms for automatic estimation and maintenance of inter-sensor relationship and geo-registration in large visual sensor networks. The proposed algorithms automatically determine whether two or more sensors have overlapping fields of view and use traffic patterns to find an inter-sensor geometric model and geo-registration. The algorithms also detect changes in sensor topology over time and adapt accordingly. The proposed algorithms have been shown to perform well in noisy scenarios and are operational at a port installation for maritime surveillance.

I am also interested in data association algorithms for finding correspondences between the observations of people in a network of cameras with overlapping and non-overlapping fields of view. Conventional tracking approaches assume proximity in space, time and appearance of objects in successive observations. However, observations of objects are often widely separated in time and space when viewed from multiple non-overlapping cameras. To address this problem, we present a novel approach for establishing object correspondence in a system of non-overlapping cameras. We observe that people or vehicles follow the well-defined paths in most cases, i.e roads, walk ways, corridors etc. Our method exploits this redundancy in paths traversed by using both motion trends and appearance of objects for tracking. Our system does not require any inter-camera calibration, instead the system learns the camera topology and path probabilities of objects using Parzen windows during a training phase. Once the training is complete, correspondences are assigned using the maximum a posteriori (MAP) estimation framework. The learned parameters are updated with changing trajectory patterns.

Related Publications

  1. "Modeling Inter-Camera Space-Time and Appearance Relationships for Tracking across Non-Overlapping Views," with Javed O., Rasheed Z., and Shah M., Computer Vision and Image Understanding Journal (Elsevier), vol. 109, issue 2, February 2008.
  2. "An Efficient Data Driven Algorithm for Multi-Sensor Alignment," with Guo F., Aggarwal G., Cao X., Rasheed Z., and Haering N., Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications, with ECCV, Marseille, France, October 2008.
  3. "Self Calibrating Visual Sensor Networks," with Hakeem A., Javed O., and Haering N., IEEE Workshop on Applications of Computer Vision, Copper Mountain, Colorado, January 2008.
  4. "Automatic Visual Analysis for Transportation Security," with Haering N, IEEE Conference on Technologies for Homeland Security, 2007.
  5. "Appearance Modeling for Tracking in Multiple Non-overlapping Cameras," with Javed O. and Shah M., IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, June 2005.
  6. "Tracking Across Multiple Cameras with Disjoint Views," with Javed O., Rasheed Z., and Shah M., The Ninth IEEE International Conference on Computer Vision (ICCV), Nice, France, October 2003. (Acceptance Rate 20%)

Return to top of the page

Aerial Video Exploitation

Aerial video analysis and exploitation is the focus of my current research. Recent advancements in sensor technology and platforms as well as the increase in the number of deployed sensors has resulted in a glut of imagery and the need for automated and efficient exploitation tools. I am investigating algorithms to enable detection and tracking of targets in challenging aerial videos that exhibit large ego-motion, parallax displacements, and clutter. The other problems of interest include ortho-rectification, video mosaicking, sensor and parallax modeling, 3D-reconstruction, detection of scene elements (such as road networks, buildings, and vegetation), extraction of scene context (such as target models of appearance, motion, and scales), video based mensuration, and event extraction.

Related Publications

  1. "Autonomous Real-time Ground Ubiquitous Surveillance-Imaging System (ARGUS-IS)," with Leininger B., Edwards J., Antoniades J., Chester D., Haas D., Liu E., Stevens M., Gershfield C., Braun M., Targove J.D., Wein S., Brewer P., and Madden D.G., SPIE Defense and Security Symposium, 2008.
  2. "Target-Tracking in Airborne Forward Looking Infrared Imagery," with Yilmaz A. and Shah M., Journal of Image and Vision Computing, vol. 21, no. 7, 2003.
  3. "Target-Tracking in FLIR Imagery Using Mean-Shift and Global Motion Compensation," with Yilmaz A., Shah M., and da-Vitoria Lobo N., Workshop on Computer Vision Beyond Visible Spectrum with CVPR 2001, Kauai, Hawaii, Dec 2001.

Return to top of the page

Multi-frame Data Association

Tracking large number of targets in a video is a challenging task due to high ambiguity in data association caused by frequent occlusions among targets, arrival and departure of targets from the scene, and the presence of detection errors and noise. The task is further complicated when there are not many distinguishing features among the targets (either because of sensor limitations or because the targets are themselves alike). These scenarios commonly occur in applications like surveillance, optical flow, feature tracking, and structure from motion. I have attempted to solve this problem using multi-frame scanning window based algorithms (delayed inference). The problem of multi-frame point correspondence is NP Hard for three or more frames. The proposed algorithms explicitly model the common problems of occlusion, missed detections, and false positives and use combinatorial analysis to reduce the search space and to find a greedy solution. The proposed algorithms have been applied (by others) in various applications including visual surveillance, feature tracking, structure from motion, and medical image analysis.

Related Publications

  1. "A Rank Constrained Continuous Formulation of Multi-frame Multi-Target Problem ," with Lee M. and Haering N., IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, Alaska, June 2008.
  2. "A Non-Iterative Greedy Algorithm for Multi-frame Point Correspondence," with Shah M., IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 27, no. 1, January 2005.

Return to top of the page

Object Based Video Coding

We present a novel object-based video coding framework for videos obtained from a static camera. As opposed to most existing methods, the proposed method does not require explicit 2D or 3D models of objects and hence is general enough to cater for varying types of objects in the scene. The proposed system learns the appearance model of each object online using incremental principal component analysis (IPCA). Due to smooth transitions between limited number of poses of an object, usually a limited number of significant principal components contribute to most of the variance in the object’s appearance space and therefore only a small number of coefficients are required to code the object. The rigid component of the object’s motion is coded in terms of its affine parameters. The framework is applied to compressing videos in surveillance and video phone domains and is shown to provide higher Peak Signal to Noise Ratio (PSNR) compared to MPEG-2 and MPEG-at comparable or better bitrates.

Related Publications

  1. "Object Based Video Coding Framework for Static Camera Videos," with Hakeem A., and Shah M., ACM Multimedia Conference, Singapore, 2005. (Nominated for Best Paper Award)

Return to top of the page

Radiometric Analysis

The mapping that relates the image irradiance to the image brightness (intensity) is known as the Radiometric Response Function or Camera Response Function. This usually unknown mapping is nonlinear and varies from one color channel to another. I have worked on the problem of estimation of the radiometric response functions (of R, G and B channels) of a color camera directly from the images of an arbitrary scene taken under different illumination conditions (The illumination conditions are not assumed to be known). The response function of a channel is modeled as a gamma curve and is recovered by using a constrained nonlinear minimization approach by exploiting the fact that the material properties of the scene remain constant in all the images. I have also investigated separation of diffuse and specular components from images.

Related Publications

  1. "Estimation of the Radiometric Response Functions of a Color Camera from Differently Illuminated Images," with Shah M., IEEE International Conference on Image Processing (ICIP), Singapore, October 2004.

Return to top of the page

Alliances in Graphs

Alliances in a graph define grouping among the vertices of the graph based on their neighborhood properties. I have studied different types of alliances in graphs, and the invariants associated with each type, for example, size of maximum and minimum alliances, partitioning and packing numbers, subgraph properties, forbidden subgraphs, etc. In particular, I am interested in the problem of partitioning a graph into specified type of alliances, for example:

  • What graphs have such partitions?
  • What is the maximum number of disjoint alliances in a given graph?
  • What is the maximum order of a partition in a given graph, so that each set in the partition is an alliance of a given type?
  • If such a partition does not exist in a graph then what is the best approximate partition?
  • What are the computational complexities for the above problems?

Related Publications

  1. "Partitioning a Graph in Alliances and its Application to Data Clustering," School of Computer Science, University of Central Florida, Orlando, FL, 2004. (Note: Proof for NP-Completeness of “Partition into Global Defensive Alliances” [Theorem 59, p. 64] is misprinted in the original version. The corrected proof is available here.
  2. "Partitioning a Graph in Alliance Free Sets," with Dutton R. D., Discrete Mathematics, to appear.
  3. "A Tight Bound on the Cardinalities of Maximum Alliance-Free and Minimum Alliance-Cover Sets," with Dutton R.D., Journal of Combinatorial Mathematics and Combinatorial Computing, vol. 56, 2006.
  4. "On X free Covers," with Dutton R. D., Congressus Numerantium, 2004.
  5. "Maximum Alliance-Free and Minimum Alliance-Cover Sets," with Dutton R. D., Congressus Numerantium 162, pp. 139-146, 2003.
  6. "On Satisfactory Partitioning of Graphs," with Dutton R. D., Congressus Numerantium 154, pp. 183-194, 2002.

Return to top of the page