My Research


Research Topics:


Manifold Models for Human Motion Analysis  
Modeling View and Posture Manifolds  
We
consider modeling data lying on multiple continuous manifolds. In
particular, we model shape manifold of a person performing a motion observed
from different view points along a view circle at fixed camera height. We
introduce a model that ties together the body configuration (kinematics)
manifold and the visual manifold (observations) in a way that facilitates
tracking the 3D configuration with continues relative view variability. The
model exploits the low dimensionality nature of both the body configuration
manifold and the view manifold where each of them are represented
separately. 

Tracking People on a Torus  
Suppose
we want to model the visual patterns of a periodic articulated motion (such
as walking) observed from any view point. Such visual patterns lie on a
product space (different body configuration X different views). We showed
that a topology preserving setting is suitable to model certain human
motions which lie intrinsically on onedimensional manifolds, whether closed
and periodic (such as walking, jogging, running, etc.) or open (such as golf
swing, kicking, tennis serve, etc.) We showed that we can represent the
visual manifold of such motions (in terms of shape) as observed from
different view points by mapping such data to a torus manifold (for
the case of a single view circle) or family of tori (for the whole
view sphere). The approach we introduced is based on learning the visual
observation manifold in a supervised manner. Traditional manifold learning
approaches are unsupervised where the goal is to find a low dimensional
embedding of the data. However, if the manifold topology is known, manifold
learning can be formulated as learning a mapping from/to a topological
structure to/from the data where that topological structure is homeomorphic
to manifold of the data. 

Multifactor Models for Style Separation  
We show
how to model several ``style'' factors using multilinear analysis in the
space of nonlinear basis functions. This way we can separate different
sources of style variations of the same motions. For example, the model can
be used to render data for different people's walking figures from different
views; or different faces performing different facial expressions. On the
other hand, given an input pattern, we introduced an optimization process
that can solve for the different factors which produced that pattern. For
example, from a single shape instance we can recover the body configuration,
the view point, and the person's shape identity. Similarly, from a single
face image, we can recover the facial expression, the face identity, and the
motion phase. 

Separating Style and Content on a Nonlinear Manifold  
Bilinear and multilinear models have been successful in decomposing static image ensembles into perceptually orthogonal sources of variations, e.g., separation of style and content. If we consider the appearance of human motion such as gait, facial expression and gesturing, most of such activities result in nonlinear manifolds in the image space. The question that we address in this research is how to separate style and content on manifolds representing dynamic objects. We learn a decomposable generative model that explicitly decomposes the intrinsic body configuration (content) as a function of time from the appearance (style) of the person performing the action as timeinvariant parameter. The framework is based on decomposing the style parameters in the space of nonlinear functions which map between a learned unified nonlinear embedding of multiple content manifolds and the visual input space.  
Inferring 3D Body Pose from Silhouettes using Activity Manifold Learning  
We aim to infer 3D body pose directly from human silhouettes. Given a visual input (silhouette), the objective is to recover the intrinsic body configuration, recover the view point, reconstruct the input and detect any spatial or temporal outliers. In order to recover intrinsic body configuration (pose) from the visual input (silhouette), we explicitly learn viewbased representations of activity manifolds as well as learn mapping functions between such central representations and both the visual input space and the 3D body pose space. The body pose can be recovered in a closed form in two steps by projecting the visual input to the learned representations of the activity manifold, i.e., finding the point on the learned manifold representation corresponding to the visual input, followed by interpolating 3D pose.  
Nonlinear Generative Models for Dynamic Shape and Dynamic Appearance  
Our objective is to learn representations for the shape and the appearance of moving (dynamic) objects that supports tasks such as synthesis, pose recovery, reconstruction and tracking. We introduce a framework that aim to learn a landmarkfree correspondencefree global representations of dynamic appearance manifolds. We use nonlinear dimensionality reduction to achieve an embedding of the global deformation manifold that preserves the geometric structure of the manifold. Given such embedding, a nonlinear mapping is learned from the embedding space into the visual input space. Therefore, any visual input is represented by a linear combination of nonlinear bases functions centered along the manifold in the embedding space. We also show how approximate solution for the inverse mapping can be obtained in a closed form which facilitate recovery of the intrinsic body configuration. We use the framework to learn the gait manifold as an example of a dynamic shape manifold, as well as to learn the manifolds for some simple gestures and facial expressions as examples of dynamic appearance manifolds.  
Bilinear and Multilinear Models for Gait Recognition  
Human Identification using gait is a challenging computer vision task due to the dynamic motion of gait and the existence of various sources of variations such as viewpoint, walking surface, clothing, etc. In this research we investigate gait recognition algorithms based on bilinear and multilinear decomposition of gait data into timeinvariant gaitstyle and timedependent gaitcontent factors. We developed a generative model by embedding gait sequences into a unit circle and learning nonlinear mapping which facilitates synthesis of temporallyaligned gait sequences. Given such synthesized gait data, bilinear model is used to separate invariant gait style which is used for recognition. We also show that the recognition can be generalized to new situations by adapting the gaitcontent factor to the new condition and therefore obtain corrected gaitstyles for recognition.  
Exemplarbased Tracking and Gesture Recognition  nonparametric HMMs  
In this research we addresses the problem of capturing the dynamics for exemplarbased recognition systems. Traditional HMM provides a probabilistic tool to capture system dynamics and in exemplar paradigm, HMM states are typically coupled with the exemplars. Alternatively, we propose a nonparametric HMM approach that uses a discrete HMM with arbitrary states (decoupled from exemplars) to capture the dynamics over a large exemplar space where a nonparametric estimation approach is used to model the exemplar distribution. This reduces the need for lengthy and nonoptimal training of the HMM observation model. We used the proposed approach for viewbased recognition of gestures. The approach is based on representing each gesture as a sequence of learned body poses (exemplars). The gestures are recognized through a probabilistic framework for matching these body poses and for imposing temporal constraints between different poses using the proposed nonparametric HMM.  
Tracking:  
Learning to Track  
Tracking
is typically posed as a search problem in a geometric transformation
parameter space as well as in the object's configuration parameter space.
Generally, tracking is based on learning an invariant representation of the
tracked object, then searching the parameter space for the best fit. The goal of this research is to achieve trackers that can directly ``infer'' such parameters from the object appearance through learned models of the visual manifolds of such parameter spaces. We show how to learn a representation of the appearance manifold of an object, given a class of geometric transformation. We learn a generative model for object appearance where the appearance of the object at each new frame is an invertible function that maps from a representation of the geometric transformation space into the visual space. By learning such generative model we can infer the geometric transformation (track) directly from the tracked object appearance. As a result tracking can be achieved in a closedform and therefore can be done very efficiently. The novelty of this work is that it showed how learning the appearance manifold of an object can play a role to achieve efficient tracking. 

AppearanceBased Generalized Kernel Tracking  
We exploit the featurespatial distribution of a region representing an object as a probabilistic constraint to track that region over time. The tracking is achieved by maximizing a similaritybased objective function over transformation space given a nonparametric representation of the joint featurespatial distribution. Such a representation imposes a probabilistic constraint on the region feature distribution coupled with the region structure which yields an appearance tracker that is robust to small local deformations and partial occlusion. We presented the approach for the general form of joint featurespatial distributions and apply it to tracking with different types of image features including row intensity, color and image gradient.  
Tracking Multiple People  
In this research we address the problem of segmenting foreground regions corresponding to a group of people given models of their appearance that were initialized before occlusion. We present a general framework that uses maximum likelihood estimation to estimate the best arrangement for people in terms of 2D translation that yields a segmentation for the foreground region. Given the segmentation result we conduct occlusion reasoning to recover relative depth information and we show how to utilize this depth information in the same segmentation framework. We also present a more practical solution for the segmentation problem that is online to avoid searching an exponential space of hypothesis. The person model is based on segmenting the body into regions in order to spatially localize the color features corresponding to the way people are dressed. Modeling these regions involves modeling their appearance (color distributions) as well as their spatial distribution with respect to the body. We use a nonparametric approach based on kernel density estimation to represent the color distribution of each region and therefore we do not restrict the clothing to be of uniform color. Instead, it can be any mixture of colors and/or patterns. We also present a method to automatically initialize these models and learn them before the occlusion.  
Scene Modeling and Background Subtraction  
Feature Selection for Background Subtraction  Boosted Background Model  
Various statistical approaches have been proposed
for modeling a given scene background. However, there is no theoretical
framework for choosing which features to use to model different regions of
the scene background. In research paper we introduce a novel framework for
feature selection for background modeling and subtraction. A oosting
algorithm, namely RealBoost, is used to choose the best combination of
features at each pixel. Given the probability estimates from a pool of
features calculated by Kernel Density Estimate (KDE) over a certain time
period, the algorithm selects the most useful ones to discriminate
foreground objects from the scene background. The results show that the
proposed framework successfully selects appropriate features for different
parts of the image.


Nonparametric Model for Background Subtraction  
In video
surveillance systems, stationary cameras are typically used to monitor activities at
outdoor or indoor sites. Since the cameras are stationary, the detection of moving objects
can be achieved by comparing each new frame with a representation of the scene background.
This process is called background subtraction and the scene representation is called the
background model. Typically, background subtraction forms the first stage in automated
visual surveillance systems. Results from background subtraction are used for further
processing, such as tracking targets and understanding events. We introduced a novel background model and a background subtraction technique based on statistical nonparametric modeling of pixel process. The model keeps a sample of intensity values for each pixel in the image and uses this sample to estimate the probability density function of the pixel intensity. The density function is estimated using kernel density estimation technique. Since this approach is quite general, the model can approximate any distribution for the pixel intensity without any assumptions about the underlying distribution shape. The model can handle situations where the background of the scene is cluttered and not completely static but contains small motions that are due to moving tree branches and bushes. The model is updated continuously and therefore adapts to changes in the scene background. The approach runs in realtime. Code is available per request


Efficient Kernel Density Estimation using Fast Gauss Transform  
Many vision algorithms depend on the estimation of a probability density function from observations. Kernel density estimation techniques are quite general and powerful methods for this problem, but have a significant disadvantage in that they are computationally intensive. In this research we explore the use of kernel density estimation with the fast Gauss transform (FGT) for problems in vision. The FGT allows the summation of a mixture of M Gaussians at N evaluation points in O(M+N) time as opposed to O(MN) time for a naive evaluation, and can be used to considerably speed up kernel density estimation. We present applications of the technique to problems from image segmentation and tracking, and show that the algorithm allows application of advanced statistical techniques to solve practical vision problems in real time with today's computers. 