Instance-specific and Class-consistent Cues

• We aim to resolve the difficulties of action recognition arising from the large intra-class variations. These unfavorable variations make it infeasible to represent one action instance by other ones of the same action. We hence propose to extract both instance-specific and class-consistent features to facilitate action recognition.
• Credits
  • Instance-specific features: Self-similarities among frames of an action sequence. Multivariate linear prediction (MLP) is adopted to aggregate all the causalities among frames.
  • Class-consistent features: Characteristics shared by instances of the same action. Support vector machines (SVMs) are used to discover these features based on the bag-of-words model.
  • We propose a generative formulation to integrate the two complementary types of features, and boost the performance.

Core Techniques

• We view actions as multivariate time signals. For signal processing in our approach, there are several essential techniques:
  • Wide-Sense Stationary Process
  • Linear Prediction
  • Support Vector Machine
• We propose a generative model. The main idea is to consider the static and dynamic information of a multivariate time signal separately. This is based on the assumption that these two information are independent for action recognition.
  • Instance-specific cue via MLP
  • Class-consistent cue via SVM
  • Linear fusion
    　

Action Recognition using Instance-specific and Class-consistent Cues

Semantic Manifold Learning for Image Retrieval

• To address the problem of CBIR with relevance feedback by manifold learning
  • Two aspects of information are being fused: Intrinsic Similarity Relations and Query & Relevance Feedback
  • User-specific Semantic Manifold
• A manifold-learning technique to effectively fuse the multi-modality of information
  • Intrinsic Similarity Relations: abundant, and auxiliary
  • User Relevance Feedback: few, but crucial

• User relevance feedback serves as augmented relations to intrinsic similarity relations
  • Augmented Relations Embedding (ARE)
  • Augmented features: global and local image features

Features

• Global
  • Color
    • Quantized HSV color histogram: 64 dimensions
    • First three moments in each color channel: 9 dimensions
    • Color coherence vector: 128 dimensions
  • Texture
    • Tamura coarseness: 10 dimensions
    • Tamura directionality: 8 dimensions
  • Wavelet
    • 3-level DWT image decomposition
    • The first two moments in each high-frequency sub-bands: 18 dimensions
• Local
  • Detecting salient regions (associated with interest points)
  • Describing the local properties of each salient region
  • Difference-of-Gaussian (DoG) for salient-region detection [Lowe 2004]

• Augmented Features
  • The proposed image representation
  • Augmenting the local-feature scheme with global image properties so that the vector quantization just described can be used to derive a k-dimensional vector

Visualization of Embedding Space

• The iterative procedure
  • Use ARE to compute a 30-D semantic manifold
  • Project data points onto a 2-D plane by multidimensional scaling (MDS) [Cox and Cox, 1994] for visualization

Firework
Office Interiors

Semantic Manifold Learning for Image Retrieval

Multiple Kernel Learning with Local Learning

• We address two unfavorable issues of local learning, i.e., high risk of overfitting and heavy computational cost, and present an efficient boosting algorithm to learn sample-specific local classifiers for object category recognition.

• Our approach
  • We cast the multiple, independent training processes of local classifiers as a correlative multi-task learning problem.
  • We establish a parametric space where these local classifiers lie and spread as a manifold-like structure.
  • By designing a new multi-task boosting algorithm, the local classifiers are obtained by completing the manifold embedding.
  • The algorithm carries out incremental multiple kernel learning.

  

Cluster-dependent Feature Selection

• A chicken-and-egg problem.
  • Clustering VS. Feature selection: Is it the clustering that contributes to the feature selection, or the feature selection that boosts the clustering?
• Again, the optimal data descriptors often vary from cluster to cluster.
• With the idea of associating each cluster with a learnable ensemble kernel, we integrate multiple kernel learning into the clustering procedure, and cast it as a joint optimization problem.

Cluster-dependent Feature Selection by Multiple Kernel Self-organizing Map

Clustering Complex Data with Group-dependent Feature Selection

Efficient Discriminative Local Learning for Object Recognition

Real-time, Multi-view Face Detection

• A real-time, multi-view face detector
• Multi-view: Profile faces, rotated faces, faces with partial occlusions, or faces under different lighting conditions
• Real-time: At least 15 frames per second on a PC
• Two key components
• A new boosting algorithm: Multi-class Bhattacharyya boost (MBHBoost)
• A new detection architecture: Multi-class cascade
• The detector leads to a computational cost sub-linear to the number of face classes (views)

Key Features / Techniques

• Vector-valued weak learners
• A new boosting algorithm: Multi-class Bhattacharyya boost (MBHBoost)
• A new detection architecture: Multi-class cascade

Rectangle Features

Problems Caused by Thresholding and Our Solution

Classifier Sharing

• An vector-valued weak learner associated with is defined by
  
• Advantages
  • Shared by all classes, no human knowledge or searching
  • Each component independently learns a decision boundary
  • Computational efficiency: The value of k is identical in all components

Multi-class Cascade

• Reduce the detection problem to a series of pattern rejection problem
• Speed up the detection process: Coarse-to-fine detection
• Deal with vector-valued outputs: Message-passing between stages

Robust Face Detection with Multi-class Boosting

Fast Object Detection with Occlusions

What is the problem in feature matching?

• Motivation: For image matching, the initial feature correspondence set
  • Can not be too large: Low recall
  • Contains corrupt matches: Low precision
  • Requires geometric checking: Time consuming

• Proposed method: Alternate Hough and inverted Hough voting
  • Correspondence mutual checking in homography space
• Main advantages:
  • High precision: Hough voting
  • High recall: Inverted Hough voting
  • Low computational cost: BPLR for voter filtering

Core Ideas

• Precision:
  • The correct matches are biased to a dense cluster in the transformation space
  • We cast the task of feature matching problem into a density estimation problem
• Recall:
  • Grouped features with high probability undergo similar transformations in matching
  • We utilize the nature of BPLR to locate non-crossboundary regions which correspond to groups of similar transformations
    　

Core Techniques

• Hough voting
  • The tentative correspondences are found via nearest-neighbor search in descriptor space and use to generate votes in the transformation space.
  • We use density of each correspondence in the transformation space to verify its correctness
• Inverted hough voting
  • Recommend each feature additional transformations by investigating density distribution of nearby features covered by the same BPLR.

Robust Feature Matching with Alternate Hough and Inverted Hough Transforms

Object Recognition with Kernel Machines

• The problems and observations
  • Diverse object categories
  • Large intra-class variations
  • The goodness of a feature representation for recognition is often category-dependent, and can even be object-dependent for the case of large intra-class variation
• Improve recognition accuracy by focusing on
  • Image features
  • Similarity measures
• Approach
  • Fusing various feature representations in the domain of kernel matrices
  • Learning a local ensemble kernel machine (e.g. local ensemble kernel + SVM) for each training sample
    • Localized kernel alignment (initialization)
    • MRF modeling and optimization
    • Associate each sample with a SVM classifier
• Feature Fusion and Kernel Alignment
  • Fusing feature representations is now combining kernels
  • The effectiveness of a possible fusion can therefore be reasonably estimated by how good an ensemble kernel is
  • We consider target kernel alignment for measuring the goodness of a kernel. [Cristianini et al. '01]
• Optimization over a MRF Model
  • Retain the effectiveness of each local classifier
  • Alleviate the possible overfitting
  • Reduce the redundancy of the local classifiers

Multiple Kernel Learning (MKL) for Dimensionality Reduction

• Observations
  • No single feature representation suffices to explain the complexity of the whole data
• Goals
  • Perform MKL for heterogeneous feature fusion
  • Generalize a set of dimensionality reduction methods to consider multiple kernels
  • Extend MKL from supervised learning to unsupervised and semi-supervised learning
  • Improve performances of vision applications by using multiple feature representations
    　

Multiple Kernel Learning for Dimensionality Reduction

Dimensionality Reduction for Data in Multiple Feature Representations

Local Ensemble Kernel Learning for Object Category Recognition

Single-Camera People Counting

• Cross-camera people counting
  • Applicable to various environments
  • Online training data acquisition and camera perspective estimation

• Occlusion handling: Coupled Gaussian processes
  • First-pass Gaussian processes: Visible part
  • Second-pass Gaussian processes: Occluded part

Why Predict Conflict Works

• An example: prediction conflict between horizontal and vertical gradients for occlusion handling

• Legend
  • ---- Ground truth
  • ---- Prediction by the feature of horizontal gradients
  • ---- Prediction by the feature of vertical gradients

Blob Representation

• We focus on foreground objects (pedestrians) in images
• Background subtraction
• Grouping spatially connected pixels

People Counting with Multiple Cameras

• Why multiple cameras?
  • Complementary information
  • Dealing with resolution issues, occlusions, ...

• Our approach
  • Ground plane matching + Visual knowledge transfer

Why Bob Matching?

• Find the same groups of pedestrians across cameras
  • Synchronized frames
    • The numbers of people are not always equal, particularly when FOVs are quite different
  • We work on corresponding blob sets: blob clusters

Observation

• Approximating the people counts in an image in two parts
  • Regular part: intra-camera visual features
  • Residual part: inter-camera visual knowledge

• Formulate it as a transfer learning problem

Visual Knowledge Transfer among Multiple Cameras for People Counting with Occlusion Handling

Cross Camera People Counting with Perspective Estimation and Occlusion Handling

Class-based Image Segmentation

• A concise annotation method for collecting training data for class based image segmentation. Two steps
  • Generate multiple tight segments, combining the multiple segment method with the concept of bounding box prior
  • Select the best segment by semi-supervised regression
• Credits
  • Present a novel algorithm which integrates the bounding box prior into the concept of multiple image segmentation, and automatically generate multiple tight segments
  • Case the segment selection as a semi-supervised regression problem
  • Demonstrate that our approach provides an effective alternative for manually labeled contours

Core Techniques

• Multiple Tight Segment Generation: Present an algorithm that automatically generates a set of tight segments for the bounding box of an object, and at least one of these tight segments would approach the object segment
• Segment Selection: Given a few contours as well as a set of bounding boxes of an object class, we illustrate how to infer the object segments of these bounding boxes by solving a semi-supervised regression problem

Knowledge Leverage from Contours to Bounding Boxes: A Concise Approach to Annotation

Introduction

• Multi-class object recognition with few labeled data
  • Goal: Learn a target classifier with low generalization errors
  • Difficulty: When only few labeled data are available, over-fitting occurs. That is, the yielded classifier has poor generalization.
• What and how prior knowledge help to learn a robust classifier without labeling new data?
  • Source task: many existed labeled data
  • Target task: few labeled data
• Motivations: to leverage the extra source knowledge, together with the target knowledge in a common domain, and consequently learn a more robust multi-class classifier
• Conventional transfer learning algorithms: lack multi-class formulation

Core Ideas

• Attribute transfer
• Multi-classes (source) to multi-classes (target) knowledge transfer
• What to transfer: a sequence of learnable, discriminant attributes
  • Commonly shared by the source and target domains
  • Converted two multi-class classification tasks to related, binary ones
• How to transfer: Two-layer multi-task variant of AdaBoost.OC
  • Boosting algorithm with error-correcting output codes (ECOC)
    • Better generalization
  • Outer layer: Attribute partition discovery
    • Discriminant:Multi-class formulation
    • Learnable:Without human effort
    • Complementary:Iterative error minimization
  • Inner layer: Attribute classifier learning
    • Employ classifier sharing principle
    • Support multiple kernel learning: Combining various low-level features

• Our goal: Attributes should be learnable

Cross-Database Transfer Learning via Learnable and Discriminant Error-correcting Output Codes