Research Areas (with selected publications)
Machine Learning and Pattern
Unsupervised and Semi-Supervised Deep Learning [pdf] [pdf] [pdf]
Multimodal Representation Learning [pdf] [pdf]
Generative Adversarial Network Theory and Applications [pdf] [pdf]
o Model Security and Privacy in Deep Learning [pdf] [pdf]
o Network Architecture Search and Model Compression [pdf] [pdf]
Vision and Multimedia Computing
2D/3D Scene Reconstruction, Mapping and Localization [pdf] [pdf]
Video Analysis and Understanding [pdf] [pdf]
Face/Gesture Recognition and Human Pose Estimation [pdf] [pdf]
Internet-of-Things (IOTs) Information Processing and Analytics
Discovery of trustworthy sources/sensors [pdf] [pdf]
Multi-source information fusion [pdf] [pdf]
Heterogeneous sensor data analysis [pdf] [pdf]
Network Pretraining] Using self-supervised methods for
unsupervised, semi-supervised and/or supervised (pre-)training of CNNs, GCNs,
We developed two novel paradigms of self-supervised methods a)
Auto-Encoding Transformations (AET)
that learns Transformation-Equivariant Representations; b)
Adversarial Contrast (AdCo) that directly self-trains negative pairs in contrastive learning approach.
- 1) Unsupervised training of
and AETv2 [link],
- 2) Variational
AET and the connection to transformation-equivariant
representation learning [link][pdf][github],
- 3) (Semi-)Supervised
AET training with an ensemble of spatial and non-spatial
- 4) GraphTER (Graph Transformation Equivariant Representation): Unsupervised
training of Graph Convolutional Networks (GCNs) for 3D Scene Understanding based on Point Cloud Analysis [pdf][github],
- 5) Transformation
by using the AET loss to train the discriminator for better
generalization to create new images [pdf].
- 6) Adversarial Contrast (AdCo) [pdf][github]:
An adversarial contrastive learning method to directly train
negative samples end-to-end. It shows high performance to
on ImageNet with
20% fewer epochs than the SOTA methods (e.g., MoCo v2, and BYOL)
achieving even better top-1 accuracy. The model is easy to implement
and can be used as a plug-in algorithm to combine with many
- 7) Multi-task AET (MAET) for Dark Object Detection [pdf]:
We propose a multi-task AET for visual representation learning in
low-light environment for object detection. It applies an orthogonal
regularity among the tangents under both spatial and low-illumination
degrading transformations to minimize the cross-task redundancy,
which delivers the SOTA performance on dark object detection.
(a) AutoEncoding Transformations (AET) [pdf
Graph TER (GTER) [pdf
Multitask AET (MAET)
Comparison of BYOL vs. AdCo. While BYOL has to learn a multi-layer of
MLP predictor (highlighted in red) to estimate the represenation of the
other branch, AdCo
instead learns a single layer of negative adversaries. For the first
time, the AdCo shows the negative samples are learnable to track the
change of represenations over the pretraining course, with superior
performances on downstream tasks.
- [Regularized GANs and Applications to Multimedia Content Synthesis and Manipulation] We
present a regularized Loss-Sensitive GAN (LS-GAN),
and extended it to a generalized version (GLS-GAN) with
many variants of regularized GANs as its special cases.
We proved both the distributional consistency and generalizability of
the LS-GAN with polynomial
sample complexity to generate new contents. See more
- 1) LS-GAN and GLS-GAN [pdf][github],
- 2) A landscape of
regularized GANs in a big picture [url],
- 3) An
extension by obtaining an encoder of
input samples directly with
margins through the loss-sensitive GAN [github: torch,
- 4) The LS-GAN has been
adopted by Microsoft CNTK (Cognitive Toolkit) as a reference
regularized GAN model [link].
Localized GAN was used to model the manifold of images along their
tangent vector spaces. It was used to capture and/or generate
local variants of input images so that their attributes can be edited
by manipulating the input noises. The local variants of
along the tangents can also be used to approximate
the Beltrami-Laplace operator for semi-supervised
The map of conventional vs. regularized GANs, in which the GLS-GAN
contains all known regularized GANs as its special cases [pdf
It provides a systematic plot of regularized GAN models found thus
far from both theoretic and practical perspectives. The proposed metric,
Minimum Recontruction Error (MRE) [pdf
] also gives a quantity measure of generalizability to generate and synthesize new
contents out of existing examples. This demonstrates regularized GANs
such as LS-GAN and GLS-GAN are models not only merely memorizing
training examples, but also being able to create contents never
- Machine Learning for Internet-Of-Things (IOTs) and Multi-Source Analysis] We
developed 1) State-Frequency Memory RNNs [pdf]
for multiple-frequency analysis of signals, 2) Spatial-Temporal
to integrate self-attentions over spatial topolgy and temporal
dynamics for traffic forecasting, and 3) First-Take-All
to efficiently index and retrieve multimodal sensor signals at
- 1) State-Frequency Memory (SFM)
RNNs for Multi-Source Signal/Financial Data Analysis. It explores
multiple frequencies of
dynamic memory for time-series analysis through SFM RNNs.
The multi-frequency memory enables more accurate signal
the LSTM in various ranges of dynamic contexts. For example, in
financial anlayis [pdf],
long-term investors use low-frequency information to
forecast asset prices, while
high-frequency traders rely more on high-frequency pricing signals to
- 2) Spatial-Temporal Transformer and Applications to Traffic
Forecasting. The spatial-temporal transformer [pdf]
is among one of the first works to apply self-attention to
graph neural networks by exploring both the network topology and
dynamics to forecast traffic flows from city-scale IOT data.
- 3) First-Take-All Hashing and
The First-Take-All (FTA) hashing was developed to
efficiently index dynamic activities captured by multimodal sensors
(cameras and depth sensors) [pdf]
fior eldercare, and image [pdf]
and cross-modal retrieval [pdf].
It is also applied to
classify singals of brain neural activities for early
which is one order of magnitude faster than
the SOTA methods on the
multi-facility dataset in a Kaggle Challenge .
- 4) Temporal alignment between Multi-Source
We propose Dynamically Programmable Layers to
automatically align signals from multiple sources/devices. We
its application to predict the brain connectivities between neurons [pdf].
- 5) Sensor Selection and Time-Series Prediction. We propose State-Stacked Sparseness [pdf] for sensor selection and the Mixture Factorized Ornstein-Uhlenbeck Process [pdf]
for time-series forecasting. The method considers the impact of both
faulty sensors (e.g., damaged and out-of-battery) and the change of
hidden states of the underlying mechanic/electric system for
time-series analysis and predictions.
- 6) E-Optimal Sensor Deployment and
We develop an optimal online sensor selection approach with the
restricted isometry property based on e-optimality [link].
successfully applied for collaborative spectrum sensing in cognitive
radio networks (CRNs), and selecting the most informative features from
a large amount of data/signals. The paper will be featured in
IEEE Computer's "Spotlight
on Transactions" Column.
(a) Comparison of RNN, LSTM and SFM for finanical analysis [pdf
(b) Spectrum by SFM [pdf
MF Ornstein-Uhlenbeck Process [pdf
- [Small Data Challenges with Limited Supervision]
Take a look at our survey of "Small
Data Challenges in Big Data Era: A
Survey of Recent Progress on Unsupervised and Semi-Supervised Methods"
and our tutorial presented at IJCAI 2019 [link]
with the presentation slides [pdf].
Also see our recent works on
- 1) Unsupervised Learning.
AutoEncoding Transformations (AET) [pdf],
Autoencoding Variational Transformations (AVT) [pdf],
GraphTER (Graph Transformation Equivariant Representations) [pdf], TrGANs
(Transformation GANs) [pdf],
- 2) Semi-Supervsied Learning.
Localized GANs (see how to compute Laplace-Beltrami operator directly
for semi-supervised learning) [pdf],
Ensemble AET [pdf],
- 3) Few-Shot Learning.
FLAT (Few-Short Learning via AET) [pdf],
knowledge Transfer for few-shot learning [pdf],
task-agnostic meta-learning [pdf]
Overview of small data methods with limited or no supervision [pdf
- [MAPLE Github] We
are releasing the source code of our research projects
at our MAPLE github homepage [url].
We are inviting everyone interested in our works to
Feedbacks and pull requests are warmly welcome.
Back to top