LAHA: Learning To Anticipate Human Actions

This research is supported by the National Research Foundation Singapore under
its AI Singapore Programme (Award Number: AISG-RP-2019-010). Duration : October 2019 to March 2023

Our ability to anticipate the behaviour of others comes naturally to us. For example, an experienced driver can often predict the behaviour of other road users. Similarly, a good table tennis player can estimate the direction of the ball just by observing the movements of the opponent. This phenomenon is called Action Anticipation, the ability to recognise actions of others before it happens in the immediate future. This is so natural to us but how to develop a computational approach to do the same remains a challenge. It is critical to transfer this ability to computers so that robots may be able to react quickly by anticipating human actions like humans. Robots’ ability to understand what humans might do in the immediate future is important for the development of assistive robotics in domains such as manufacturing and healthcare. The objective of this project is to investigate a novel approach to anticipate human actions, specifically one to five seconds before action happens using visual information in human-robot engagement scenarios. Current action anticipation approaches primarily rely on new loss functions that allow us to tackle the uncertainty in future predictions. However, these approaches assume observed data contains rich information about the future human actions. Other approaches overcome this limitation by generating appearance information such as objects, environment and context for the future, and then classify those to anticipate future human actions. We argue that movement of objects and humans are the key indicators to anticipate human actions before they happen. Unfortunately, these approaches are not able to generate most useful information to anticipate actions because they only consider appearance and do not explicitly generate movement information such as motion.

*** We are the ranked 1 team in EPIC-KITCHEN 100 for action anticipation!

*** We are the ranked 1 team in EPIC-KITCHEN 55 for action anticipation!

*** All codes will be released here!


Interaction Visual Transformer for Egocentric Action Anticipation
Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando
IEEE/CVF Winter Conference on Applications of Computer Vision - WACV (2024)
Ranked 1 team in EPIC-KITCHEN 100
PDF Code Bibtex
Predicting the Next Action by Modeling the Abstract Goal
Debaditya Roy and Basura Fernando
Under Review
Ranked 1 team in EPIC-KITCHEN 55
PDF Code Bibtex
TDAM: Top-Down Attention Module for Contextually Guided Feature Selection in CNNs
Shantanu Jaiswal and Basura Fernando and Cheston Tan
ECCV 2022
PDF Bibtex
Anticipating human actions by correlating past with the future with Jaccard similarity measures
Basura Fernando and Samitha Herath
CVPR 2021
PDF Bibtex
Action Anticipation using Pairwise Human-Object Interactions and Transformers
Debaditya Roy and Basura Fernando
IEEE Transactions on Image Processing 2021
Impact factor 10.856
PDF Bibtex
Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting
Yan Bin Ng, and Basura Fernando
IEEE Transactions on Image Processing 2020
Impact factor 10.856
PDF Web Bibtex
Weakly supervised action segmentation with effective use of attention and self-attention.
Yan Bin Ng and Basura Fernando
Computer Vision and Image Understanding 2021
PDF Bibtex
Action anticipation using latent goal learning
Debaditya Roy and Basura Fernando
WACV 2022
PDF Code Bibtex
Long-term Action Forecasting Using Multi-headed Attention-based Variational Recurrent Neural Networks
Siyuan Brandon Loh, Debaditya Roy and Basura Fernando
CVPR 2022 (Workshop)
PDF Bibtex
A Log-likelihood Regularized KL Divergence for Video Prediction with A 3D Convolutional Variational Recurrent Network
Haziq Razali and Basura Fernando
WACV 2021
Generation of Human Behavior Workshop

PDF Bibtex
FlowCaps: Optical Flow Estimation with Capsule Networks For Action Recognition
Vinoj Jayasundara, Debaditya Roy and Basura Fernando
WACV 2021
PDF Bibtex
What do CNNs gain by imitating the visual development of primate infants?
Shantanu Jaiswal, Dongkyu Choi, Basura Fernando
BMVC 2020
PDF Bibtex
Weakly Supervised Gaussian Networks for Action Detection
Basura Fernando and Cheston Tan and Hakan Bilen
WACV 2020
PDF arxiv Bibtex
Human Action Sequence Classification
Yan Bin Ng and Basura Fernando
PDF Bibtex

Team Members

1. 2019-2021 Yan Bin Ng (Research Engineer)

2. 2019-2021 Haziq Razali (Research Engineer)

3. 2019-2020 Vinoj Jayasundara (Research Engineer)

4. 2020-now Debaditya Roy (Research Scientist)

5. 2020-2021 Brandon Loh (Research Engineer)

6. 2021-2022 Johnathon Toh (Research Engineer)

7. 2022-2022 Yang Hong (Research Engineer)

8. 2022-2022 Venkata Sai Vijay Kumar (Research Engineer)