Learning why do we do what we do - Understanding human actions using Neurosymbolic AI

Recognizing what we do or what we will do has been well investigated under action recognition and anticipation literature in the Computer Vision (CV) and Machine Learning (ML) research communities. However, computational learning of why do we do what we do is not well investigated. The objective of this project is to develop Artificial Intelligent (AI) models to process videos and learn why do humans do what they do? by reducing the gap between neural and symbolic representation through novel neurosymbolic AI. These neurosymbolic AI models can see what we do and then reason about our behavior to interpret, justify, explain and understand our actions.

Publications

	Who are you referring to? Coreference resolution in image narrations Arushi Goel and Basura Fernando and Frank Keller and Hakan Bilen International Conference on Computer Vision - ICCV (2023) PDF CIN Dataset Code Bibtex @inproceedings{arushi2023, title={Who are you referring to? Coreference resolution in image narrations}, author={Arushi Goel and Basura Fernando and Frank Keller and Hakan Bilen}, booktitle={International Conference on Computer Vision 2023}, pages={}, year={2023} } ×
	Semi-supervised multimodal coreference resolution in image narrations Arushi Goel and Basura Fernando and Frank Keller and Hakan Bilen Empirical Methods in Natural Language Processing - EMNLP (2023) PDF Bibtex @inproceedings{emnlp2023, title={Semi-supervised multimodal coreference resolution in image narrations}, author={Arushi Goel and Basura Fernando and Frank Keller and Hakan Bilen}, booktitle={Empirical Methods in Natural Language Processing 2023}, pages={}, year={2023} } ×
	Energy-based Self-Training and Normalization for Unsupervised Domain Adaptation Samitha Herath, Basura Fernando, Ehsan Abbasnejad, Munawar Hayat, Shahram Khadivi, Mehrtash Harandi, Hamid Rezatofighi, and Reza Haffari International Conference on Computer Vision - ICCV (2023) PDF Bibtex @inproceedings{samith2023, title={Energy-based Self-Training and Normalization for Unsupervised Domain Adaptation}, author={Samitha Herath and Basura Fernando and Ehsan Abbasnejad and Munawar Hayat and Shahram Khadivi and Mehrtash Harandi and Hamid Rezatofighi and Reza Haffari}, booktitle={International Conference on Computer Vision 2023}, pages={}, year={2023} } ×
	ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition Debaditya Roy and Dhruv Verma and Basura Fernando IEEE/CVF Winter Conference on Applications of Computer Vision - WACV (2024) Best results in SWiG - 2024 Best results in imSitu - 2024 Code PDF Bibtex @inproceedings{clipsitu2024, title={ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition}, author={Debaditya Roy and Dhruv Verma and Basura Fernando}, booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision WACV 2024}, pages={}, year={2024} } ×
	A Region-Prompted Adapter Tuning for Visual Abductive Reasoning Hao Zhang, Yeo Keat Ee, Basura Fernando Preprint PDF
	Abductive Action Inference Clement Tan and Chai Kiat Yeo and Cheston Tan and Basura Fernando Preprint PDF
	Learning to Visually Connect Actions and their Effect Eric Peh and Paritosh Parmar and Basura Fernando Preprint PDF