Sarosij Bose

email:  ojnsmemsearb.sosoce@ii.sunscramble

I am a Ph.D. student at University of California, Riverside advised by Dr. Amit K. Roy Chowdhury and Dr. Konstantinos Karydis. My research revolves around how autonomous systems perceive this world and building pipelines that can enable them to understand scenes using minimal supervision just as humans do.

I received my Bachelor in Technology (BTech.) degree in Computer Science and Engineering from University of Calcutta, India. I am fortunate to be advised by some amazing researchers over the years. I interned at Siemens with the Future of Automation (FoA) division on designing sensor realistic LiDAR simulation systems. I am a recipient of the MITACS Globalink Fellowship and worked with Prof. Jiju Poovvancheri at GSC Lab, Saint Mary's University on skeleton extraction for motion capture. Earlier, I spent consecutive semester breaks at Indian Institute of Science working with Prof. Anirban Chakraborty at VCL Lab and Prof. Kunal Narayan Chaudhury at LISA Lab on person-reidentification and lipschitz regularization respectively. I have also worked extensively with Prof. Amlan Chakrabarti on action recognition from videos.

Email /  Scholar  /  Twitter  /  Github  /  LinkedIn

I am currently focused on enhancing the perception capabilities of robotic systems by developing robust and generalizable world representations, with a particular interest in leveraging foundation models for comprehensive scene understanding. I have also worked on pose estimation under occlusions. My previous work encompassed various sub-domains, including video understanding, person-reid and shape approximation methods with an emphasis on utilizing unlabeled data for real-world applications.


I am always open to new collaborations and research ideas. Feel free to reach out if you are interested in working together!

Scene Graph Generation (SGG) aims to represent visual scenes by identifying objects and their pairwise relationships, providing a structured understanding of image content. However, inherent challenges like long-tailed class distributions and prediction variability necessitate uncertainty quantification in SGG for its practical viability. In this paper, we introduce a novel Conformal Prediction (CP) based framework, adaptive to any existing SGG method, for quantifying their predictive uncertainty by constructing well-calibrated prediction sets over their generated scene graphs. These scene graph prediction sets are designed to achieve statistically rigorous coverage guarantees. Additionally, to ensure these prediction sets contain the most practically interpretable scene graphs, we design an effective MLLM-based post-processing strategy for selecting the most visually and semantically plausible scene graphs within these prediction sets. We show that our proposed approach can produce diverse possible scene graphs from an image, assess the reliability of SGG methods, and improve019 overall SGG performance.

Occlusions are a significant challenge to human pose estimation algorithms, often resulting in inaccurate and anatomically implausible poses. Although current occlusion-robust human pose estimation algorithms exhibit impressive performance on existing datasets, their success is largely attributed to supervised training and the availability of additional information, such as multiple views or temporal continuity. Furthermore, these algorithms typically suffer from performance degradation under distribution shifts. While existing domain adaptive human pose estimation algorithms address this bottleneck, they tend to perform suboptimally when the target domain images are occluded, a common occurrence in real-life scenarios. To address these challenges, we propose OR-POSE: Unsupervised Domain Adaptation for Occlusion Resilient Human POSE Estimation. OR-POSE is an innovative unsupervised domain adaptation algorithm which effectively mitigates domain shifts and overcomes occlusion challenges by employing the mean teacher framework for iterative pseudo-label refinement. Additionally, OR-POSE reinforces realistic pose prediction by leveraging a learned human pose prior which incorporates the anatomical constraints of humans in the adaptation process. Lastly, OR-POSE avoids overfitting to inaccurate pseudo labels generated from heavily occluded images by employing a novel visibility-based curriculum learning approach. This enables the model to gradually transition from training samples with relatively less occlusion to more challenging, heavily occluded samples. Extensive experiments show that OR-POSE outperforms existing analogous state-of-the-art algorithms by ˜ 7% on challenging occluded human pose estimation datasets.

Classifying player actions from soccer videos is a challenging problem, which has become increasingly important in sports analytics over the years. Most state-of-the-art methods employ highly complex of- fline networks, which makes it difficult to deploy such models in resource constrained scenarios. Here, in this paper we propose a novel end-to-end knowledge distillation based transfer learning network pre-trained on the Kinetics400 dataset and then perform extensive analysis on the learned framework by introducing a unique loss parameterization. We also introduce a new dataset named "SoccerDB1" containing 448 videos and consisting of 4 diverse classes each of players playing soccer. Furthermore, we introduce an unique loss parameter that help us linearly weigh the extent to which the predictions of each network are utilized. Finally, we also perform a thorough performance study using various changed hyperparameters. We also benchmark the first classification results on the new SoccerDB1 dataset obtaining 67.20% validation accuracy. The dataset has been made publicly available at: https://bit.ly/soccerdb1

We present a novel shape approximation method using a pill decomposition approach given the surface points and their corresponding normals at each point on the surface. We first extract the maximal empty sphere representation of a given input shape and then construct the `pill`: consisting of two sphere meshes. These collection of pills are progressively decomposed to obtain a good approximation of the original shape. Our algorithm is easy to reuse and implement and is currently available in a multi-processing setup. To ensure reproducibility and further research, the source code and raw data has also been released.

The wide array of terrestrial forest and wooded lands are one of the richest sources because of their inherent structural diversity. For the diversity indicator of a forest, structure plays a significant role in it. We propose a transfer learning framework consisting of the ResNet-50 architecture using which we obtained a test accuracy of 75.86%. In this paper the analysis of the structural diversity of the Kejimkujik National Park is done with the help of a drone and using deep learning methods which predict the structural class of the forest. We have used a novel forest structural diversity dataset which was collected using DJI Mavic drone to train the deep learning model.

Lipschitz Bound Estimation is an effective method of regularizing deep neural networks to make them robust against adversarial attacks. This is useful in a variety of applications ranging from reinforcement learning to autonomous systems. In this paper, we highlight the significant gap in obtaining a non-trivial Lipschitz bound certificate for Convolutional Neural Networks (CNNs) and empirically support it with extensive graphical analysis. We also show that unrolling Convolutional layers or Toeplitz matrices can be employed to convert Convolutional Neural Networks (CNNs) to a Fully Connected Network. Further, we propose a simple algorithm to show the existing 20x-50x gap in a particular data distribution between the actual lipschitz constant and the obtained tight bound. We also ran sets of thorough experiments on various network architectures and benchmark them on datasets like MNIST and CIFAR-10. All these proposals are supported by extensive testing, graphs, histograms and comparative analysis.

Human Activity Recognition (HAR) is a domain of increasing interest with several two-stream architectures being suggested in recent years. However, such models have a huge number of parameters and storage needs due to the presence of a dedicated temporal stream. In this paper, we propose an architecture comprising of the weighted late fusion between the Softmax scores of the spatiotemporal stream (I3D) and another 2D convolutional neural network stream (Xception). We show that our model produces competitive performance w.r.t to other existing spatial and two-stream architectures along with reducing the number of parameters significantly and minimizing storage costs.

Convolutional Neural Networks (CNN) have been used for long for feature extraction from images in deep learning. Here we introduce ResilientCNN or ResCNN for short where we show that when convolution is implemented as an matrix-matrix operation coupled with some image processing techniques like Singular Value Decomposition (SVD) can be used as an better alternative to traditional convolution. We show that our ResCNN learns with bigger batch sizes and at much higher learning rates (7x) without compromising on accuracy compared to traditional convolutional networks by implementing both models on the MNIST Dataset.

TSCLite: A powerful and lightweight Traffic Sign Classification model Implementation
Sarosij Bose*, Avirup Dey*

Won 1st position in AI Entrepre-Neural, 2021 by GES, IIT Kharagpur and Intel

This work focuses on two lightweight Traffic sign classification implementations which can predict Traffic signs from any real time video feed. Here, a model based on an slightly enhanced LeNet architecture has been used and trained on the German Traffic Sign Dataset (GTSD) which has over 70000 images of traffic signs and over 40 various classes. Our model achieves a validation accuracy of over 98% and a training accuracy of over 97%. This saved model is then optimized over the Intel OpenVINO Model Optimizer + Inference Engine and run directly for predicting Traffic signs live from any video source(we have used webcam for our run). We have also provided a non optimized solution for comparison purposes.

RobustFreqCNN
Sarosij Bose

PyTorch implementation of this paper

This project is the unofficial implementation of the paper "Towards Frequency-Based Explanation for Robust CNN". It primarly deals with the extent to which image features are robust in the frequency domain. Here, the DCT Transform, the pre-trained ResNet 18 model and the RCT maps are generated from the adversarial as well as the normal images.

© Sarosij Bose (2023) | When in Rome, do as the romans do