Abstract

Measuring social connectedness with feedback is a large challenge involving a great potential for socially adaptive interfaces. However, having such measures running in real-time to differentiate from mimicry is a bigger challenge that requires more work. The experimental hub is an open-source research platform for such research and the Offset-Adjusted SImilarity Score (OASIS) algorithm [Hen+23] provides a solution to score social connectedness with lower time complexity from Duchenne smiles [EDF90] extracted with OpenFace [ALS+16]. A group filter infrastructure to enable researchers to focus on experimental design and run their research in real-time in a group context during video calls is built. Later, this infrastructure is adapted for synchrony score to be real-time with an end-to-end pipeline as a group filter to extract Duchenne smiles from frames and process them with OASIS algorithm. Real-time synchrony score computation is achieved through the modular design of the group filter infrastructure along with algorithmic run-time optimizations on OpenFace and OASIS. Ultimately, the experimental results indicate that the group filter infrastructure has an overhead of 1 milliseconds (ms) or less at 65.93 % of the time and of 33 ms or less at 99.96 % of the time which can be stated as a light-weight overhead. Moreover, the synchrony score experiments showed that the synchrony score group filter runs with 10 or higher Frame per Second (FPS) at 91.47 % of the time. This thesis work concludes with a discussion of the implications of this group filter on real-time synchrony feedback for more genuine engagement in human interactions.

Results/Implementation/Project Description

Conclusion

[ PDF (optional) ] 

[ Slides Kickoff/Final (optional)]