Fu, HongyuLiu, ChenQi, XingqunLin, BeibeiLi, LinchengZhang, LiYu, Xin2025-03-222025-03-2297830312508420302-9743Scopus:85151330375https://dspace-test.anu.edu.au/handle/1885/733728359This work aims to locate a query isolated sign in a continuous sign video. In this task, the domain gap between the isolated and continuous sign videos often handicaps the localization performance. To address this issue, we propose a parallel multi-modal sign spotting framework. In a nutshell, our framework firstly takes advantage of multi-modal information (including RGB frames, 2D key-points and 3D key-points) to achieve representative sign features. The multi-modal features are employed to complement each other and thus compensate for the deficiency of a single modality, thus leading to informative representations for sign spotting. Moreover, we introduce a testing time top-k transferring technique into our framework to reduce the aforementioned domain gap. Concretely, we first compare the query sign with extracted sign video clips, and then update the feature of the query sign with the features of the top-k best matching video clips. In this manner, the updated query feature will exhibit a smaller domain gap with respect to continuous signs, facilitating feature matching in the following iterations. Experiments on the challenging OSLWL-Test-Set benchmark demonstrate that our method achieves superior performance (0.559 F1-score) compared to the baseline (0.395 F1-score). Our code is available at https://github.com/bb12346/OpenSLR.Acknowledgement. The research was supported by the Australian Research Council DP220100800 and Google Research Scholar Program.17EnglishPublisher Copyright: © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.Multi-modal fusion frameworkPose-based feature extractionRGB-based feature extractionSign language spottingTop-K transferring technique202310.1007/978-3-031-25085-9_16http://www.scopus.com/inward/record.url?scp=85151330375&partnerID=8YFLogxK