CorrNet+

This repo holds codes of the paper: CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation [paper], which is an extension of our previous work (CVPR 2023) [paper]

For the code supporting continuous sign language recognition and sign language translation, refer to CorrNet_Plus_CSLR and CorrNet_Plus_SLT for their codes, respectively.

The code of CorrNet_Plus_SLT is updating and will be released later.

Performance

• On the continuous sign language cognition task, CorrNet+ achieves superior performance on PHOENIX14, PHOENIX14-T, CSL-Daily and CSL datasets.

Method	PHOENIX2014				PHOENIX2014-T		CSL-Daily
	Dev(%)		Test(%)		Dev(%)	Test(%)	Dev(%)	Test(%)
	del/ins	WER	del/ins	WER
CVT-SLR (CVPR2023)	6.4/2.6	19.8	6.1/2.3	20.1	19.4	20.3	-	-
CoSign-2s (ICCV2023)	-	19.7	-	20.1	19.5	20.1	-	-
AdaSize (PR2024)	7.0/2.6	19.7	7.2/3.1	20.9	19.7	21.2	31.3	30.9
AdaBrowse+ (ACMMM2023)	6.0/2.5	19.6	5.9/2.6	20.7	19.5	20.6	31.2	30.7
SEN (AAAI2023)	5.8/2.6	19.5	7.3/4.0	21.0	19.3	20.7	31.1	30.7
CTCA (CVPR2023)	6.2/2.9	19.5	6.1/2.6	20.1	19.3	20.3	31.3	29.4
C2SLR (CVPR2022)	-	20.5	-	20.4	20.2	20.4	-	-
CorrNet+	5.3/2.7	18.0	5.6/2.4	18.2	17.2	19.1	28.6	28.2

• On the sign language translation task, CorrNet+ achieves superior performance on PHOENIX14, PHOENIX14-T and CSL-Daily datasets.

PHOENIX2014-T
Method	Dev(%)					Test(%)
	Rouge	BLEU1	BLEU2	BLEU3	BLEU4	Rouge	BLEU1	BLEU2	BLEU3	BLEU4
SignBT (CVPR2021)	50.29	51.11	37.90	29.80	24.45	49.54	50.80	37.75	29.72	24.32
MMTLB (CVPR2022)	53.10	53.95	41.12	33.14	27.61	52.65	53.97	41.75	33.84	28.39
SLTUNET (ICLR2023)	52.23	-	-	-	27.87	52.11	52.92	41.76	33.99	28.47
TwoStream-SLT (NeuIPS2023)	54.08	54.32	41.99	34.15	28.66	53.48	54.90	42.43	34.46	28.95
CorrNet+	54.54	54.56	42.31	34.48	29.13	53.76	55.32	42.74	34.86	29.42
CSL-Daily
Method	Dev(%)					Test(%)
	Rouge	BLEU1	BLEU2	BLEU3	BLEU4	Rouge	BLEU1	BLEU2	BLEU3	BLEU4
SignBT (CVPR2021)	49.49	51.46	37.23	27.51	20.80	49.31	51.42	37.26	27.76	21.34
MMTLB (CVPR2022)	53.38	53.81	40.84	31.29	24.42	53.25	53.31	40.41	30.87	23.92
SLTUNET (ICLR2023)	53.58	-	-	-	23.99	54.08	54.98	41.44	31.84	25.01
TwoStream-SLT (NeuIPS2023)	55.10	55.21	42.31	32.71	25.76	55.72	55.44	42.59	32.87	25.79
CorrNet+	55.52	55.64	42.78	33.13	26.14	55.84	55.82	42.96	33.26	26.14

Visualizations

As shown below, our method intelligently models the human body trajectories across adjacent frames and pays special attention to the moving human body parts.

Data preparation, Environment, Training, Inference and Visualizations

For detailed instructions of data preparation, environment, training, inference and visualizations, please refer to each sub-repo for guidance.