Skip to content

Latest commit

 

History

History
132 lines (125 loc) · 3.25 KB

README.md

File metadata and controls

132 lines (125 loc) · 3.25 KB

A Study of Bilingual Chinese-Tibetan Named Entity Recognition for Traditional Tibetan Festivals（面向藏族传统节日的汉藏双语命名实体识别研究)

Overview

This repository contains the official implementation of "A Study of Bilingual Chinese-Tibetan Named Entity Recognition for Traditional Tibetan Festivals" paper. Additionaly, detailed implement guides are provided.

In this study, Chinese-Tibetan bilingual text data containing information on Tibetan traditional festivals from news websites such as People's Daily Online and People's Daily Tibetan Edition were collected and manually annotated. This study compares the performance of multiple pretrained models and word vectors for Tibetan traditional festival named entity recognition task in the Chinese-Tibetan bilingual scenario and analyzes the impact of two feature processing layers of the model, BiLSTM layer and CRF layer, on the experimental results.

Environment

  • Python 3.8
  • Pytorch 1.10
  • beautifulsoup4 4.11.1
  • Flask 2.0.2
  • html2text 2020.1.16
  • joeynmt 1.5.1
  • transformers 4.18.0

To install the environment using Conda:

$ conda env create -f requirements.txt

Running

Train

To train the models in this study, run the command below. Detailed configs are in the config folder.

$ python main.py

Deployment

To deploy trained model to server, run the command below.

$ python app.py

Metrics

语言

模型

准确率

召回率

F1

汉语

汉语fastText词向量-BiLSTM-CRF模型

94.65%

89.64%

91.97%

汉语RoBERTa预训练模型-BiLSTM-CRF模型

93.97%

92.32%

93.05%*

藏语

藏语fastText词向量-BiLSTM-CRF模型

84.97%

76.68%

80.37%

藏语RoBERTa预训练模型-BiLSTM-CRF模型

83.40%

89.60%

86.27%*

Citation

If you make use of this code, please cite the following paper:

@article{__nodate,
	title = {面向藏族传统节日的汉藏双语命名实体识别研究},
	issn = {2096-3467},
	url = {https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CAPJ&dbname=CAPJLAST&filename=XDTQ20220919000&uniplatform=NZKPT&v=E0vy1-y12agMc69tk-2GtR8p4fYnElMzKx2NKvq_UAb22I3wPYyc87DvVxQNCyxI},
	language = {中文},
	urldate = {2022-09-29},
	journal = {数据分析与知识发现},
	author = {邓, 宇扬 and 吴, 丹},
	keywords = {Named Entity Recognition, Pretrained Language Model, Tibetan Traditional Culture, 命名实体识别, 藏族传统文化, 预训练语言模型},
	pages = {1--15},
}