|
Hokkaido University Collection of Scholarly and Academic Papers >
Graduate School of Information Science and Technology / Faculty of Information Science and Technology >
Peer-reviewed Journal Articles, etc >
Parameter-efficient feature-based transfer for paraphrase identification
| Title: | Parameter-efficient feature-based transfer for paraphrase identification |
| Authors: | Liu, Xiaodong Browse this author | | Rzepka, Rafal Browse this author →KAKEN DB | | Araki, Kenji Browse this author →KAKEN DB |
| Keywords: | Parameter-efficient feature-based transfer | | Paraphrase identification | | Natural language inference | | Semantic textual similarity | | Continual learning |
| Issue Date: | 19-Dec-2022 |
| Publisher: | Cambridge University Press |
| Journal Title: | Natural Language Engineering |
| Start Page: | 1 |
| End Page: | 31 |
| Publisher DOI: | 10.1017/S135132492200050X |
| Abstract: | There are many types of approaches for Paraphrase Identification (PI), an NLP task of determining whether a sentence pair has equivalent semantics. Traditional approaches mainly consist of unsupervised learning and feature engineering, which are computationally inexpensive. However, their task performance is moderate nowadays. To seek a method that can preserve the low computational costs of traditional approaches but yield better task performance, we take an investigation into neural network-based transfer learning approaches. We discover that by improving the usage of parameters efficiently for feature-based transfer, our research goal can be accomplished. Regarding the improvement, we propose a pre-trained task-specific architecture. The fixed parameters of the pre-trained architecture can be shared by multiple classifiers with small additional parameters. As a result, the computational cost left involving parameter update is only generated from classifier-tuning: the features output from the architecture combined with lexical overlap features are fed into a single classifier for tuning. Furthermore, the pre-trained task-specific architecture can be applied to natural language inference and semantic textual similarity tasks as well. Such technical novelty leads to slight consumption of computational and memory resources for each task and is also conducive to power-efficient continual learning. The experimental results show that our proposed method is competitive with adapter-BERT (a parameter-efficient fine-tuning approach) over some tasks while consuming only 16% trainable parameters and saving 69-96% time for parameter update. |
| Type: | article (author version) |
| URI: | http://hdl.handle.net/2115/89905 |
| Appears in Collections: | 情報科学院・情報科学研究院 (Graduate School of Information Science and Technology / Faculty of Information Science and Technology) > 雑誌発表論文等 (Peer-reviewed Journal Articles, etc)
|
Submitter: RZEPKA Rafal
|
