open access

https://hdl.handle.net/2115/96109

Expert Evaluation of Export Control-Related Question Answering Capabilities of LLMs

英語

2024-04-05

IEEE

2023 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE)

1-6

2023 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) Dec. 4 2023 to Dec. 6 2023 Nadi, Fiji

In this paper we introduce evaluation experiments performed by an expert to assess Large Language Models ability to answer questions related to export control in Japanese language. We compare outputs of two popular models, namely ChatGPT and GPT-4, and measure their accuracy in predicting intention of the question and label of the answer. Prediction results indicate that both models were better in guessing question intention but failed to recognize what type of an answer would be most probable. Furthermore, expert evaluation of outputs generated by both models show that most of the answers contain mistakes which can mislead users asking a dialog system for an expertise. We analyze these outputs and enumerate problems to be tackled in the future.

会議発表論文

Accepted Manuscript

© 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.