Unsupervised Spam Detection by Document Probability Estimation with Maximal Overlap Method

Uemura, Takashi; Ikeda, Daisuke; Kida, Takuya et al.

Transactions of the Japanese Society for Artificial Intelligence, 2011, 26(1), 297-306

Permalink : https://hdl.handle.net/2115/47125

このアイテムのアクセス数:1,043件（2026-03-16 18:55 集計）

閲覧可能ファイル 

ファイル フォーマット サイズ 閲覧回数 説明
26_297-1 pdf 555 KB 944

論文情報

ファイル出力 EndNote Basic出力 Mendeley出力

アクセス権
open access
URI
https://hdl.handle.net/2115/47125
タイトル
Unsupervised Spam Detection by Document Probability Estimation with Maximal Overlap Method
著者
Uemura, Takashi ; Ikeda, Daisuke ; Kida, Takuya ; Arimura, Hiroki
著者名
Uemura, Takashi
著者名
Ikeda, Daisuke
e-Rad_Researcher
70343316
著者名
Kida, Takuya
著者名
Arimura, Hiroki
言語
英語
キーワード
unsupervised spam detection
document complexity
suffix tree
maximal overlap method
word salad
発行日
2011
出版者
人工知能学会
収録物名
Transactions of the Japanese Society for Artificial Intelligence
巻(号)
26(1)
ページ
297-306
抄録
In this paper, we study content-based spam detection for spams that are generated by copying a seed document with some random perturbations. We propose an unsupervised detection algorithm based on an entropy-like measure called document complexity, which reflects how many similar documents exist in the input collection of documents. As the document complexity, however, is an ideal measure like Kolmogorov complexity, we substitute an estimated occurrence probability of each document for its complexity. We also present an efficient algorithm that estimates the probabilities of all documents in the collection in linear time to its total length. Experimental results showed that our algorithm especially works well for word salad spams, which are believed to be difficult to detect automatically.
資源タイプ
学術雑誌論文
出版タイプ
Version of Record
PISSN
1346-0714
EISSN
1346-8030
関連情報 (isIdenticalTo)
出版社版DOI
https://doi.org/10.1527/tjsai.26.297