Abstract

Anomaly detection has been an important research topic in data mining and machine learning. Many real-world applications such as intrusion or credit card fraud detection require an effective and efficient framework to identify deviated data instances. However, most anomaly detection methods are typically implemented in batch mode, and thus cannot be easily extended to large-scale problems without sacrificing computation and memory requirements. We introduce an online over-sampling principal component analysis (osPCA) algorithm for detecting the presence of outliers from a large amount of data via an online updating technique. Unlike prior PCA based approaches, we do not store the entire data matrix or covariance matrix, and thus our approach is especially of interest in online or large-scale problems. Since our osPCA needs not performing eigen analysis explicitly, the proposed framework is favored for online applications which have computation or memory limitations. This advantage will allow applying our method to distributed anomaly detection systems such as M2M communication systems and the Internet of Things (IoT). We believe that it will be an important technique for the Big Data era.