Few-Shot Learning for Effective Pattern Discovery: Overcoming Limitations of Traditional Data Mining in Micro Datasets

Abstract

The large datasets called “Big Data” is heavily favored in traditional researches due to its large volume velocity and variety which has led to the development and optimization of different algorithms. And due to this reason unique challenges that are presented by smaller data volumes are usually ignored. This creates a significant gap in systematic analysis in relation with performance, robustness and optimal parameter setting of the classic data mining algorithms when applied to smaller data sets. Modern domains such as industrial IoT, Rare diseases diagnosis etc. usually generates small datasets and the performance and optimal configuration of “micro datasets” are prone to overfitting and poor result generation.

The aim of the study is to analyze the limitations and underperformance of traditional data mining algorithms such as Decision Trees, Apriori, and Clustering. And propose Few- Shot Learning (FSL) as an effective alternative approach in pattern discovery of smaller datasets “micro datasets”. Methodology is to conduct the experiment in two phases. First, applying traditional algorithm in “micro dataset” to document the performance and the reason that leads to the failure. Secondly, evaluate the prototypic FSL model on the same dataset and pen down the observations. Then we compare both the approaches based on generalized accuracy