5

我试图想出一些代码，让我可以搜索我的 ArrayList 并检测“好值”的公共范围之外的任何值。

示例：100 105 102 13 104 22 101

我如何编写代码来检测（在这种情况下）13 和 22 不属于 100 左右的“好值”？

4

9 回答 9

7

There are several criteria for detecting outliers. The simplest ones, like Chauvenet's criterion, use the mean and standard deviation calculated from the sample to determine a "normal" range for values. Any value outside of this range is deemed an outlier.

Other criterions are Grubb's test and Dixon's Q test and may give better results than Chauvenet's for example if the sample comes from a skew distribution.

于 2013-09-14T18:48:15.390 回答
5 
package test;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class Main {
    public static void main(String[] args) {
        List<Double> data = new ArrayList<Double>();
        data.add((double) 20);
        data.add((double) 65);
        data.add((double) 72);
        data.add((double) 75);
        data.add((double) 77);
        data.add((double) 78);
        data.add((double) 80);
        data.add((double) 81);
        data.add((double) 82);
        data.add((double) 83);
        Collections.sort(data);
        System.out.println(getOutliers(data));
    }

    public static List<Double> getOutliers(List<Double> input) {
        List<Double> output = new ArrayList<Double>();
        List<Double> data1 = new ArrayList<Double>();
        List<Double> data2 = new ArrayList<Double>();
        if (input.size() % 2 == 0) {
            data1 = input.subList(0, input.size() / 2);
            data2 = input.subList(input.size() / 2, input.size());
        } else {
            data1 = input.subList(0, input.size() / 2);
            data2 = input.subList(input.size() / 2 + 1, input.size());
        }
        double q1 = getMedian(data1);
        double q3 = getMedian(data2);
        double iqr = q3 - q1;
        double lowerFence = q1 - 1.5 * iqr;
        double upperFence = q3 + 1.5 * iqr;
        for (int i = 0; i < input.size(); i++) {
            if (input.get(i) < lowerFence || input.get(i) > upperFence)
                output.add(input.get(i));
        }
        return output;
    }

    private static double getMedian(List<Double> data) {
        if (data.size() % 2 == 0)
            return (data.get(data.size() / 2) + data.get(data.size() / 2 - 1)) / 2;
        else
            return data.get(data.size() / 2);
    }
}

输出：[20.0]

解释：

  • 对整数列表进行排序，从低到高
  • 将整数列表拆分为 2 部分（中间）并将它们放入 2 个新的单独的 ArrayList 中（称它们为“左”和“右”）
  • 在这两个新的 ArrayLists 中找到一个中间数（中位数）
  • Q1 是左侧的中位数，Q3 是右侧的中位数
  • 应用数学公式：
  • IQR = Q3 - Q1
  • LowerFence = Q1 - 1.5*IQR
  • UpperFence = Q3 + 1.5*IQR
  • 有关此公式的更多信息：http: //www.mathwords.com/o/outlier.htm
  • 循环遍历我所有的原始元素，如果它们中的任何一个低于下栅栏或高于上栅栏，则将它们添加到“输出”ArrayList
  • 这个新的“输出”ArrayList 包含异常值
于 2017-12-19T17:18:10.173 回答
3

可以在MathUtil.java中找到Grubb 测试的实现。它将找到一个异常值，您可以将其从列表中删除并重复，直到您删除了所有异常值。

取决于commons-math，所以如果你使用 Gradle：

dependencies {
  compile 'org.apache.commons:commons-math:2.2'
}
于 2016-09-26T20:54:14.277 回答
1
  • find the mean value for your list
  • create a Map that maps the number to the distance from mean
  • sort values by the distance from mean
  • and differentiate last n number, making sure there is no injustice with distance
于 2013-09-14T18:45:20.783 回答
1

使用这个算法。该算法使用平均值和标准差。这 2 个数字可选值（2 * 标准偏差）。

 public static List<int> StatisticalOutLierAnalysis(List<int> allNumbers)
            {
                if (allNumbers.Count == 0)
                    return null;

                List<int> normalNumbers = new List<int>();
                List<int> outLierNumbers = new List<int>();
                double avg = allNumbers.Average();
                double standardDeviation = Math.Sqrt(allNumbers.Average(v => Math.Pow(v - avg, 2)));
                foreach (int number in allNumbers)
                {
                    if ((Math.Abs(number - avg)) > (2 * standardDeviation))
                        outLierNumbers.Add(number);
                    else
                        normalNumbers.Add(number);
                }

                return normalNumbers;
            }
于 2016-06-19T23:09:33.463 回答
1

正如Joni已经指出的那样，您可以借助标准偏差和均值消除异常值。这是我的代码，您可以将其用于您的目的。

    public static void main(String[] args) {

    List<Integer> values = new ArrayList<>();
    values.add(100);
    values.add(105);
    values.add(102);
    values.add(13);
    values.add(104);
    values.add(22);
    values.add(101);

    System.out.println("Before: " + values);
    System.out.println("After: " + eliminateOutliers(values,1.5f));

}

protected static double getMean(List<Integer> values) {
    int sum = 0;
    for (int value : values) {
        sum += value;
    }

    return (sum / values.size());
}

public static double getVariance(List<Integer> values) {
    double mean = getMean(values);
    int temp = 0;

    for (int a : values) {
        temp += (a - mean) * (a - mean);
    }

    return temp / (values.size() - 1);
}

public static double getStdDev(List<Integer> values) {
    return Math.sqrt(getVariance(values));
}

public static List<Integer> eliminateOutliers(List<Integer> values, float scaleOfElimination) {
    double mean = getMean(values);
    double stdDev = getStdDev(values);

    final List<Integer> newList = new ArrayList<>();

    for (int value : values) {
        boolean isLessThanLowerBound = value < mean - stdDev * scaleOfElimination;
        boolean isGreaterThanUpperBound = value > mean + stdDev * scaleOfElimination;
        boolean isOutOfBounds = isLessThanLowerBound || isGreaterThanUpperBound;

        if (!isOutOfBounds) {
            newList.add(value);
        }
    }

    int countOfOutliers = values.size() - newList.size();
    if (countOfOutliers == 0) {
        return values;
    }

    return eliminateOutliers(newList,scaleOfElimination);
}
  • removeOutliers() 方法正在完成所有工作
  • 它是一种递归方法，每次递归调用都会修改列表
  • 传递给方法的 scaleOfElimination 变量定义了要删除异常值的比例：通常我选择 1.5f-2f，变量越大，删除的异常值越少

代码的输出：

之前：[100、105、102、13、104、22、101]

之后：[100、105、102、104、101]

于 2018-08-10T09:11:01.403 回答
0

感谢 @Emil_Wozniak 发布完整代码。我挣扎了一段时间，没有意识到eliminateOutliers()实际上返回了异常值，而不是消除了它们的列表。该isOutOfBounds()方法也令人困惑，因为当值为 IN 边界时它实际上返回 TRUE。以下是我对一些（恕我直言）改进的更新：

  • removeOutliers() 方法返回删除了异常值的输入列表
  • 添加了 getOutliers() 方法来获取异常值列表
  • 删除了令人困惑的 isOutOfBounds() 方法以支持简单的过滤表达式
  • 扩展 N 列表以支持多达 30 个输入值
  • 防止输入列表太大或太小时出现越界错误
  • 制作统计方法（均值、标准差、方差）静态实用方法
  • 只计算一次上限/下限，而不是每次比较
  • 在 ctor 上提供输入列表并存储为实例变量
  • 重构以避免使用与实例和局部变量相同的变量名

代码：

/**
 * Implements an outlier removal algorithm based on https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/dixon.htm#:~:text=It%20can%20be%20used%20to,but%20one%20or%20two%20observations).
 * Original Java code by Emil Wozniak at https://stackoverflow.com/questions/18805178/how-to-detect-outliers-in-an-arraylist
 * 
 * Reorganized, made more robust, and clarified many of the methods.
 */

import java.util.List;
import java.util.stream.Collectors;

public class DixonTest {
    protected List<Double> criticalValues = 
            List.of( // Taken from https://sebastianraschka.com/Articles/2014_dixon_test.html#2-calculate-q
                    // Alfa level of 0.1 (90% confidence)
                    0.941,  // N=3
                    0.765,  // N=4
                    0.642,  // ...
                    0.56,
                    0.507,
                    0.468,
                    0.437,
                    0.412,
                    0.392,
                    0.376,
                    0.361,
                    0.349,
                    0.338,
                    0.329,
                    0.32,
                    0.313,
                    0.306,
                    0.3,
                    0.295,
                    0.29,
                    0.285,
                    0.281,
                    0.277,
                    0.273,
                    0.269,
                    0.266,
                    0.263,
                    0.26     // N=30
                    );
    
    // Stats calculated on original input data (including outliers)
    private double scaleOfElimination;
    private double mean;
    private double stdDev;
    private double UB;
    private double LB;
    private List<Double> input;
    
    /**
     * Ctor taking a list of values to be analyzed. 
     * @param input
     */
    public DixonTest(List<Double> input) {
        this.input = input;
        
        // Create statistics on the original input data
        calcStats();
    }

    /**
     * Utility method returns the mean of a list of values.
     * @param valueList
     * @return
     */
    public static double getMean(final List<Double> valueList) {
        double sum = valueList.stream()
                .mapToDouble(value -> value)
                .sum();
        return (sum / valueList.size());
    }

    /**
     * Utility method returns the variance of a list of values.
     * @param valueList
     * @return
     */
    public static double getVariance(List<Double> valueList) {
        double listMean = getMean(valueList);
        double temp = valueList.stream()
                .mapToDouble(a -> a)
                .map(a -> (a - listMean) * (a - listMean))
                .sum();
        return temp / (valueList.size() - 1);
    }

    /**
     * Utility method returns the std deviation of a list of values.
     * @param input
     * @return
     */
    public static double getStdDev(List<Double> valueList) {
        return Math.sqrt(getVariance(valueList));
    }
    
    /**
     * Calculate statistics and bounds from the input values and store
     * them in class variables.
     * @param input
     */
    private void calcStats() {
        int N = Math.min(Math.max(0, input.size() - 3), criticalValues.size()-1); // Changed to protect against too-small or too-large lists
        scaleOfElimination = criticalValues.get(N).floatValue();
        mean = getMean(input);
        stdDev = getStdDev(input);
        UB = mean + stdDev * scaleOfElimination;
        LB = mean - stdDev * scaleOfElimination;        
    }

    /**
     * Returns the input values with outliers removed.
     * @param input
     * @return
     */
    public List<Double> eliminateOutliers() {

        return input.stream()
                .filter(value -> value>=LB && value <=UB)
                .collect(Collectors.toList());
    }

    /**
     * Returns the outliers found in the input list.
     * @param input
     * @return
     */
    public List<Double> getOutliers() {

        return input.stream()
                .filter(value -> value<LB || value>UB)
                .collect(Collectors.toList());
    }

    /**
     * Test and sample usage
     * @param args
     */
    public static void main(String[] args) {
        List<Double> testValues = List.of(1200.0,1205.0,1220.0,1194.0,1212.0);
        
        DixonTest outlierDetector = new DixonTest(testValues);
        List<Double> goodValues = outlierDetector.eliminateOutliers();
        List<Double> badValues = outlierDetector.getOutliers();
        
        System.out.println(goodValues.size()+ " good values:");
        for (double v: goodValues) {
            System.out.println(v);
        }
        System.out.println(badValues.size()+" outliers detected:");
        for (double v: badValues) {
            System.out.println(v);
        }
        
        // Get stats on remaining (good) values
        System.out.println("\nMean of good values is "+DixonTest.getMean(goodValues));
    }
}
于 2021-10-28T15:51:49.747 回答
0

我很高兴并感谢Valiyev。他的解决方案对我帮助很大。我想把我的小 SRP 介绍给他的作品。

请注意，我用于List.of()存储 Dixon 的临界值，因此需要使用高于 8 的 Java。

public class DixonTest {
protected List<Double> criticalValues = 
    List.of(0.941, 0.765, 0.642, 0.56, 0.507, 0.468, 0.437);
private double scaleOfElimination;
private double mean;
private double stdDev;

private double getMean(final List<Double> input) {
    double sum = input.stream()
            .mapToDouble(value -> value)
            .sum();
    return (sum / input.size());
}

  private double getVariance(List<Double> input) {
    double mean = getMean(input);
    double temp = input.stream()
            .mapToDouble(a -> a)
            .map(a -> (a - mean) * (a - mean))
            .sum();
    return temp / (input.size() - 1);
}

private double getStdDev(List<Double> input) {
    return Math.sqrt(getVariance(input));
}

protected List<Double> eliminateOutliers(List<Double> input) {
    int N = input.size() - 3;
    scaleOfElimination = criticalValues.get(N).floatValue();
    mean = getMean(input);
    stdDev = getStdDev(input);

    return input.stream()
            .filter(this::isOutOfBounds)
            .collect(Collectors.toList());
}

private boolean isOutOfBounds(Double value) {
    return !(isLessThanLowerBound(value)
            || isGreaterThanUpperBound(value));
}

private boolean isGreaterThanUpperBound(Double value) {
    return value > mean + stdDev * scaleOfElimination;
}

private boolean isLessThanLowerBound(Double value) {
    return value < mean - stdDev * scaleOfElimination;
}
}

我希望它会帮助别人。

最良好的问候

于 2020-03-26T12:17:09.057 回答
-1

这只是一个非常简单的实现，它获取不在范围内的数字的信息：

List<Integer> notInRangeNumbers = new ArrayList<Integer>();
for (Integer number : numbers) {
    if (!isInRange(number)) {
        // call with a predefined factor value, here example value = 5
        notInRangeNumbers.add(number, 5);
    }
}

此外，在isInRange方法中，您必须定义 'good values' 是什么意思。您将在下面找到一个示例实现。

private boolean isInRange(Integer number, int aroundFactor) {
   //TODO the implementation of the 'in range condition'
   // here the example implementation
   return number <= 100 + aroundFactor && number >= 100 - aroundFactor;
}
于 2013-09-14T18:52:14.847 回答