错误：java。lang.IllegalArgumentException：比较方法违反了其总合同，甚至使用了变通方法

Question

我已经花了两天时间来缩短这个错误，甚至我尝试了几个stackoverflow帖子中建议的解决方法“-Djava.util.Arrays.useLegacyMergeSort=true”，但它也不起作用。

以下是我的命令及其返回错误的详细信息：

命令：

hadoop jar CloudBrush.jar -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -reads /Ec10k -asm Ec10k_Brush -k 21 -readlen 36

错误：

    Error: java.lang.IllegalArgumentException: Comparison method violates its general contract!
    at java.util.TimSort.mergeHi(TimSort.java:895)
    at java.util.TimSort.mergeAt(TimSort.java:512)
    at java.util.TimSort.mergeCollapse(TimSort.java:437)
    at java.util.TimSort.sort(TimSort.java:241)
    at java.util.Arrays.sort(Arrays.java:1512)
    at java.util.ArrayList.sort(ArrayList.java:1454)
    at java.util.Collections.sort(Collections.java:175)
    at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:252)
    at Brush.VerifyOverlap$VerifyOverlapReducer.reduce(VerifyOverlap.java:1)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
    at Brush.VerifyOverlap.run(VerifyOverlap.java:381)
    at Brush.BrushAssembler.buildOverlap(BrushAssembler.java:326)
    at Brush.BrushAssembler.run(BrushAssembler.java:838)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at Brush.BrushAssembler.main(BrushAssembler.java:913)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这是比较器：

    class OvelapSizeComparator implements Comparator {
        public int compare(Object element1, Object element2) {
            OverlapInfo obj1 = (OverlapInfo) element1;
            OverlapInfo obj2 = (OverlapInfo) element2;
            if ((int)(obj1.overlap_size - obj2.overlap_size) >= 0) {
                return -1;
            } else {
                return 1;
            }
        }
    }

Answer 1

我修复了错误。我以为是Hadoop的问题，我错了。这是Java版本的问题。我们还升级了Java版本和Hadoop一起。如果要排序的元素数量超过32个，java 7和java 8的rrays.sort（）方法使用TimSort。蒂姆排序在比较中强制执行严格的传递属性。

如果（比较（x，y）

您应该更改比较方法以与传递属性保持一致，或者使用旧版本排序，将“映射任务Java选择库”或“减少任务Java选择库”的“java.util.rrays.useLegacyMergeSort”设置为true，然后它应该应用于所有为映射或减少而启动的JVM。

对于2.6.0-cdh5.4.2 Haddop，您可以通过添加

-Dmapreduce.map.java.opts="-Djava.util.Arrays.useLegacyMergeSort=true"
-Dmapreduce.reduce.java.opts="-Djava.util.Arrays.useLegacyMergeSort=true"

或者通过密码

工作getConfiguration（）。set（“mapreduce.map.java.opts”，“-Djava.util.Arrays.useLegacyMergeSort=true”）；

job.get配置（）. set（"mapreduce.reduce.java.opts"，"-Djava.util.Arrays.useLegacyMergeSort=true"）；

Answer 2

您可以通过在测试中运行main方法来重现此错误。JVM7或更高版本上的java代码。总结一下这段代码的作用。这段代码尝试使用不具有传递属性的比较方法对40个人对象进行排序

//人。JAVA

公共阶层人士{

public String name;
public int age ;
public int salary;

@Override
public int compareTo(Person o) {
    if(o instanceof Person){
        int ret=0;
        if(age == 25 && ((Person)o).age ==27) ret = 1;
        else if(age == 27 && ((Person)o).age ==29) ret = 1;
        else if(age == 25 && ((Person)o).age ==29) ret = -1;
        else{
                if( age < ((Person)o).age) ret = -1;
                if(age > ((Person)o).age) ret = 1;
                if(salary < ((Person)o).salary) ret = -1;
                if(salary > ((Person)o).salary) ret = 1;           
        }
        return ret; 

    }
    return 0;
}

@Override
public String toString(){
 return "name="+name+":age="+age+";";     
}

}

//测试。JAVA

导入java.util.数组；

公开课考试{

public static void main(String args[]) {
    Test t = new Test();
    t.sortPersons(args);        
}

public void sortPersons(String args[]) {        

    Person p1 = new Person();
    p1.age = 25;
    p1.name = "ABC";

    Person p2 = new Person();
    p2.age = 29;
    p2.name = "ABZ";

    Person p3 = new Person();
    p3.age = 27;
    p3.name = "AZ";

    Person p4 = new Person();
    p4.age = 27;
    p4.name = "AZ";

    Person p5 = new Person();
    //p5.age = 22;
    //p5.name="ZZ";

    Person[] p = new Person[40];

    p[0] = p2;
    p[1] = p3;
    p[2] = p4;
    p[3] = p1;
    p[4] = p5;
    for (int i = 1; i < 8; i++) {
        p[i * 5] = p[0];
        p[i * 5 + 1] = p[1];
        p[i * 5 + 2] = p[2];
        p[i * 5 + 3] = p[3];
        p[i * 5 + 4] = p[4];
    }

    System.out.println("\nSortingInput\n");           
    Arrays.sort(p);       
    System.out.println("\nSorting complete\n");
}

}

Answer 3

真正的问题是您的OvelapSize比较器[sic]有缺陷。如果两个对象的overlap_size值相等，它会返回-1，而它应该返回0。如果它们不相等，它会返回一个符号不正确的值。

要解决此问题，请替换以下内容：

if ((int)(obj1.overlap_size - obj2.overlap_size) >= 0) {
   return -1;
} else {
   return 1;
}

...有了这个：

return obj1.overlap_size - obj2.overlap_size;