0

嗨，我正在尝试进行递归 .pdf url 收获，但遇到 ConcurrentModificationException .. 我不明白这是怎么发生的，我对并发也不太了解；我将非常感谢您对这是如何发生的以及如何解决它的一些见解。

public class urlHarvester {
    private URL rootURL;
    private String fileExt;
    private int depth;
    private HashSet<String> targets;
    private HashMap<Integer, LinkedList<String>> toVisit;

public urlHarvester(URL rootURL, String fileExt, int depth) {
    this.rootURL = rootURL;
    this.fileExt = fileExt;
    this.depth = depth;
    targets = new HashSet<String>();
    toVisit = new HashMap<Integer, LinkedList<String>>();
    for (int i = 1; i < depth + 1; i++) {
        toVisit.put(i, new LinkedList<String>());
    }
    doHarvest();
}

private void doHarvest() {
    try {
        harvest(rootURL, depth);
        while (depth > 0) {
            for (String s : toVisit.get(depth)) {
                toVisit.get(depth).remove(s);
                harvest(new URL(s),depth-1);
            }
            depth--;
        }   
    } catch (Exception e) {
        System.err.println(e);
        e.printStackTrace();
    }   
    for (String s : targets) {
        System.out.println(s);
    }

}

private void harvest(URL url, int depth) {
    try {
        URLConnection urlConnection = url.openConnection();
        InputStream inputStream = urlConnection.getInputStream();
        Scanner scanner = new Scanner(new BufferedInputStream(inputStream));
        java.lang.String source = "";
        while (scanner.hasNext()) {
            source = source + scanner.next();
        }   
        inputStream.close();
        scanner.close();

        Matcher matcher = Pattern.compile("ahref=\"(.+?)\"").matcher(source);
        while(matcher.find()) {
            java.lang.String matched = matcher.group(1);
            if (!matched.startsWith("http")) {
                if (matched.startsWith("/") && url.toString().endsWith("/")) {
                    matched = url.toString() + matched.substring(1);
                } else if ((matched.startsWith("/") && !url.toString().endsWith("/"))
                        || (!matched.startsWith("/") && url.toString().endsWith("/"))) {
                    matched = url.toString() + matched;
                } else if (!matched.startsWith("/") && !url.toString().endsWith("/")) {
                    matched = url.toString() + "/" + matched;
                }
            }
            if (matched.endsWith(".pdf") && !targets.contains(matched)) {
                targets.add(matched);System.out.println("ADDED");
            }
            if (!toVisit.get(depth).contains(matched)) {
                toVisit.get(depth).add(matched);
            }
        }
    } catch (Exception e) {
        System.err.println(e);
    }
}

主要调用类：

urlHarvester harvester = new urlHarvester(new URL("http://anyasdf.com"), ".pdf", 5);
4

2 回答 2

5

该错误可能与并发无关，而是由此循环引起的：

for (String s : toVisit.get(depth)) {
    toVisit.get(depth).remove(s);
    harvest(new URL(s),depth-1);
}

要在迭代时从集合中删除项目，您需要使用remove迭代器中的方法：

List<String> list = toVisit.get(depth); //I assume list is not null
for (Iterator<String> it = list.iterator(); it.hasNext();) {
    String s = it.next();
    it.remove();
    harvest(new URL(s),depth-1);
}
于 2012-10-21T20:05:21.567 回答
1

ConcurrentModificationException尝试在迭代对象时直接从集合中删除对象时抛出A。

当您尝试从以下位置删除条目时会发生这种情况toVisit HashMap：

for (String s : toVisit.get(depth)) {
   toVisit.get(depth).remove(s); <----
   ...

您可以使用迭代器而不是尝试直接从您的集合中删除：

Iterator<String> iterator = toVisit.get(depth).iterator();
while (iterator.hasNext()) {
   String s = iterator.next();
   iterator.remove();
   harvest(new URL(s),depth-1);
}
于 2012-10-21T20:08:50.683 回答