-2

这是我的正则表达式匹配代码，适用于网页：

public class RegexTestHarness {

    public static void main(String[] args) {

        File aFile = new File("/home/darshan/Desktop/test.txt");
        FileInputStream inFile = null;
        try {
            inFile = new FileInputStream(aFile);
        } catch (FileNotFoundException e) {
            e.printStackTrace(System.err);
            System.exit(1);
        }

        BufferedInputStream in = new BufferedInputStream(inFile);
        DataInputStream data = new DataInputStream(in);
        String string = new String();
        try {
            while (data.read() != -1) {
                string += data.readLine();
            }

        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Pattern pattern = Pattern
                .compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>");
        Matcher matcher = pattern.matcher(string);
        boolean found = false;
        while (matcher.find()) {
            System.out.println("Name: " + matcher.group(1) );
            found = true;
        }
        if(!found){
            System.out.println("Pattern Not found");
        }
    }
}

但是相同的代码不适用于我正在测试正则表达式的 crwaler 代码，我的爬虫代码是：（我正在使用 Websphinx）

// Our own Crawler class extends the WebSphinx Crawler
public class MyCrawler extends Crawler {

    MyCrawler() {
        super(); // Do what the parent crawler would do
    }

    // We could choose not to visit a link based on certain circumstances
    // For now we always visit the link
    public boolean shouldVisit(Link l) {
        // String host = l.getHost();
        return false; // always visit a link
    }

    // What to do when we visit the page
    public void visit(Page page) {
        System.out.println("Visiting: " + page.getTitle());
        String content = page.getContent();

        System.out.println(content);

        Pattern pattern = Pattern.compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>");
        Matcher matcher = pattern.matcher(content);
        boolean found = false;
        while (matcher.find()) {
            System.out.println("Name: " + matcher.group(1) );
            found = true;
        }
        if(!found){
            System.out.println("Pattern Not found");
        }
    }
}

这是我运行爬虫的代码：

public class WebSphinxTest {

    public static void main(String[] args) throws MalformedURLException, InterruptedException {

        System.out.println("Testing Websphinx. . .");

        // Make an instance of own our crawler
        Crawler crawler = new MyCrawler();
        // Create a "Link" object and set it as the crawler's root
        Link link = new Link("http://justeat.in/restaurant/spices/5633/indian-tandoor-chinese-and-seafood/sarjapur-road/bangalore");
        crawler.setRoot(link);

        // Start running the crawler!
        System.out.println("Starting crawler. . .");
        crawler.run(); // Blocking function, could implement a thread, etc.

    }

}

关于爬虫代码的一点细节。shouldvisit(Link link)过滤是否访问链接。visit(Page page)决定当我们得到页面时要做什么。

在上面的例子中，test.txt 和 content 包含相同的 String

4

1 回答 1

3

在您RegexTestHarness中，您正在从文件中读取行并连接没有换行符的行，然后进行匹配（readLine()返回没有换行符的行的内容！）。

所以在你的MyCrawler类的输入中，输入中可能有换行符。而且由于默认情况下正则表达式元字符.与换行符不匹配，因此它不适用于MyCrawler.

(?s)要解决此问题，请从包含.元字符的所有模式中追加。所以：

Pattern.compile("<div class=\"rest_title\">.*?<h1>(.*?)</h1>")

会成为：

Pattern.compile("(?s)<div class=\"rest_title\">.*?<h1>(.*?)</h1>")

DOT-ALL 标志 ,(?s)将导致.匹配任何字符，包括换行符。

于 2011-09-07T19:07:49.087 回答