java - 运行网络爬虫程序时出现“线程“主”java.lang.NullPointerException 中的异常”错误

Question

我对网络抓取相当陌生，对 Java 的了解有限。

每次运行此代码时，我都会收到错误消息：

Exception in thread "main" java.lang.NullPointerException

    at sws.SWS.scrapeTopic(SWS.java:38)
    at sws.SWS.main(SWS.java:26)
Java Result: 1
BUILD SUCCESSFUL (total time: 0 seconds)

我的代码是：

import java.io.*;

import java.net.*;

import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;

public class SWS
{

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args)
    {
        scrapeTopic("wiki/Python");

    }

    public static void scrapeTopic(String url)
    {
        String html = getUrl("http://www.wikipedia.org/" + url);



        Document doc = Jsoup.parse(html);

        String contentText = doc.select("#mw-content-text > p").first().text();

        System.out.println(contentText);


    }


    public static String getUrl(String Url)
    {
        URL urlObj = null;

        try 
        {
            urlObj = new URL(Url);

        }

        catch(MalformedURLException e)
        {
            System.out.println("The url was malformed");

            return "";
        }


        URLConnection urlCon = null;

        BufferedReader in = null;

        String outputText = "";

        try
        {
            urlCon = urlObj.openConnection();

            in = new BufferedReader(new InputStreamReader(urlCon.getInputStream()));
             String line = "";

             while ((line = in.readLine()) != null)
             {
                 outputText += line;

             }

             in.close();
        }

         catch(IOException e)
         {
             System.out.println("There was a problem connecting to the url");

             return "";

         }

        return outputText;



    }

}

我一直盯着我的屏幕有一段时间了，需要帮助！

提前致谢。

Answer 1

在以下代码中：

 String contentText = doc.select("#mw-content-text > p").first().text()

如果doc.select("#mw-content-text > p") 没有找到与查询匹配的任何元素并返回一个空元素，则调用first()此类元素应给出NullPointerException.

查看Element.select和Elements.first()的 jsoup 文档页面

Answer 2

In this line

doc.select("#mw-content-text > p").first().text();

Your doc.select obviously dont find anything, so it returns null. You then call first() method on null and thats why it ends with error.

Answer 3

您的代码完全适合我。

为了调试和诊断其他答案是否指出可能的错误，您最好使用一些临时变量并在调试器中逐步执行代码。

public static void scrapeTopic(String url)
{
    String html = getUrl("http://www.wikipedia.org/" + url);
    Document doc = Jsoup.parse(html);

    Elements select = doc.select("#mw-content-text > p");
    Element first = select.first();
    String contentText = first.text();

    System.out.println(contentText);
}