2

如何使用 Java 从 HTML 中提取文本的问题已被查看和重复了无数次: 从 HTML Java 中提取文本

感谢在 Stackoverflow 上找到的答案,我目前的情况是我正在使用JSoup

<!-- Jsoup maven dependency -->
<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.7.3</version>
</dependency>

和这段或代码:

// parse the html from the givne string
Document doc = Jsoup.parse(html);
// loop over children elements of the body tag
for (Element el:doc.select("body").select("*")) {
  // loop over all textnodes of these children
  for (TextNode textNode:el.textNodes()) {
    // make sure there is some text other than whitespace
    if (textNode.text().trim().length()>0) {
        // show:
        //    the original node name
        //    the name of the subnode witht the text 
        //    the text 
        System.out.println(el.nodeName()+"."+textNode.nodeName()+":"+textNode.text());
    }
  }
}

现在我还想显示手头的 textNode 来自的行号和原始 html 源代码。我怀疑 JSoup 可以做到这一点(例如参见

并尝试解决方法:

int pos = html.indexOf(textNode.outerHtml());

无法可靠地找到原始 html。所以我想我可能不得不切换到另一个库或方法。Jericho-html:是否可以参考源文件中的位置提取文本?正如上面的链接也指出的那样,有一个答案说“杰里科可以做到”。但是缺少指向实际工作代码的指针。

在杰里科,我做到了:

Source htmlSource=new Source(html);
boolean bodyFound=false;
// loop over all elements
for (net.htmlparser.jericho.Element el:htmlSource.getAllElements()) {
    if (el.getName().equals("body")) {
        bodyFound=true;
    }
    if (bodyFound) {
        TagType tagType = el.getStartTag().getTagType();
        if (tagType==StartTagType.NORMAL) {
            String text=el.getTextExtractor().toString();
            if (!text.trim().equals("")) {
                int cpos = el.getBegin();               
                System.out.println(el.getName()+"("+tagType.toString()+") line "+   htmlSource.getRow(cpos)+":"+text);
            }
        } // if
    } // if
} // for

这已经很好了,因为它会给你这样的输出:

body(normal) line 91: Some Header. Some Text
div(normal) line 93: Some Header
div(normal) line 95: Some Text

但现在的后续问题是 TextExtractor 递归输出所有子节点的整个文本,以便文本多次显示。

什么是过滤以及上述 JSoup 解决方案(请注意文本元素的正确顺序)但显示源代码行的有效解决方案,如上述 Jericho 代码片段一样?

4

2 回答 2

3

您需要和 jsoup 缺乏的功能恕我直言更难以实现。使用 Jericho 并实现类似的东西,以找到直接的文本节点。

package main.java.com.adacom.task;

import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;

import net.htmlparser.jericho.Element;
import net.htmlparser.jericho.EndTag;
import net.htmlparser.jericho.Segment;
import net.htmlparser.jericho.Source;
import net.htmlparser.jericho.StartTag;
import net.htmlparser.jericho.StartTagType;
import net.htmlparser.jericho.Tag;
import net.htmlparser.jericho.TagType;

public class MainParser {

    /**
     * @param args
     */
    public static void main(String[] args) {

        String html = "<body><div>divtextA<span>spanTextA<p>pText</p>spanTextB</span>divTextB</div></body>";

        Source htmlSource=new Source(html);
        boolean bodyFound=false;
        // loop over all elements
        for (net.htmlparser.jericho.Element el:htmlSource.getAllElements()) {
            if (el.getName().equals("body")) {
                bodyFound=true;
            }
            if (bodyFound) {
                TagType tagType = el.getStartTag().getTagType();
                if (tagType==StartTagType.NORMAL) {
                    String text = getOwnTextSegmentsString(el);
                    if (!text.trim().equals("")) {
                        int cpos = el.getBegin();               
                        System.out.println(el.getName()+"("+tagType.toString()+") line "+   htmlSource.getRow(cpos)+":"+text);
                    }
                } // if
            } // if
        } // for

    }

    /**
     * this function is not used it's shown here only for reference
     */ 
    public static Iterator<Segment> getOwnTextSegmentsIterator(Element elem) {
        final Iterator<Segment> it = elem.getContent().getNodeIterator();
        final List<Segment> results = new LinkedList<Segment>();
        int tagCounter = 0;
        while (it.hasNext()) {
            Segment cur = it.next();            
            if(cur instanceof StartTag) 
                tagCounter++;
            else if(cur instanceof EndTag) 
                tagCounter--;

            if (!(cur instanceof Tag) && tagCounter == 0) {
                System.out.println(cur);
                results.add(cur);
            }
        }
        return results.iterator();
    }

    public static String getOwnTextSegmentsString(Element elem) {
        final Iterator<Segment> it = elem.getContent().getNodeIterator();
        StringBuilder strBuilder = new StringBuilder();
        int tagCounter = 0;
        while (it.hasNext()) {
            Segment cur = it.next();            
            if(cur instanceof StartTag) 
                tagCounter++;
            else if(cur instanceof EndTag) 
                tagCounter--;

            if (!(cur instanceof Tag) && tagCounter == 0) {
                strBuilder.append(cur.toString() + ' ');
            }
        }
        return strBuilder.toString().trim();
    }

}
于 2014-09-30T09:22:56.593 回答
1

这是一个测试预期输出的 Junit 测试和一个基于 Jericho 的 SourceTextExtractor,它使基于原始 Jericho TextExtractor 源代码的 JUnit 测试工作。

@Test
public void testTextExtract() {
    // https://github.com/paepcke/CorEx/blob/master/src/extraction/HTMLUtils.java
    String htmls[] = {
            "<!DOCTYPE html>\n" + "<html>\n" + "<body>\n" + "\n"
                    + "<h1>My First Heading</h1>\n" + "\n"
                    + "<p>My first paragraph.</p>\n" + "\n" + "</body>\n" + "</html>",
            "<html>\n"
                    + "<body>\n"
                    + "\n"
                    + "<div id=\"myDiv\" name=\"myDiv\" title=\"Example Div Element\">\n"
                    + "  <h5>Subtitle</h5>\n"
                    + "  <p>This paragraph would be your content paragraph...</p>\n"
                    + "  <p>Here's another content article right here.</p>\n"
                    + "</div>" + "\n" + "Text at end of body</body>\n" + "</html>" };
    int expectedSize[] = { 2, 4 };
    String expectedInfo[][]={
        { 
            "line 5 col 5 to  line 5 col 21: My First Heading",
            "line 7 col 4 to  line 7 col 23: My first paragraph."
        },
        { 
            "line 5 col 7 to  line 5 col 15: Subtitle",
            "line 6 col 6 to  line 6 col 55: This paragraph would be your content paragraph...",
            "line 7 col 6 to  line 7 col 48: Here's another content article right here.",
            "line 8 col 7 to  line 9 col 20: Text at end of body"
        }
    };
    int i = 0;
    for (String html : htmls) {
        SourceTextExtractor extractor=new SourceTextExtractor();
        List<TextResult> textParts = extractor.extractTextSegments(html);
        // List<String> textParts = HTMLCleanerTextExtractor.extractText(html);
        int j=0;
        for (TextResult textPart : textParts) {
            System.out.println(textPart.getInfo());
            assertTrue(textPart.getInfo().startsWith(expectedInfo[i][j]));
            j++;
        }
        assertEquals(expectedSize[i], textParts.size());
        i++;
    }
}

这是一个改编的 TextExtractor 见 http://grepcode.com/file_/repo1.maven.org/maven2/net.htmlparser.jericho/jericho-html/3.3/net/htmlparser/jericho/TextExtractor.java/?v=source

/**
 * TextExtractor that makes source line and col references available
 * http://grepcode.com/file_/repo1.maven.org/maven2/net.htmlparser.jericho/jericho-html/3.3/net/htmlparser/jericho/TextExtractor.java/?v=source
 */
public class SourceTextExtractor {

    public static class TextResult {
        private String text;
        private Source root;
        private Segment segment;
        private int line;
        private int col;

        /**
         * get a textResult
         * @param root
         * @param segment
         */
        public TextResult(Source root,Segment segment) {
            this.root=root;
            this.segment=segment;
            final StringBuilder sb=new StringBuilder(segment.length());
            sb.append(segment);
            setText(CharacterReference.decodeCollapseWhiteSpace(sb));
            int spos = segment.getBegin();  
            line=root.getRow(spos);
            col=root.getColumn(spos);

        }

        /**
         * gets info about this TextResult
         * @return
         */
        public String getInfo() {
            int epos=segment.getEnd();

            String result=
                    " line "+   line+" col "+col+
                    " to "+
                    " line "+   root.getRow(epos)+" col "+root.getColumn(epos)+
                    ":"+getText();
            return result;
        }

        /**
         * @return the text
         */
        public String getText() {
            return text;
        }

        /**
         * @param text the text to set
         */
        public void setText(String text) {
            this.text = text;
        }

        public int getLine() {
            return line;
        }

        public int getCol() {
            return col;
        }

    }

    /**
     * extract textSegments from the given html
     * @param html
     * @return
     */
    public List<TextResult> extractTextSegments(String html) {
        Source htmlSource=new Source(html);
        List<TextResult> result = extractTextSegments(htmlSource);
        return result;
    }

    /**
     * get the TextSegments from the given root segment
     * @param root
     * @return
     */
    public List<TextResult> extractTextSegments(Source root) {
        List<TextResult> result=new ArrayList<TextResult>();
        for (NodeIterator nodeIterator=new NodeIterator(root); nodeIterator.hasNext();) {
            Segment segment=nodeIterator.next();
            if (segment instanceof Tag) {
                final Tag tag=(Tag)segment;
                if (tag.getTagType().isServerTag()) {
                    // elementContainsMarkup should be made into a TagType property one day.
                    // for the time being assume all server element content is code, although this is not true for some Mason elements.
                    final boolean elementContainsMarkup=false;
                    if (!elementContainsMarkup) {
                        final net.htmlparser.jericho.Element element=tag.getElement();
                        if (element!=null && element.getEnd()>tag.getEnd()) nodeIterator.skipToPos(element.getEnd());
                    }
                    continue;
                }
                if (tag.getTagType()==StartTagType.NORMAL) {
                    final StartTag startTag=(StartTag)tag;
                    if (tag.name==HTMLElementName.SCRIPT || tag.name==HTMLElementName.STYLE ||  (!HTMLElements.getElementNames().contains(tag.name))) {
                        nodeIterator.skipToPos(startTag.getElement().getEnd());
                        continue;
                    }

                }
                // Treat both start and end tags not belonging to inline-level elements as whitespace:
                if (tag.getName()==HTMLElementName.BR || !HTMLElements.getInlineLevelElementNames().contains(tag.getName())) {
                    // sb.append(' ');
                }
            } else {
                if (!segment.isWhiteSpace())
                    result.add(new TextResult(root,segment));
            }
        }
        return result;
    }

    /**
     * extract the text from the given segment
     * @param segment
     * @return
     */
    public String extractText(net.htmlparser.jericho.Segment pSegment) {

        // http://grepcode.com/file_/repo1.maven.org/maven2/net.htmlparser.jericho/jericho-html/3.3/net/htmlparser/jericho/TextExtractor.java/?v=source
        // this would call the code above
        // String result=segment.getTextExtractor().toString();
        final StringBuilder sb=new StringBuilder(pSegment.length());
        for (NodeIterator nodeIterator=new NodeIterator(pSegment); nodeIterator.hasNext();) {
            Segment segment=nodeIterator.next();
            if (segment instanceof Tag) {
                final Tag tag=(Tag)segment;
                if (tag.getTagType().isServerTag()) {
                    // elementContainsMarkup should be made into a TagType property one day.
                    // for the time being assume all server element content is code, although this is not true for some Mason elements.
                    final boolean elementContainsMarkup=false;
                    if (!elementContainsMarkup) {
                        final net.htmlparser.jericho.Element element=tag.getElement();
                        if (element!=null && element.getEnd()>tag.getEnd()) nodeIterator.skipToPos(element.getEnd());
                    }
                    continue;
                }
                if (tag.getTagType()==StartTagType.NORMAL) {
                    final StartTag startTag=(StartTag)tag;
                    if (tag.name==HTMLElementName.SCRIPT || tag.name==HTMLElementName.STYLE ||  (!HTMLElements.getElementNames().contains(tag.name))) {
                        nodeIterator.skipToPos(startTag.getElement().getEnd());
                        continue;
                    }

                }
                // Treat both start and end tags not belonging to inline-level elements as whitespace:
                if (tag.getName()==HTMLElementName.BR || !HTMLElements.getInlineLevelElementNames().contains(tag.getName())) {
                    sb.append(' ');
                }
            } else {
                sb.append(segment);
            }
        }
        final String result=net.htmlparser.jericho.CharacterReference.decodeCollapseWhiteSpace(sb);
        return result;
    }
}
于 2014-10-04T05:47:44.057 回答