1

我已经在 Apple 的应用商店中发布了一个移动应用程序。

此 SPI 客户端应用程序使用服务器端的 Rest API 来检索有关巴士到达特定巴士站的实时信息。

该应用程序像魅力一样运行了 6 个月。

Rest API 使用 WebHarvest 从网站（例如：http ://www.metlink.org.nz/stop/4912/departures ）中提取真实数据信息。

几天前，从我的服务器端代码中抓取的 HTML 页面通过添加以下行进行了更改：

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">

从那时起，我的应用程序就停止了工作。

我知道我可以使用 regExp 剥离上面的行，但我想知道是否有办法通知 WebHarvest 禁用 XML 验证。禁用 XML 验证，我不需要进入我拥有的每个配置并将我的 xpath 表达式更改为 regExp 以剥离上面的行。

这是我的配置文件：

<config charset="UTF-8">

    <var-def name="pageContentStr">
        <html-to-xml>
            <http url="http://www.metlink.org.nz/stop/${stationID.toString()}/departures" />
        </html-to-xml>
    </var-def>

    <var-def name="serverTime">
        <xpath expression="/html/body/ul/li/span/text()">
            <var name="pageContentStr" />
        </xpath>
    </var-def>

    <var-def name="busRTI">
            <xpath expression="//tbody/tr[@data-code]/concat(td[1]/a[starts-with(@href,'timetables/')]/span/text(),'::',td[1]/a[starts-with(@href,'timetables/bus/')]/span/attribute::style,'::',td[2]/span/text(),'::',td[3]/span/text())">
                <var name="pageContentStr" />
            </xpath>
    </var-def>
</config>

如果我在 WebHarvest GUI 中运行它，上面插入的配置文件工作正常（奇怪）。但是，在我的 Rest API 中运行它时收到错误消息。这是我收到的错误：

exception

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.webharvest.exception.ScraperXPathException: Error parsing XPath expression (XPath = [/html/body/ul/li/span/text()])!
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:948)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

org.webharvest.exception.ScraperXPathException: Error parsing XPath expression (XPath = [/html/body/ul/li/span/text()])!
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:70)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

net.sf.saxon.trans.XPathException: org.xml.sax.SAXParseExceptionpublicId: -//W3C//DTD HTML 4.0 Transitional//EN; systemId: `http://www.w3.org/TR/REC-html40/loose.dtd`; lineNumber: 31; columnNumber: 3; The declaration for the entity "HTML.Version" must end with '>'.
    net.sf.saxon.event.Sender.sendSAXSource(Sender.java:420)
    net.sf.saxon.event.Sender.send(Sender.java:169)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3346)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3288)
    net.sf.saxon.query.StaticQueryContext.buildDocument(StaticQueryContext.java:327)
    org.webharvest.utils.XmlUtil.evaluateXPath(XmlUtil.java:77)
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:68)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
root cause

org.xml.sax.SAXParseExceptionpublicId: -//W3C//DTD HTML 4.0 Transitional//EN; systemId: http://www.w3.org/TR/REC-html40/loose.dtd; lineNumber: 31; columnNumber: 3; The declaration for the entity "HTML.Version" must end with '>'.
    com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
    com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
    com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
    com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
    com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1388)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanEntityDecl(XMLDTDScannerImpl.java:1562)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDecls(XMLDTDScannerImpl.java:1964)
    com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.scanDTDExternalSubset(XMLDTDScannerImpl.java:297)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1162)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1049)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:962)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
    com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
    com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
    com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
    com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
    com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
    com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
    net.sf.saxon.event.Sender.sendSAXSource(Sender.java:396)
    net.sf.saxon.event.Sender.send(Sender.java:169)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3346)
    net.sf.saxon.Configuration.buildDocument(Configuration.java:3288)
    net.sf.saxon.query.StaticQueryContext.buildDocument(StaticQueryContext.java:327)
    org.webharvest.utils.XmlUtil.evaluateXPath(XmlUtil.java:77)
    org.webharvest.runtime.processors.XPathProcessor.execute(XPathProcessor.java:68)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.processors.BodyProcessor.execute(BodyProcessor.java:25)
    org.webharvest.runtime.processors.VarDefProcessor.execute(VarDefProcessor.java:59)
    org.webharvest.runtime.processors.BaseProcessor.run(BaseProcessor.java:115)
    org.webharvest.runtime.Scraper.execute(Scraper.java:166)
    org.webharvest.runtime.Scraper.execute(Scraper.java:179)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.scrapeBusesForStation(MetLinkAdapterImpl.java:147)
    com.didibaba.services.adapters.metlink.MetLinkAdapterImpl.getStationBuses(MetLinkAdapterImpl.java:118)
    com.didibaba.services.BusStationServiceImpl.getBusStationInfoByName(BusStationServiceImpl.java:80)
    com.didibaba.web.controllers.BusStationController.getBusStationInfo(BusStationController.java:36)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

提前致谢。

4

2 回答 2

0

与 xpath 评估器抛出类似的问题org.xml.sax.SAXParseException

White spaces are required between publicId and systemId.

当您可以更改原始xml时，此问题的解决方案已在此处解决。

Webharvest 在后台使用html 清洁器。我使用 Complete Web-Harvest 项目，因此可能会阻止在html-to-xml.

我使用 html 清洁器 2.6.1 版，并进行了修改org.webharvest.runtime.processors.HtmlToXmlProcessor以支持这个较新的版本

HtmlCleaner cleaner = new HtmlCleaner( );
CleanerProperties cleanerProperties = cleaner.getProperties();

由于 html 清洁器支持omitDoctypeDeclaration -完全省略 doctype 的参数，我添加了这一点（将来可能通过带有刮板 xml 的额外属性来完成）。

cleanerProperties.setOmitDoctypeDeclaration(true);

希望它有所帮助，感谢 webharvest 的创建者，它是一个很棒且非常可靠的工具！

于 2014-12-28T21:25:51.620 回答
0

您可以尝试omithtmlenvelope="true"在html-to-xml调用中使用 a 。

<var-def name="pageContentStr">
    <html-to-xml omithtmlenvelope="true">
        <http url="http://www.metlink.org.nz/stop/${stationID.toString()}/departures" />
    </html-to-xml>
</var-def>

但是，不幸的是，正如您所说，我无法重现您遇到的错误，也无法测试结果......

于 2014-08-21T12:01:38.607 回答