我想从 html 文档中提取有意义的文本,并且我正在使用 html-agility-pack 来做同样的事情。这是我的代码:
string convertedContent = HttpUtility.HtmlDecode(
ConvertHtml(HtmlAgilityPack.HtmlEntity.DeEntitize(htmlAsString))
);
转换HTML:
public string ConvertHtml(string html)
{
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
StringWriter sw = new StringWriter();
ConvertTo(doc.DocumentNode, sw);
sw.Flush();
return sw.ToString();
}
转换成:
public void ConvertTo(HtmlAgilityPack.HtmlNode node, TextWriter outText)
{
string html;
switch (node.NodeType)
{
case HtmlAgilityPack.HtmlNodeType.Comment:
// don't output comments
break;
case HtmlAgilityPack.HtmlNodeType.Document:
foreach (HtmlNode subnode in node.ChildNodes)
{
ConvertTo(subnode, outText);
}
break;
case HtmlAgilityPack.HtmlNodeType.Text:
// script and style must not be output
string parentName = node.ParentNode.Name;
if ((parentName == "script") || (parentName == "style"))
break;
// get text
html = ((HtmlTextNode)node).Text;
// is it in fact a special closing node output as text?
if (HtmlNode.IsOverlappedClosingElement(html))
break;
// check the text is meaningful and not a bunch of whitespaces
if (html.Trim().Length > 0)
{
outText.Write(HtmlEntity.DeEntitize(html) + " ");
}
break;
case HtmlAgilityPack.HtmlNodeType.Element:
switch (node.Name)
{
case "p":
// treat paragraphs as crlf
outText.Write("\r\n");
break;
}
if (node.HasChildNodes)
{
foreach (HtmlNode subnode in node.ChildNodes)
{
ConvertTo(subnode, outText);
}
}
break;
}
}
现在在某些情况下,当 html 页面格式错误时(例如以下页面 - http://rareseeds.com/cart/products/Purple_of_Romagna_Artichoke-646-72.html有一个格式错误的元标记,例如<meta content="text/html; charset=uft-8" http-equiv="Content-Type">
)[注意“uft” utf] 我的代码在我尝试加载 html 文档时正在呕吐。
有人可以建议我如何克服这些格式错误的 html 页面并仍然从 html 文档中提取相关文本吗?
谢谢,卡皮尔