0

抱歉，在这里将抓取的内容存储到 MYSQL 数据库时遇到了一些麻烦。

所以我想要做的是从这个站点 [http://www.ucc.ie/modules/descriptions/page014.html][1] 保存模块代码和模块标题到 MYSQL 数据库中。我可以从站点获取内容，但我似乎无法将抓取的内容保存到数据库中。我不断收到错误“查询为空”，而查询不是。

花了一些时间做这件事，似乎无法绕过它。解决此问题的任何帮助将不胜感激。

<?php

//Here is a simple web scraping example using the PHP DOM that tries to get the largest      text body of a HTML document. I needed it for a spider 
//that had to show a short description for a page. It assumes that document annotation can be the largest <div>, <td> or <p> element in the //page.
//In the example I show a way to prevent a bug in the DOM as it sometimes just doesn't recognize html encoding. It seems to work if you put 
//charset meta tag right after the head tag of the document.

$host="localhost";
$user="root";
$password="";
mysql_connect($host,$user,$password) or die("could not connect to the host");
mysql_select_db("plot_a_coursedb");


$ch= curl_init();
curl_setopt ($ch, CURLOPT_URL, 'http://www.ucc.ie/modules/descriptions/page014.html' );
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch,CURLOPT_VERBOSE,1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt ($ch, CURLOPT_REFERER, 'http://localhost:8080/extractsite/index2.html');     //just a fake referer
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch,CURLOPT_POST,0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 20);

$html= curl_exec($ch);
$html1= curl_getinfo($ch);

//try to get page encoding as it was sent from server
if ($html1['content_type']){
$arr= explode('charset=',$html1['content_type']);
$csethdr= strtolower(trim($arr[1]));
} else {
$csethdr= false;
}

$cset= false;
$arr= array();

//This has to replace page meta tags for charset with utf-8, but it doesn't actually help(see the bug info).
if (preg_match_all('/(<meta\s*http-equiv="Content-Type"\s*content="[^;]*;
\s*charset=([^"]*?)(?:"|\;)[^>]*>)/' //merge this line
,$html,$arr,PREG_PATTERN_ORDER)){
$cset= strtolower(trim($arr[2][0]));
if ($cset!='utf-8'||$cset!=$csethdr){
    $new= str_replace($arr[2][0],'utf-8',$arr[1][0]);
    $html= str_replace($arr[1][0],$new,$html);
    $cset= $csethdr;
} else {
    $cset= false;
}

if ($cset=='utf-8'){
    $cset= false;
}
}
unset($arr);
if ($cset){
$html= iconv($cset,'utf-8',$html);
}
unset($cset);

//solve dom bug
$html=preg_replace('/<head[^>]*>/','<head><META HTTP-EQUIV="Content-Type"   CONTENT="text/html; charset=utf-8">',$html);

@$dom= new DOMDocument();
@$dom->loadHTML($html);
@$dom->preserveWhiteSpace = false;

function getMaxTextBody($dom){
$content = $dom->getElementsByTagname('div');
$content2= $dom->getElementsByTagname('td');
$content3= $dom->getElementsByTagname('p');
$content4 = $dom->getElementsByTagname('B');

$new = array();
foreach ($content as $value) {
    $new[]= $value;
    unset($value);
}
unset($content);

foreach ($content2 as $value) {
    $new[]= $value;
    unset($value);
}
unset($content2);

foreach ($content3 as $value) {
    $new[]= $value;
    unset($value);
}
unset($content3);

foreach ($content4 as $value) {
    $new[]= $value;
    unset($value);
}
unset($content4);

$maxlen= 0;
$result= '';
foreach ($new as $item)
{
    $str= $item->nodeValue;
    if (strlen($str)>$maxlen){
        $content1= $item->getElementsByTagName('div');
        $content2= $item->getElementsByTagname('td');
        $content3= $item->getElementsByTagname('p');
        $content4 = $dom->getElementsByTagname('b');
        
        $contentnew= array();
        foreach ($content1 as $value) {
            $contentnew[]= $value;
            unset($value);
        }
        unset($content1);
        
        foreach ($content2 as $value) {
            $contentnew[]= $value;
            unset($value);
        }
        unset($content2);
        
        foreach ($content3 as $value) {
            $contentnew[]= $value;
            unset($value);
        }
        unset($content3);
        
        foreach ($content4 as $value) {
            $contentnew[]= $value;
            unset($value);
        }
        unset($content4);



        // Insert data into database query
        $query = mysql_query("INSERT INTO data (div,td,p,b) VALUES ('$content1','$content2','$content3','$content4')"); 
        mysql_query($query) or die (mysql_error());

        // Close the database connection
        mysql_close();


        if (count($contentnew)==0){
            $result= $str;
        } else {
            foreach ($contentnew as $value) {
                $str1= getMaxTextBody($value);
                $str2= $value->nodeValue;
                    //let's say largest body has more than 50% of the text in its parent
                                if (strlen($str1)*2<strlen($str2)){
                    $str1= $str2;
                }
                if (strlen($str1)*2>strlen($str)&&strlen($str1)>$maxlen){
                    $result= $str1;
                } elseif (strlen($str1)>$maxlen){
                    $result= $str1;
                }
                $maxlen= strlen($result);
            }
        }
        $maxlen= strlen($result);
        unset($contnentnew);
    }
}

unset($new);
return $result;

}
print getMaxTextBody($dom);


?>

下面是我创建的用于存储内容的 MYSQL 表

DROP TABLE IF EXISTS `data`;
CREATE TABLE `data` (
`div`  varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`td`  varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`p`  varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`b`  varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
PRIMARY KEY (`div`)
)
ENGINE=InnoDB
DEFAULT CHARACTER SET=utf8 COLLATE=utf8_unicode_ci;
}

任何帮助找出我的内容没有保存到数据库的原因将不胜感激。

4

4 回答 4

2

在您构造 sql 查询时，您的内容变量确实为空 - 它们在上面未设置。 

$query = mysql_query("INSERT INTO data (div,td,p,b) VALUES \
('$content1','$content2','$content3','$content4')");

我认为您可能希望将每个 $content 中的 $value 的 nodeValue 内容作为 $value 字符串放置在该 mysql 查询中（您可以使用 $value->nodeValue 访问节点的文本内容）。

例如，如果您想要一个节点的文本内容，例如示例中的这个 P 节点，当您 print_r 节点时，它看起来像这样：

DOMElement Object
(
[tagName] => p
[schemaTypeInfo] =>
[nodeName] => p
[nodeValue] => Students should note that all of the modules below may not 
be available to them.
[nodeType] => 1
[parentNode] => (object value omitted)
[childNodes] => (object value omitted)
[firstChild] => (object value omitted)
[lastChild] => (object value omitted)
[previousSibling] => (object value omitted)
[nextSibling] => (object value omitted)
[attributes] => (object value omitted)
[ownerDocument] => (object value omitted)
[namespaceURI] =>
[prefix] =>
[localName] => p
[baseURI] =>
[textContent] => Students should note that all of the modules below may \\
not be available to them.
)

您可以看到该节点中有两个您可能感兴趣的值 - textContent 和 nodeValue。

您可以通过执行以下操作从您的代码中访问这些：

  foreach ($content3 as $value) { // content3 contains the p nodes, I think?
           // let's see what the node looks like
          print_r($value); 
          // let's get hold of the text value from the node
          $mytempvariable=$value->nodeValue;
          print "CONTENT OF P NODE: \n\n$mytempvariable\n\n\n";
  }

这将从所有 P 节点打印出文本。

于 2012-06-08T12:54:14.243 回答
0

我认为使用 memcache 会更快。也只获取必要的 div 并将其存储在内存缓存中，并在需要时获取唯一的页面名称

于 2012-06-08T12:54:42.307 回答
0

您没有在查询中转义变量 $contentX。我怀疑您启动的查询比您想象的要复杂。

检查http://php.net/manual/en/function.mysql-escape-string.php了解更多信息。

更广泛地说，您应该为查询使用 PDO 扩展和准备好的语句。

于 2012-06-08T12:55:19.390 回答
0

它没有插入$content1到的原因$content4是因为你在你的 foreach 循环之后取消了它们。所以不会插入任何值。

于 2012-06-08T13:12:51.460 回答