0

我被要求从页面中抓取某行，但似乎该站点已阻止 CURL 请求？

有问题的网站是http://www.habbo.com/home/Intricat

我尝试更改 UserAgent 以查看他们是否阻止了它，但它似乎没有起到作用。

我正在使用的代码如下：

<?php

$curl_handle=curl_init();
//This is the URL you would like the content grabbed from
curl_setopt($curl_handle, CURLOPT_USERAGENT, "Mozilla/5.0");
curl_setopt($curl_handle,CURLOPT_URL,'http://www.habbo.com/home/Intricat');
//This is the amount of time in seconds until it times out, this is useful if the server you are requesting data from is down. This way you can offer a "sorry page"
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2);

curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1);
$buffer = curl_exec($curl_handle);
//This Keeps everything running smoothly
curl_close($curl_handle);

// Change the message bellow as you wish, please keep in mind you must have your message within the " " Quotes.
if (empty($buffer))
{
    print "Sorry, It seems our weather resources are currently unavailable, please check back later.";
}
else
{
    print $buffer;
}
?>

如果他们阻止了 CURL 请求，我可以从该页面获取一行代码的任何想法？

编辑：在通过我的服务器运行 curl -i 时，该站点似乎首先设置了一个 cookie？

4

4 回答 4

1

你对你所说的块类型不是很具体。有问题的网站http://www.habbo.com/home/Intricat首先检查浏览器是否启用了 javascript：

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    <meta http-equiv="Content-Script-Type" content="text/javascript">
    <script type="text/javascript">function setCookie(c_name, value, expiredays) {
        var exdate = new Date();
        exdate.setDate(exdate.getDate() + expiredays);
        document.cookie = c_name + "=" + escape(value) + ((expiredays == null) ? "" : ";expires=" + exdate.toGMTString()) + ";path=/";
    }
    function getHostUri() {
        var loc = document.location;
        return loc.toString();
    }
    setCookie('YPF8827340282Jdskjhfiw_928937459182JAX666', '179.222.19.192', 10);
    setCookie('DOAReferrer', document.referrer, 10);
    location.href = getHostUri();</script>
</head>
<body>
<noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your
    browser.
</noscript>
</body>
</html>

由于 curl 没有 javascript 支持，您要么需要使用具有 - 或者 - 您需要模仿该脚本并创建自己的 cookie 和新请求 URI 的 HTTP 客户端。

于 2012-11-02T16:38:37.223 回答
1

进入您的浏览器并复制正在发送的确切标头，该站点将无法判断您正在尝试卷曲，因为请求看起来完全相同。如果使用 cookie - 将它们作为标题附加。

于 2012-11-02T16:37:32.250 回答
1

这是我几年前做的 Curl 课的剪切和粘贴，希望你能从中挑选一些宝石。

function get_url($url)
{ 
    curl_setopt ($this->ch, CURLOPT_URL, $url); 
    curl_setopt ($this->ch, CURLOPT_USERAGENT, $this->user_agent);
    curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $this->cookie_name);
    curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $this->cookie_name);
    if(!is_null($this->referer))
    {
        curl_setopt ($this->ch, CURLOPT_REFERER, $this->referer);  
    }
    curl_setopt ($this->ch, CURLOPT_SSL_VERIFYHOST, 2);
    curl_setopt ($this->ch, CURLOPT_HEADER, 0); 
    if($this->follow)
    {
        curl_setopt ($this->ch, CURLOPT_FOLLOWLOCATION, 1);
    }
    else
    {
        curl_setopt ($this->ch, CURLOPT_FOLLOWLOCATION, 0);
    }
    curl_setopt ($this->ch, CURLOPT_RETURNTRANSFER, 1); 
    curl_setopt ($this->ch, CURLOPT_HTTPHEADER, array("Accept: text/html,text/vnd.wap.wml,*.*"));
    curl_setopt ($this->ch, CURLOPT_SSL_VERIFYPEER, FALSE);  // this line makes it work under https

    $try=0;
    $result="";
    while( ($try<=$this->retry_attempts) && (empty($result)) )  // force a retry upto 5 times
    {
        $try++;
        $result = curl_exec($this->ch);
        $this->response=curl_getinfo($this->ch);
        // $response['http_code'] 4xx is an error
    }
    // set refering URL to current url for next page.
    if($this->referer_to_last) $this->set_referer($url);

    return $result; 
}
于 2012-11-02T16:38:04.583 回答
0

我知道这是一个很老的帖子，但是由于我今天必须回答自己同样的问题，所以我在这里分享给来的人，它可能对他们有用。我也完全了解 OP 的curl具体要求，但是 - 就像我一样 - 可能有人对解决方案感兴趣，无论是否curl。

我想获得的页面curl阻止了它。如果阻止不是因为javascript，而是因为代理（这是我的情况，并且设置代理curl没有帮助），那么wget可能是一个解决方案：

wget -o output.txt --no-check-certificate --user-agent="Mozilla/5.0 (Windows NT 5.2; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" "http://example.com/page"
于 2015-08-10T22:20:34.780 回答