1

我正在尝试使用 Prerender 服务为爬虫的服务器端渲染设置 Angular 1.5 应用程序。

内页一切正常，但主页的渲染存在问题 - 爬虫看到 404 页面而不是主页。

我想我的 .htaccess 中的其他一些规则存在问题 - 除了 Prerender 的规则，我对所有页面都使用了另外两个规则：

  • 将不带斜杠的网址重写到带有斜杠的网址上
  • 在不带 www 的 url 上重写带 www 的 url

任何提示将不胜感激！

这是我的 Apache 服务器的 .htaccess 文件

RequestHeader set X-Prerender-Token "MyToken"

  RewriteEngine On

  RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
  RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]


  # If an existing asset or directory is requested go to it as it is
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f [OR]
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d
  RewriteRule ^ - [L]

  RewriteCond %{REQUEST_URI}  ^/$
  RewriteCond %{QUERY_STRING} ^_escaped_fragment_=/?(.*)$
  RewriteRule ^(.*)$ /snapshots/%1? [NC,L]

  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteRule ^(.*[^/])$ /$1/ [L,R=301]

  <IfModule mod_proxy_http.c>
      RewriteCond %{HTTP_USER_AGENT} Googlebot|bingbot|Googlebot-Mobile|Baiduspider|Yahoo|YahooSeeker|DoCoMo|Twitterbot|TweetmemeBot|Twikle|Netseer|Daumoa|SeznamBot|Ezooms|MSNBot|Exabot|MJ12bot|sogou\sspider|YandexBot|bitlybot|ia_archiver|proximic|spbot|ChangeDetection|NaverBot|MetaJobBot|magpie-crawler|Genieo\sWeb\sfilter|Qualidator.com\sBot|Woko|Vagabondo|360Spider|ExB\sLanguage\sCrawler|AddThis.com|aiHitBot|Spinn3r|BingPreview|GrapeshotCrawler|CareerBot|ZumBot|ShopWiki|bixocrawler|uMBot|sistrix|linkdexbot|AhrefsBot|archive.org_bot|SeoCheckBot|TurnitinBot|VoilaBot|SearchmetricsBot|Butterfly|Yahoo!|Plukkie|yacybot|trendictionbot|UASlinkChecker|Blekkobot|Wotbox|YioopBot|meanpathbot|TinEye|LuminateBot|FyberSpider|Infohelfer|linkdex.com|Curious\sGeorge|Fetch-Guess|ichiro|MojeekBot|SBSearch|WebThumbnail|socialbm_bot|SemrushBot|Vedma|alexa\ssite\saudit|SEOkicks-Robot|Browsershots|BLEXBot|woriobot|AMZNKAssocBot|Speedy|oBot|HostTracker|OpenWebSpider|WBSearchBot|FacebookExternalHit [NC,OR]
      RewriteCond %{QUERY_STRING} _escaped_fragment_

      # Only proxy the request to Prerender if it's a request for HTML
      RewriteRule ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff))(.*) http://service.prerender.io/http://example.com/$2 [P,L]
  </IfModule>

  # If the requested resource doesn't exist, use index.html
  RewriteRule ^ /index.html
4

1 回答 1

0

你有这个部分：

  RewriteCond %{REQUEST_URI}  ^/$
  RewriteCond %{QUERY_STRING} ^_escaped_fragment_=/?(.*)$
  RewriteRule ^(.*)$ /snapshots/%1? [NC,L]

_escaped_fragment_如果在 URL 中，它将尝试从 /snapshots/ 目录中提供文件。这与 Prerender.io 没有任何关系，因此您可能希望删除该部分，因为它可能是 404 的原因。

您还通过他们的用户代理检查 Googlebot 和 Bingbot，这是一个坏主意，因为他们可能会因伪装而惩罚您。

于 2016-12-13T19:44:48.893 回答