0

在客户的网站上有一系列位于不同页面上的导师弹出窗口。

http://launcheducation.com/

问题是它们不需要被编入索引,它们只是主页上的弹出窗口。如果您直接加载链接,则它们的样式不正确。它们没有适当的 html 或 head 标签,因为它们只是通过 AJAX 加载的。

因为他们没有头标签,所以我不能给它添加元标签。我是 SEO n00b,我只知道基础知识。但我当然不认为我可以添加没有头标签的元标签。所以我将它们添加到 robots.txt 中。

我以为这就结束了,但我的客户正在使用 SEOmoz 工具栏,上面说他们仍在被跟踪。我认为这根本不是问题,我认为只是工具栏没有检查 robots.txt,但他不断反复提出。

我不知道该怎么做,所以我想澄清一下将链接放在 robots.txt 中是否可行。这是我的 robots.txt:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-content/themes/launcheducation/tutor-ajax/*?
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669

是的,我确实从“禁止:/wp-content/themes/launcheducation/tutor-ajax/*?”开始。当插件仍然没有确认更改时,我将它们全部添加。robots.txt 也可以清楚地加载到根目录中:

http://launcheducation.com/robots.txt

任何帮助将不胜感激,即使它只是确认我的陈述是正确的并且它只是不检查 robots.txt 的插件。如果我错了,当然,我想知道如何解决这个问题。

提前致谢。

4

1 回答 1

1

看起来你在正确的轨道上。我不确定 SEOMoz 工具,但您应该使用的是 Google Webmaster Tool。在该工具中,有一项功能可让您使用 Googlebot 抓取任何给定的 URL。这将明确地告诉您您的 AJAX 部分是否被 robots.txt 正确阻止

如果存在 URL 模式与您希望可抓取的内容匹配的任何极端情况,您始终可以通过 http 标头发送 NOINDEX 命令,因为您无法添加元标记。X-Robots-Tag http 标头用于此目的,主要用于 PDF 和图像无法呈现 HTML 的情况......它适用于您的情况。

完整的规格在这里:https ://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

顺便说一句,像 SEOMoz 这样的 3rd 方工具很棒,但没有什么比引擎本身提供的工具更好的了。:)

于 2013-01-12T01:59:18.907 回答