Comments on: Robots.txt for WordPress Blog

By: international travel

international travel — Thu, 10 Nov 2011 21:34:46 +0000

today I read this post and i check Google robot.txt ,there added many parameter, what about site map?

By: SEO Services Pakistan

SEO Services Pakistan — Sat, 27 Aug 2011 22:12:26 +0000

“it is good to learn from experienced people, also like you.
Thanks for sharing this useful information!
“

By: Object Synergy

Object Synergy — Tue, 09 Aug 2011 10:02:08 +0000

I totally agree with SEO GURU, If we do not want to restrict any folder or file of our website then no need to use robots txt file.

By: dana

dana — Thu, 21 Apr 2011 13:49:20 +0000

Yeah, we can surely play with robots.txt if we find that it is too hard to fix the crawls errors directly — we just tell Google to not crawl the bad part.

By: Michael Brown | Cutting Edge

Michael Brown | Cutting Edge — Mon, 11 Apr 2011 03:13:21 +0000

Dana;
Thanks for this article, Google Webmaster is giving me fits trying to correct crawl errors, most of which are the result of making links out of internal site tags and categories. This should help to get rid of some of the worst offenders.
Thanks, Michael Brown

By: dana

dana — Tue, 11 Jan 2011 13:39:09 +0000

Yes, it can be restricted by robots.txt (forgot the detail though). However, you can restrict it easily by using AIO or Platinum SEO.

By: taranulsef

taranulsef — Sat, 08 Jan 2011 11:55:01 +0000

Great article that clarifies some things…
I’ve got one question: I get duplicate content from my comments pages such as “……./?replytocom=2″. Can this be restricted by robots.txt?

By: dana

dana — Wed, 22 Dec 2010 15:08:08 +0000

Thanks mate.

By: Seo

Seo — Wed, 22 Dec 2010 13:48:32 +0000

Very important to use the robots.txt file. Great post.

By: dana

dana — Tue, 14 Dec 2010 11:22:59 +0000

Yes, you are right mate.

By: SEO GURU

SEO GURU — Tue, 14 Dec 2010 11:16:05 +0000

In general, we prefer that our web pages are indexed by the search engines. But there may be some content that we don’t want to be crawled & indexed. Like the personal images folder, website administration folder, customer’s test folder of a web developer, no search value folders like cgi-bin, and many more. The main idea is we don’t want them to be indexed.

By: dana

dana — Sat, 04 Sep 2010 00:16:59 +0000

Thanks for suggestion.

By: aljuk

aljuk — Fri, 03 Sep 2010 12:45:43 +0000

Don’t forget that you need to set your canonical links, otherwise your content is still accessible, like

mysite.com/my-post/0
mysite.com/my-post/1
mysite.com/my-post/2
etc. …
mysite.com/my-post/100000000000

All it takes is one enemy to generate a link farm pointing to a few thousand of these addresses, and your domain will never get good serps again, and will be permanently marked as spam.

By: aljuk

aljuk — Fri, 03 Sep 2010 12:40:33 +0000

Use /category/*/*

That way you index your category names (good seo) but don’t duplicate the posts they point to.

By: dana

dana — Sat, 30 Jan 2010 00:46:06 +0000

Yes, it is.

By: Nimit Kashyap

Nimit Kashyap — Fri, 29 Jan 2010 14:51:09 +0000

using robot.txt file is essential in order to avoid the duplicate contnet issue.

By: dana

dana — Fri, 29 Jan 2010 14:35:26 +0000

It is because i my self is in basic state of blogging.

By: Nhoel

Nhoel — Fri, 29 Jan 2010 13:11:57 +0000

As usual… you always make post for basic things, but the good thing is.. they hit.. good thing there’s someone bothering to do this..

thanks Dana!

-Nhoel of keywordspeak.com

By: dana

dana — Fri, 29 Jan 2010 10:41:40 +0000

You should be not afraid of it because you always can delete it if something wrong.

By: Everyday Techies

Everyday Techies — Thu, 28 Jan 2010 15:22:16 +0000

Nice explanation about robot.txt, it always looked technical to me so i never dared mess with it will try to set it up now with the help of your post,