Whats the difference - spider.txt, robots.txt, .htaccess

[queenniara]

I just installed the mod: Automated Sitemaps for Google, Yahoo, Ask etc.

It asked me to ask add this line Sitemap: http://mydomain.com/myshop/sitemap-google.php to my robots.txt file to allow search engines to auto-discover the sitemap.

My problem here is I have Cubecart 4.2.2. running and I have enabled the SEO mod included with it.

I looked in my root directory. There is NO robots.txt file, there is only a spiders.txt file and a .htaccess file. Which one of these do I put that line into, or do I need to get a robots.txt file?

Thanks for your help in advance!

[Guest PaulD]

I just installed the mod: Automated Sitemaps for Google, Yahoo, Ask etc.

It asked me to ask add this line Sitemap: http://mydomain.com/myshop/sitemap-google.php to my robots.txt file to allow search engines to auto-discover the sitemap.

My problem here is I have Cubecart 4.2.2. running and I have enabled the SEO mod included with it.

I looked in my root directory. There is NO robots.txt file, there is only a spiders.txt file and a .htaccess file. Which one of these do I put that line into, or do I need to get a robots.txt file?

Thanks for your help in advance!

I am not familiar with the mod you mention, but spider.txt and robots.txt are both small fry compared to your htaccess file. (W3C schools explain the differences very well)

Just create a txt file called robots, and add your line. You could also add instructions for google to follow all (google for robots.txt and there are loads of good sites about it). Whether search engines pay any attention to this is a different matter, although google claims to obey them, others certainly do not.

Google will have no trouble finding your site map. If you are concerned search for your site map on google. If it comes up, then you know google knows about it already, and any robots file instruction becomes useless anyway. If it does not appear, then add a link at the bottom of your site to the site map, google will definitely find it then.

Hope this helps,

Paul.

[queenniara]

I just installed the mod: Automated Sitemaps for Google, Yahoo, Ask etc.

It asked me to ask add this line Sitemap: http://mydomain.com/myshop/sitemap-google.php to my robots.txt file to allow search engines to auto-discover the sitemap.

My problem here is I have Cubecart 4.2.2. running and I have enabled the SEO mod included with it.

I looked in my root directory. There is NO robots.txt file, there is only a spiders.txt file and a .htaccess file. Which one of these do I put that line into, or do I need to get a robots.txt file?

Thanks for your help in advance!

I am not familiar with the mod you mention, but spider.txt and robots.txt are both small fry compared to your htaccess file. (W3C schools explain the differences very well)

Just create a txt file called robots, and add your line. You could also add instructions for google to follow all (google for robots.txt and there are loads of good sites about it). Whether search engines pay any attention to this is a different matter, although google claims to obey them, others certainly do not.

Google will have no trouble finding your site map. If you are concerned search for your site map on google. If it comes up, then you know google knows about it already, and any robots file instruction becomes useless anyway. If it does not appear, then add a link at the bottom of your site to the site map, google will definitely find it then.

Hope this helps,

Paul.

Thanks Paul!