Guest llchrisb Posted January 31, 2007 Share Posted January 31, 2007 Searched the forum, but didn't see much for specifics on the topic. What parts of my shop should I disallow spiders to crawl? thanks chris Quote Link to comment Share on other sites More sharing options...
convict Posted January 31, 2007 Share Posted January 31, 2007 User-agent: * Disallow: /index.php?act=login Disallow: /index.php?act=taf Disallow: /admin/ Disallow: /classes/ Disallow: /docs/ Disallow: /extra/ Disallow: /images/ Disallow: /includes/ Disallow: /js/ Disallow: /language/ Disallow: /modules/ Disallow: /pear/ Disallow: /skins/ Disallow: /cart.php Disallow: /confirmed.php Disallow: /download.php Disallow: /offLine.php Disallow: /switch.php Quote Link to comment Share on other sites More sharing options...
roban Posted January 31, 2007 Share Posted January 31, 2007 It's controversial as to whether robots.txt is even of any value. Only a few robots support it. Using it cannot hurt. Quote Link to comment Share on other sites More sharing options...
Guest llchrisb Posted January 31, 2007 Share Posted January 31, 2007 It's controversial as to whether robots.txt is even of any value. Only a few robots support it. Using it cannot hurt. That was my understanding, and what I was going for. Thanks for the heads up. User-agent: * Disallow: /index.php?act=login Disallow: /index.php?act=taf Disallow: /admin/ Disallow: /classes/ Disallow: /docs/ Disallow: /extra/ Disallow: /images/ Disallow: /includes/ Disallow: /js/ Disallow: /language/ Disallow: /modules/ Disallow: /pear/ Disallow: /skins/ Disallow: /cart.php Disallow: /confirmed.php Disallow: /download.php Disallow: /offLine.php Disallow: /switch.php thanks convict, that's exactly what I was looking for. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.