Jump to content
vidmarc

Robots.txt question

Recommended Posts

When I test my robots.txt using the Google console robots.txt Tester, I'm seeing errors for every line.

Any suggestions as to how to fix them?

 

robotscheck.jpg

Share this post


Link to post
Share on other sites

You shouldn't really have a .xml.gz AND a .xml version

You dont have an image sitemap file - but then the built in function in CubeCart doesnt create one (we have a fantastic sitemap generator that creates images / news / video sitemaps as well as the standard sitemap and the program can be setup to run via cron so it is automatically created and has a large amount of configuration possible)

Why would you want to disallow indexing of images ?  Even without an image sitemap, the Google crawler can eventually pick up some images

Ian

Share this post


Link to post
Share on other sites

I know this is a little old, but is the sitemap generator a freely available one, or bespoke and/or paid for?

 

Share this post


Link to post
Share on other sites

It is a feature of CubeCart. In admin, Maintenance, Rebuild tab, Miscellaneous section, check the box for Rebuild Sitemap and click Save.

If PHP has a ZipArchive extension installed (and this is a normal requirement for CubeCart), CubeCart will create a zipped version of the XML file.

CubeCart will then immediately notify Google that this file exists.

  • Like 1

Share this post


Link to post
Share on other sites
8 hours ago, Nelly111s said:

I know this is a little old, but is the sitemap generator a freely available one, or bespoke and/or paid for?

 

As bsmither has said, there is a (relatively basic) xml sitemap generation function built into CubeCart.  This is better than nothing but has to be run manually (which most store owners forget to do thereby presenting very old sitemap files to crawlers).  The software that I mentioned has considerably more features, creates better sitemap files including separate image, video, news sitemap files if relevant ; can be scheduled to run automatically via cron say once per week ; and also has a lot of other features such as reporting broken links anywhere within your site - something which Google and other crawlers really dont like.

If you would like more details, please contact me via PM or better still directly via our website

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...