Jump to content

unspecific Errorpage while recreating the sitemap


CHPG

Recommended Posts

Hi All,

i have currently the Problem that when i try to rebuild the sitemap it takes a long time and than ends with the attached screen.

I have alread activated the error_log using the file ini-custom.inc.php as suggested by bsmither in a different Topic.

The Shop is running CC 6.1.1 and has currently about 39000 Products loaded.

Since it is planed to increase the number of products to about 800.000 I´m a little bit worried about this issue.

Hopefully somebody has an idea how this issue can be fixed.

kind regards

Christian

 

Error-Screen.jpg

Link to comment
Share on other sites

Hi @CHPG Welcome to the forums! It sounds like what you need is a way to create sections of the sitemap, much like the Export size can be set in Admin - creating a series of exported files. Or maybe it's a matter of changing the server memory limits???? Sorry I can't help, but hopefully someone will be along shortly who can be of help.

What kind of hosting setup do you have? How much control over settings do you have?

Link to comment
Share on other sites

I would also agree that it sounds like it is either hitting a memory or process time limit although your hosting company should be able to give you more details.  With an inventory of that size, I would suggest a specialist xml Sitemap generation script which we generally use for most of our larger customers

Ian

Link to comment
Share on other sites

Hi Ditry Butter,

thanks for the quick reply.  Since the site is hosted at 1und1.de i don´t have much contol over the settings.

Is there maybe a way to exclude the products from the sitemap ? For my oppinion it would be OK to have only the catagories and the documents in the sitemap.

kind regards

Link to comment
Share on other sites

7 minutes ago, CHPG said:

Is there maybe a way to exclude the products from the sitemap ? For my opinion it would be OK to have only the catagories and the documents in the sitemap.

That sounds doable. But it would certainly not work for our stores - in fact we don't use the categories in our urls at all. Would that trimmed down sitemap not be a problem SEO wise?

Link to comment
Share on other sites

Looking at the structure of the sitemap that CubeCart creates, it seems to me to be incredibly easy to create one manually.

1. Have phpMyAdmin give you a dump of the contents of the CubeCart_seo_urls table. A CSV file will be fine.
2. Load that file into a spreadsheet app. Delete all columns except 'path'. Save as a text file.
3. Load that text file into a programmer's text editor. Prepend and append common stuff to each line. Save with an .xml file extension.

You will need to have some proficiency with the programmer's text editor in order to add the common stuff across hundreds of thousands of lines, and a programmer's text editor is built to do exactly those kinds of things.

Link to comment
Share on other sites

47 minutes ago, bsmither said:

Looking at the structure of the sitemap that CubeCart creates, it seems to me to be incredibly easy to create one manually.

That is actually the whole point !  The Sitemap file that CubeCart creates is pretty basic and while it is much better to have something rather than nothing, a comprehensive Sitemap file (and that is before you start considering separate image, video, news Sitemaps !) contains so much more information than this and in addition, a good Sitemap creation system will create this automatically on a weekly, daily or hourly basis as needed.  This is not a criticism of the CubeCart created Sitemap - it is what it is

Ian

Link to comment
Share on other sites

Has there been anything logged in the error_log file? We would expect to see a Fatal Error if PHP ran out of memory or exceeded it's own time limits.

However, depending on how your hosting account is setup, there will also be a time limit enforced by the environment that PHP runs in (CGI, FastCGI, PHP-FPM), and the web server can have a timeout. The error page posted above seems to imply that the web server sent this message, probably because FastCGI did not make any output for some time.

So, even if we tell PHP to have more time to run (max_execution_time in PHP.INI), there is maybe little one can do to have the FastCGI or web server allow for more time.

The code that creates the sitemap is not optimized at all. For every product, category, and document, a database call is made. Now, PHP does not accrue time expended when the system is communicating with the database (except for Windows environments), but FastCGI and the web server is beyond PHP's ability to control.

Therefore, as I said, if there is nothing logged in the error_log, then we need to start looking at optimizing the sitemap code, or bumping up the hosting account's service level so that control of the other timeouts can be managed.

Link to comment
Share on other sites

In Wordpress I use a sitemap plugin that makes several sitemaps - one for posts, another for pages, another for directories that are not part of the actual WP install. It creates a sitemap with links to each sitemap section in it. Would it be possible to split the sitemap into sections, like the Export File page does for number of products in the export?

Link to comment
Share on other sites

Hi All,

thanks for your comments/hints.

@bsmither - unfortunately the error_log does not contain an error message. So i think the webserver killed the process before it was ready with the sitemap.

So based on the informations received here it seems that i will have to create my own sitemap generator that creates several "small" sitemap files which are connected via a main sitemap file

Link to comment
Share on other sites

26 minutes ago, CHPG said:

So based on the informations received here it seems that i will have to create my own sitemap generator that creates several "small" sitemap files which are connected via a main sitemap file

That depends on how skilled you are writing php and complex sql queries and how much spare time you have to then keep it updated in line with constantly changing search engine requirements.  There are several very good Sitemap generation software packages that are cheap (some free but with no support) that will do everything that has been raised here and a lot, lot more and all automatically so you never need to worry about Sitemaps again.  We supply one such third party software to many customers and they are all very happy with it for minimal outlay

Ian

Link to comment
Share on other sites

  • 2 weeks later...
On 18.11.2016 at 9:26 PM, havenswift-hosting said:

There are several very good Sitemap generation software packages that are cheap (some free but with no support) that will do everything that has been raised here and a lot, lot more and all automatically so you never need to worry about Sitemaps again.  We supply one such third party software to many customers and they are all very happy with it for minimal outlay

can you give me more informations about these packages, at least about the one you are selling ?

kind regards

Christian

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...