Crawl errors
Autor: Terry R.
Visitado 847,
Seguidores 1,
Compartilhado 0
Hi.
I have built a website with x5 11. www.discountelements.com When I go into google webmaster and click on crawl errors. It brings this up. 1. downloads/ga-001-web.html, 2. products/indium/index.html, 3.products/index.html, 4.products/gallium/index.html, 5.products/lithium/index.html. Code 404 for all of them. but these urls are not in my website. What is it?
Regards.
Publicado em
Hello Terry,
In your sitemap file created by the program these pages are not present. Can it be these pages existed before and where submitted with another sitemap file?
If this is the case remove the other sitemap file you added in google webmaster and leave only the one created by WebSite X5.
Many thanks!
Autor
Hi.
There is no other sitemap in my website files. The sitemap was created with x5 11 and uploaded by x5 11. I don't understand it.
Regards.
Hello Terry,
Can you post a screenshot of what you see in Google Webmaster Tools? It would be helpful to have more information.
What do you have in your robots.txt file?
Kind regards,
Paul
Mystery solved... your domain used to belong to someone else... see the following link for the proof:
https://web.archive.org/web/20110202231428/http://discountelements.com/
You need to tidy up the robots.txt file to prevent the Googlebot trying to crawl these old pages which no longer exist.
This issue is not connected with WebSite X5 in any way.
Kind regards,
Paul
Autor
You know, that's exactly what I thought it might be, that someone else owned it.
I have attached the robots text file. I don't know how to resolve this issue.
Thanks.
You need to append the following lines to the end of the 'robots.txt' file, Terry:
Disallow: /downloads/ga-001-web.html
Disallow: /products/indium/index.html
Disallow: /products/index.html
Disallow: /products/gallium/index.html
Disallow: /products/lithium/index.html
or a shorter, simpler version would be this:
Disallow: /downloads
Disallow: /products
The disadvantage of the shorter version is that it effectively prohibits you from using sub-folders called 'downloads' and 'products' in your X5 project, which might be a problem for an online shop!
Reupload the modified 'robots.txt' file to your server, and the next time the Googlebot comes around he won't try to follow those dead links
Kind regards,
Paul