WebSite X5Help Center

 
Andy W.
Andy W.
User

No Crawl Results from Screaming Frog  en

Author: Andy W.
Visited 1260, Followers 1, Shared 0  

Hi,

I'm about to launch a new web site for https://expertemployment.co.uk/ (this is the old one) and thought I would run a Screaming Frog (SF) crawl to check for 404 errors.  It came back with just the one indexable page.

I then tested it on the current site, same result.

There are plenty of internal links and menu links, but SF doesn't seem to be able to follow them.  Is there a checkbox I've missed somewhere to enable these links to be crawled as obviously this would be a massive SEO problem to relaunch without a crawlable menu and internal links.

Thanks for your help.

Andy

Posted on the
7 ANSWERS - 1 USEFUL
Aleksej H.
Aleksej H.
Moderator

Hello. It seems that hosting settings or third party code may be the cause.

A similar situation is shown not only by Screaming Frog but by WebSite Auditor. WebSite Auditor also scans only one page. When I try to add another page manually (from your menu), I get a strange error.

If you enter into Google this query site:expertemployment.co.uk

You will get almost 600 pages. But they don't seem to be made by the Website X5.

https://expertemployment.co.uk/Apply.asp?JobRef=1247

https://expertemployment.co.uk/Apply.asp?JobRef=1205

https://expertemployment.co.uk/Apply.asp?JobRef=1482

...

As far as I can see, you do not have a robots.txt file. It would be nice to create this tfile.

I sent a notification about your question to the company employees, expect an answer from them here in the comments.

Read more
Posted on the from Aleksej H.
Andy W.
Andy W.
User
Author

Hi,

Thanks for the quick replies.  Same result for the full URL.

Checking with the hosting company now.

Thanks

Read more
Posted on the from Andy W.
Andy W.
Andy W.
User
Author

Hi,

I've checked with the hosting company and they say it is crawling fine.

Is it someting to do with the X5 platform, do I have a checkbox not checked somewhere? I know I can do a sitemap but internal links are massive for SEO so if X5 doesn't support this it would be really odd and a fatal flaw for the CMS - so i'm guessing i've missed something.

Thanks

Andy

Read more
Posted on the from Andy W.
Andy W.
Andy W.
User
Author

I think i'm mistaken about this post, didn't realise it was so old. I'm using 2022.1.2

Read more
Posted on the from Andy W.
Incomedia
Stefano G.
Incomedia

Good afternoon Andy

I believe I'm missing something about these tests. For example, why does the screenshot you provide indicate that the test is being done in the web folder instead of the main one?

Also, I can see some files that do not seem to belong to website X5, such as this one:

The fact that Aleksej analysis via Google shows pages that do not belong to the website leads me to believe that there might be some other issue here.

Can you please attempt to perform a full reset of the hosting space, delete all of the files inside, then reupload the entire website and try again?

Please proceed like this and let me know if the result is any different

Thank you

Stefano

Read more
Posted on the from Stefano G.