No Crawl Results from Screaming Frog 
Author: Andy W.
Visited 1536,
Followers 1,
Shared 0
Hi,
I'm about to launch a new web site for https://expertemployment.co.uk/ (this is the old one) and thought I would run a Screaming Frog (SF) crawl to check for 404 errors. It came back with just the one indexable page.
I then tested it on the current site, same result.
There are plenty of internal links and menu links, but SF doesn't seem to be able to follow them. Is there a checkbox I've missed somewhere to enable these links to be crawled as obviously this would be a massive SEO problem to relaunch without a crawlable menu and internal links.
Thanks for your help.
Andy
Posted on the
Hello. It seems that hosting settings or third party code may be the cause.
A similar situation is shown not only by Screaming Frog but by WebSite Auditor. WebSite Auditor also scans only one page. When I try to add another page manually (from your menu), I get a strange error.
If you enter into Google this query site:expertemployment.co.uk
You will get almost 600 pages. But they don't seem to be made by the Website X5.
https://expertemployment.co.uk/Apply.asp?JobRef=1247
https://expertemployment.co.uk/Apply.asp?JobRef=1205
https://expertemployment.co.uk/Apply.asp?JobRef=1482
...
As far as I can see, you do not have a robots.txt file. It would be nice to create this tfile.
I sent a notification about your question to the company employees, expect an answer from them here in the comments.
use: https://expertemployment.co.uk/web/index.html
Author
Hi,
Thanks for the quick replies. Same result for the full URL.
Checking with the hosting company now.
Thanks
Author
Hi,
I've checked with the hosting company and they say it is crawling fine.
Is it someting to do with the X5 platform, do I have a checkbox not checked somewhere? I know I can do a sitemap but internal links are massive for SEO so if X5 doesn't support this it would be really odd and a fatal flaw for the CMS - so i'm guessing i've missed something.
Thanks
Andy
Author
https://helpcenter.websitex5.com/en/post/157983
This seems to indicate something similar with V13 which I'm using. Tested a V12 on the same server and it is working fine.
Author
I think i'm mistaken about this post, didn't realise it was so old. I'm using 2022.1.2
Good afternoon Andy
I believe I'm missing something about these tests. For example, why does the screenshot you provide indicate that the test is being done in the web folder instead of the main one?
Also, I can see some files that do not seem to belong to website X5, such as this one:
The fact that Aleksej analysis via Google shows pages that do not belong to the website leads me to believe that there might be some other issue here.
Can you please attempt to perform a full reset of the hosting space, delete all of the files inside, then reupload the entire website and try again?
Please proceed like this and let me know if the result is any different
Thank you
Stefano