Code entered in wrong place 
Author: Graham O.In another thread now closed by you I entered the robots.txt code in the wrong place as spotted by claudio. I have now removed the code and put a txt file in the root of my site, but there is still a 'banner' with User-Agent: GBTBot Disallow showing as in the screenshot in the next paragraph.
Further down there are screenshots of my cleared and the root showint robots.txt file.
how can I get rid of the 'banne
if the site is this
http://www.lutondunstablepc.org/
the file
robots.txt
is not present !!!
And you need to delete the code you put in stats and code as from this error
---
PS: also activate SSL certificate and redirect from http to https (you have to entire your Host)
Read morePosted on the 08/21/2023 10:45:36 from Claudio D.Graham O. User
Author
Yes - I've done that when uploaded the txt file I createsd into the root. It is showing in the attached screenshot.
Posted on the 08/21/2023 11:38:33 from Graham O. Attachment: L&DPC Root.jpgGraham O. User
Author
you need to delete the code you put in stats and code as from this error
I removed code when I uploaded the txt file soo screenshot. That error should not ne there.
Posted on the 08/21/2023 11:45:05 from Graham O. Attachment: L&DPC SEO Code.jpgGraham O. User
Author
I just checked with Google robots.txt Tester and got a 404 witha message that my site does not have a robots.txt file!
Posted on the 08/21/2023 13:44:41 from Graham O.
Since the "User agent" error appears on every page, this section must be deleted in WebSite X5 at:
1 Settings > Statistics and Code / Code
Then click on "Preview" and wait until the progress bar has run through completely.
Then export the entire website, i.e. upload it to the web space.
About the .htaccess file for redirecting from http to https there is this post and there I also have a small guide to FTP windows in WebSite X5, see
>> https://helpcenter.websitex5.com/en/post/256090
You can also use this FTP window to upload your own robots.txt file.
----- File robots.txt -----
User-agent: GPTBot
Disallow: /
The two files .htaccess and robots.txt are uploaded to the same directory in the FTP window as the index.html (homepage) file is located.
Author
Thanks for the replies guys.
I have the
User-agent: GPTBot
Disallow: /
in the robots.txt file in the root, but when I used the Google checking tool it said that it was not visible! I think I'll put this in a seperate thread as this will get complicated for this old dinosaur!
I've done What Daniel W. suggested and I also did what Franx-Josef H. suggested and moved the GA tracking code to after opening body tag. Now in the preview the banner is gone, but after export, it is still visible when looking at the site on the web asfter restarting mu computer!
I'll look at the .htaccess when I've sorted out the unwanted banner.
The code may still be in step 1 Settings, selection "Statistics and Code" in the "Code" area and there in the "Custom code" field in the area "before the </HEAD> tag".
But there is another problem with the settings in your project. In Step 5 Export, select "Export the website to the Internet" in the "URL of the website" field contains http://ftp.lutondunstablepc.org/ This one is wrong. The entry must be correct
http://www.lutondunstablepc.org/
or if you use an SSL certificate
https://www.lutondunstablepc.org/
This must also apply if you export with the selection "Export to a drive" in step 5 and upload with another ftp program.
If the field is changed, the project must be rebuilt to correct the internal links. To do this, click on the preview button while holding down the Ctrl key and wait until the progress bar has run through completely. Then upload everything, not just the changes.
When that is done we can deal with the .htaccess file.
Author
Thanks for this Franz-Josef, but I deleted the code that was in the wrong place and it is only in the 'after opening the body tag' area. I only see a custom code area in my Evo, no "code" area. The banner is not showing in preview, only on the actual website. I have restarted my computer too.
It is a very long time since I set this website up and as I know very little about the behind the scenes stuff and terminology now, (which is why I use Website X5) back then I was stumbling about in the dark so I might have got the url with the ftp in it from the hosting site. The website has been working just fine with this incorrect urtl though. I have now changed it.
Author
Sorry there is an error imy last post. The
User-agent: GPTBot
Disallow: /
Is nowhere in the step 2 Code area - I deleted it. What is in there only in the 'after opening the body tag' area is the GA tracking code.
The
User-agent: GPTBot
Disallow: /
is now only in the robots.txt file in the site root.
1) you closed the other post !!! Certainly not the STAFF
2) the file
robots.txt
as I already told you is NOT present in the root of your site !!!
3) there is still that error I had pointed out to you !!!
the code to block ChatGPT should NOT be put there in the code but ONLY in the file
robots.txt
4) are you sure you are exporting to the correct folder of your Host ?
Author
Claudio - I did not mark the previous thread closed, I marked your reply as correct. However, when I carried out what you said I found that it did not solve the issue. I returned to the thread only to find that it was closed. I guess your system did that when I marked your reply as correct.
At the bottom of the thread there is this text:
This post was signed as solved by the author or the Incomedia staff.
So I clicked on the bottom link and posted 3 more posts including 2 screenshots, one showing the SEO code area had the robots code deleted and the other that clearly shows the robots.txt file in the root. But I had no reply so I had to start this thread. In my first post here I re-posted the 2 screenshots and that I'd used a Google tool that told me that the robots file was not visible.
In your last reply:
2) the file
robots.txt
as I already told you is NOT present in the root of your site !!!
I attach the screenshot again.
In your last reply:
4) are you sure you are exporting to the correct folder of your Host ?
But previously you said that the robots.txt file should be in the root and the screenshot that I have now posted twice clearly shows it in the root. Now you are talking about a folder?
Since my last post here I've been trying to find the answer elsewhere, but with no luck so I've had to come back here.
by putting the correct to my reply there, you closed the post ;-)
From your pictures it looks okay...
but basically the file
robots.txt
doesn't show up online
and there is still that error...
meanwhile correct here
the URL is
http://www.lutondunstablepc.org
without "ftp"
and try Upload all website file to the internet
Can it be that you are uploading the project to the wrong folder? Your last upload is from 08/18/2023... In the target directory field you only have / entered, try the entry /public_html there
Exactly !
as I had already told him ...
Author
Claudio in every other post in these 2 threads you have said that the robots.txt has to go in the root. You have seen my screenshot and said it looks OK. Now you are talking about a folder. Which is it?
I deleted the robots.txt file and uploaded the site but the error in the banner did not go away so I uploaded it to the root again. I also tried hitting the all website files radio button and uploading again but no change. The error is still there and Again the Google robots.txt tester tool shows:
robots.txt not found (404)It seems like you don't have a robots.txt file. In such cases we assume that there are no restrictions and crawl all content on your site. Learn more.
This made me think that it is in the wrong place Claudio is saying in the root but agrees with Franz-Josef it should be in the /public_html folder.
I'm confused.
There seems to be some terminology confusion...
Root usually means the top-level directory. In the case of the provision of websites, however, the provider specifies a directory into which the files of the website or the WebsiteX5 project are to be uploaded. This is the target directory that must be entered in WebsiteX5 in step 5 when exporting the website to the internet. The robots.txt file must also be loaded into this directory. In your case, you probably interpreted 'root' as the top level and you only entered / in the target directory field (instead of /public_html). Please change that and then we'll see what happens.
Author
Will do Franz-Josef. Thanks :)
Author
Hey Franz-Josef - I just uploaded the robots.txt file into the public folder and the error has gone! Thank you.
The Google robots.txt tester tool is still showing "robots.txt not found (404)It seems like you don't have a robots.txt file. In such cases we assume that there are no restrictions and crawl all content on your site. Learn more."
So is this due to it being too early to test, the site needing to be crawled by Google again?
Can I leve the robots.txt file in the root or best to delete it?
Yes, the website looks ok now. If I were you, I would delete the robots.txt in the root... However, you have not yet followed the advice described on 08/24/2023:
"... But there is another problem with the settings in your project. In Step 5 Export, select "Export the website to the Internet" in the "URL of the website" field contains http://ftp.lutondunstablepc.org/ This one is wrong. The entry must be correct
http://www.lutondunstablepc.org/
or if you use an SSL certificate
https://www.lutondunstablepc.org/
This must also apply if you export with the selection "Export to a drive" in step 5 and upload with another ftp program.
If the field is changed, the project must be rebuilt to correct the internal links. To do this, click on the preview button while holding down the Ctrl key and wait until the progress bar has run through completely. Then upload everything, not just the changes. ..."
Oh, you just did it...
Author
Thank you! Will do.