WebSite X5Help Center

 
Tommy Du Preez
Tommy Du Preez
User

I have a robot.txt file i wana ask you to look at. I'm sure mine is too long. Please help me get the best one for my sit  en

Author: Tommy Du Preez
Visited 2306, Followers 1, Shared 0  

Please help me to tweek this robot.txt

I'm sure it's an overkill but wana get the best search results possible out of the internet:)

User-Agent: *

User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:

Allow: Googlebot
Disallow:
Allow: Googlebot-image
Disallow:

User-agent: APIs-Google

Disallow:

User-agent: mediapartners-Google

Disallow:

User-agent: AdsBot-Google-Mobile

Disallow:

User-agent: AdsBot-Google-Mobile

Disallow:

User-agent: AdsBot-Google

Disallow:

User-agent: (Various mobile device types) (compatible; Mediapartners-Google/2.1; http://www.google.com/bot.html)

Disallow:

User-agent: AdsBot-Google-Mobile-Apps

Disallow:

Allow: /*?hl=

Allow: /?cd=

Allow: /?hl=

Allow: /adwords/

Allow: /apt/AdPreview

Allow: /apt/anon/AdPreview

Allow: /d/AdPreview/

Allow: /home*

Allow: /keywordplanner

Allow: /ko/KeywordPlanner/Home

Allow: /support/

Allow: /um/home$

Allow: /um/identity

Allow: /um/KeywordPlanner

Allow: /um/signin

Allow: /um/Signup/Home$

Allow: /um/StartNewLogin

Allow: /um/StartNow

Allow: /um/Welcome

Allow: /um/Welcome/

Disallow: /*?

Disallow: /api

Disallow: /um/*

Disallow: /um/StartAccountSetup

Sitemap: https://adwords.google.com/home/sitemap.xml

Posted on the
1 ANSWERS