How to prevent Google from crawling UserDir URLs (but not the real domain)?

Go To


We have clients who build their site on a UserDir URL before their real domain goes live. The UserDir URL is always in the format:

Sometimes, Google crawls these UserDir URLs and the temporary site will show up in results even after the site is live on

So, once a client is live on, how can I prevent Google from crawling the UserDir address?

(of course, I need Google to crawl the real domain because SEO is important to our clients)

2012-04-04 19:03
by Callmeed
Have you tried using a robots.txt file on - Adam Mihalcin 2012-04-04 19:06
Well, both the temp URL and real domain point to the same httpdocs .. - Callmeed 2012-04-04 19:23


I use the canonical tag for this purpose. If you put the canonical tag on the index.html file like such:

<link rel="canonical" href="" />

Then when Googlebot finds it at it will know that it is a duplicate of and Google will index the correct one. Googlebot will see the same tag when it crawls the real site and not have a problem with the self-referential canonical.

2012-04-06 15:30
by Stephen Ostermiller