Technical SEO Tip: HTTPS and the Robots.txt File

This is a technical SEO tip -- it's a problem that appears fairly often on ecommerce sites.

I just came across this hostel-related website with this problem.

Here's the basic anatomy of a URL:

http://www.example.com/

Here's what the different parts mean:

• http = the protocol (method) that your browser connects to the remote server with
• www = a subdomain
• example.com = the domain name
• / = the root directory of the server at www.example.com. If you leave it off, the server will redirect you.

Sometimes URLs have https instead of http -- that means it's connecting to the server with a secure encrypted connection.

There are two problems on the site I just found:

Invalid Security Certificate:
If there's something wrong with the security certificate, Firefox 3 will warn you like this:

Non-tech users might not know how to load the page, because it involves several steps. A user landing on that page probably won't make the booking.

Duplicate Content Over HTTPS:
In the case of this hostel website, Google has also indexed both the http and https versions of the website. Two URLs with the same content means that search engines are encountering duplicate content.

To prevent search engines from indexing duplicate content over https a website needs a separate robots.txt file.

The regular http section of the site should have the normal robots.txt file here:
http://www.example.com/robots.txt

Another robots.txt file should be served when the following file is requested:
https://www.example.com/robots.txt

The https version should include the following lines:

That will prevent search engines from crawling https URLs. The two lines of code should only be shown when someone requests the robots.txt file over https or it will block your entire site from search engines.

If my explanation doesn't make sense, please leave a comment below.

Google also explains it here.