As you might know that if your staging or development site has been crawled by search engines then they will index them and show them in Search Engine Result Pages (SERPs) which is not good for your live website because it will affect your live site page ranking. This can lead to dropping your organic traffic which ultimately affects your sales. How can you remove staging site from Google?
There are following ways you can prevent search engines to crawl or index your test or staging Magento site
1) Add the following entry in robots.txt and upload this file on root of your test or staging sites
User-agent: * Disallow: /
This robots.txt is respected by all search engines and by adding the above entry you are telling them that don’t crawl any URL of this website.
2) Add password to your development or staging sites
Step 1 – Create .htpasswd file with the following entries
username:encrypted password
Encrypted Password for .htpasswd can be generated by using this link http://www.htaccesstools.com/htpasswd-generator/. Once generated update your .htpasswd with the generated line and upload .htpasswd to root folder of your staging or test site.
Step 2 – Update .htaccess file with the following entries
AuthUserFile /www/public_html/mystaging/.htpasswd AuthType Basic AuthName "Enter username and password" Require valid-user
Just make sure you change the path of your .htpasswd file and leave rest of the lines as it is.
3) Set NO INDEX, NO FOLLOW for all of Magento URLs
Step 1 – Login to your Magento Admin
Step 2 – Go to System -> Configuration -> General -> Design -> HTML Head -> Default Robots -> NOINDEX, NOFOLLOW
Step 3 – Go to System -> Cache Management -> Refresh Cache
Please make sure you don’t apply any of the above prevention methods on your live site
The above will help you to prevent your staging site to appear in Google but what if you forgot to put any of the above prevention at the first place. And Google and other search engines have crawled and indexed your staging or development sites.
The most easiest way to remove your staging site from search engines is to add any of the above preventions and request Google to remove your staging site.
To remove your already indexed site from Google search engine pages you need to follow the following steps -:
Step 1 Create your staging site in Google webmaster tool
Step 2 Get your staging site verified
Step 3 Once the verification is done then click on your staging site in Google webmaster tool
Step 4 Select Google Index -> Remove URLs from left hand navigation
Step 5 Click “Create a new removal request” button
Step 6 Enter “/” in the URL text box provided
The above will create “Site removal” entry for Google to remove your site from search result pages. It generally takes 24 hours for Google to remove all your staging site links from search results.
Hope this article helped you in some way. Please leave us your comment and let us know what do you think? Thanks.
