It has come time to make the WordPress sites I made for my kids disappear off the public internet. They are getting old enough that their friends are finding their sites using google searches.
Here is what I did.
- Password protect the entire WordPress site
- allow a hole in the password protection to allow robots.txt to still be accessible.
If you password protect the whole site, then archived copies of the site will contine to show at the Internet Wayback Machine. So you still need the robots.txt to be accessible, but it should block all web archivers from indexing the site.
Sample robots.txt
User-agent: * Disallow: /
This tells all web robots to go away, and indeed once this is active, the internet archive stops showing any of the snapshots that it has collected over the years.
I used SiteGround’s cpanel tool to password protect the whole tree containing the blog.
That resulted in the following Sample .htaccess contents
AuthType Basic AuthName "EP" AuthUserFile "/home/xxx/.htpasswds/public_html/elliot.pascoe.biz/passwd" Require valid-user <Files "robots.txt"> Allow from all Satisfy any </Files>
I added the Files paragraph to allow the robots.txt to be visible. cf;
user@shell: curl https://elliot.pascoe.biz/robots.txt User-agent: * Disallow: /
So far so good, the site will soon disappear out of Google and be only visible to those to whom I will share the password.
