Remove old URLs from Google, Bing, etc.

[alanorth]

Our current nginx configuration bends over backwards to serve https://dspace.ilri.org and https://mahider.ilri.org.

We only need to handle those links for as long as they keep coming, so should actually be pro-active and de-register links containing those domains from search indexes (ie in Google Webmaster tools).

[alanorth]

Verified as an owner of dspace.ilri.org and mahider.ilri.org and submitted a change of address request for cgspace.cgiar.org. Currently the old URLs have nearly 10,000 links in Google's index!

This is great because the handles are the same, but the domain name will now be directed to CGSpace.

[alanorth]

Still waiting for these URLs to disappear. inurl:dspace.ilri.org shows 5,300 results right now.

Maybe I should also add the X-Robots-Tag: none HTTP Header...

[alanorth]

Added to nginx vhost template: ilri/rmg-ansible-public@15a2bd1

[alanorth]

Currently there are 1,600 items in Google results for dspace.ilri.org.

I need to re-visit the Google webmaster tools to see if there's anything else I can do to remove these.

[alanorth]

The redirect to cgspace.cgiar.org has been completed since May, so I'm not sure why these links are still in the index. I have added the CGSpace sitemap to dspace.ilri.org's console to see if Google will learn that these have moved.

Currently requests to dspace.ilri.org are handled like this:

$ http --print Hh https://dspace.ilri.org/handle/10568/33832
GET /handle/10568/33832 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: dspace.ilri.org
User-Agent: HTTPie/0.9.2

HTTP/1.1 301 Moved Permanently
Connection: keep-alive
Content-Length: 178
Content-Type: text/html
Date: Fri, 02 Oct 2015 08:34:03 GMT
Location: https://cgspace.cgiar.org/handle/10568/33832
Server: nginx
X-Robots-Tag: none