Scroogle is gone.
I originally asked "Why would Google do this?", but that seems pretty clear in the wake of the surveillance they've been involved in.
Scroogle, Privacy-First Search Engine, Shuts Down for Good
November 3, 2010: Here we go again...
We regret to announce that Google changed their output format once again. The last time this happened was in July, and we were down for five days. During that time we looked for the simplest remaining Google format we could find, reprogrammed our parser, and ended up with something that worked. However, the file we fetched from Google was three times more bloated for the same information, as compared to the previous format we used, and we are still not happy about this.
Now it looks like even more bloat. We have to take a closer look at the new format and see if we can program around it. Check back in a day or two.
UPDATE: One day later, Scroogle is back up online. Scroogle hasn't released any information about what's going on. Perhaps Google simply provided their service at a new address or via some new method?
my original post ∞
Metasearch engines are just one of the effected users. There are also alternate presentation engines - perhaps just for usability, or in the case of Scroogle, for privacy. It's literally been just a few days since I switched over to using Scroogle from Google. On that note, I was getting pissed off at Firefox's hidden use of Google with
control-k in my configuration.
So when searching, I came upon this notice:
We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we've been scraping since Scroogle started five years ago. Actually, we've been using that interface for scraping since Google-Watch.org began in 2002.
This interface was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.
That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn't show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google's basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched "Universal Search," which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.
Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google's main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.
Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.
Thank you for your support during these past five years. Check back in a week or so; if we don't hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.
— Daniel Brandt, Public Information Research, scroogle AT lavabit.com
(as an aside, Lavabit is gone)
Why would Google do this? I'm very surprised it wasn't done a long long time ago. Many meta search engines act like a parasite by leveraging their host search engines. They do some work, sure, but the host search engines do all the internet crawling and other heavy lifting.
So aside from Scroogle, which you may never have even heard of, who else does this effect? Off the top of my head it would include:
.. and perhaps both:
Wikipedia has a small list of meta search engines which have some more affected users.
I've been putting it off for a while now, but I think I'm going to change the way I use the internet and stop relying on search engines so much, especially to stop using Google.
Since Radar magazine doesn't have it anymore, and Scroogle isn't showing it, here is the short story which inspired Scroogle: