Believe it or not, there was once a world\h of search engines that did not contain AI\h\h
summaries filled with nonsensical garbage.\h There was once a world of search engines that\h\h
did not always serve up a Wikipedia article or\h Reddit thread. There was once a world of search\h\h
engines where exciting things could happen\h and dreams could be forged, all at the press\h\h
of a button. Well close enough anyway. In fact the real excitement was the wealth\h\h
of choice we had available, and the battle\h between these companies, forging their path\h\h
on this brand new landscape. On the 15th March 1985, the first\h\h
internet .com domain was registered;\h symbolics.com. Symbolics was a computer\h\h
manufacturer located in Cambridge, Massachusetts,\h that from 1980 had created a range of computers\h\h
utilising the Lisp programming language; Lisp\h Machines. Technically the first commercial\h\h
computer workstations. So it’s no surprise\h they wanted the first internet domain too. \h
By the end of 1985, there was only a handful of\h domains registered. Indeed, even by 1989, pickings\h\h
were sparse. But we’re talking a time before even\h the World Wide Web. A working version of which\h\h
wasn’t ready until the end of 1990. So before this\h point, searching the internet, in the way we know\h\h
it now, wasn’t really necessary. You tended to\h know which domains you wanted to connect to. \h
Even so, there were tools, such as\h Wide Area Information Server Systems,\h\h
using Archie to index FTP archives, and the using\h the TCP/IP Application Layer protocol Gopher,\h\h
which predated HTTP and had its own search\h engines such as Jughead and Veronica. \h
But the web as we know it was started\h here. This NeXT machine, used by Sir Tim\h\h
Berners-Lee was actually the world’s first\h web server, hosting the first web page at\h\h
nxoc01.cern.ch/hypertext/WWW/TheProject.html. This webpage would soon include a list of other\h\h
pages that you could navigate to through\h HTML. The link “What’s out there” breaking\h\h
down various pages by Subject and Type. We\h could look at the US weather, some politics,\h\h
and of course, an online link to the Bible. Our\h lord and saviour is always ahead of the curve. \h
But this was a limiting way to explore the web. It\h depended on these links being updated by whoever\h\h
was looking after the page. So if you created\h a new website, on a new domain, you might have\h\h
to kindly request that your website be added. This form of page would quickly become known as a\h\h
link page, or for the larger versions, organised\h by category; a directory. A hand picked list of\h\h
sites that you could visit and explore from. That’s fine when there’s a few thousand websites\h\h
in the world, but with the Mosaic\h web browser launching in late 1993,\h\h
things were taking off at a rapid pace. But it wasn’t commercial enterprise that\h\h
attempted to create a tool to shine a light\h into the unknown, it was University of Stirling\h\h
graduate Jonathon Fletcher, who decided to put the\h Scottish university’s internet server to use. \h
Search engines of sorts already existed,\h such as W3Catalog and Aliweb. However,\h\h
neither of these are like the search engines\h we know today. W3Catalog just referenced\h\h
existing link pages on the web and updated\h from there, whilst Aliweb required website\h\h
owners to notify of their existence. However\h Jonathon Fletcher’s JumpStation combined the\h\h
three essential elements of a modern search\h engine; crawling, indexing and searching. \h
You see, a search engine isn’t some magic\h portal to the web. It’s just a database,\h\h
held on a server, that periodically gets updated\h as and when new content is found. It’s the job\h\h
of the crawl-bot to find this new content,\h and it does that by visiting its known sites\h\h
and following the links within. If a site has\h linked to a new page or site then it’s added\h\h
to the search engine’s database. Each page added\h to the database is indexed, in these early days,\h\h
usually just by the words found within it. Then\h a query form is made available to a visitor,\h\h
so that they can search this indexed database.\h From which results are then presented to the\h\h
searcher in the form of a URL list. JumpStation, launched on 12th December\h\h
1993 did all of this. Although, in a bid\h to save resources, only document titles\h\h
and webpage headings were indexed at this point,\h which kinda did the job back then. By the time\h\h
the site closed down in late 1994, the database\h held 275,000 entries, spanning 1,500 servers. \h
Other search tools popped up in that time,\h but WebCrawler, released in April 1994 was the\h\h
first search engine that actually indexed entire\h webpages, allowing users to search for any words\h\h
within those pages. This proved so useful that it\h became the standard for search going forward. \h
By this time Infoseek, founded by Steve\h Kirsch, had also rolled out, although\h\h
initially as a pay to use service. Something it\h would later drop, becoming the default search\h\h
engine for Netscape Navigator in 1995. Lycos, what began as a research project\h\h
headed by Michael Loren Mauldin of Carnegie\h Mellon University appeared in July 1994,\h\h
and quickly gained popularity. The name\h Lycos was actually derived from Lycosidae,\h\h
the scientific name for the “wolf spider”1; an\h excellent hunter that catch prey by finding it,\h\h
instead of luring it into a web. Lycos has\h full crawling, and indexing capability, but was\h\h
heralded for it’s simplicity. Even the results\h page was just a plain list of links relevant\h\h
for the keyword. Within a month of launch Lycos\h recorded more than 400,000 pages and documents. \h
By the start of 95, Yahoo! had built upon\h their earlier directory feature by launching\h\h
Yahoo Search, which allowed users to search\h the Yahoo! Directory, and so actually not a\h\h
true search engine, BUT due to the quality of\h its directory, it became popular quickly. \h
But these engines would face serious\h competition from Altavista. Which\h\h
by it’s own reckoning became the most\h comprehensive search engine on the web. \h
Even before it’s launch, there was intense hype\h around AltaVista. Within weeks, hundreds of\h\h
articles were written on how it’s “Super-Spider”\h could help you find long lost friends or list\h\h
all sites pointing to your page. It seemed that a\h small group of friends and colleagues had indexed\h\h
the entire web and were ready to serve it to\h millions of users within fractions of a second. \h
The “Super-Spider” in question was actually called\h “Scooter”. Scooter was AltaVista’s multi-threaded\h\h
web crawler, and rather than collecting just\h titles, or even key words and sentences, it would\h\h
instead collect complete pages, which were then\h indexed in their entirety, ready for searching.\h\h
What’s more, it did this continuously, and\h quickly. Meaning that if a new website appeared\h\h
on the web, it could be retrieved from AltaVista\h in hours rather than days or even weeks. \h
Most search engines at the time were running\h out of university research labs, utilising their\h\h
powerful machines, but AltaVista was a venture\h launched by Digital Equipment Corporation, and\h\h
they had the technology to boot. This included the\h Alpha 8400 TurboLaser server. A beast of a system\h\h
which contained 12 EV5 processors, each running\h at 300MHz, alongside a whopping 28GB of RAM.\h\h
When you bear in mind the average desktop PC then\h was a 486 running at 50MHz with 8MB of RAM, this\h\h
is obscene. But then, so was the price, with each\h server costing around half a million dollars. \h
This then was when search really started\h to get serious. Altavista had started a\h\h
race which meant Lycos and Yahoo had to\h step up their game. But it also meant\h\h
that competition came thicker and faster. By the end of 1996, AltaVista was handling\h\h
19 million requests per day, and had a\h database covering over 30 million pages.\h\h
But Lycos was still dominating, in terms of\h volume at least, with its crawlers having\h\h
apparently indexed 60 million pages. This was starting to become a war of stats,\h\h
but really, this isn’t what people\h wanted. People wanted clarity and\h\h
they wanted results that were relevant. That’s where RankDex came onto it’s own2.\h\h
Defining an important part of search, and\h that is, how results are ranked. Originally,\h\h
results were just delivered how they fell out\h of the database. Altavista started using other\h\h
factors, such as the how frequently the keyword\h being searched for appeared on the page, serving\h\h
the pages with the highest frequency first. But\h this quickly led to problems, such as websites\h\h
spamming their pages with popular keywords in\h order to artificially inflate their rankings. \h
Rankdex then would begin using backlinks to\h measure the quality of websites. That is,\h\h
if a website or page has more hyperlinks directed\h to it, then it’s more likely to be a reputable\h\h
source of information, and therefore the content\h it contains is likely to be more relevant to the\h\h
searcher. In these early days especially, this was\h an incredibly effective way of ranking content. \h
At around the same time, Larry Page and\h Sergey Brin were working on a little search\h\h
service called BackRub; the predecessor to\h Google. With it’s crawl bots starting to\h\h
scour the web in March 96, running from\h Stanford’s Sun Ultra II web servers. \h
But by May, the online division of Wired\h magazine, HotWired, in collaboration with\h\h
American Internet Service Provider Inktomi\h would launch the HotBot Search Engine. Now\h\h
HotBot claimed to be even better than Altavista.\h Having indexed 54 million pages and documents,\h\h
and being able to refresh this entire index\h once a week, this was probably the most up to\h\h
date database of the web the 90s would see. Things were indeed hotting up. Even before1997,\h\h
the search engine world was saturated, and this\h isn’t even including the Metasearch engines. \h
Otherwise known as Search Aggregator, these sites\h used the databases and tools of search engines\h\h
to create their own results. This included\h sites like Dogpile, excite! and MetaCrawler,\h\h
that could arguably serve up better results\h without building their own foundations. \h
Dogpile pulled queries from Yahoo, Lycos,\h Excite, Webcrawler, Infoseek, AltaVista,\h\h
HotBot, WhatUSeek and the World Wide Web Worm,\h as well as having the ability so earch Usenet,\h\h
making it an incredibly comprehensive tool.\h Whilst Excite offered services such as email\h\h
and customisation from the go, giving them\h distinct advantages over competitors. \h
Like most of these sites, Metacrawler was\h again, created by a university graduate,\h\h
Erik Selberg. By the end of 1996 Metacrawler was\h receiving 150,000 queries per day, which was far\h\h
too much for the University of Washington’s\h DEC AlphaStation servers to keep up with,\h\h
and so it become a commercial entity under NetBot\h Inc, which was shortly after purchased by Excite!\h\h
Excite! utilised features of Metacrawler to\h create the Network Shopping Channel, demonstrating\h\h
increasingly commercial ways these engines were\h taking to make money, as some morphed more into\h\h
web portals, rather than just search engines. But even with this crowded area, the internet was\h\h
growing at an exponential rate, and more casual\h users demanded a more casual experience. So,\h\h
in April 97 Ask Jeeves saw a gap in the market.\h Unlike other engines which had their own query\h\h
formatting, Jeeves was a natural language\h engine. The idea here then was that you\h\h
could type your question in a very human form,\h and get search results that were appropriate.\h\h
Ask Jeeves quickly began to build up an entire\h brand around their search, including books and\h\h
butler themed promotional items, capturing\h a niche but stable portion of the market. \h
Of course this was a time for innovation,\h and there was plenty still to be done. \h
On the 15th September 97 Google.com was\h registered. Larry and Sergey having already\h\h
tested the premise from Stanford’s website\h and decided that BackRub was probably not\h\h
the best name for their engine. But Google wouldn’t actually launch\h\h
until September 1998, around the same\h time that Microsoft decided to get in\h\h
on the act with the MSN Search Portal, which\h actually used search results from Inktomi,\h\h
the same database that HotBot ran on. This database was also used by Direct Hit\h\h
Technologies for their Popularity Search engine\h which actually incorporated previous search\h\h
activity to serve up more relevant results.\h What we were seeing here was really the\h\h
emergence of features that would at some point,\h be common across all the main search engines. \h
But by this point, things were crazy. We not only had a plethora of search engines,\h\h
and search engines specific for other\h languages, such as Russia’s Yandex,\h\h
we also had much more comprehensive directories.\h These directories, such as DMOZ, tried to\h\h
concentrate of quality content. A curated batch\h of sites and pages, rather than opening you up\h\h
to the slurry of content available on the web, and\h sometimes, these were actually the best choice. \h
But even putting the directories aside, if you\h combine all the Metasearch engines and search\h\h
engines there were hundreds to choose from. Not\h to mention the spinoffs such as Yahooligans for\h\h
kids and web portals that ISPs used, which were\h usually either Metasearch engines themselves\h\h
or using the technology of a product such\h as Lycos, Inktomi or Altavista. Usually,\h\h
people would flock to whichever ones were\h suggested to them, a whole set business of\h\h
websites even appeared specifically to investigate\h and recommend which search engine to use…. \h
This was a big money industry, one of the\h biggest… these pages were indeed the portal\h\h
to the internet, and so the main contenders\h fired shots with big marketing campaigns.\h\h
Offering free services, better features and\h whatever else they could think of. However,\h\h
it was probably Yahoo who struck the chord\h most, with their humorous TV adverts. \h
As we went into the millennium, the\h main contenders were shining through. \h
AltaVisa, Lycos, HotBot and Yahoo, with Excite\h dominating the Metasearch world. MSN were also\h\h
packing punches, mainly because their homepage was\h the default landing page for Internet Explorer,\h\h
those cheeky rascals…. but regardless,\h that little company called Google were\h\h
starting to get people’s attention,\h and they were getting it fast. \h
In late 1999, Sergey Brin and Larry Page had\h decided that Google was taking up too much of\h\h
their study time and offered the business\h to Excite for $750,000. The CEO of Excite,\h\h
George Bell decided it was too expensive\h and unnecessary, with Google having less\h\h
than 10% of the market share at the time. Google noticed that not everyone could win\h\h
this race, this was a time of consolidation.\h Much like the early micro wars of the 80s,\h\h
a few would come out dominant. Deals such as\h Lycos acquiring Wired Digital, and therefore\h\h
the HotBot engine would become the norm. But with the deal with excite dead,\h\h
Google realised that people were starting to hone\h on on their advantages. The first was it’s simple\h\h
design, much like Lycos originally started, but\h by now had progressed into mug’s eyeful. The\h\h
second was the unintrusive advertising, Brin\h and Page were against advertising altogether,\h\h
but allowed some simple text ads (based on\h searched keywords) to pay for their venture.\h\h
The third was Google’s pagerank algorithm that\h used various factors to rank search results. This\h\h
method actually borrowed a lot from Robin Li’s\h RankDex, which is actually cited in their 1998\h\h
patent, but added in various other factors, that\h would grow and adapt over time. This quickly put\h\h
Google ahead of competitors in terms of relevant\h search results and this did not go unnoticed by\h\h
Search comparison sites and the general media. The fourth reason however, was the name. Google.\h\h
It was Larry Page himself who first wrote\h “Have fun and keep googling” on an mailing\h\h
list email from July 1998, two months before\h the company had even launched. From there\h\h
the verb quickly took off, giving the world a\h concise word for “searching the internet”. \h
From there, things really did progress quickly. By 2001, Google’s main competition was really msn\h\h
and Yahoo! Although Microsoft’s business\h practices and their antitrust case for\h\h
monopolising internet browsers had left\h a bitter taste in the mouths of some. \h
In 2002 Yahoo! would purchase Inktomi, and\h Overture Services inc. which by that point\h\h
already owned Altavista. Essentially eating\h it’s competition apart from Google. In fact,\h\h
up until that point Yahoo! was using Google’s\h search functionality to power it’s own search,\h\h
but in 2003 started using their new Yahoo\h Slurp web crawler to power their own search.\h\h
By leaning on all their acquisitions, they\h hoped to use the technology to beat Google\h\h
at their own game and retake dominance. This switch actually dented Google’s market\h\h
share for several months, but the damage was not\h long term. Although Yahoo had plenty of resources\h\h
and money to invest in their search, they also\h had other arms to their business. For Google,\h\h
it was really their only driving force, and at\h that point, they already had over 30% share of the\h\h
market. This enabled them to invest more in their\h core functionality, creating a runaway success,\h\h
that left most of their competition for dead. In March 2004, the dependable Ask.com, formerly\h\h
Ask Jeeves purchased Excite for 9.3million common\h shares and $144million in cash. This was part of\h\h
a strategy to rejuvenate their search services\h across the board. However, it was an expensive\h\h
and futile attempt, and they couldn’t keep up with\h the search arms race between Google and Yahoo. \h
In June 2005 Google was valued at almost $52\h billion, making it one of the world’s biggest\h\h
media companies, and leaving tears in the eyes\h for Excite who turned down the $750,000 deal. \h
Most of the prior search companies were just\h strung out versions of their former selves\h\h
at this point. Some tried to adapt. Lycos for\h example, introduced several media services,\h\h
including the Lycos phone which combined video\h chat, real-time on demand video and an MP3 player.\h\h
Lycos would eventually be sold to an internet\h marketing company based in India in 2010. \h
In 2008 Yahoo were faced with laying off\h hundreds of people, having become bloated\h\h
over the years and now struggling in the face\h of competition, mainly from Google. Especially\h\h
with the launch of Google’s Chrome browser,\h flipping the bird to Microsoft’s earlier\h\h
business practices and starting the process\h of toppling Internet Explorer’s dominance. \h
In February 2008, Microsoft made a\h bid to acquire Yahoo for $47 billion,\h\h
which was rejected by Yahoo, leaving Microsoft to\h focus on building a new search engine, Bing. \h
In a cruel irony, by 2012 Yahoo\h were unable to sustain their own\h\h
search model and switched to using Bing’s\h technology to power their own site. \h
Other search engines would come and go. Some\h stuck around, focusing on ethics, a lack of\h\h
advertising or just being a company\h that wasn’t Google or Microsoft. \h
But, we never really regained the excitement\h of those initial Search Engine battles,\h\h
simply because Google got so good at serving up\h what people wanted, that no one really needed\h\h
to go anywhere else. It’s inevitable. \h
The early days of products are always\h an exciting playground of opportunity.\h\h
Companies fighting each other, and in doing so\h driving each other forward. But sooner or later,\h\h
it’s just easier to go with the masses. Good ole’ reliable convenience. \h
But should you wish to hark back to the days of\h old and experience search like it was, there are\h\h
still a few of the original search engines\h running. A lot are just fronts using Google\h\h
or Bing, but some, some are still doing their own\h thing, and for that, they get my full respect. \h
Until next time, I’ve been Nostalgia Nerd. Toodleoo.