History of Search Engines

The first website ever created, http://info.cern.ch/, was created by Tim Berners-Lee at CERN, and was launch on August 6, 1991[i]. Its content was simple: it provided instructions on what the World Wide Web was, how to set up a web server, and other relevant details. And the rest, they say, is history. From there websites, web servers, and an entire industry began.

It’s hard to imagine, but there was a time before a single search engine existed. The Web had a very small amount of websites, with only a little over 620 in December of 1993. What necessitated better ways of categorizing and navigating the Web was the rapid growth that soon happened as the 1990s progressed.

As you can see from the table below, the number of websites in the world went from less than 150 in 1993 to over 650,000 in 1997 – that’s a 500,000% increase in less than four years! From there, websites continued to proliferate, and so did things like social networks which created even more information to search, sort and browse.

Web Growth from 1993-1997[ii]
Month# of Web sites% .com sitesHosts* per Web server
6/931301.513,000 (3,846)
12/936234.63,475 (963)
6/942,73813.51,095 (255)
12/9410,02218.3451 (99)
6/9523,50031.3270 (46)
1/96100,00050.094 (17)
6/96230,000 (est)68.041
1/97650,000 (est)62.6N/A

Yahoo! and other Web Directories

The first solution to the problem of searching the Web was the directory, with the first one being started and curated by Tim Berners-Lee in 1992[iii]. It makes perfect sense, because directories actually have a lot in common with the way libraries were using methods such as the Dewey Decimal System. But with the rapid growth of websites, it put such a massive strain on this growing number of manually-updated directories that an alternative solution needed to be found. Yahoo!, one of the most popular Web directories launched in 1995 (alongside Looksmart and Lycos), and was one of the last to close its doors, which finally happened at the end of 2014.

The First Search Engines

“Between the summer of 1993 and the end of 1994, an explosion of Internet search services changed the way people use the Internet. Before then, users could only browse pages, clicking on likely looking hypertext links: a process that David Eichmann likened to starting in one city and driving around until you found a road that leads to your destination. Since 1994, most people start their Internet surfing at either a search service or a directory service, and these have become the roadmaps to the Internet.”

Started at McGill University in Montreal in 1986[iv], Archie (named for “archive” without the letter “v”) was written by Peter Deutsch as a method of searching public FTP sites and is widely acknowledged as the first true search engine. The software was almost instantly successful and led Peter Deutsch to partner with Alan Emtage to form a company called Bunyip which went on to support Archie for many years. Still, it was not a search engine made for the Web itself, only for FTP file directors.

Web search engines would start to appear at the beginning of the 1990s. While several search engines started appearing after Archie, including W3Catalog and WebCrawler among others, the first commercially successful one to appear was Lycos, which was created at Carnegie Mellon University. Over the next several years, a rash of search engines were released, some with a specific take on search, such as AskJeeves.com (later Ask.com), which prompted users to perform queries in the form of a question, and many more that failed to distinguish themselves and have since disappeared.

A Brief Timeline of Search Engines:

As you can see below, there used to be a lot of competition among search engines:

  • 1993: Excite
  • 1994: Yahoo!, Webcrawler, Lycos, Infoseek
  • 1995: AltaVista
  • 1996: Inktomi
  • 1997: Google, Ask Jeeves
  • 1998: MSN Search
  • 1999: Alltheweb

The Google Era

Which brings us to the modern era. While there are still many ways to search on the Web, as of April 2017, Google holds just over 77% of the world’s search market share[v]. This leaves Bing and a few other competitors with what little market share is left, no more than st 8% for the second place competitor. So when we talk about the modern era of search (since at least 2005), we’re primarily talking about Google’s search engine.

As time has passed between the 1990s and today, several things have converged to allow our world to become more and more searchable every day:

  • An increasing amount of content is being created and stored on the Web
  • Existing (or pre-existing) content is being stored on the Web
  • Search technologies are growing more and more sophisticated

This all adds up to a nearly endless amount of information out there waiting to be found. In fact, there are so many results out there to be found, Linda Evans made the following statement that rings particularly true, saying, “I have Googled approximately 50,000 times over the years, and only once have I received no result. I repeat: only once have I put together a thought that no one else has thought to ask or publish. Which evidently means I have only had one original thought in all those years.[vi]” Can it be true that there’s almost no question you can ask Google for which there isn’t already at least a handful of answers waiting for you?

This means that not only are our search engines sophisticated enough to find and crawl all of this information, but that all of the information has been created, captured, and put up on the Web for consumption.

Why Google?

To complete our history of “standard” search engines, it’s important to note that there has truly been a winner in the search engine wars. Google far and away outlasted and beat the competition with over three quarters of searches being performed by “googling.” But how and why did this happen?

There are many explanations for this. Much of their success can be attributed to the sheer quality of the Google search algorithm, which we’ll talk about in the next chapter of this book. But some of the success comes from the single-minded, almost simple approach that Google took to their interface and their user experience. While many of their competitors made revenue based on the “stickiness” of their websites[vii], Google focused on having the best results and instead suffered from a website that got users answers almost too quickly, thus leaving their site sooner than others whose search results were not as good and required performing multiple searches, or paging through results, thus increasing advertising revenue and time on the site, but not necessarily making users happy.

An interesting anecdote about Excite.com was shared in Steven Levy’s book:

“[Google] was too good. If Excite were to host a search engine that instantly gave people information they sought, [Excite’s CEO] explained, the users would leave the site instantly. Since his ad revenue came from people staying on the site—“stickiness” was the most desired metric in websites at the time—using Google’s technology would be counterproductive. “He told us he wanted Excite’s search engine to be 80 percent as good as the other search engines,” … and we were like, “Wow, these guys don’t know what they’re talking about.”

Steven Levy, In The Plex Obviously, Google found a way to generate revenue without increasing “stickiness” on their site.

[i] Wall, Aaron. “Search Engine History.” SearchEngineHistory.com 2017

[ii] Gray, Matthew. Web Growth Summary. MIT.edu

[iii] Veradyn, Jim. A History of Web Directories – The Past, Present and Future. November 24, 2014.

[iv] Savetz, Kevin. Article by Kevin Savetz.

[v] Allen, Robert. “Search Engine Statistics 2017.” April 13, 2017. Smart Insights.

[vi] Evans, Linda. “What did we do before Google and the Internet?” Awhautkee Foothill News

[vii] Dixon, Chris. “Why Google Succeeded Where Other Search Engines Failed.” Business Insider. May 16, 2011.