We cannot know where we are going without knowing where we have been. This is the young history of web search engines.
Not many inventions can claim to have such a tremendous impact on mankind as much as the Internet has had. The Internet as we know it today was created by Tim Berners-Lee, while working for the European Organization for Nuclear Research (CERN) as an independent contractor.
Lee developed a framework that would be the foundation of the World Wide Web (www). He launched the first website in 1991, which contained explanatory content on what the www was.
During this time the website also included a directory of other websites available on the www. To find information on the web at the time, one would only need to visit Berners-Lee’s website to look at available destinations.
As the Internet rapidly developed and more information was uploaded online, the need of a method to sort and deliver the information was created. The first basic search engine was created by Alan Emtage, named Archie, and it worked by matching search queries with file names.
Archie did not index the content of a webpage but indexed the file name of a webpage, so it was imperative you knew the file name of the document you were searching for. Even though Archie was a basic search engine and would be viewed as primitive today, at the time it provided a way for users to search and find new information. It was a novel concept.
The first generation of search engines has many similarities with today’s search engines, particularly when it comes to the user interface. However, they are completely different when you take a look under the hood.
Some of the first-generation search engines include Ask, Altavista, and Lycos. Even though these search engines were a lot more useful than Archie, their algorithm was still limited and relatively easy to manipulate.
Altavista was groundbreaking in the search engine industry due to its ability to accumulate a high amount of webpages and handle natural language search queries. However, manipulation and irrelevant search results reduced the popularity of the search engine.
The Altavista search engine and other first-generation search engines all went through similar issues. For example, Altavista had a market share of 11.4% as late as 1999.
A factor that contributed to the stagnation of market share for these search engines was that they were “punished” by being created too early and were destined to be short term since they did not anticipate the major amount of websites that would try to “game” the search engine to rank better.
The bottom line is that none of the first-generation search engines have any relevant market share today due to not continuously evolving their product and adapting to their constantly changing environment.
The outcome would often be that these search engines provided irrelevant results because of the high amount of websites that were manipulating the search. The fact that none of the major search engines today were created before 1995 supports this statement.
This opened up the opportunity for the second generation of search engines and the entrance of Google. Some of the second-generation search engines include Yahoo, Google, Baidu, Yandex, and Bing.
The arrival of these search engines opened up another completely new aspect of the web. No longer was it necessary to memorize the domain name of specific websites. Users could now use search engines to find relevant information almost immediately.
By using a more advanced algorithm and including more factors to determine website ranking in the search engine, it made it more challenging for websites to manipulate their ranking. This led to an improvement in user experience as relevant results increased dramatically and manipulative sites were pushed down in the search results.
Another aspect that was heavily improved for the second generation of search engines was the ability to handle advanced queries. First-generation search engines only indexed some parts of a web file, while second-generation search engines indexed the entire webpage, including all of the text. This meant that search engines could now match search queries with text from the whole page.
All these improvements have led to a higher adoption rate and search engines quickly became more popular to use. The search engine advertising market has become a multibillion-dollar industry, and terms such as SEO, SEM and link building have become common within the advertising industry.
Today search engines are moving toward providing a more personalized experience when searching the web. This transition could be considered the beginning of the third generation of search engines.
The future direction of search engines seems to be heading toward social search, where the search results will be influenced by what people in your ecosystem have recommended.
The social search revolution is in its beginning phase but has already made a major impact on how businesses approach their marketing efforts.
What do you think will be the future direction of web search engines?