Searching on the Fly

There is only one way to search live pages: browsing. It is impossible to access live information without browsing. Browsing consists of entering URLs manually in a browser's address box or clicking links to live pages. The ability to browse effectively depends on a combination of skill and luck. The skills involved are the abilities to 1) scan material to 2) find keywords that are more closely related to the search objective. If a link proves not to move a search in the desired direction, then the ability to stop and retrace steps becomes important. Luck is heavily involved in browsing because promising keywords do not always lead to better information and sometimes the search dead ends. Browsing is the most primitive form of speculative searching, like lifting one rock at a time looking for gold.

Searching live pages is recommended to locate up-to-the-minute information. For breaking news, a favorite news site would be the best place to search, although a crawler will inevitably come across breaking news somewhere. For example, within an hour of the first plane hitting the World Trade Center on 9-11-01, it was possible to find news of it by searching Google. One of the first sites indexed where it was possible to see pictures of what was unfolding in New York City that morning was an Australian news source. Over the next several hours, more and more relevant snippets could be found as crawlers came across sites carrying news of the tragedy. An rss feed from a news organization is the fastest way to learn about current events on the Internet.
knowing where to search for information

Problems searching live pages

The main difficulties searching live pages are knowing what and where they are and how to navigate them to find the information needed.

1. Finding live pages requires that a user knows either a specific URL or how to do a Deep Web search. Typing bbc.com in a browser window is one way to get to that site. Another way is to search a database such as Google with a specific query like BBC news. A simple Deep Web query, news, returns a variety of online news organizations to choose from: cnn.com, msnbc.com, foxnews.com, usatoday.com, and others (bbc.news is on the second page of results). Normally, a Deep Web search uses keywords sparingly, looking for specialized types of sites: movie database, weather almanac, stock market prices. The search for information continues at one of the sites retrieved.

2. Navigating live pages depends on what the page provides and the ability to select good leads and stay on track. If a page provides a search box for the site or a subject directory (usually in the form of a navigation menu) that will make the search easier. If not, the only way to track down information is to select links that have good potential. This is accomplished by a combination of a) knowing what keywords may lead in the direction of the information needed, b) luck and c) being able to back out of dead-ends.

Keywords are the best way to browse, just as they are the best way to use a search engine. The main difference is that browsing allows only the keywords that the author used. Since the 1 in 5 rule also applies to browsing, there may be a slim chance that the author of the page used the same word(s) you have in mind. Finding the closest word is required and in many cases the author's word will be more general than your word. This is where it really helps to have a good grasp of vocabulary.

Luck factors in because a page may or may not have links to the information you need. Sometimes words you would never think to use are the link you need. Sometimes you have to take a chance to get where you need to go.

If you are not lucky--or the page has no relevant links--you have to be willing to back up to a place where there were other choices to try. This requires patience, persistence and time.

Searching the past

The most powerful search strategies combine search engines and browsing. To return to the prospecting metaphor, using a search engine is like having a machine do the laborious work of collecting and isolating all the rocks that (as defined by the keywords) have gold-bearing characteristics. This is why searching with a query is so much more powerful than searching with a browser: there's so much more potentially rich material to choose from.

After the search engine returns one or more snippets, browsing takes over. Browsing snippets is not much different than browsing links or pages. The task is to scan the results, find the keywords, see how they are used and decide whether they are relevant to the original question.

The typical search experience is to compose a query based on the information needed, review the archived results and then start browsing by clicking the title link associated with a snippet. The Web page reached will contain the information previewed in the snippet unless the page has been updated and not re-crawled recently.

From what we observe, very few searchers click links associated with a cache, mainly because they don't know it's there and what it is.

To learn more about using a Search Engine, check out Searching the Cache: Three Choices