<-- Previous**

**Next -->

FAQs

What is the nearly invisible or opaque web? The opaque or nearly visible web is information on a public website that has not been indexed by the robotic 'crawlers' or 'spiders' sent out by the search engine.  The information is 'indexible' but it hasn't yet been indexed.  Crawlers may just miss the page because they have limited the number of pages they index from each site.  Also webmasters can exclude pages by using special HTML codes.

How can I search the opaque web?

By knowing that important information may be hidden behind the next click on a web page, you'll be more disposed to look deeply into the sites you visit.  If you find a good information source, spend time exploring it at depth. If the website has a sitemap use it to dig into the information.  If a site provides a search box, use keywords to quickly find what you are looking for.  (For more on these topics see the IMSA Micro Modules: How Can You Search An Individual Web Site In Depth? & What is a Sitemap?

How does search engine 'depth of crawl' create the opaque web?

Some engines impose limits on the number of pages they record at any given site.  With a limited 'depth of crawl' the robotic spider might copy part of a site, while leaving other pages out of the index for that site.  If a website has a thousand pages, but only 100 are crawled and indexed, the depth of crawl has created a good deal of 'opaque web' content.

Are their ways to make a webpage intentionally 'opaque'?

A webmaster may choose to 'hide' a page from search engine crawlers by using special html code that instructs crawlers to skip pages or sub-directories of information.  This code is placed in a file called robots.txt.  Additionally the HTML NOINDEX meta tag can be added to a page, which will then be automatically skipped by a search engine crawler.  The HTML NOFOLLOW meta tag allows a page to be indexed, but blocks the spider from following links on that page.

Are the same pages nearly invisible or opaque to all search engines?

Each search engine has its own unique index.  What is opaque to one search engine might be indexed and highly visible to another search engine.   This is another good reason to always use three different search engines when looking for information.  Also, how long a page will remain hidden is hard to determine.  Search engines are constantly updating and revising their index systems.  What's hidden today may be visible tomorrow.