The following is based on an Investigative Reporters and Editors seminar this weekend in Birmingham, Alabama.

IRE talked about how the Web--both the Surface ("visible") and Deep ("invisible") Webs--can help reporters address the occupational hazard of having to know everything about anything at any given moment.

The hour-long presentation, Effective use of the Internet, was fittingly framed by the first word in the title. Mark Horvit, IRE’s executive director, began by emphasizing that reporters should approach online research armed with a strategy (i.e., key words to search and a general idea of what’s available and desirable) to avoid getting distracted by the Web’s potentially cavernous detours. Step one, Horvit said, is not to log on, but to sketch out a plan.

Important for every investigative journalist to know about search engines is that a Google search, for example, does not look through the actual Internet, per se. It searches Google’s servers, which are stocked with information that the search engine company’s Web “crawlers” have found and stored.

What they’re missing – eye-opening stats:

• Google searches far less than half of what’s out there
• Total shared results of any two search engines: 8.9 percent
• Any three search engines: 2.2 percent
• Above figures from 2007 study by Dogpile, Penn State and Queensland University of Technology
• Some estimate the “invisible” Web is 550 times bigger than the “visible” Web.
• Google says more than 1,000 federal government sites can’t be crawled.

If (way) more than half the Web isn’t showing up in a search engine result, then it is important for investigative reporters to know where to go to find it. Here are some of the principles behind efficiently conducting those searches, with both superficial tools and subterraneous means.

Surface Web – Savvy searching tips:

• Treat info online as one would any source (confirm)
• Find out who owns the Web site
• Know Google advanced search options (esp. domain and file type)
• Archived Web: Gone doesn’t mean forever. (Google cache, Wayback Machine)
• Consult at least two other search engines–each has its own strengths and weaknesses.
• People finders (i.e., www.pipl.com, www.whitepages.com, etc.)
• Social media searches (i.e., www.whostalkin.com… Who’s Talkin’, not Who Stalkin’… or so they say)
• Use Wikipedia for the footnotes only

The session then took Web searches to the next level… well, at least a step above what amateur voyeurs might use to get information.

Deep Web – Search like a pro:

• Know what search engines typically miss (databases, content behind firewalls and registration screens, ASP/dynamically generated pages, Robo.txt excluded pages)
• The information is out there, but the key is to find organizations that make is more easily accessible. Bookmark these!
• Directories by and for journalists (‘Net Tour and Reporter’s Desktop)
• Know the gateways to public records
• Pipl actually claims to access the Deep Web. Try it. Pipl yourself. It’s scary how much information it digs up with just a name.
• The census is your friend, especially in 2010
• To get fully submerged… go to IRE’s Web site!

I’m not going to copy-paste in this post all of the useful links for discovering the “hidden Web” and the “dead Web,” which were hyper-linked in the PowerPoint presentation that Mark offered to send out to anybody at the day-long seminar who asked for it. All of this stuff is available at the organization’s site, and I can see what the nominal membership fees pay for, seriously.

For this project, you will be evaluating and reviewing an internet research tool.  You will find the name of and link to your assigned tool on the class wiki (ENG 1020, ENG 122).

Write a review of the research tool assigned to you and post the review in your blog.  Your review should include all of the following information that is relevant for the type of tool you were assigned:

• Name of and link to the tool
• Summary or description of the tool
• Strengths
• Weaknesses
• Search engines, directories, and other applications searched
• Databases
• Operators
• Case sensitivity
• Stop words
• Advanced search function
• Limits
• Sorting
• Display
• Help function
• Special features

See the column headings of the Search Engine Features Chart for explanations if needed.  You can see sample reviews by clicking on links here and here.

Use the department and course number (ENG 1020 or ENG 122) and other appropriate tags (”Labels” on Blogger) for the post.

כשאנו תרים אחר מידע ברשת, ניצבים בפנינו שני אתגרים גדולים עיקריים: האתגר הראשון הוא הקושי לאתר מידע במהירות וביעילות – דבר שאמור להיות קל יותר בשימוש באופרטורים עליהם כבר כתבתי כאן. האתגר השני – שאנחנו בד”כ פחות מודעים אליו – הוא החיפוש ב”רשת הבלתי נראית“.

אפתח בהסבר פשטני על אופן העבודה של מנועי חיפוש כגון גוגל. ובכן, תוכנה שמכונה בשמות המסתוריים והמעקצצים “זוחלים”, “רובוטים” או “עכבישים”, עושה את דרכה ברשת האינטרנט, ומתייגת תכנים. אותם “זוחלים” עוברים בין הלינקים, ומאחסנים דפים לשליפה מהירה בזיכרון (בעבר היו הזוחלים הללו מסוגלים לעבור רק בדפים הכתובים ב-HTML או בטקסט, אבל כיום מנועי החיפוש יודעים להמיר גם פורמטים כגון PDF ולאחסן אותם).

ה”רשת הבלתי נראית” מורכבת מכל אותם דפים שה”זוחלים” של מנועי החיפוש לא מציגים לנו. המדובר במסה עצומה של דפים (ויש הסבורים כי “הרשת הבלתי נראית” גדולה אף פי 500 מהיקף הדפים הזמינים דרך מנועי חיפוש). כך למשל, הזוחלים לא יודעים להיכנס דרך מסך כניסה של בסיסי נתונים, קטלוגים, מאגרי מאמרים ועוד. הם גם מתעלמים בכוונה מחלק מהדפים בחיפוש, שהם סבורים שסתם יעמיסו על הגולש, ומתעלמים גם מהדפים שבעליהם לא רוצים שמנוע החיפוש יאתר אותם.

לכן, כשאנו ניגשים למחקר ברשת, אסור לנו להסתמך רק על חיפוש בגוגל, ואנחנו צריכים לשלב בחיפוש שלנו גם חיפוש ברשת הבלתי נראית.

הנה כמה טיפים:

1. בין מכרות הזהב ברשת ניתן למנות את הבאים:

• http://www.intute.ac.uk/ , http://infomine.ucr.edu/, http://lii.org/ – ספריות מקורות בנושאים שונים
• הספריות של גוגל ו-יאהו
• Complete Planet ו-Oaister – מנועי חיפוש לרשת הבלתי נראית.

2. הכנת רשימת מקורות שימושית וחיפוש נפרד ברשימת האתרים הללו בכל חיפוש, בנוסף לחיפוש ברשת. למשל: בלוגים מקצועיים או פרטיים בנושא החיפוש, פורומים, אתרי חברות, חברות יעוץ ועוד.

איך מאתרים כאלה? באופן משולב, הכולל:

• חיפוש כללי של הנושא ואיתור חברות שהאתרים שלהם עולים בהקשר זה, וכן בחינה בתשומת לב של שמות חברות הנזכרות במאמרים / פרופילים על התעשיה או הנושא הרלוונטיים. במקרה שאחת החברות היא חברה ציבורית, כדאי לקרוא דוחות שנתיים שלה, על מנת לצבור ידע נוסף בנושא.
• איתור של מקור לפי מאמר אחד שאיתרתם – לעתים יש דפים שמאופסנים על גבי הרשת וניתן להגיע אליהם באמצעות מנועי חיפוש, אך חיפוש ישיר באתר יניב הרבה יותר תוצאות מאלה שהגעתם אליהן דרך מנוע החיפוש, מסיבות שונות. מומלץ תמיד לבחון את האתר שהוא המקור למאמר המעניין שאיתרתם, בין אם ע”י לחיצה על עמוד הבית של האתר, ובין אם ע”י מחיקה של סיומת ה-URL עד שתגיעו ל-directory שיש גישה אליה (לעתים זה יהיה רק מהעמוד הראשי, כלומר תצטרכו למחוק את כל מה שמופיע אחרי הסיומת com או org או co.uk וכיו”ב).
• חיפוש של מקור ספציפי באמצעות גוגל – לאחר שזיהיתם אתרים רלוונטיים, ניתן לנסות לערוך בהם חיפוש עמוק ע”י שימוש באופרטור :site. כלומר, להכניס את מילות החיפוש ואז להכניס את האופרטור ובצמוד לו את כתובת האתר שחשפתם, רצוי בלי ה-www משום שלעתים במקום www  יש שם של ספריה ספציפית (כגון finance.google.com, ir.nestle.com…) כך: site:XYZ.com
• איתור של פורום/ ספריה / בלוג באמצעות חיפוש המלים הרלוונטיות, והוספת המילה directory, forum, database, blog וכיו”ב.

כמובן שבאופן אידאלי, מי שעורך מחקר באופן מקצועי, בין אם כפונקציה בחברה או כמידען או חוקר שוק (כמוני, למשל), צריך גם לרכוש מנוי למאגר אחד הכולל מאמרים ממקורות שונים בתשלום כגון מאגר המידע העסקי Nexis.com, וכן מנויים למאגרים מקצועיים אחרים בתשלום, משום שאין דרך לעקוף את דרישת התשלום על מאמרים איכותיים יותר.

אשמח לקבל מכם תגובות והפניות למקורות נוספים, וכמובן לענות על שאלות.

Video:  “[OSTI] Deep Web Video“

Office of Scientific & Technical Information

Experts say search engines such as Yahoo! and Google only pick up about 1% of the information available on the Internet. The rest of that information is considered to be hidden in the deep web, also referred to as the invisible web. So how can you find all the rest of this information? This list - 100 Useful Tips and Tools to Research the Deep Web offers 100 tips and tools to help you get the most out of your Internet searches.

Experts say search engines such as Yahoo! and Google only pick up about 1% of the information available on the Internet. The rest of that information is considered to be hidden in the deep web, also referred to as the invisible web. So how can you find all the rest of this information? This list - 100 Useful Tips and Tools to Research the Deep Web offers 100 tips and tools to help you get the most out of your Internet searches.

Deep Web, is also know as ‘Hidden Web’, ‘Dark Web’, or the ‘Invisible Web’.  It is mostly generated by database-driven pages that are usually visible to authorized members. Who can access the hidden information after logging into the system.

Such data is totally hidden from search enging crawlers (or the ’spiders’), that means Google, Yahoo, MSN, Altavista and other search endinges cannot search data from these pages.

Web crawlers of todays technology can only read ‘surface web‘.

Internet Research Presentation

When we perform Google searches how do we know that Google has searched in every database on the Internet to come up with suggested matches for what we’re looking for?  The easy answer is that Google can’t do that and we are presented only with hits for Web pages that Google knows about.   Considering that this past summer Google added the one trillionth address to its list of Web pages, wouldn’t one think that’s far and away enough?

That’s where the Deep Web or the Invisible Web comes into play.  Chris Sherman and Gary Price wrote a terrific book in 2001, The Invisible Web, that covered dozens and dozens of information sources that search engines haven’t found.  There have been a few more books on this topic since then but I wish Sherman & Price would update their book. 

Most of the Web pages in the Deep Web are from associations, businesses, libraries, universities and government agencies.  The amazing amount of information, statistics, data, etc. that can be found within these is enormous.

 There are some great developments in deeper searching that have popped up recently.  Kosmix (www.kosmix.com) started out as a search engine for health and travel information.  It has since developed a platform for a universal search engine that snags data from a lots of sources – Flickr, Google, Wikipedia, Yahoo Answers, YouTube, and others. 

It then creates sort of a customized web page that breaks your search into segments.  I searched for the topic “Burma” and Kosmix returned more information than I knew was available.  Everything from reference, media, news & blogs, to ethnic groups, history, shopping, and books.  Sources included Wikipedia, BBC & CNN, Shopping.com, Flickr, SeeqPod, the blog Backtype, and Slideshare.net.  Yes, my search did uncover Burma Shave but the other riches outshone it.  And, yes, Kosmix is one of those Mountain View, CA companies.

Another Deep Web crawler is DeepPeep (www.deeppeep.org) which is being developed by a professor at the University of Utah.  When I entered my search term, “Burma,” I received 143 documents.  I initially thought the search was totally off the wall but when I investigated each retrieved document I discovered what DeepPeep is trying to do.

I had also told DeepPeep to search in “all domains” rather than the more selective subjects airfare, book, rental, job, or biology.  Therefore, I hit the mother lode of stuff.  One of the first hits was for horse jobs and, sure enough, Burma is one of the countries listed in the horse-jobs.biz web site.  I couldn’t figure out how the Hotel Oscar in Athens could be related to Burma but a very close look at the bottom of its home page listed links to other hotels.  What russiamaritime.com had to do with Burma (and just who knew there was a russiamaritme.com?) was also easily discovered.  This is really deep web searching and totally fascinating for those of us who love to bounce around the web discovering Web databases.

So, is there one search engine that does it all?  Obviously, no.  It’s great to have multiple search engines which create search strategies so differently.  It makes searchers think harder about how to formulate their keyword strategies.  Now, if I could just whittle down my favorite search engines to a five or six from several dozens.

10, 9, 8, 7, 6, 5, 4, 3, 2, 1… Happy New Year! 2009 is here. Getting off to a fast start in 2009 is paramount for everyone in your sales team. Every Sales person is looking for an edge to shorten a sales cycle or build a tighter relationship with a prospect.

Recently, FIVE STAR Business Speakers had the opportunity to hear Sam Richter. Sam is a Business and Sales Speaker who wrote Take The Cold Out of Cold Calling. Sam has developed a vast knowledge on how to leverage the web efficiently to gain a deep understanding and knowledge about a client or prospect. The tools are practical, ethical, and available to all who will use them. The tools will help you be more knowledgeable about, connect quicker with, and build confidence with your client or prospect.

Read the review on Business, Marketing and Sales Speaker Sam Richter book’s. Take The Cold Call out of Cold Calling

To book Sam Richter – Business, Marketing, and Sales Speaker contact FIVE STAR Speakers @ 913.648.6480.

Timeline of selected events related to the Deep Web (work in progress)

1980 Tim Berners-Lee “developed his first hypertext system, “Enquire” for his own use (although unaware of the existence of the term HyperText). With a background in text processing, real-time software and communications, Tim decided that high energy physics needed a networked hypertext system and CERN was an ideal site for the development of wide-area hypertext ideas (CERN).”

1989 Tim Berners-Lee started the WorldWideWeb project at CERN.

1992-09 Arthur Secret at the CERN created the first web gateway to a relational database system RDB (Shestakov 2008-05).

1994 Dr. Jill Ellsworth “first coined the phrase “invisible Web” to refer to information content that was “invisible” to conventional search engines (Bergman 2001 citing Garcia 1996).” See also

1996 Frank Garcia (1996) claimed Texas-based university professor Jill H. Ellsworth (d.2002), Internet consultant for Fortune 500 companies, coined the term “Invisible Web” in 1996 to refer to websites that are not registered with any search engine. ” “Ellsworth is co-author with her husband, Matthew V. Ellsworth, of The Internet Business Book (John Wiley & Sons, Inc., 1994), Marketing on the Internet: Multimedia Strategies for the World Wide Web (John Wiley & Sons, Inc.), and Using CompuServe. She has also explored education on the Internet, and contributed chapters on business and education to the massive tome, The Internet Unleashed.”

[S]igns of an unsuccessful or poor site are easily identified, says Jill Ellsworth. “Without picking on any particular sites, I’ll give you a couple of characteristics. It would be a site that’s possibly reasonably designed, but they didn’t bother to register it with any of the search engines. So, no one can find them! You’re hidden. I call that the invisible Web. Ellsworth also makes reference to the “dead Web,” which no one has visited for a long time, and which hasn’t been regularly updated (Garcia 1996).

1996-12-01 “The first commercial Deep Web tool (although they referred to it as the “Invisible Web”) was @1, announced December 12th, 1996 in partnership with large content providers. According to a December 12th, 1996 press release, @1 started with 5.7 terabytes of content which was estimated to be 30 times the size of the nascent World Wide Web. ( “America Online to Place AT1 from PLS in Internet Search Area: New AT1 Service Allows AOL Members to Search “The Invisible Web”).”See (Choi 2008-01-07).”

1996-12-12 “Personal Library Software, Inc. (PLS), the leading supplier of search and retrieval software to the online publishing industry, ushered in the next generation of Internet search engines with the introduction of a new Internet based service, AT1 which combines the best of PLS’s search, agent and database extraction technology to offer publishers and users something they have never had before: the ability to search for content residing in “hidden” databases — those large collections of documents managed by publishers not viewable by Web spiders. AT1 also allows users to create intelligent agents to search newsgroups and websites with E-Mail notification of results (Press release).”

1997 Michael Lesk wrote an unpublished paper entitled ”How much information is there in the world?”], in which he estimated that in 1997, the Library of Congress had between 20 terabytes and 3 petabytes.” See Choi (2008).

1999-02 Lawrence and Giles (1999) claimed that the publicly indexable World Wide Web (PIW) contained about 800 million pages; the search engine with the largest index, Northern Light, indexed roughly 16% of the publicly indexable World Wide Web; the combined index of 11 large search engines covered (very) roughly 42% of the publicly indexable World Wide Web.

2000-03 c. 43,000–96,000 Deep Web sites existed (Bergman 2001).

2000-07-26 BrightPlanet released a study documenting the Deep Web (a massive storehouse of databases and information that was invisible to search engines in 2000) claiming that the Deep Web was 500 times larger than the indexed Web accessible by most search engines. BrightPlanet researchers also released their direct-query search technology called LexiBot™ which automatically identifies, retrieves, qualifies, and classifies content from Deep Web sites. They listed c. 20,000 Deep Web searchable sites. Direct-query search technology that can access searchable databases unlike most search engines, implies that the Invisible Web is not really Invisible just harder to reach. BrightPlanet Unveils the ‘Deep’ Web: 500 Times Larger than the Existing Web.

2001BrightPlanet

“quantified the size and relevancy of the deep Web in a study based on data collected between March 13 and 30, 2000. Our key findings include: Public information on the deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web; The deep Web contains 7,500 terabytes of information compared to nineteen terabytes of information in the surface Web; The deep Web contains nearly 550 billion individual documents compared to the one billion of the surface Web; More than 200,000 deep Web sites presently exist; Sixty of the largest deep-Web sites collectively contain about 750 terabytes of information — sufficient by themselves to exceed the size of the surface Web forty times; On average, deep Web sites receive fifty per cent greater monthly traffic than surface sites and are more highly linked to than surface sites; however, the typical (median) deep Web site is not well known to the Internet-searching public; The deep Web is the largest growing category of new information on the Internet; Deep Web sites tend to be narrower, with deeper content, than conventional surface sites; Total quality content of the deep Web is 1,000 to 2,000 times greater than that of the surface Web; Deep Web content is highly relevant to every information need, market, and domain; More than half of the deep Web content resides in topic-specific databases; A full ninety-five per cent of the deep Web is publicly accessible information — not subject to fees or subscriptions (Bergman 2001).”

2001 AlltheWeb, public search engine was launched. (AlltheWeb is now owned by Yahoo.com). It was a redesign of Fast (1999-05 to 2001). Fast Search & Transfer is a Microsoft Subsidiary.

2000 Shestakov (2008) cites Bergman (2001) as the source for the claim that the term deep Web was coined in 2000. Bergman distinguished the Surface Web from the Deep Web using the metaphor of Surface and Deep water fishing or trawling. Deep Web is preferred over the term Invisible Web.

2000 UC-Berkeley Biologist Michael Eisen, Nobel Laureate Harold Varmus and Stanford biochemist Patrick Brown helped start the Public Library of Science, PLoS is a “nonprofit organization of scientists and physicians committed to making the world’s scientific and medical literature a freely available public resource” by encouraging scientists to insist on open-access publishing models rather than being forced to sign over their (often publicly-funded research) to expensive scientific journals. Wright (2004) cited Eisen, Varmus and Brown as examples of scientists who are making making some areas of the Deep Web more accessible to the public.

2001 Raghavan and Garcia-Molina (2001) “presented an architectural model for a hidden-Web crawler that used key terms provided by users or collected from the query interfaces to query a Web form and crawl the deep Web resources (Choi 2008-01-07).”

2002-02 StumbleUpon began to use human crawlers or human-based computation techniques to uncover data on the Deep Web. Human crawlers can find relevant links that algorithmic crawlers miss (Choi 2008-01-07).”

2002-12 There were c. 130,000 Deep Web sites (He, Patel, Mitesh, Zhang and Chang 2007, Shestakov 2008).

2003-06-01 Dorner and Curtis (2003-06-01) conducted a survey (data collected from 2002-12 through 2003-04) of librarians in New Zealand to compare their common user interface software products supplied by vendors: Endeavour, ExLibris, Follet, Fretwell-Downing, Innovative Interfaces, MuseGlobal, OCLC, SIRSI, WebFeat and VTLS. MuseSearch, ENCompass, MetaLib, Single Search and WebFeat received the highest scores in 2003 (Dorner and Curtis 2003-06-01:2). SingleSearch was noted as having the added cost advantage to librairies since it was open access, open source (Dorner and Curtis 2003-06-01:2). In 2002-2003 a successful common user interface technology software should support formats and protocols other than Z39.50 such as OpenURL, HTTP, SQL, XML, MARC, CrossRef, DOI, EAD, Dublin Core and Telnet (Dorner and Curtis 2003-06-01:8).

2004-04 There were c. 310,000 Deep Web sites (He, Patel, Mitesh, Zhang and Chang 2007, Shestakov 2008).

2004 Between 2000 and 2004 the Deep Web increased in size by 3-7 times (He, Patel, Mitesh, Zhang and Chang 2007, Shestakov 2008).

2004-03-02 Yahoo announced its Content Acquisition Program users paid for enhanced search coverage by “unlocking” the deep Web (Wright 2004).

2005 Yahoo released Yahoo! Subscriptions which searched a few of the Deep Web’s subscription-only web sites.

2005 Ntoulas et al. (2005) “created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms. Their crawler generated promising results, but the problem is far from being solved. Since a large amount of useful data and information resides in the deep Web, search engines have begun exploring alternative methods to crawl the deep Web (Choi 2008-01-07).”

The search engine Pipl crawlers can identify, interact and retrieve some information from the deep Web.

Deep Web “search engines like CloserLookSearch and Northern Light Group|Northern Light create specialty engines by topic to search the deep Web. Because these engines are narrow in their data focus, they are built to access specified deep Web content by topic. These engines can search dynamic or password protected databases that are otherwise closed to search engines (Choi 2008-01-07).”

Google’s “Sitemap and mod oai are mechanisms that allow search engines and other interested parties to discover deep Web resources on particular Web servers. Both mechanisms allow Web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not directly linked to the surface Web (Choi 2008-01-07).”

2007-06 WorldWideScience was created to provide access to the Deep Web. When it began it linked to 12 databases from 10 countries. It is a “science portal developed and maintained by the Office of Scientific and Technical Information (OSTI), an element of the Office of Science within the U.S. Department of Energy. The WorldWideScience Alliance, a partnership consisting of participating member countries provides the governance structure for the WorldWideScience.org portal (RWW).”

2007-07-27 “Indiana University faculty member Javed Mustafa appeared on National Public Radio’s Science Friday, and drawing on information in a published study from University of California, Berkeley entitled ”How much information is there?”, estimated that the deep web consists of about 91,000 terabytes. By contrast, the surface web, which is easily reached by search engines, is only about 167 terabytes. The Library of Congress contains about 11 terabytes, for comparison. Mustafa noted that these numbers were a bit dated and were just rough estimates (Choi 2008-01-07).”

2008-05-14 ReadWriteWeb contributor Sarah Perez listed a number of “Digital Image Resources on the Deep Web.“

2008-06 WorldWideScience portal to the Deep Web linked to 32 national, scientific databases and portals from 44 different countries. RWW.

2008 Several “Deep Web directories are under development such as OAIster by the University of Michigan, INFOMINE] at the University of California at Riverside and DirectSearch by Gary Price to name a few (Choi 2008-01-07).”

2008-09-22 Infovell launched its research engine for the Deep Web. “Available initially on a subscription basis, Infovell gives users access to hard to find, in-depth, expert information spanning Life Sciences, Medicines, Patents, and other reference categories with more to be added over time.” “Infovell’s research engine will be available beginning September 22 as a premium service for individual researchers and corporations who are seeking more affordable access to expert information. The Company is offering a risk-free trial through its website www.infovell.com. Later this year, Infovell will be beta-releasing a free version of its research engine on a limited basis for those individuals who want to search the Deep Web but don’t have the need for some of the advanced features available in the premium version.”

2009- United States “Congressional Representative John Conyers (D-MI) re-introduced a bill (HR801) that essentially would negate the  National Institutes of Health (NIH) policy concerning depositing research in Open Access (OA) repositories. The bill goes further than prohibiting open access requirements, however, as the bill also prohibits government agencies from obtaining a license to publicly distribute, perform, or display such work by, for example, placing it on the Internet, and would repeal the longstanding ‘federal purpose’ doctrine, under which all federal agencies that fund the creation of a copyrighted work reserve the ‘royalty-free, nonexclusive right to reproduce, publish, or otherwise use the work’ for any federal purpose. The National Institutes of Health require NIH-funded research to be published in open-access repositories (Doctorwo 2009).” HR801 would benefit for-profit science publishers and increase challenges for making the Deep Web more accessible. See Doctorwo, Cory. 2009-02-16. “Scientific publishers get a law introduced to end free publication of govt-funded research.” >> Boing Boing. 

Notes

“Metasearch technology, also known as federated search or broadcast search, creates a portal that could allow the library to become the one-stop shop their users and potential users find so attractive (Luther 2003-10-01).”

Joo-Won Choi’s (2008-01) useful categories of Deep Web resources include:

Dynamic content: “Dynamic Web page and/or dynamic pages, which are returned in response to a submitted query or accessed only through a form (especially if open-domain input elements e.g. text fields are used; such fields are hard to navigate without domain knowledge). “

Unlinked content: “pages which are not linked to by other pages, which may prevent Web crawling programs from accessing the content. This content is referred to as pages without backlinks or inlinks. “

Private Web: “sites that require registration and login (password-protected resources).

Contextual Web: “pages with content varying for different access contexts (e.g. ranges of client IP addresses or previous navigation sequence).

Limited access content: “sites that limit access to their pages in a technical way (e.g., using the Robots Exclusion Standard, CAPTCHAs or HTTP headers, prohibiting search engines from browsing them and creating cached copies.”

Scripted content: “pages that are only accessible through links produced by JavaScript as well as content dynamically downloaded from Web servers via Macromedia Flash or AJAX solutions.”

Non-HTML/text content: “textual content encoded in multimedia (image or video) files or specific file formats not handled by search engines.” For more see Choi (2008-01).

Webliography and Bibliography

Bergman, Michael K. 2001-09-24. “The Deep Web: Surfacing Hidden Value.” White Paper.

Bergman, Michael. 2001. “The Deep Web: Surfacing Hidden Value.” Journal of Electronic Publishing. 7:1.

Choi, Joo-Won. 2008-01-07 “Deep Web.” KAIST

Dorner, Daniel G.; Curtis, Anne Marie. 2003-06-01. “A comparative review of common user interface software products for libraries.” National Library of New Zealand. 67 pp.

Ellsworth, Jill H.; Ellsworth, Matthew V. 1994. The Internet Business Book. John Wiley & Sons, Inc.

Ellsworth, Jill H.; Ellsworth, Matthew V. 1997. The Internet Business Book. John Wiley & Sons, Inc.

Ellsworth, Jill H.; Ellsworth, Matthew V. 1995. Marketing on the Internet: Multimedia Strategies for the World Wide Web. John Wiley & Sons, Inc.

Ellsworth, Jill H.; Ellsworth, Matthew V. 1996. Marketing on the Internet: Multimedia Strategies for the World Wide Web. 2nd Edition. John Wiley & Sons, Inc.

Ellsworth, Jill H.; Ellsworth, Matthew V. Using CompuServe. John Wiley & Sons, Inc.

Ellsworth, Jill H. Chapters? The Internet Unleashed.

Garcia, Frank. 1996. “Business and Marketing on the Internet.” Masthead. 9:1. January. Alternate url @ web.archive.org

Guernsey, Lisa. 2001-01-25. “Mining the deep web with sharper shovels“. New York Times, No.25: pp.G1.

Lawrence, Steve: Giles, C. Lee. 1999-07-08.”Accessibility of Information on the Web.” Nature. 400:6740:107 – 109. See http://www.wwwmetrics.com.

Luther, Judy. 2003-10-01. “Trumping Google? Metasearching’s Promise.” Library Journal.

PLS. 1996-12-01. “America Online to Place AT1 from PLS in Internet Search Area: New AT1 Service Allows AOL Members to Search “The Invisible Web”.” Press Release.

Shestakov, Dennis. 2008-05. deep web

Smith, Richard. 2008-10-07. “More evidence on why we need radical reform of science publishing.” PLoS.

Wright, Alex. 2004-03-09. “In search of the deep Web: The next generation of Web search engines will do more than give you a longer list of search results. They will disrupt the information economy.” Salon.

He, Bin; Patel, Mitesh; Zhang, Zhen; Chang, Kevin Chen-Chuan. 2007. “Accessing the deep Web. Communications. ACM. 50:5:94–101.

See also http://papergirls.wordpress.com/the-ultimate-guide-to-the-invisible-web

Joo-Won Choi’s Bibliography:

Panagiotis Ipeirotis, Luis Gravano, and Mehran Sahami. 2001. “Probe, Count, and Classify: Categorizing Hidden-Web Databases.”Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data. pp. 67-78.

Gary Price & Chris Sherman. July 2001. ”The Invisible Web : Uncovering Information Sources Search Engines Can’t See.” CyberAge Books, ISBN 0-910965-51-X.

Michael K. Bergman. 2001-08. “The Deep Web: Surfacing Hidden Value.” The Journal of Electronic Publishing. 7:1.

Sriram Raghavan and Hector Garcia-Molina. 2001. “Crawling the Hidden Web.” In Proceedings of the 27th International Conference on Very Large Data Bases (VLDB). pp. 129-138

Nigel Hamilton (2003). ”The Mechanics of a Deep Net Metasearch Engine.” 12th World Wide Web Conference poster.

Bin He and Kevin Chen-Chuan Chang. 2003. “Statistical Schema Matching across Web Query Interfaces.” In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data

Joe Barker (Jan 2004). ”[ Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity. UC Berkeley - Teaching Library Internet Workshops.

Alex Wright (Mar 2004). ''In Search of the Deep Web'' Salon.com

Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho. 2005. "Downloading Textual Hidden Web Content Through Keyword Queries." In Proceedings of the Joint Conference on Digital Libraries (JCDL). pp 100-109.

Extended version]

Frank McCown, Xiaoming Liu, Michael L. Nelson, and Mohammad Zubair. 2006.-03/4. “Search Engine Coverage of the OAI-PMH Corpus.” IEEE Internet Computing. pp. 66-73. 10:2.

Lesk’s bibliography:

[Bell 1994]. Alan Bell; IBM Academy Digital Library Workshop (Sept 12-13, 1994).

[Census 1995]. United States Census Bureau Statistical Abstract of the United States Government Printing Office (1995).

[Fargion 1996]. G. S. Fargion, R. Harberts, and J. G. Masek An Emerging Technology Becomes an Opportunity for EOS From the online file.

[Landauer 1986]. T. K. Landauer; “How much do people remember? Some estimates of the quantity of learned information in long-term memory,” Cognitive Science, 10 (4) pp. 477-493 (Oct-Dec 1986).

[Louis 1996 ]. Steve Louis Cooperative High-Performance Storage in the Accelerated Strategic Computing Initiative 5th NASA Goddard Conference on Mass Storage Systems and Technologies (Sept. 17-19, 1996 ). As reported by Ron Van Meter, .

[Markoff 1997]. John Markoff; “When Big Brother is a Librarian,” The New York Times pp. 3, sec. 4 (March 9, 1997).

[Mauldin 1995]. Matt Mauldin, “Measuring the Web with Lycos,” Third International World-Wide Web Conference, April 1995.

[Mills 1996]. Mike Mills; “Photo Opportunity,” Washington Post pp. H01 (January 28, 1996).

[Optitek]. The Need for Holographic Storage http://www.optitek.com/hdss_competition.htm.

[Radding 1990]. Alan Radding; “Putting data in its proper place,” Computerworld pp. 61 (August 13, 1990).

[Tenopir 1997]. Carol Tenopir, and Jeff Barry; “The Data Dealers,” Library Journal pp. 28-36 (May 15, 1997).

[UNESCO 1995]. UNESCO Statistical Yearbook Bernan Press (1995).

[Wells 1938]. H. G. Wells World Brain Methuen (1938).

Like most people, my search engine of choice is Google. It’s the world’s largest search engine, accessing over 8 million web pages. If Google can’t find what I’m looking for, it doesn’t exist, right?

Wrong. There’s a whole other internet out there that Google just can’t reach. It’s called the deep web, invisible web or hidden web. Invisible because most search engines can’t see it. And it’s about 500 times bigger than the internet we all know and love.

So, what’s in the invisible web? Contents of searchable databases such as library catalogues and article databases. There’s a wealth of information held in these databases that you’re missing out on by only using Google or other similar search engines.

The good news is that accessing the invisible web is easy. You can ‘google’ for databases on a particular subject by typing, for example, ‘plane crash database’ into the search bar. Once you’ve accessed the database itself, you can then interrogate it for the information you’re looking for.

There are also some dedicated tools out there that specialise in searching the invisible web. Try some of the following next time you come up emptyhanded with Google.

• Clusty: a metasearch engine ie it searches several search engines at once
• Librarians Internet Index: sites compiled by real librarians, not a computer
• Infomine: excellent for academic research
• Directory of Journal Articles: free access to full text articles from over 1,000 journals in the humanities/social sciences and science fields

They are important ‘creatures’ in the virtual world, the web spiders; more so in the search engine optimization and search engine marketing industry. What these crawlers do is recognize web pages existing in their database using a particular system, and include them in a web search.

Obtaining new pages, updating existing ones, and deleting obsolete pages are some of the functions they perform. If there has been no linking to the page, then another way to bring a web page to the notice of the crawlers is by sending the site URL to search engine companies and requesting them to include the new page into their search.

The next step is indexing, where the pages are sent to another computer program, where links, keywords, etc. play a major role in identifying the relevancy and the relation of a web page to the search. These are then featured in SERPs.

Some pages, however, do not come up in search engine result pages. They form that part of the web known as the ‘Invisible Web’ or ‘Deep Web’. In a study, the University of California, Berkeley, estimated that the ‘Deep Web’ contained approximately 91,000 terabytes of data and 550 billion individual documents.

One reason this happens is due to pages having random content or it could be irrelevant and badly conceptualized. Another reason is the presence of technical barriers where spiders cannot perform by themselves. For instance, pages where access is granted only by typing manually, member’s only sites, etc.

At Cosmos Creative Boston, our SEO team submits your website using accurate techniques that make it easy for algorithmic crawlers [spiders] to notice and provide links to and ultimately result in being picked up by a search engine quickly.

This U C Berkeley Website offers information about differences between the three main search engines: Google, Yahoo and Ask.com. But these major search engines generally function in the same way.

There are specialized and unique search engines, for example, RollYo and Mammathat eliminate duplication and make searching easier.  Dogpile adds another layer of efficiency as it brings all the search engines together on one convenient page.

Now there is what is called the Deepnet or the Invisible Web which needs attention when we talk about search engines. Search engines are only able to see certain documents on the Internet.  U C Berkeley and Wikipedia both have articles on the hidden web although a lot of the documents that used to be hidden are becoming more and more visible.

As I was saying, there is so much that other search engines are unable to see or crawl. A new search engine, Cuil pronouced to rhyme with cool, has come up claiming to can “search 121,617,892,992 web pages” a lot more websites than Google, the current top search engine. Check out this Cuil search engine.

What the heck is the surface web? It refers to the thin top layer of the Net that the major search engines are able to spider. So, when you’re working on the surface web, that is, searching only Google, or Yahoo, or MSN, you are only “scratching the surface” so to speak.

<A HREF=”http://searchthehiddenweb.com” TARGET=”_blank”>There’s a lot more available to you on the hidden web</A>… but first, are you sure you’re getting everything you can out of the surface web?

Here are some basic ways you can better search the Surface Web. If you’re doing market research for example,using something like Google TRends, it’s a good idea to start here anyway, since the vast number of online users are using the “big three”.

Once you learn some of these search techniques, you can apply them to the Hidden Web too. First, let’s examine some of the major search engines out there right now.

First is of course is Google. They are growing exponentially, every day indexing more pages, scanning in more documents, and generally sucking up everything they can on the Web. Google’s even trying to add ways to dig through the “deep web” too. (Stay tuned, we’ll keep you up to date on these new features.)

Next up are Yahoo! and MSN’s Live Search as the next biggest competitors. Following those three, there are many others, such as Ask!, Alta Vista, and others, however, Google, Yahoo and MSN Live have acquired and now are the power behind many of the search engines that started out early on the Web.

These three together account for the vast majority of searches conducted on the Web, and webmasters tailor their pages to show up in these results. But sometimes you will get different results for your search depending on which engine you use, and what search terms you use.

Next post, we’ll talk about some basic search terms.

<A HREF=”http://searchthehiddenweb.com” TARGET=”_self”>The “hidden web” contains the fastest growing storehouse of information on the Internet</A>. Think you’re getting everything you need when you search Google? We hear how massive it is but it is only about 1/10th of the full amount of data available, for free, on the Internet.

Google and Yahoo, so-called “surface web” engines, are big, yes, but their indexing software (called crawlers, spiders, bots, etc.) is unable to access the vast majority of websites and databases.

Some examples of pages these spiders can’t reach are: databases, dynamically created pages (using Java or other software), newspaper archives, any site that requires a login, certain file formats that are not index-friendly, and so on.

You may have heard of the “deep web” or the “hidden web” – these are the names given to the 99% of information that is available on the Internet but not easily found using the major search engines. It has by far the greatest amount of best quality information available.

And guess what? Luckily for the smart researcher, marketer, or product developer, nearly 95% of this content is accessible by the public, for free!

There is an amazing wealth of ideas here, for new hot products, for high-quality market research and analyses by academic experts, massive stores of government information, both national and international, and tools for finding detailed information on competitors – both their businesses and their personal information, scary as that may sound. And that’s just the beginning; there are materials available to the public for the taking, in the public domain, which you can use to create new and valuable content for your customers.

If you have an ounce of creativity, just try scanning these sources for one of your market niches, and you will begin to see the possibilities immediately. It’s an instant brainstorm!

With students back for first semester classes in our part of the world, Internet cafes are abuzz with students doing their literature search for assignments, term papers, or thesis review of literature. At the moment, the most popular search engine appears to be Google. As of June 14, 2008, the estimated size of Google’s index is about 20 billion web pages, making it the largest crawler-based search engine, based on reported numbers.

So you think that with an Internet search engine like Google or Google Scholar, you’ve done a comprehensive review of all available information, besides those articles which are pay-per-view or for paid subscribers only. Think again. Studies have shown that the hidden web has as much as 500 billion web pages.

Search engines crawl only a small portion or the shallow part of the web. “Invisible web” or deep web refers to information available on the world wide web but is not accessible to general all-purpose search engines. Some materials hidden from the usual search engines include dynamic content, unlinked content, private web, and limited access content.

How to find the invisible web

To search the invisible web, here’s a list of some notable databases that we should check out (see Robert Lackie’s “Those Dark Hiding Places: Invisible Web Revealed, Wendy Boswell):

• Librarians’ Internet Index – websites you can trust
• FindLaw – “The highest-trafficked legal Web site”
• About.com
• Direct Search site put together by Gary Price
• Invisible Web Directory -put together by Gary Price and search guru Chris Sherman. This site is a directory of searchable databases, organized by subject
• Resource Discovery Network – has resources mostly from the United Kingdom, and is extremely well-organized and very searchable
• InfoMine – an incredible resource that at last count included over 100,000 links and access to hundreds, if not thousands, of databases
• Virtual Library
• Intute – a free online service providing access to the very best Web resources for education and research.
• Internet archive – a digital library of internet sites and other cultural artifacts in digital form.
• Beaucoup! – a search spot to help search the invisible web.
• Digital Librarian – a librarian’s choice of the best of the web.
• ScienceResearch.com – A portal allowing searchable access to numerous scientific journals and databases.
• Agricola Database – provides citations to agriculture literature.
• Energy Citations Database – provides free access to science research to over 2.3 million science research citations.
• Envirofacts – EPA’s one-stop source for environmental information.
• Plants Database provides standardized information about the vascular plants, mosses, liverworts, hornworts, and lichens of the U.S. and its territories.
• PlantFacts – an international knowledge bank and multimedia learning center on plants.
• Window to My Environment database – provides a wide range of federal, state, and local information about environmental conditions and features in an area of your choice.

This years AADLIS (Alumni Association of Dept. of Library and Information Science, Univ. of Kerala) was on 24th of May, 2008.

Although the annual meeting is the only activity of the Association fo the whole year, they have been doing a good work by honouring the old BLISc batch students and PhD and MLISc rank holders. Some endovments are also instituted such as, one by Smt. K.K.Lalitha Bai.

The concept of Deep web (more suitable to be known as “invisible web”) is not clear to most of the professionals working in the field. More classes should be there.

The possibilities of harnessing the information or knowledge hidden in the  deep areas of the surface web (kown or currently searchable web) is enormous. Academic invisible web is a separate area of study.

It is true that, the lion’s share of authentic information such as in databases, catalogues, digital libraries , etc are only available in the deep web which are not indexed and hence not available to all (search engines).

More explorations must be there in the field.