Semantifi, Google, Microsoft, Kosmix, DeepPeep, DeepDyve, Socrata, Infochimps, Data.gov and many others are trying to search the Deep Web. Each has different technologies, tools, and most importantly some very different approaches. All their approaches can be broadly put into three categories.

1. Consolidate Datasets on the Web

Technically, this is not an approach to searching the Deep Web but these efforts are a valuable first step in organizing datasets and making them available online. They include various government repositories like Data.gov, SEC.gov, SFData.gov, DC Data Catalog, Infochimps.org, Socrata.com, etc. and are a significant force behind Deep Web Search. In the absence of data search capabilities these properties/websites remain as Data Catalogs only. Socrata goes one step further to allow users to browse datasets but browsing 1000s of rows can be challenging.

2. Look / Peek thru Web Forms or APIs to search Underlying Databases

This approach is driven by the belief that the Deep Web is simply the Web behind HTML forms when in fact it is a lot more than that. If you looking to buy a car, you might visit Edmunds.com and fill the search form by selecting the Manufacturer, Model, Price Range, Zip Code, etc. The information filled into the form is used to compose a database query which is then submitted to the one or more databases to present the results back to you as an HTML page. Because this page is created on demand current search engine can’t see the page.

Google’s approach to the Deep Web is to find HTML forms, send input to these forms, and index the resulting HTML pages. Google’s approach is fully automated, can easily scale and fits nicely with its crawl infrastructure. For further insights, read Alon’s VLDB paper published in 2008. Kosmix takes a similar approach of tapping into web forms as Google does but using API calls instead. DeepPeep follows a similar approach of tapping into web forms to search underlying databases.

While this approach offers some benefits, it has severe limitations in the scope of content that can be searched and level of analytics that can be conducted. Given the simplicity of this approach, it can be easily scaled to a large number of Web Forms or APIs.  Coming to the challenges: first, this approach leaves out a huge portion of the Deep Web comprising of datasets that do not have a Form or API in front of them as is the case with millions of government, finance, research, etc. datasets on the web; second, Forms & APIs offer only a limited window into the underlying databases hence allow only simple queries but not advanced analytics.

3. Make Datasets Searchable

Tim Berners-Lee explains in this TED video that the real value of data is in directly searching them. This he envisions will drive the next innovation on the web.

Semantifi is pursuing this vision. It uses pioneering technology to search databases. It can crawl any type of database including XML data sources, index multi-terabyte databases, search based on relevance, present automatic charts & tables. Then Semantifi is built like a marketplace for “Data Search Apps” on the lines of App Store where

• Anyone can publish “Data Search Apps”
• Users can search at Semantifi.com or on their website
• Apps can be shared publicly or within groups
• Apps can be FREE or FEE based
• Revenue is shared from commercial Apps

All three paths add value.  We are big fans of the Data Catalogs, Government Transparency Initiatives and others who are organizing datasets and making them available on the Web. Google, Kosmix, DeepPeep, etc. have demonstrated that we can get “answers to more search questions” by indirectly searching databases via the Forms / APIs. Semantifi is focusing the directly searching databases and has built initial Apps to demonstrate that anyone can build data search apps and that the Deep Web can be wired one dataset at a time by a community of Publishers.

According to Digvijay Lamba of Kosmix, “for search to be viable as a platform in the long-term, it must offer value to everyone involved: the platform creator, the application providers, and the users. The platform creator and application providers need to make money, obviously, while at the same time offering consumers high-quality, easily accessible content for free.  This can be achieved in one of three ways:  The Amazon Model, the eBay Model, or the Facebook model.”

Later he adds, “All three of these models have merit, and it remains to be seen which direction a search platform will take. One thing remains clear: as search evolves it will become harder and harder for one company to do it all. A platform that connects hundreds of search engines together can become a powerful source of innovation allowing it to build a deeper and richer experience for its users. The innovation in search has only just started. We need millions of connected search engines.”

Click here to see the full article.

We at Semantifi believe that this philosophy it is especially true of Deep Web Search which is orders of magnitude larger than the surface Web and hence no one company can search it all. Semantifi has just begun to search this vast Web by

1. building a platform like App Store by hosting its search technology online for anyone to use to build “Data Search Apps” for free
2. engaging a community of App Publishers, and
3. creating value for everyone involved: the platform owner, publishers and users.

With Semantifi as the platform owner, a diverse community of publishers and enough number of Apps covering most web verticals, the structured content can be wired together one dataset at a time making Deep Web Search a reality.

Campos de búsqueda :

- Kosmix -

El campo es muy sencillo y no tiene la opción de delimitar la búsqueda.

- Altavista -

El campo es my completo ya que te permite delimitar tu búsqueda a españa, o a todo el mundo, como se prefiera, además, cuenta con una búsqueda avanzada.

 Uso de los operadores lógicos (cuántos y cuáles):

- Kosmix -

Yo pude utilizar tres

Not, Or, And.

- Altavista-

Y, o ,  no.

Cantidad de documentos obtenidos:

- Kosmix -

33

- Altavista -

166,000

Calidad de los documentos obtenidos.:

- Kosmix -

Regular, ya que algunos documentos pertenecen a blogs y otros a paginas de televisoras, muchos de ellos no tenían nombre de autor, solo la fecha de publicación.

- Altavista-

De los que pude revisar, regular, ya que al igual que en Kosmix, existen artículos buenos y otros que solo tienen la fecha de publicación, por mencionar solo un criterio.

By crossing the billion downloads mark, Mozilla is well on its way to become the dominant Web browser on computer desktops.

But being the #1 Web browser does not appear to be the “be all, end all” goal for the Mountain View non-profit maker of Firefox.

“We don’t have an idea of what the end goal is but… we feel we are successful if the Internet feels vibrant, fun and exciting and still feel relevant to you as an individual.

An oddity at Mozilla: our mission is to bring choice and innovation so having dominant marketshare is not a success condition by definition… so being the monopoly browser is not in the cards for us… but being able to shape the future of the Internet is,” explains Mozilla’s Chief Innovation Officer Chris Beard during a panel discussion at the Lunch 2.0 event hosted today at Kosmix in Mountain View, Calif.

Spearheading Mozilla’s innovation programme, Beard’s mission is to bring more people/volunteers into Mozilla’s design and developing process to help the 200-people organisation build Internet technologies for desktops, mobile, etc.

“Our Mozilla Labs programme where we invite anybody who has an idea for how the Web or the Web experience could be better and to come and participate. From there, we then prototype, then incubate new ideas that potentially become product features or new capabilities for the Web.”

However, Beard was also quick to point out that despite everything that Mozilla does is open, “we do not do design by commitee, we do not vote on what the user interface should be, because that barely works. So we still need to exercise leadership and vision but also figure out how to engage our users.”

On the software patent issue, Beard had this to say: “there’s not a whole lot we can protect, or go after people for stealing. In fact, please come take our stuff, it’s all open source license!”

Here’s a video excerpt (more videos after the jump) where Chris Beard explains his role at Mozilla, as well the reasons behind developing Fennec, Mozilla’s future mobile browser expected to publicly launch in September.

In this video, Mozilla’s Chief Innovation Officer Chris Beard explains how the non-profit deals with a big competitor like Microsoft:

Beard on upcoming Fennec mobile browser due this Fall:

For Beard, fostering a community of developers is key to Mozilla’s and Firefox’s successes:

Sometimes too much experience is bad for innovation says Mozilla’s Chief Innovator:

Finally, why Beard does not believe in software patents:

Kosmix es un buscador que al buscar en el explora por la web y te reune todo hacerca de ese tema ofreciendote todo un catalogo de datos como definición, datos de wikipedia, blog, musica, videos, libros, páginas, consejos, etc.

Aun esta en base beta, pero aun asi lo convierte en uno de los mejores buscadores y lo recomiendo mucho.

║Enlace: Kosmix

Most of you probably know about the big players like Google, Yahoo and now Bing and you probably use them everyday. There are others out there though! Check them out and you may be surprised with their usefulness. Once engine I just stumbled upon is kosmix. So, what is different about this engine when comparing it to the big players? Several things.

There is not just a search box like Google or Bing. The search box is at the top of the screen and below it are groups(basically widgets) of related items such as videos, blogs, web,  etc. When you type a query, there results are displayed in those groups (web, videos, images, etc). The results list looks more like a social media site with information coming from mlive, meehive, digg, facebook, wikis, etc. Grouping the information by type is very helpful because there is more organization  instead of looking at a big results list or limiting the search through a tab. The group boxes can also be moved around the page so the user can change the look of the site. 

I found the “related in the kosmos” group to be very useful. A simple search of Lansing, Michigan will yield results in the “related in the kosmos” section for Ingham and Eaton Counties and sports and education in Lansing.

As you probably know I’m always looking at how companies in our space do their stuff.  A few days ago Ted Dziuba provided some interesting information on the Kosmix backend, which I found through Josh Young’s nice “What I’m Reading” feed:

…Kosmix… wrote its own data store in C++.  It’s basically a clone of Google’s GFS…

After I edited out the snark and obscenities, that was all that’s left of his post.  The rest of it, calling them a search engine, would only rankle the Kosmix co-founders.  Dziuba is an interesting read, and he has a good sense of humor.  But he’s never going to make it as a pro linebacker.  He’s just too small and weak.  He needs to spend less time at the keyboard and more at the gym, and get his hands on some steroids.

Today’s Indian Express carried an article about Kosmix, another search engine but with a revolutionary new idea. We all know that Google has become synonymous to web search that we often call it as googling. May be that verb should get added to the dictionaries.

But, as the article rightly points out, the outcome of a Google search is nothing but a list of blue links, sorted by its PageRank mechanism. Thus, what we see as the result is just the top scoring web pages and not a mixture of relevant and different types of information. The reason is still most top web search engines consider the web to be made of pages, which is not entirely true.

Though we still access a web page to see a content or in original terms the information, the information is not just a bunch of text. And even information as a bunch of text is not of the same type. In this web2.0 era, information is not just in web pages, it is in blogs, it is in Orkut and Facebook profiles and groups, it is in microblogging especially tweets, it is in images posted in Flickr and lot other places, it is in discussion forums, it is in Youtube and other video sharing sites, it is in iTunes and other music sharing sites, and a hell lot of other places and in a hell lot of formats.

For example, when we are searching about a person we just do not want to know his home page and wherever his name is mentioned. We might want to know his personal information, we might want to know his Facebook profile, we might want to know his digital albums, we might want to know the music he hears to or uploads, we might want to know the video he has just uploaded or a video about him or presented by him, we might want to know his last tweet, we might want to know lot of other things. Just giving them as a list of blue links in a page doesn’t help people anymore.

The search results need to be organized and categorized so that the audience can get a big screen picture and not the boring links to click one after another in an endeavor to find the information that one wants. This is what Kosmix tries to do. It organizes the information into news, video, books, blogs, tweets, photos…

For example, search for Manmohan Singh gives me the wiki article about him, videos of him and about him, links for information pertaining to him, images of him, discussion about him, questions asked about him,  news and blogs, and all you want to know about him neatly organized.

As the article mentions in the end, it is now upto Google, Yahoo and other search engine beasts to wake up and take the web search to the next era. We are sick of blue links..

The Wolfram Alpha, launched on May 15, 2009, claims to be a computational engine that has all the answers to your queries.  It has initiated much discussion on whether it will kill Google, taking over as the ultimate search engine, or will simply be a ‘nerdy’ assistant on someone’s science project.

The Atlantic carries an article on the useful features of Wolfram Alpha, and how it really provides answers while search engines like Google and Kosmix only claim to do so.  The writer says that Google only shows us the direction to pages that it thinks are relevant.

ZDNet wonders if Wolfram Alpha is a Wikipedia Killer, while SearchEngineLand.com says that WA is what they would call “Un-Google”. Pretty high praise, also seconded by ITProPortal.com and DailySEOblog.com.

Others are not so sure.  Epicenter says WA simply fails the cool test.  Ryan Singel describes it as “the nerdy kid the other kids only talk to when they need help with a physics exam…”!  I have to agree on that.  WA brings up only facts and figures, no matter what the search term is.  Google, on the other hand, brings up results from all spectrums, from facts and trivia to entertainment and Twitter updates.  If facts are all you want, we do have Wikipedia, which is the input of many people, whereas WA is one entity’s research.  Blogs.com says that Wolfram Alpha is a huge nerd, in spite of its winning computation moments.

Mashable’s post on Wolfram Alpha lists out 5 things that WA does better than Google: complex queries, localization, precision, calculation, and comparison.  I can only imagine a silver-haired science professor working on his post-doctoral, staring with admiration at Wolfram Alpha, lapping up the facts and figures it doles out.  I simply cannot picture me or any other Internet surfer looking for some normal useful info on Wolfram Alpha.

So, who is WA for? CNet News carries the comments of some users who say that WA is for ‘the tech crowd – the kind of people who want to dig into the data’.  Another user says he would not recommend it to normal users as WA is very picky about search terms and accuracy of words and spellings, which difficulty you would never face on Google (italics is mine).

In a lighter vein, one user says how he entered his own name into Wolfram Alpha, and the engine responded with a very curt “Wolfram Alpha isn’t sure what to do with your input”!  I received the same response when I searched for this ‘Ad Libitum’, whereas on Google I would have found my Twitter account, a post from this blog, and at least one other human being who has a blog with the same name.

In conclusion, go to Wolfram Alpha by all means, but only when you have some serious facts and figures to crunch, or when your science paper is due tomorrow.  For a more thorough search of the entire online world use Google, or if you want them arranged according to your interests, use Kosmix.  If you want a scholarly look at the topic, look up Wikipedia.

Wolfram Alpha has a long way to go if it wants to nudge out these big fellas…!

_______________________

Other sites I referred to:

Technology Review

Geeks are Sexy

LockerGnome.com

This past weekend more than 3,500 attendees flocked Silicon Valley’s Santa Clara convention center to attend the 16th edition of Tiecon, billed as the world’s largest conference for entrepreneurs.

And unlike its peers, Tiecon – which theme focused on the “The Bold Entrepreneur” – has done better this year than last, attracting about 20 percent more people according to the organizers.

The energy at the conference was high and networking opportunities abundant, with more than 150 speakers attending (from entrepreneurs to VCs) in more than 50 sessions.

“This is the year to innovate and start a company. The current macro-economic environment is a unique opportunity for the bold entrepreneur,” said the conference c0-chairs, Shaukat Shamim and Gara Gauba.

For the first time this year, Tiecon organisers honored their top 50 “hottest” emerging startups in 5 segments (Tie50), out of 1,200 nominated and after 50,000 votes!

Featured Topic. Kosmix. Wikipedia. Star Trek. Star Trek fever has reached beyond Trekkies and into the ….. Kosmix Sites. Kosmix » � RightHealth » � MeeHive ». About. About Kosmix » � Contact Us » � Our Team » � Jobs » � Media » � FAQ …[More..]


Kosmix, the universal search engine that dynamically generates guides to search queries using dozens of different content sources, is quickly gaining momentum. According …[More..]

Techno broadcast hosted by Miss Kosmix live from London This edition of Bluster presents a newly welcomed Austrian duo whom will take you by storm with their extreme and celebrated liveset. FLOTEK