Ho accennato nel precedente post al fatto che stanno nascendo numerosi motori di ricerca che si dicono semantici basati sulla linguistica computazionale che pretendono di migliorare sostanzialmente la ricerca di contenuti, rispetto alla ricerca per parole chiave che utilizza criteri statistici o di popolarità.

Oltre ad Hakia ho preso in esame anche Lexxe, apparentemente un efficace motore di ricerca che si propone di  rispondere a domande formulate in linguaggio naturale e non solo con parole chiave. La caratteristica più marcata di Lexxe è quella di presentare accanto ai risultati di una ricerca, categorizzazioni o raggruppamenti ad hoc che servono a specificare o filtrare i risultati per ottenere maggior precisione. Tali raggruppamenti non sono precostituiti al momento della presentazione dei risultati, bensì si costruiscono on the fly se vengono cliccati. Il risultato di tale ricerca è una nuova lista di risultati ma anche il suggerimento di nuovi raggruppamenti da esplorare  (dinamic clustering) .

Si tratta di un modo efficace per esplorare un argomento da diversi punti di vista, attraverso la classificazione ontologica, che si basa non sul matching delle parole della query con quelle dell’indice del motore di ricerca, valutate poi in modo statistico quantitativo,  bensì sulla identificazione dei concetti rappresentati dalle parole della query e il loro confronto con una ontologia o mappa di concetti, cioè di parole dotate di relazioni tra di se, costruita in modo semiautomatico.

Di questa categoria fa parte anche Powerset un motore di ricerca semantico  recentemente acquistato da Microsoft per integrare i risultati del proprio motore di ricerca, Bing. La sua tecnologia risiede soprattutto sull’analisi sintattica tramite un software di parsing su licenza di Xerox Parc. Come spiega Lorenzo Thione co-fondatore di Powerset, il software scompone il testo in frasi e fa

l’analisi logica e grammaticale della frase, scomponendola in soggetto, predicati verbali e complementi, e cercando di analizzare il ruolo di ciascun complemento nell’ambito del testo, come per esempio, individuare se un complemento e’ finale, causale o di moto a luogo. Una volta che il software ha fatto questa analisi, delle componenti semantiche (dette features, o caratteristiche) vengono estratte e inserite nell’algoritmo di selezione e ordinamento dei risultati, in armonia con le altre componenti meno linguistiche, come il PageRank o la prossimita’ delle parole chiave.

Thione  sostiene che lo spider di Powerset è in grado di indicizzare il web (naturalmente solo pagine in inglese) ma “per adesso l’indice di Powerset e’ ristretto a Wikipedia“  (ibidem) e al database di Freebase (basato sull’analisi di Wikipedia e su contenuto sociale).

Ad esempio la ricerca di “Al Gore” conduce ad una risposta univoca tratta da Freebase ma presenta anche le possibili triple:

Al Gore:

won                 district, country, vote, internet, primari, states, seat, nomination …

carried             disctrict, stated and Nashville

received          percent, vote, primetime Emmy Avard, a number, electors …..

Inoltre gli articoli di Wikipedia relativi alla ricerca e rimaneggiati da Powerset, presentano sulla destra un utile quadro riassuntivo del contenuto dell’articolo che rimane stabile allo scorrere dello schermo, che si suddivide in Overview (dati sintetici); outline (l’indice dell’articolo); images (le immagine pubblicate nell’articolo); locations (le mappe dei luoghi connessi al contenuto dell’articolo); Twitter (recent tweets di Al Gore). Powerset è quantomeno un ottimo motore di ricerca sui dati tratti fattuali da Wikipedia e di Freebase.

Cognition si basa come Hakia sulla classificazione ontologica ma, affermano i suoi ideatori, con una mappa semantica (cioè parole e concetti classificati) più comprensiva e quindi più funzionale. Inoltre sostiene di utilizzare anche l’analisi morfologia e di gestire i sinonimi. Non risulta tuttavia che a oggi il motore di ricerca abbia suscitato particolare interesse (maggiori informazioni qui)

In fase di declino dell’efficacia dell’algoritmo di Google, tra le tendenze di ricerca proposte per ovviare alla scarsa rilevanza dei risultati, figurano i motori di ricerca semantici capaci cioè di distinguere e riconoscere i significati. L’idea non è nuova dato che sottende la ricerca sul semantic web condotta del w3c, che consiste, in estrema sintesi, nell’introdurre strutture semantiche fin dalla costruzione delle pagine web. Questo approccio, come è noto, è considerato complessivamente irrealistico perché richiede l’accettazione estesa di standard condivisi e la padronanza delle complesse regole della semantica. Una via alternativa è invece quella di spostare l’analisi semantica negli algoritmi dei motori di ricerca rendendo interpretabile ogni testo: “We need algorithms that match the meaning of concepts (rather than mere words) and emulate “understanding.”” (Riza C. Berkan, Founder & CEO, hakia.com) () Questo è il percorso intrapreso dai cosiddetti motori di ricerca semantici quali Hakia, Powerset, Cognition search, Lexxe che si avvale di un nuovo approccio delle tecniche di Natural Language Processing.

Hakia si presenta come alternativa ai motori di ricerca tradizionali e mira a sostituire al criterio quantitativo quello qualitativo: “a popular Web site may not always be credible, and a credible Web site may not always be popular”. Inoltre i motori di ricerca basati sulla popolarità richiedono una costruzione della query con poche, semplici parole. Ciò significa che le query più lunghe e complesse che rispecchiano spesso bisogni informativi specifici o che richiedono risposte in tempo reale rispetto agli eventi cercati, non vengono soddisfatte.

Tramite la analisi della query (meglio se formulata con frasi che non solo con  parole chiave) l’algoritmo di Hakia sarebbe invece in grado di trovare corrispondenze col contenuto delle pagine web e non solo sulle frasi chiave che esse condividono o le statistiche di popolarità. Questi risultati sono ottenuti implementando l’ontologia semantica (OntoSem) che traduce in modo automatico ogni frase che le viene sottoposta, in Text meaning representation (TMR). A questo scopo utilizza complesse procedure di analisi linguistica del testo pieno che mirano a identificarne i componenti grammaticali e sintattici e i concetti contenuti e a collocarli in un contesto semantico di sinonimi e concetti correlati, costruito attraverso l’analisi di migliaia di pagine di testo pieno. In tal modo si emula il processo mentale di un umano che legge un testo. Queste metodologie sono poi utilizzate insieme a tecniche non semantiche per essere implementate nel motore di ricerca (Hempelmann, , Christian F. Hakia inc.).

Attraverso l’algoritmo QDEX (Quality Detection and Extraction) i ricercatori di Hakia sarebbero riusciti a implementare procedure veloci ed efficienti per indicizzare le risorse di rete. Sembra anche di capire che il primo passo è stato creare la struttura semantica dei TMRs analizzando siti di provata qualità come benchmarck per la verifica dei concetti. Il risultato sono una organizzazione dei risultati per categorie, cioè per contesti semantici.

I risultati delle ricerche sono presentati non come liste standard bensì per schede (tabs) dal nome:  All results - Credible sites – News – Images – Meet Others a suggerire i diversi approcci, uno generalista, uno limitato a siti scelti da bibliotecari o specialisti dell’informazione (ma sembra che non vi sia alcuna vera barriera contro lo spam e tentativi di influenzare i risultati), uno dedicato alle news e l’ultimo è una sorta di forum, suddiviso in diverse stanze di conversazione, in cui gli utenti possono articolare conversazioni stabilire contatti, esprimere valutazioni e scambiarsi informazioni.

L’analisi semantica permette l’implementazione di una serie di nuove funzionalità, quali:

evidenziazione di frasi coerenti: i risultati della ricerca riportano selezionata nella descrizione del sito la frase semanticamente corrispondente alla query.

la categorizzazione: per richieste formulate in modo semplice e popolare (p. e. cancer oppure Madonna), hakia presenta i risultati organizzati in categorie, costruendo aggregazioni chiamate Galleries composte da circa 10 categorie, corrispondenti, secondo gli ideatori ad altrettante queries più specifiche nei motori di ricerca convenzionali.

Parallelismo: è una innovativa funzionalità che rinvia all’uso di termini equivalenti o associati nella ricerca (ad esempio “kill=murder”).

Generalizzazione: in molte domande è necessario applicare forme di generalizzazione: ad esempio una ricerca con il termine car può dover essere riconosciuto come termine generale per le singole marche che devono quindi essere contenute nei risultati della ricerca

Nella Home page di Hakia viene proposto un confronto con google. La lista dei risultati della stessa query suggerita (naval academy) in Hakia (mi limito alla scheda All results per Hakia) è organizzata in  categorie (p. e.; college profile, Headline news, Admissions, Athletics, Applying for Financial Aid, Faculty Directory ….etc) mentre in google i risultati sono correlati ai “rich snippets” che raggruppano i dati per formato (videos, news, blogs, forums), cronologicamente (risultati recenti, un’ora fa …), per pagine visitate o no, secondo visualizzazioni particolari (Standard, Related Searches, Wonder Wheel, timeline), e infine: risultati standard oppure con immagini tratte dalla pagina, con più o meno siti di shopping, con la preview. Il confronto dovrebbe evidenziare il valore semantico della struttura categoriale proposta da Hakia per organizzare i risultati, rispetto alla suddivisione più occasionale dei “rich snippet” (Per uno studio sulle performance di Hakia in confronto con Yahoo, Google e Msn che tuttavia non presenterebbero grandi diversità, si veda qui),

Leo un artículo en barrapunto acerca de que Bing (el buscador que reemplazó a Windows Live Search) ha “perdido cuota del mercado de buscadores (mi pregunta personal fue: ¿cuando tuvo cuota?), en fín, recordé la lucha entre Google y Powerset (el buscador semántico del cual hice un review el año pasado) y se me ocurrió hacer una rápida revisión.

Lo básico:

Google nació para lo que és!, Tarda 990 ms en cargar en mi conexión (sin caché habilitada, con la sesión iGoogle que carga más cosas) y solamente pesa 36kb.

Bing por el contrario tarda en mi conexión 2.74 segundos en cargar, posee 92Kb en archivos y hace 11 solicitudes HTTP, demasiado para algo que debería ser un simple “buscador”. (y ojo!, no tengo Silverlight, que si lo huele, se baja más cosas!).

Sobreviviendo a Powerset:

Recordemos el hecho que Powerset es un buscador por semántica, analiza la frase como un todo y obtiene los resultados que deseas, recordé el ejemplo anterior que utilicé y volví a realizar la búsqueda:

Stanley Kubrick a la carta …

La búsqueda a realizar es “a movie with twins tricycle hotel” (recordando la famosa escena de las gemelas, el chico y el triciclo en el Hotel de “El Resplandor”), según Powerset:

http://www.powerset.com/explore/go/a-movie-with-twins-tricycle-hotel

Este hace un “mashup” interesante de los artículos de la Wikipedia, mostrando información de como Stanley Kubrick ha sido “imitado” en la cultura popular (como repetir esta escena en “Family Guy”); si mejoramos un la frase:

“a terror movie with twins, a tricycle in hotel”

Simplemente su incapacidad de salir de la Wikipedia (es lo único que han logrado indexar) lo hace responder menos cosas aún y solamente mostrar un artículo de la Wikipedia sobre “Stanley Kubrick en la cultura popular”.

Bing:

Se “supone” que Microsoft compró parte de la tecnología semántica de Powerset y más el “tomar prestado” búsquedas del motor de Google, de Yahoo y de Ask.com (Bing es más un meta-buscador que un buscador-indexador en sí); pues debería funcionar muchisimo mejor:

Como vemos; esta “capacidad” le permite a Bing responder a la pregunta 1:

http://www.bing.com/search?q=a+movie+with+twins+tricycle+hotel&go=&form=QBLH&qs=n

Con el video  YouTube – The Shining hallway scene.

Sin embargo, la búsqueda solamente posee 36 mil resultados; a la segunda pregunta:

http://www.bing.com/search?q=a+terror+movie+with+twins%2C+a+tricycle+in+hotel&go=&form=QBLH&qs=n

La respuesta “carece” del video de Youtube y agrega el registro de IMDB como primera opción.

Lo peor: 7 de los 10 resultados agregados a la “potente tecnología Powerset comprada por Microsoft” son sitios de venta de películas, dejando de lado sitios de interés general por lo comercial (algo que obviamente Microsoft sabe hacer muy bien, vender su posición como dueño de Microsoft Windows y de IE que te obliga a usar Bing como opción por defecto).

Google:

Google obviamente sigue siendo el líder indiscutible, no solamente por demostrar que puede tener en catálogo 336 mil búsquedas (obviamente 10 veces más que Bing); sino que además en ambas búsquedas:

http://www.google.com/search?hl=en&safe=off&ei=d8rLSsyTA4i-lAfH9bHeBQ&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=a+horror+movie+with+twins,+a+tricycle+in+hotel&spell=1

La resolución de la misma fué bastante más rápida que la tecnología powerset y la información retornada es más “informativa” que comercial; incluyendo artículos como análisis de la película y el libro de Stephen King y un par de reviews, solo 1 es un enlace comercial; se extraña la ausencia del artículo oficial de la Wikipedia (que Bing si la muestra, gracias a la colaboración del mashup de Powerset); pero se agradece con creces el “preview” del video que Google siempre acompaña a la izquierda de todas sus entradas Youtube.

Google Preview de Youtube en las busquedas

Nota: Desconozco si Bing tendrá esa opción en la versión “Windows”, ya que Bing usa SilverLight y yo obviamente no puedo (uso Linux) instalar Silverlight.

Búsqueda clásica de Bing (no-preview)

también se agradece además que la búsqueda anterior:

http://www.google.co.ve/search?hl=es&q=a+movie+with+twins+tricycle+hotel&btnG=Buscar&meta=

Sea personalizada por su capacidad “multi-idioma” (y no solo inglés) de Google, con lo cual yo aparezco como quinta opción de la búsqueda en Google Venezuela.-

Capacidades multi-idioma:

Recordando el hecho que Powerset aún no entiende semántica en español, pero por ser una corporación Mundial Microsoft “debería” comprar esa tecnología, hice una prueba en versiones “traducidas” de las búsquedas, a ver que pasaba:

Bing:

Tuve que ponerme “manualmente” en un país con idioma español (no se encuentra Venezuela) y además “hacer logon” con un usuario de Windows Live, a partir de esa “molesta” “conversión” a búsquedas en español (que Google hace años que las hace automáticas):

La búsqueda:

“Una pelicula de horror de gemelas, un triciclo en un hotel”

Arroja 193 resultados en Bing; de los primeros 10, solo uno es relativo a lo que estoy preguntando, por lo que la “semántica” de powerset se la ha pasado por el forro e hizo una búsqueda simple por palabras clave, “tokenizando” la frase y buscando palabra por palabra (que hombre!, Altavista hacia eso hace ya 10 años y no tenía tantos millones de dólares detrás como Bing).

Google:

Google me ha “redireccionado” desde mi navegador (en inglés por cierto, porque uso Swiftfox y no Firefox oficial) a la versión “Venezolana” de Google (www.google.co.ve) y al realizar la búsqueda; esto ha encontrado:

http://www.google.co.ve/search?hl=es&safe=off&ei=XM3LSvG6KomtlAfsiZHRBQ&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=Una+pelicula+de+terror+de+gemelas,+un+triciclo+en+un+hotel&spell=1

No solamente ha encontrado 817 búsquedas coincidentes, y aunque también ha “tokenizado” la frase, obviamente ha podido relacionar “mejor” las frases y logrado mostrar un increíble “10 de 10″ coincidencias verdaderamente relacionadas con la película “El resplandor” de Stanley Kubrick.-

Widgets:

Bing y Google soportan ambos calculadora inline en el buscador; escribes cosas como:

(2^2)+4

Y obtendrás “8″ como resultado.

Sin embargo, Google soporta “conversión directa entre medidas” y mientras Bing a esto:

10 inches in cm

Me enviará a sitios relacionados con conversión de medidas, Google me devolverá:

10 inches = 25.4 centimeters

Que se supone era lo que estaba inicialmente buscando.

Buscando imágenes ando yo …

Se supone que Microsoft ha comprado un par de empresas de donde obtuvo uno de los “mejores algorítmos de indexación y búsqueda de imágenes”; es menester aclarar que en la búsqueda en español anterior, para imágenes Bing no ha arrojado absolutamente nada, pero Google ha mostrado esto:

http://images.google.co.ve/images?hl=es&safe=off&resnum=0&q=Una%20pelicula%20de%20terror%20de%20gemelas%2C%20un%20triciclo%20en%20un%20hotel&um=1&ie=UTF-8&sa=N&tab=wi

Incluso la escena oficial de las gemelas está ahi! …

Buscando complejidad en las imágenes:

La búsqueda se refirió a:

“creepy twins hotel shining”

Bing:

La búsqueda me permitió además agregar “que fueran solamente imágenes en color”; por lo que la búsqueda terminó así:

http://www.bing.com/images/search?q=creepy+twins+hotel+shining+filterui%3acolor-color&scope=images&filt=all&qpvt=creepy+twins+hotel+shining&FORM=R5FD7

Perfecto, son solo 16 imágenes, todas en color, ahí están las gemelitas y WTF!, ¿qué rayos hace Michael Jackson ahi? …

¬¬ (creo que me han espantado!!).

Google:

http://images.google.co.ve/images?hl=es&source=hp&q=creepy+twins+hotel+shining&btnG=Buscar+im%C3%A1genes&gbv=2&aq=f&oq=

Qué bueno que Michael Jackson no está en ningún lado!, la búsqueda ha devuelto 301 mil coincidencias!, lo más interesante es que en Bing solamente 4 de 16 son relativas EXACTAMENTE a la escena de las gemelitas, mientras en Google 13 de las 20 iniciales son EXACTAMENTE referidas a las gemelitas …

Y como va eso del color? …

Claro que el filtro “Color y Blanco y negro” funciona para las búsquedas en Google como en Bing, pero google agrega algo más, ¿Alguien recuerda el color del vestido de las gemelitas en El Resplandor? … azul verdad?

Adoro ver lo maravillo que es decir “quiero solo las imágenes que contengan azul” y obtener esto:

http://images.google.co.ve/images?hl=es&source=hp&q=creepy+twins+hotel+shining&btnG=Buscar+im%C3%A1genes&gbv=2&aq=f&oq=#start=0&imgc=specific&imgcolor=blue

Lo más interesante es que si pido “Rojo”:

http://images.google.co.ve/images?hl=es&source=hp&q=creepy+twins+hotel+shining&btnG=Buscar+im%C3%A1genes&gbv=2&aq=f&oq=#start=0&imgtbs=cl&imgc=specific&imgcolor=red

Sale la escena de sangre y las puertas de los ascensores! (recuerdan su color no?).

La pantalla de hoy es “capibara!” …

Me gusta esa idea de poner una gran foto de inicio en Bing (hoy me ha tocado un Capibara, Chigüire para los Venezolanos), haciendo honor al animalito, buscamos “Capibara”:

http://www.bing.com/images/search?q=Capibara++filterui%3acolor-color+filterui%3aphoto-photo&scope=images&filt=all&qpvt=Capibara&FORM=R5FD10

La opciones son: Solo en Color: y que solo sean fotos; vean que devuelve 4800 resultados; me sorprende notablemente que:

1. La primera opción o alguna de las primeras, no sea “precisamente” la imágen que tienen de fondo de pantalla en Bing.
2. En las primeras 20 búsquedas iniciales, se les han colado 2 imágenes (ilustraciones) que no son precisamente fotos.

Pues este es el “fuerte” de Bing cuando nació, búsqueda de imágenes.

Google:

La búsqueda exacta en Google de Capibara, retorna 17400 resultados; y aunque tambien retorna el “dibujo” del capibara que retorna Bing, no comete el cachazo de ponerlo entre “los primeros 20″ y no lo verás sino hasta después de ver 80 fotografías.

Bing solamente filtrará entre “Color” y “Blanco y Negro”, ¿qué pasará si le digo a Google que me busque Capibaras, pero solamente imágenes que contengan blanco?, ¿qué creen?, pues no sé como hizo pero las dos primeras opciones son ¡cráneos de Capibaras! blanquitos blanquitos!.

interesante, ¿no?.

Conclusiones

No me extraña que Bing pierda mercado (¿alguna vez lo tuvo?), la gente se vió obligada al actualizar a tener a Bing como buscador por defecto (además que los 100 mil empleados de Microsoft a nivel mundial están obligados a usarlo), pero la gente volverá a google porque siempre será (y por mucho tiempo) un excelente buscador, conciso, preciso, sin tanta “basura” comercial.

En cambio, a Bing le ha ocurrido lo mismo que a Windows Vista, prometieron tantas cosas y solo han hecho un par de “cosillas” que ya todo el mundo hace o hacía y que solamente los más fieles seguidores de Microsoft (y alguno que otro loco) podrían decir que “esto es mejor que Google”.

Y como para terminar de “desmentir” esas cosas de que “Bing” es mejor que google; por qué no nos hacemos una pregunta interesante?

¿Qué disco, de Pink Floyd, fue editado “sin tener en cuenta a Roger Waters”, de la época del disco “The Final Cut” y que contiene como Productor a Bob Ezrin?

+”Pink Floyd” -”Roger Waters” AND “The Final Cut” +”Producer Bob Ezrin”.

Google muestra magníficamente y como primera opción, la portada del disco “A momentary Lapse of Reason”, Disco de Pink Floyd con David Gilmour de 1987.

Quisiera que Bing hubiera devuelto algo, pero no retornó nada …

Aun quitando el productor, Bing mostrará portadas del dico “The Final Cut” (donde si estaba Roger Waters) y no del disco que andamos buscando.

Es que uno va a lo que va!, Buscar!, para las demás “Chorradas” existe Facebook!

Check out this follow-up story to the feature I wrote on super-angels back in May.

Mint.com: Nurtured by Super-Angel VCs
Intuit is acquiring a startup conceived by a “25-year-old kid” and funded by First Round Capital, a new breed of high-risk, early-stage venture capitalists

By Spencer E. Ante

Intuit’s purchase of Mint.com was a big win for Mint CEO Aaron Patzer and the rest of the 38-person staff he assembled to run the personal finance Web site. Yet the $170 million acquisition also vindicated a new breed of early-stage investor that is betting aggressively on startups while big-name venture capital firms conserve capital and shy away from risk. These so-called super angels are trying to reinvigorate venture capital by taking it back to its roots, when firms were smaller, nimbler, and more adept at helping to build companies from the ground up.

Mint.com owes much of its success to one such investor, First Round Capital, which opted to back the fledgling company at a time when other VCs demurred. Indeed, the Mint.com acquisition is First Round Capital’s largest exit, beating out the $100 million sale of portfolio company Powerset to Microsoft (MSFT). And although First Round Capital would not quantify the return on its investment, co-founder Josh Kopelman says the Mint.com deal generated the highest return of any deal the firm has done. Previously its best return came when eBay (EBAY) acquired StumbleUpon for $75 million, which generated more than 14 times First Round Capital’s original investment. “I don’t think this changes our strategy,” Kopelman says. “It is continued validation for our approach.”

Read the rest of my BusinessWeek story here.

Bing’s First Month A Bust

The praise for Microsoft’s new search platform called Bing seems even and unanimous. In this video Paul Kedrosky calls it the triumph over low expectations and doubts if Bing will make any difference as it appears only incrementally better. He adds, products must be significantly better especially if the cost is zero. Search Wars Forget Bing Google Has More to Worry About from Twitter?

I agree, this argument is key to zero cost products, and especially key to new products taking on established ones. Some notable examples are Google Search taking on “85% search-market share” Yahoo just a few years ago, Google Maps taking on Mapquest.

Further more, size really doesn’t matter — by size, I mean size of the new product company. To support this argument consider, Google Finance introduced with serious buzz could not shake the big 3 finance portals or Google Video or Yahoo Video with all their”might could not shake up YouTube and in the end resorted to a buyout.

This analysis should encourage all budding garage ventures to aspire to take on any big established “zero cost” product. Cheers.

With Bing, Microsoft argues it is not only creating yet another search engine, but also helping users “accomplishing their tasks,” like buying a plane ticket or shop.

40% of users queries go unanswered

To do so, the computer giant’s latest “decision” engine employs semantic technologies to both try to make sense of the content of Web pages and the users’ queries.

“We use the term semantic technologies in a more loosely fashion than some would,” said Mark Johnson, Microsoft’s Senior Program Manager for Bing (and formerly at startup Powerset acquired by Microsoft) speaking this week at the Semantic Technology Conference in San Jose, Calif.

For Johnson, vertical search is what really matters for users, not a series of blue links or search results spit out by the search engine.

Microsoft plans to expand the number of verticals (to perhaps around a dozen) that it now offers with Bing (travel, shoping, news, maps).

Follows a video excerpt of our conversation with Johnson at the Semantic Technology Conference:

Lately, many people have been experimenting with Bing, Microsoft’s (s msft) new search engine offering, which we covered here. I agree with the many people who are noting improvements that need to arrive in Bing, such as blog searching and more varied search results for basic keywords. However, not everyone realizes that Bing is built on a powerful search engine technology from an open source-focused company that Microsoft acquired last year: Powerset. As I covered in this post, the Powerset technology underlying Bing introduces some powerful features that many people aren’t trying. You may find them useful.

Powerset originally got noticed in the search community because it leveraged Hadoop, which is an open-source software framework that allows clusters of computers to make very quick work of mining large data sets. (Hadoop also powers Yahoo’s (s yhoo) search engine.) Powerset’s idea was to leverage Hadoop to improve natural language searching, where you type in questions in sentence form instead of using keywords. If you’ve followed natural language searching,  you’ll know that it’s had a rocky, and generally unsuccessful, history.

Hadoop brought Powerset more speed at mining through the mountains of possibly relevant search returns that come back in natural language search engines. Powerset doesn’t underlie all of Bing, but this unique facility with natural language searching is in Bing, and you may find it useful if you haven’t tried it.

Here are some example queries to try at Bing.com, to get a sense for how it works:

• Which companies has Google acquired?
• Which Firefox extensions block online ads?
• What is the semantic web?
• How much was MySQL acquired for?

You’ll notice that many of the search results that come back in these types of searches at Bing are mined from Wikipedia, because Powerset has always specialized in Wikipedia search. Entries on Wikipedia aren’t always correct, so you have to take results with a grain of salt, but Microsoft has extended Bing’s facility with natural language search beyond just Wikipedia searches, and it can be quite good at providing quick answers to natural language questions.

What do you think of Bing’s natural language search capabilities?

While trying to think of good, quick interview questions I stumbled across power set (thanks to Mike DiStaula for suggesting it). After 10 minutes of hacking in Ruby I discovered that it’s not a reasonable replacement for our current starter question (it takes too long), but it’s also a fun problem to solve.

Then I wandered across this definition of power set in Haskell:

powerset = filterM (const [True, False])

Well, that was easy. So I set out to write that in Ruby. That is, I wanted to write this:

class Array
  def powerset
    self.filterM {|x| [true, false]}
  end
end

First thing I need is a list monad:

class Array
  def self.unit(x)
    [x]
  end

  def bind(&f)
    self.inject([]) do |acc, e|
      acc + f.call(e)
    end
  end
end

I quickly hit my weekly reminder that Ruby does not have Array#rest, so I added that:

class Array
  def rest
    r = self[1..-1]
    r.nil? ? [] : r
  end
end

Whew; now things are easier. Next I needed filterM, so I added it to Enumerable as a faithful copy from Haskell (expanding the do notation into the >>= notation):

module Enumerable
  def filterM(&p)
    return self.class.unit([]) if self.empty?
    p.call(self.first).bind do |flg|
      self.rest.filterM(&p).bind do |ys|
        self.class.unit(flg ? ([self.first] + ys) : ys)
      end
    end
  end
end

Now my original Array#powerset method works:

irb(main):013:0> [1,2,3].powerset
=> [[1, 2, 3], [1, 2], [1, 3], [1], [2, 3], [2], [3], []]

I expect you to write that quickly in your next interview.

Non avevamo ancora fattoa tempo a digerire le novità di WOLFRAM ALPHA che la MICROSOFT finalmente annuncia l’immininente arrivo di BING (che fino a qualche giorno fa si chiamava KUMO) il motore di ricerca destinato non solo a sostituire l’attuale LIVE SEARCH, ma che grazie alle innumerevoli novità (in testa il motore semantico POWERSET) si propone di recuperare il terreno perso in questi anni rispetto a GOOGLE.

As promised last week, Microsoft has trotted out its new search engine. It’s called Bing and you can find it here. PCMag has a great review and slideshow of the search engine here. True to its semantic genetics (including the Powerset search engine that Microsoft previously purchased and incorporated), Microsoft is calling Bing a “decision engine” rather than a search engine. Bing offers suggestions for related concepts and information, with answers to inquiries, rather than just links to other pages, offered right on the search results page. You can get even more information from suggested links when you hover over a result entry. You can get “deep” links, such as search boxes for FedEx or UPS tracking right on a Bing results page. Bing’s pages are not spare, but can include images of current events, landscapes or other “bling” (sorry – couldn’t resist ).

Rather than review each element of Bing and how it handles matters such as travel, shopping, images and video and news, I recommend you go try it yourself. Check out the PC Mag article linked above for a comprehensive list of features and the page slide show. While many web commenters are echoing the famous refrain “it’s not a Google killer”, Bing looks to be a promising hike along the evolutionary path away from the popular-by-number-of-links search option.

Related articles by Zemanta

• Ballmer says “search deserves a good feature war” as Bing nears release (downloadsquad.com)
• Bing – the full press release (thenextweb.com)
• Microsoft Wants You To ‘Verb Up’ And ‘Bing It’ (gawker.com)
• Bing: Microsoft Reveals its New Search Engine [Video] (mashable.com)
• Bing? – no need for Google to worry – just yet… (cheezedmg.com)
• Bing takes on Google (telegraph.co.uk)
• Just Bing It (marketing.fm)

Question Answering system is, as I said in the previous article, a method based on natural language, whose aim is to locate a piece of information within a large document.

As I’m not going to talk again about the same topic, I think is a good idea to give examples of Question-Answering systems, as I didn’t do it before, and it can be a good way to understand better the concept.

If we take a look on the internet, we will be able to find many different Question Answering systems. Let’s see some of them:

• Powerset: A system that wants people to interact with technology by making the computer understand our language. You can ask for information by making questions (“Who is the President of the USA?”), by using keywords (“Usa president”), phrases (“The president of the USA”)… This system will give you an answer immediately. Take a look at it here.
• Asimov the Chatbot: Asimov is a chatbot that learns from people. He is able to learn the facts that people tell him, so his answers are more clear everytime. It was first developed to just answer to questions, but you are able to mantain a conversation with him nowadays. Take a look at it here.
• Answers.com: It works thesame way as Powerset does. You can make questions, write keywords, phrases… this Question Answering system will find an answer for you. Take a look at it here.
• AnswerBus: It works as Powerset and Answers.com. Take a look at it here.
• Start: It also works as Powerset and most of the other Question Answering systems. Take a look at it here.
• Anna: Anna is an assistant of Ikea. She will help you through the Ikea website and she will answer to any doubt you have. Take a look at it here.

So, this are some great examples of what we call Question-Answering systems which may help you understand better what this system is and how it works.

References:

• Powerset. In Powerset.com. Retrieved 19:05, May 21, 2009, from: http://www.powerset.com/
• About Powerset. In Powerset.com. Retrieved 19:15, May 21, 2009, from: http://www.powerset.com/about/
• Asimov the Chatbot. In Chatbots.org. Retrieved 20:03, May 21, 2009, from: http://www.chatbots.org/chatterbot/asimov
• The START Natural Language Language Question Answering System. In Start webpage. Retrieved 20:14, May 21, 2009, from: http://start.csail.mit.edu/
• Question Answering Engine. In Answers.com. Retrieved 20:30, May 21, 2009, from: http://www.answers.com/bb/
• AnswerBus Question Answering system. In Answerbus.com. Retrieved 20:40, May 21, 2009, from: http://www.answerbus.com/index.shtml
• IKEA te ayuda. In Ikea.es. Retrieved 20:50, May 21, 2009, from: http://193.108.42.79/ikea-es/cgi-bin/ikea-es.cgi

Not to be outdown by the likes of Google and Wolfram Alpha, Microsoft appears to be unveiling its new search engine next week at the Wall Street Journal’s D: All Things D tech conference in Carlsbad, California, according to Resource Shelf.

The engine’s code name is Kumo, but it is really a rebranding of Microsoft Live Search conjoined with its new semantic ally, Powerset. And guess what? It is going to help us find more relevant results! The screenshots over at All Things Digital / Boomtown show a clean, spare look. PC World reports a three-column search results page with useful tools like related searches, a “single-session search history for quick backtracking”, and other related categories tied to your search inquiry. PC World uses an example of searching for a recording artist with results that include song lyrics, tickets, albums and the artist’s biography. Or searching for a product with results including images, reviews and product manuals.

Will Kumo stand or fall amidst the search stars? Not sure, but I can say this: more semantic competitors add up to us edging closer to a truly semantic on-line world! Kudos to Kumo!!!!

I can be a freak for organizing.  No, not organizing things (although I hate clutter); what I have in mind is organizing information.  I used to be a dynamite legal researcher, back in the dark days when legal research was done almost exclusively on paper, and heavily reliant on meticulously written indices (days when, I suspect, legal research was done much better).

So this info from the latest print edition of the Christian Science Monitor (May 3, 2009 issue, p. 35) caught my eye.  It seems a number of fun, quirky, and occasionally helpful internet search sites are making some waves.  

Just took three of them for a test drive.  Searched three topics on each one.   My topics were cherry pie recipes, the equal rights amendment, and digestive disorders (my stomach hurts, and, no, I have not been munching on cherries).

Far and above, the new, highly visual Viewzi is the best.  According to the article, Viewzi relies on Google results, but takes those results and offers a slew of new ways to organize, view, and manage the material.  The photo tag cloud option is getting kudos, but I found it confusing and a bit slow.  What I did like was the much more readable basic text results option, and the visual screen shot option.  Timelines also looked useful.

Mahalo, evidently an attempt to blend a web directory and a search site, seemed way too much ’still under construction’ to be of much use.   Powerset relies entirely on Wikipedia articles–I mean, come on.  And finally ChaCha is a FREE cell phone search service, which sounds helpful, although at the moment I can’t think of much use for it.  I guess that shows how much I rely on my cell.

When you need to kill a few minutes, try these sites out.  Or, far better, put your head to the task of coming up with a way to improve on Google.  God knows the search engine could use improvement, and–oh boy–think of the money to be made!

Fascinating post by Louis Gray, over louisgray.com, one of my favorite blogs about all things Web and social media-related. Apparently, Microsoft hosted a get-together Tuesday evening about semantic search engine Powerset and its incorporation into Microsoft’s Live Search. One of the topics discussed was how search engine reviewers perform their “craft” and how ineffective their process really is in gleaning an understanding of this complex endeavor. And the effect of this inefficient review might be to sink the better option.

This result seems particularly true as search engines become more complex in design and practice. To truly understand how effective a search engine might be, one needs to spend some time with the engine, put it through its paces and delve deep into the results. “Teaching” search engines to “think” like humans takes time, and recognizing when the engine “gets it right” should also take time.

As I have said before here, when search engine’s compete to grab out attention, we the researchers stand to win the grand prize. In Louis Gray’s words, for Microsoft, “building the better mousetrap” will only be half the battle in the war of the ‘engines. Can’t wait to see the “results.”

• Microsoft Bottoms Out in Search Race
• Microsoft to start testing ‘Kumo’ search service
• Microsoft to start testing ‘Kumo’ search service
• Microsoft’s search must begin in Redmond

Por IDG News Service/EUA

Publicada em 24 de março de 2009 às 15h54

Atualizada em 24 de março de 2009 às 16h01

Miami – Buscador inaugura 1ª ferramenta semântica em sua busca, oferecendo conceitos e termos relacionados ao usuário na lista de resultados.

This post was originally published at onlinejournalismblog.com.

Many of the services that are being developed as part of the ’semantic web’ are necessarily works in progress, but they all contribute to extending the success of this burgeoning area of technology. There are plenty more popping up all the time, but for the purposes of this post I have loosely grouped some prominent sites into specialities – social networking, search and browsing – before briefly explaining their uses.

BROWSING

OpenCalais is a way to tag people, places, facts and events in pre-existing content to increase its value and accessibility. It makes use of RDF to annotate content intelligently and automatically so that it can be used in more meaningful ways. Developed by Thomson Reuters, the service now has a preview tool that can take any document and provide a display of the results of tagging and linking the semantic data. It provides an immediate and useful example of the way the technology works and is fun to play around with. OpenCalais is also available as a WordPress plugin which uses the service for auto tagging posts and archives with the correct themes.

headup is a Firefox plugin that enables semantic capabilities within any web page. Extra data is displayed fully in context as the service just alerts the user with a ‘+’ symbol when there is something else of interest to them. On encountering data about a band, headup might highlight the latest YouTube videos, tour dates and official blog-posts next to their name. This data can all be viewed without ever navigating away from the original page. Impressively headup’s semantic engine promises to provide a personalised service by retrieving information that specifically interests the individual user. Watch a demonstration video below.

SEARCH

SemantiFind claims to return more relevant results than traditional search engines, yet users can still continue using them as it is compatible with Google, Yahoo and Microsoft Live Search. You have to download and install a free browser plug-in, but SemantiFind results are displayed alongside normal search engine results, offering some familiarity. You can watch a demonstration video here.

Powerset is among those services applying natural language processing to the web and Wikipedia already benefits from its approach. Powerset displays an interface alongside the Wiki itself so users can navigate quickly and seamlessly using the keywords, themes and sections which have been stripped out of the original article. You can watch a demonstration video here.

iGlue is a search engine that tries to identify and manage entities, not keywords. The service finds relevant information even if the given element appears in a form different from that used in the original search. It understands that corresponding words can sometimes be substituted. You can see a demonstration of the technology here.

SOCIAL NETWORKING

Twine seems to be the pre-eminent ’semantic web’ service when it comes to social networking. It acts as a means of collecting and sharing all kinds of online content, learning more about you as you fill it up and link to other content. Twine aims to build on the principles of developing communities of interest. You can even interact with Nova Spinick, the site’s creator, and see what things have captured his attention.

It is clear that all of the services - whether targeted at browsing, search or social networking - foster more advances in the field and my final post on the ’semantic web’ explores the revolutionary uses of these new technologys for the benefit of journalism. My previous post called ‘The next step to the ’semantic web” can be found here.

Microsoft parece que una vez más quiere intentar desbancar a Google como buscador número uno, esta vez el intento va a ser a través de Kumo.com, su nuevo buscador. Algunos medios indican que es un lavado de cara de su actual Live.com, cuya marca engloba Hotmail, Messenger o Xbox Live, pero parece ser que quiere ir más allá y ser el primer buscador semántico a gran nivel basado en la tecnología de Powerset, la compañía que adquirió recientemente y que Internet había proclamado como el posible “mata gigantes”, eso si, de momento sólo sirve para la Wikipedia.

Hasta el momento la página no muestra contenido y el buscador sólo está accesible para una serie de empleados que están testando el producto.

Esperemos que está vez si que se plante cara a Google, no como otros intentos (Noxtrum, Quaero…) y pueda dar más opciones de búsqueda y elección a los usuarios.

Veremos en que queda, aunque de momento no lo veo…

Search is key to the popularity and success of the internet, this much is certain. Could you even imagine an internet without search? But as more and more content is added the task facing search engines become more and more difficult and they need to adapt.

On the surface search engines don’t look to have changed much over the years, inside they are rapidly evolving almost like a virus, tweaking it shape and structure rapidly to adapting to changing environment outside. Generally though they have been moving along the same old tracks, using vast databases to store details of information on the internet and complex maths to rank those results into some sort of order.

The idea of a semantic (meaningful) web and semantic searching has been around for a while. The exact definition of semantic of course changes from provider to provider, but its usually taken to mean getting clear, concise disambiguous information.

Most of the big providers have been chasing semantics like it was the holy grail. Google, Yahoo and Microsoft have all been racing to get something out first.

There seem to be a few common methods for approaching semantics; yahoo are asking content providers to include ‘specific semantic’ data within their pages that their search technology can pick up and work with. The other approach is to creating a database of objects like people, places, events then creating structured relationships between them. This approach currently both labour intensive and slow as there is not yet away to extract this for the web.

According to an article in read write web, Google who have dominated the search market for the past decade have already introduced some semantic technology into their search engine. If you ask “how old is Harrison ford” you will get the answer “Harrison Ford — Age: 66 years (born 13 July 1942) – according to http://www.netglimse.com/celebs/pages/harrison_ford/index.shtml” This currently only seems to work for a small number of searches, largely based around public personalities.

There are many other fantastic projects out there like twittersearch , trueknowledge, Haika, clusty, that work on everything from natural language searches and real time live searches to visual and clustered results.

One of the most exciting projects to be revealed recently is Stephen Wolfram’s Wolfram|Alpha. This takes an entirely different approach to knowledge management. Wolfram has created something he calls a ‘knowledge engine’ that will actually compute results using a huge system, with trillions of pieces of curated data and millions of lines of algorithms. It’s amazing that a project as advanced as the Wolfram project managed to remain so quite, for so long, with so many people working on so much data.

The site is due to go live in May 2009, Wolfram is quoted as having said “I wasn’t at all sure it was going to work. But I’m happy to say that with a mixture of many clever algorithms and heuristics, lots of linguistic discovery and linguistic curation, and what probably amount to some serious theoretical breakthroughs, we’re actually managing to make it work.”

Its obvious that we could be on the brink of a paradigm shift in search technology, but which way it will go, and how that will affect the way we use the internet is still to be determined.

Sources: Nova Spivack’s twine , Wolfram blog , Read Write Web blog , Smartsearching Presentation

References: Search Technology, Search history Privacy

What are your views? Have you had experiences of semantic search? What do you think is the future of Search? Do you use an alternate search engine? As always we’d love to hear your opinions leave a comment and share your thoughts!

Primeira grande mudança no Live Search após compra da PowerSet deverá ocorrer até o final do ano, com novo engine e reorganização visual.