Lucene 2.4.1 provides a convenient method for you to update a Document in your Index, namely the updateDocument method of IndexWriter (shown below) but what do you do if you want to update the Fields of an existing document?

public void updateDocument(Term term, Document doc)
                    throws CorruptIndexException, IOException

Lucene’s updateDocument operation is basically delete and insert wrapped into a single function. All documents matching the Term parameter are deleted from the Lucene index and the supplied Document instance is then inserted into the index. While Lucene allows multiple copies of the same document to exist in the index, the behavior of the update operation does not insert a copy of the supplied document for every match. In other words, if your Term matches 5 documents in the index then 5 documents are deleted and a single document is inserted in its place.

As you can see, it is a very good idea for you to design your documents so they have a field that uniquely identifies them in the entire index. In the database world, this is called a primary key field.

At times, it is helpful to think of the Lucene index as a database having a single table and the Documents as rows. It is a good analogy when you frame it in terms of searching. Boolean Queries seem to fit this concept nicely.

However, there are many differences between a Lucene index and a database.

• Lucene does not provide a way to enforce field uniqueness. It is up to you to achieve the concept.
• Lucene does not require a predefined document schema for the documents in the index. This means all documents in the index do not need to have the same number of fields or use the same field names. As an example, some documents can have the fields (id, url, contents) and other documents can have the fields (productid, manufacturer, summary, review).
• Fields can be repeated in a document. For example, a document can have 3 product review fields (productid, manufacturer, summary, review, review, review). We will revist this later in the code example.

Lucene’s updateDocument method overwrites the document(s) matching the given Term with the given Document. This is a problem if you only want to update a few fields and keep the remainder.

In the scenario pictured below, you can uniquely identify a document in the index whose author field you would like to update. So you then call updateDocument and pass in the Term and a Document instance populated with the new author field value. The result is an updated author field and the loss of the 3 other fields previously stored – the title, publisher, and contents fields.

What do you do when you need to update a subset of the fields in a document but cannot re-create the remaining fields? There can be many reasons for this dilemma. Perhaps you are unable to re-create the fields because the original text is not available or perhaps the operations to re-create the fields are very costly.

One approach to resolve this dilemma is to search for the current document in the index, change the desired fields, and use the modified document as the input to the updateDocument call. This idea is illustrated below. UpdateUtil.java contains the full source.

int docId = hits.scoreDocs[0].doc;

//retrieve the old document
Document doc = searcher.doc(docId);

List<Field> replacementFields = updateDoc.getFields();
for (Field field : replacementFields) {
	String name = field.name();
	String currentValue = doc.get(name);
	if (currentValue != null) {
		//replacement field value

		//remove all occurrences of the old field
		doc.removeFields(name);

		//insert the replacement
		doc.add(field);
	} else {
		//new field
		doc.add(field);
	}
}

//write the old document to the index with the modifications
writer.updateDocument(term, doc);

Here we pass in a Document that can have both replacement fields and additional fields for the document identified by a search using the term parameter as the basis for a TermQuery.. First we obtain the list of Fields from the document parameter. If the matched document already has a field by that name, it is considered a replacement otherwise it is a new field to be added to the document.

Notice the method first removes all fields in the Document having the same name as the replacement prior to inserting the replacement field. As mentioned earlier, a Lucene document can have multiple fields with the same name.

Without the remove call, you would be adding another value for the field instead of replacing the existing value.

A great tool to view what is actually in your Lucene index is Luke, the Lucene Index Toolbox. It is very helpful tool to answer “what if” questions when you read the Lucene API.

Out of the box, Lucene does not provide a way to update the individual fields of a document in the index. However, it is relatively easy to achieve this functionality by grouping together the available API calls.

You can browse the full source at google code and download a copy of the entire project via svn.
svn checkout http://hrycan-blog.googlecode.com/svn/trunk/lucene-highlight/

Here’s a small tip I found out while troubleshooting why my JVM was exiting in the middle of optimizing my index.

I created a utility class to create and close FullTextSessions for me that also created and committed a transaction at the same time for convenience throughout my code. But this meant that the optimize() method returned while Lucene was still working, thus leaving my index locked even after the JVM exited:

FullTextSession s = Search.getFullTextSession(factory.openSession());
s.beginTransaction();
s.getSearchFactory().optimize();
s.getTransaction().commit();
s.close();

To get the method to block until the optimization work is completed, do not create a transaction:

FullTextSession s = Search.getFullTextSession(factory.openSession());
s.getSearchFactory().optimize();
s.close();

Apache Mahout

For the past two years, I have been working with this amazing bunch of people whilst, being paid by Google in their summer of code program in a project called Mahout. And like the name says, it is trying to tame the young beast known as Hadoop. I have received a lot from the community. Being part of the project, I have got some real exposure to Java, data mining, machine learning and hands on experience over distributed systems like Hadoop, Hbase, Pig.  The project is still in its infancy, but, its ambitions are high in the sky. I am happy to announce the second release of the project, and proud to be a part of it. I hope people will adapt it in their projects and that it becomes the defacto standard machine learning library the way lucene and hadoop has become in their respective focus areas.

If you are already excited and want to take it for a ride, read Grant’s article on IBM developerworks here
The release announcement below

Salve a tutti, in questi giorni mi sono addentrato ancora nel mondo del continuous integration perchè sto lavorando su un progetto in Java, gestito in outsourcing, dove aiuto un gruppo di sviluppatori ad utilizzare strumenti di sviluppo che facilitano il lavoro quali un sistema di continuos integration.

Sono partito da un progetto con subversion installato su una macchina e so che nel progetto usano Jboss con JDK 1.5 (ancora non so perchè)

Su un’altra macchina remota vado e installo JDK 1.5.0.22, Ant 1.7.1, Maven 2.2.1 (fino ad ora (2 mesi di progetto) hanno usato solo Ant e vorrebbero passare a Maven), Jboss 5.1.0, Webclient for SVN 3.1.0 e Artifactory 2.1.2.

Piccola nota: volevo usare Sventon ma l’ultima versione aveva problemi con le librerie e le precedenti mi davano anch’esse problemi.

Non sapete cosa è Artifactory ??? Allora è possibile che siete ancora freschi su Maven (non è che io non lo sia )

Artifactory è una applicazione web che permette di gestire repository Maven, sviluppata in Java e rilasciata con licenza LGPL 3.0. L’applicazione è realizzata da JFrog Ltd, una società privata isreliana.

Strutturalmente si poggia su Apache Jackrabbit per l’implemetazione delle specifiche JSR 170 per il cosiddetto Java Content Repository (anche altri content manager come Nuxeo si appoggiano su Jackrabbit).

Piccola nota: la JSR 170 (rilasciata nel 2005) è seguita dalla JSR 283 da poco rilasciata (25 Settempre 2009) ed entrambe sono condotte da David Nuescheler, CTO della società svizzera Day Software.

Inoltre Artifactory si basa su Apache Lucene per indicizzare i file e Apache Wicket per l’interfaccia utente.

Ora l’applicazione è rilasciata come standalone ma esiste la versione war che può essere deployata su un application server e quindi l’ho messa su Jboss insieme ad Hudson e Webclient for SVN (questi 2 li ho poi legati con il plugin di hudson per Polarion che permette di linkare la lista di file modificati ai file presenti sul webclient e si possono anche vedere le differenze con le versioni precedenti).

La struttura è basata anche su database, di default Apache Derby ma che può essere cambiato (vedi anche qui).

A parte il fatto che Artifactory permette di base una autenticazione LDAP, quello che lascia senza parole è la semplicità di uso (andrebbe confrontato con Nexus ma non ne ho tempo, di sicuro nella versione base manca LDAP (anche se ho scoperto di recente che esiste un plugin di terze parti) e di sicuro gli sviluppatori hanno realizzato una bella guida a maven e un plugin per Eclipse). Una volta andato su

http://localhost:8080/artifactory

e loggato, sono andato sul pannello amministrativo e nella sezione repository ho creato un repository locale dove fare il deploy dei propri jar file e un repository virtuale in cui aggiungo quello locale più tutti i remoti quali quello di Maven, Jboss ecc ecc. Fatto ! Ora potete generare anche la sezione del file settings.xml da aggiungere nel vostro progetto Maven.

Per una vecchia guida vi rimando qui. Per un confronto con Nexus e Archiva vi rimando a questo link (attenzione è scritto dallo stesso autore della guida su Artifactory ma sembra serio)

Buon maven a tutti !

Nota: Webclient for SVN si basa su Subversion 1.4 (con versioni successive ho avuto problemi perchè usa un SVNKit vecchio che non posso aggiornare perchè c’è una particolare classe non più presente nelle ultime versioni), potete trovare Subversion 1.4 qui:

http://downloads-guests.open.collab.net/servlets/ProjectDocumentList?folderID=6

It’s been a little while since the last Open Source Search Social, so we’re getting really imaginative and holding another one, this time on Wednesday the 18th of November. As usual the event is in the Pelican pub just off London’s face-bleedingly trendy Portobello Road.

The format is staying roughly the same. No agenda, no attitude, just some geeks talking about search and related topics in the presence of intoxicating substances.

Please come along if you can, just get in touch or sign up on the Upcoming page.

Since the new Documentum Search Services beta program just started last week, we thought we would share some of TSG’s thoughts on full-text search and our plans to add Lucene capabilities to our open source offerings.

Documentum Search Services (DSS), was tentatively called Enterprise Search Services (ESS) early in the product development.    DSS promises to be “the next generation of search in EMC and will be built upon xDB with Apache Lucene as the underlying indices”.  Specific highlights from EMC World included:

• Relevance Sorting
• Advanced Query Processing
  • Parallel, Native Facet computation, Xquery for structured and unstructured search
  • Lower Hardware and Storage Costs
• Native VMWare, NAS, SAN support and Advanced Data Placement

At the present time, DSS is targeted for heavy testing through the end of 2009 with a release in 2010.

TSG Thoughts on DSS

At the present time, we are very encouraged with the progress and the direction of DSS.  We have been using Lucene for a couple of clients and can safely say that the tool will address many of the shortcomings of FAST including index rebuild, overall performance and server requirements.  That being said, the scope of DSS needs to encompass all of the Documentum API level functionality that FAST or Verity have addressed in the past.  As the beta progresses, truly the “devil is in the details” of how DSS evolves so we will with hold our final thoughts until the beta is complete.

Other Tools (Autonomy, Google Appliance, SearchBlox, Vivisimo….)

As an integrator, we do get asked to integrate in different search tools.  We began working with Autonomy for EMC on an internal Documentum project (pre-Documentum purchase) back in the late 90’s.  Overall, most search tools meet full-text needs but are typically built as “crawlers” focused on the web.  As a crawler, the tool needs to scan a directory/website for changes and then update the full text index.  We have found this approach difficult when Documentum clients want to do true “Documentum  Searches” of combining attribute, security and full text.  For example – one client wanted to search on secure documents a certain plant (attribute), create date (attribute) and containing this part-number (full-text).

Also, a couple of clients have had concerns in regards to latency of when a document is stored in Documentum and indexed (after the crawler runs) in the full-text search engine.  One client complained that with FAST, sometimes the latency was 2 minutes and other times it was 2 hours.

Our last concern with the crawler approach is how to get the index data and security added to the index to avoid having to run the query against Documentum (plant, create date, security), against full-text (part-number) and then only displaying the results that are on both lists.

Native Lucene with Documentum?

One scenario we are building out for clients is a Documentum 5.3 or 6.5 application that indexes documents into Lucene from either Documentum or a cached copy (whitepaper here).   To differentiate from DSS, our approach won’t provide support for inline DQL but rather a pure web services approach.

In the diagram below, both OpenMigrate and HPI use OpenContent web services to communicate with Lucene.  OpenMigrate is used to keep the Lucene index up to date, and HPI is used to query the index for full text searches and optionally metadata searches as well:

A couple of key factors:

• 5.3 Support – we are focused on supporting both 5.3, 6.0, 6.5 and future releases.  Many of our clients have chosen to delay their upgrades due to variety of reasons.  By implementing Lucene now, clients can remove FAST in their current environment and from an eventual D6.5 upgrade.
• Attributes – we are focused on storing both the content, attributes and security in Lucene to avoid having to search both the Documentum attributes and the Lucene full-text index.
• Indexing – we are leveraging OpenMigrate to index/delete content and meta data to Lucene on a real-time, multi-threaded push basis to avoid a crawler approach.   We think the push approach can better control updates to the index, reduce server load on the full-text index and improve audit control to insure everything is indexed.
• Security – One issue we addressed was how to manage security concerns versus high-performance search.  Verifying that the user has access to browse each document retrieved from the search (Documentum lookup) is expensive and would hurt performance as identified in the crawler discussion above.  One approach was to cache document ACL information with each document in Lucene and update as ACL’s are updated.  Since Documentum ACL’s don’t change often, we would leverage one lookup to retrieve the users ACL access and add that information to the Lucene query.

So far our results have been favorable.  Please contact us if you are interested in this type of solution as we are looking for additional case studies.

Background
When you perform a search at Google or Bing, you enter your search terms, click a search button, and your search results appear. Each search result displays the title, the URL, and a text fragment containing your search terms in bold.

Consider what happens when you search for ‘Apache’ at Google. Your results would include the Apache server, the Apache Software Foundation, the Apache Helicopter, and Apache County. The contextual fragments displayed with each search result helps you judge if a search result is an appropriate match and if you need to add additional search terms to narrow the search result space. Search would not be as user friendly as it is today without these fragments.

This post covers version 2.4.1 of Apache Lucene, the popular open source search engine library written in Java. It may not be widely known, but Lucene provides a way to generate these contextual fragments so your system can display them with each search result. The functionality is not found in lucene-core-2.4.1.jar but in the contrib library lucene-highlighter-2.4.1.jar. The contrib libraries are included with the Lucene download and are located in the contrib folder once the download is unzipped.

If you are not familiar with Lucene, you can think of it as a library which provides

• a way to create a search index from multiple text items
• a way to quickly search the index and return the best matches.

A more thorough explanation of Lucene can be found at the Apache Lucene FAQ.

As an example of what the Lucene Highlighter can do, here is what appears when I search for ‘queue’ in an index of PDF documents.

e14510.pdf
Oracle Coherence Getting Started Guide
of the ways that Coherence can eliminate bottlenecks is to queue up transactions that have occurred…
duration of an item within the queue is configurable, and is referred to as the Write-Behind Delay. When data changes, it is added to the write-behind queue (if it is not already in the queue), and the queue entry is set to ripen after the configured Write-Behind Delay has passed…

The Steps
First, before you can display highlighted fragments with each search result, the text to highlight must be available. Shown below is a snippet of indexing code. We are storing the text that will be used to generate the fragment in the contents field.

Document doc = new Document();
doc.add(new Field("contents", contents, Field.Store.COMPRESS,
    Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS));
doc.add(new Field("title", bookTitle, Field.Store.YES,
    Field.Index.NOT_ANALYZED));
doc.add(new Field("filepath", f.getCanonicalPath(), Field.Store.YES,
    Field.Index.NOT_ANALYZED));
doc.add(new Field("filename", f.getName(), Field.Store.YES,
    Field.Index.NOT_ANALYZED));
writer.addDocument(doc);

The values Field.Store.COMPRESS or Field.Store.Yes tell Lucene to store the the field in the index for later retrieval with a doc.get() invocation.

Field.Store.COMPRESS causes Lucene to store the contents field in a compressed form in the index. Lucene automatically uncompresses it when it is retrieved.

Field.Index.ANALYZED indicates the field is searchable and an Analyzer will be applied to its contents. An example of an Analyzer is StandardAnalyzer. One of the things done by StandardAnalyzer is to remove stopwords (a, as, it, the, to, …) from the text being indexed.

Note: You should use the same analyzer type (like StandardAnalyzer) for your indexing and searching operations otherwise you will not get the results you are seeking.

Last part of the indexing side is the TermVectors. From the Lucene Javadocs:
“A term vector is a list of the document’s terms and their number of occurrences in that document.”

For the Highlighter, TermVectors need to be available and you have a choice of either computing and storing them with the index at index time or computing them as you need them when the search is performed. Above, Field.TermVector.WITH_POSITIONS_OFFSETS indicates were are computing and storing them in the index at index time.

With the index ready for presenting contextual fragments, lets move on to generating them while processing a search request. Below is a typical “Hello World” type search block.

QueryParser qp = new QueryParser(“contents”, analyzer);
Query query = qp.parse(searchInput);
TopDocs hits = searcher.search(query, 10);

for (int i = 0; i < hits.scoreDocs.length; i++) {
	int docId = hits.scoreDocs[i].doc;
	Document doc = searcher.doc(docId);
	String filename = doc.get("filename");
	String contents =  doc.get(“contents”);

	String[] fragments = hlu.getFragmentsWithHighlightedTerms(analyzer,
                   query, “contents”, contents, 5, 100);
}

Starting at the top, we create a query based on the user supplied search string, searchInput, using the QueryParser. Lucene supports a sophisticated query language and QueryParser simplifies transforming the supplied string to a query object. Next, we get the top 10 results matching the query. This is pretty standard so far, but now in the loop we come to the getFragmentsWithHighlightedTerms call.

Here is the code to generate the fragments:

TokenStream stream = TokenSources.getTokenStream(fieldName, fieldContents,
                      analyzer);
SpanScorer scorer = new SpanScorer(query, fieldName,
				new CachingTokenFilter(stream));
Fragmenter fragmenter = new SimpleSpanFragmenter(scorer, 100);

Highlighter highlighter = new Highlighter(scorer);
highlighter.setTextFragmenter(fragmenter);
String[] fragments = highlighter.getBestFragments(stream, fieldContents, 5);

First we obtain the TokenStream. The call shown above assumes term vectors were not stored in the index at index time.

Next is the SpanScorer and SimpleSpanFragmenter. These work to break the contents into 100 character fragments and rank them by relevancy. You can use SpanScorer and SimpleSpanFragmenter or QueryScorer and SimpleFragmenter. The full details can be found in the Javadocs.

Note: when indexing large files, like the full contents of PDF manuals, you might need to tell the Highlighter object to look at the full text by calling the setMaxDocCharsToAnalyze method with Integer.MAX_VALUE or a more appropriate value. In my case, the default value specified by Lucene was too small, thus Highlighter did not look at the full text to generate the fragments. This was not good because the match I was seeking was near the end of the contents.

Finally, we tell the Highlighter to return the best 5 fragments.

The full code for this example can be downloaded from my Google Code project. The source file that makes use of the Highlighter is HighligherUtil.java

You can also find examples of using Highlighter in the Lucene SVN Repository, specifically HighlighterTest.java

As you can see, returning search results with contextual fragments containing your search terms is very easy with the Lucene Highlighter contrib library once you know the steps to follow.

I’m working on a web project which uses Zend’s implementation of Lucene in PHP. Fyi, Lucene is an open source text retrieval library that you can use to provide full text search. So the project I’m working on uses Lucene to provide full-text search on the site and the problem they were having was that searches for say, the word “orange” would not bring up articles that had the word “oranges”. The problem was that no stemming was being done. Stemming makes it so that if you search for a word like “happy”, it also matches on similar terms like “happily”, “happiness”, etc.

So I thought it would be easy. Just turn on stemming. But I was a bit surprised that Zend’s Lucene implementation does not include a stemmer. After a quick search, I found someone who had implemented a stemming analyzer for Zend’s Lucene. Tossed that in and voila, search is instantly more usable. Anyway, Zend should really package this with their Lucene Search.

Our client in Redwood City, CA is in need of a seasoned system administrator and operations engineer to manage, maintain and support multiple environments supporting several different architectures and environments. The successful candidate will have strong exposure to large web site deployments with multiple tiers include web front-ends, middleware components, backend services, data storage systems and 2nd/3rd tier components like caching and search layers.

Requirements:

• 7 years Linux admin experience
• 5 years RHEL admin experience
• 5 years RPM management and admin experience
• 3 years working with MySQL
• Sound understanding of web services and related technologies (HTTP, REST, XML, JSON, etc)
• Experience with Linux web servers (Apache2, nginx, lighttpd, etc)
• Experience with monitoring systems and components (nagios, cacti, etc)
• Exposure to scripting languages such as Perl, Python, etc.

Really nice-to-haves but not required:

• Experience with Erlang
• Experience with Ruby on Rails
• Exposure to Jabber/XMPP
• Exposure to distributed / grid computing
• ejabberd administration and configuration
• Experience with second and third tier services and components like memcached (or like systems), search applications (lucene, xapian, sphinx, etc), etc.

Realistically, these are also nice to haves but not required:

• Exposure to functional programming languages.
• Exposure to the Facebook Platform, Open Social and or other web services and systems
• Comfortable working in an Agile work environment.
• Plays games and occasionally keeps tabs on the gaming world/industry.

If interested, please send us your resume along with the rate per hour, contact number, and availability for a phone interview to raj@mindsource.com.

必要なもの

• PyLucene
  これがなければ始まらない。ソースアーカイブをダウンロードしておいてください。

• Visual Studio 2008
  Visual Studio 2008 Express Edtionでは試していません。

• Apache Ant
  PyLuceneのビルドに使用します。バイナリをダウンロードして、適当にインストールしてください。私は c:\Program Files\apache-antにインストールしました。

• Java SE Development Kit

• Cygwin
  インストールする際、Develカテゴリの SubVersion と make を必ず選択してください。

• setuptools

環境設定

まず、PATHをJavaランタイムのjvm.dllがあるディレクトリとJDK、cygwinのbinディレクトリに通しておきます。私の場合だと

c:\Program Files\Java\jre6\bin\client;C:\Program Files\Java\jdk1.6.0_16\bin;c:\cygwin\bin;

を追加しました。

次に環境変数JAVA_HOMEを設定します。私の環境では

JAVA_HOME = C:\Program Files\Java\jre6

としました。

JCCのビルド

まず、JavaライブラリのC++インターフェースを作成する JCC モジュールをビルドします。

1. setup.pyを修正する

<pylucene>\jcc にある setup.py を環境にあわせて修正します。私の環境では次のように修正しました。

Index: setup.py
===================================================================
--- setup.py    (revision 820678)
+++ setup.py    (working copy)
@@ -39,7 +39,7 @@
'ipod': '/usr/include/gcc',
'linux2': '/usr/lib/jvm/java-6-openjdk',
'sunos5': '/usr/jdk/instances/jdk1.6.0',
-    'win32': 'o:/Java/jdk1.6.0_02',
+    'win32': 'C:/Program Files/Java/jdk1.6.0_16',
}
if 'JCC_JDK' in os.environ:
JDK[platform] = os.environ['JCC_JDK']
@@ -61,7 +61,7 @@
'linux2': ['-fno-strict-aliasing', '-Wno-write-strings'],
'sunos5': ['-features=iddollar',
'-erroff=badargtypel2w,wbadinitl,wvarhidemem'],
-    'win32': [],
+    'win32': ["/O2", "/FD", "/EHsc", "/MD", "/GR", "/c", "/Zi", "/TP"],
}

# added to CFLAGS when JCC is invoked with --debug

2. ビルド&インストール

あとは普通のPythonモジュールと同様に

cd <pylucene>\jcc
python setup.py install

で完了です。

PyLuceneのビルド

1. Makefileを修正する

環境に合わせて、Makefileを修正します。私の環境では次のように修正しました。

Index: Makefile
===================================================================
--- Makefile    (revision 820678)
+++ Makefile    (working copy)
@@ -105,12 +105,14 @@
#NUM_FILES=2

# Windows   (Win32, Python 2.5.1, Java 1.6, ant 1.7.0)
-#PREFIX_PYTHON=/cygdrive/o/Python-2.5.2/PCbuild
-#ANT=JAVA_HOME=o:\Java\jdk1.6.0_02 /cygdrive/o/java/apache-ant-1.7.0/bin/ant
-#PYTHON=$(PREFIX_PYTHON)/python.exe
-#JCC=$(PYTHON) -m jcc --shared
-#NUM_FILES=2
+PREFIX_PYTHON=/cygdrive/c/Python26
+JAVA_HOME=C:\Progra~1\Java\jdk1.6.0_16\
+ANT=/c/Progra~1/apache-ant/bin/ant.bat
+PYTHON=$(PREFIX_PYTHON)/python.exe
+JCC=$(PYTHON) -m jcc.__main__ --shared
+NUM_FILES=2

+
#
# No edits required below
#

2. ビルド&インストール

cd <pylucene>
make
make install

で完了です。インストールしたら

make test

でちゃんと動作するか、確認してみましょう。

Das beliebte Java-Sucheframework Apache Lucene ist in der Version 2.9 erschienen. Neben einem überarbeiteten Unicode-Support gibt es ein neues Query-Parser-Framework. Umsteiger von Version 2.8 sollten jedoch aufpassen, da sich die API zum Teil geändert hat. Auf JAXEnter wurde übrigens eine Artikelserie zum Thema Lucene eingerichtet.

Anda mungkin bertanya: berbeda dari apa? Tentu saja dari Lucene.Net hehehehe karena saya belum pernah pakai yang Java.
Pada prinsipnya baik yang di dotnet maupun yang di PHP, sama. Bahkan, katanya nih, soalnya aku belum coba, hasil index bisa dibaca oleh siapa saja.
Trus bedanya apa? Ada beberapa catatan, tentu saja nanti bisa nambah:
1. Di dotnet dibedakan mendaji dua class: membaca dan menulis, sementara di PHP cukup satu saja.
2. Dalam pencarian, by default, PHP akan mencari kesemua field yang diindex. Jadi tidak repot dengan term dan query.

I’m about to implement the technique for assigning Medical Subject Heading (MeSH) terms to text described in this paper from the TLAs EBI, HMI and TNO ICT:

• Trieschnigg, Pezik, Lee, de Jong, Kraaij, and Rebhoz-Schumann (2009) MeSH Up: Effective MeSH Text Classification for Improved Document Retrieval. Bioinformatics. [open access home, including link to the Methodology Supplement]

The same technique could be applied to assign Gene Ontology (GO) terms to texts, tags to tweets or blog posts or consumer products, or keywords to scientific articles.

k-NN via Search

To assign MeSH terms to a text chunk, they:

1. convert the text chunk to a search query,
2. run the query against a relevance-based MEDLINE index, then
3. rank MeSH terms by frequency in the top k (k=10) hits.

In other words, k-nearest-neighbors (k-NN) where “distance” is implemented by a relevance-based search.

Just the Text, Ma’am

Trieschnigg et al. concatenated the title and abstract of MEDLINE citations into a single field for both document indexing and query creation.

k-NN implemented as search could be faceted to include journals, years, authors, etc. For products, this could include all the facets seen on sites like Amazon or New Egg.

Language-Model-Based Search

They use language-model-based search, though I doubt that’s critical for success. Specifically, they estimate the maximum-likelihood unigram language model for the query and interpolated (with a model trained on all documents) model for each document, and then rank documents versus that query by cross-entropy of the query model given the document model (given the MLE estimate for the query, this is just the log probability of the query in the document’s LM.) Other LM-based search systems measure similarity by KL-divergence.

There weren’t any details on stemming, stoplisting, case normalization or tokenization in the paper or supplement; just a pointer to author Wessel Kraaij’s Ph.D. thesis on LM-based IR.

Application to MEDLINE

The text being assigned MeSH terms was another MEDLINE title-plus-abstract. This may seem redundant given that MEDLINE citations are already MeSH annotated, but it’s useful if you’re the one at NLM who has to assign the MeSH terms, or if you want a deeper set of terms (NLM only assigns a few per document).

It’s easy to apply the authors’ approach to arbitrary texts, such as paragraphs out of textbooks or full text articles or long queries of the form favored by TREC.

Efficient k-NN!

I did a double-take when I saw k-nearest-neighbors and efficiency together. As we all know, k-NN scales linearly with training set size and MEDLINE is huge. But, in this case, the search toolkit can do the heavy lifting. The advantage of doing k-NN here is that it reproduces the same kind of sparse assignment of MeSH terms as are found in MEDLINE itself.

Error Analysis

The authors did a nice job in the little space they devoted to error analysis, with more info in the supplements (PR curves and some more parameter evaluations and the top few hits for one example). They reported that k-NN was better than some other systems (e.g. thesaurus/dictionary-based tagging and direct search with MeSH descriptors as pseudocuments) at assigning the sparse set of MeSH terms found in actual MEDLINE citations.

Errors tended to be more general MeSH terms that just happened to show up in related documents. I’d also like to see how sensitive performance is to the parameter setting of k=10, as it was chosen to optimize F measure against the sparse terms in MEDLINE. (All results here are for optimal operating points (aka oracle results), which means the results are almost certainly over-optimistic.)

What You’ll Need for an Implementation

It should be particularly easy to reproduce for anyone with:

• half a clue about Lucene (search),
• a MEDLINE parser, and
• optionally, a MeSH parser.

Look for a demo soon.

Discriminative Semi-Supervised Classifiers?

It’d be nice to see an evaluation with text generated from MeSH and articles referencing those terms that used any of the semi-supervisied or positive-only training algorithms (even just sampling negative training instances randomly) with some kind of discriminative classifier like logistic regression or SVMs.

Improving IR with MeSH (?)

I didn’t quite follow this part of the paper as I wasn’t sure what exactly they indexed and what exactly they used as queries. I think they’re assigning MeSH terms to the query and then adding them to the query. Presumably they also index the MeSH terms for this.

I did get that only the best MeSH-term assigner improved search performance (quantized to a single number using TREC’s mean average precision metric).

Alternatives like generating a 24,000-category multinomial classifier are possible, but won’t run very fast (though if it’s our tight naive Bayes, it might be as fast as the authors

A lot has been written about why Enterprise 2.0 projects fail (the latest I’ve read: 14 Reasons Why Enterprise 2.0 Projects Fail).

Most of the articles I’ve read seem to ignore one point that I think is critical. In this post I will attempt to highlight that point.

In a previous incarnation with an international non-profit research organization, I piloted and implemented a “rogue” internal social networking application. It had to be rogue for the usual reasons (initial apathy from colleagues, hostility from IT types, zero resource-allocation, among others).

How hard is it to be social?

The biggest predictor of success, from my experience, was how well the application made it easy to be social. You’d be surprised that a vast majority of enterprise 2.0 applications make it quite difficult to be social in the way users already are. No wonder a lot of (re)training is required. No wonder user uptake is slow, if at all it takes off.

Instead of leveraging what should be the greatest asset out there, namely that humans already are incredibly social and want to further their social circles, most “social applications” toss out ways that work in the analog “real world” and try to digitally recreate unfamiliar ways of being.

Consider the following all-too-common mind-bending scenarios:

• File-centric social applications:  I don’t know anyone who voluntarily shares files with colleagues just for the sake of sharing the file. People share information, seek comments, show-off their knowledge etc. Sharing the file is a means, not an end in itself. It still reeks of file-ism even after you add a few social widgets and re-brand.

• The requirement for voodoo markup in some wiki applications: who wants to do that when there are other, albeit non-social, ways that have the added benefit of WYSIWYG editors?

• The expectation that browsing to a website will be the primary interface for interaction with the application: what ever happened to email/IM integration? I purposefully only include email and IM as they already are accepted as legitimate in most enterprises (email more so than IM).

• Context-aware search: it isn’t enough that search is content-aware. If users are to meaningfully engage with the torrent of generated information, context-aware search engines should display relevant information without the user necessarily searching for it. This is important as one cannot possibly search for information they are not aware exists!

Social like I already am

Despite a deep desire both philosophically and practically to go the open-source route, I finally had to make do with a quasi-open platform (Clearspace from Jive Software, now confusingly re-branded Jive Social Business Software) principally because Clearspace made it easy to be social in a way that users already were.

This was pleasantly borne out when users, some with little training but most with no training, took to the platform with surprising gusto. Infact, some users developed such a personal affinity for the platform that they considered it “our space and HR shouldn’t mess with it”.

The sum total of tools that a user is expected to use in order to engage socially is what I’d call a Social User Interface (SUIT). The net-effect of a conscious design decision to engage users in a socially intuitive manner, both at the visible and invisible level, is what makes for a well-designed Social User Interface (SUIT). On this metric, a good number of enterprise 2.0 applications need some dressing-up.

How one application suits up

Here’s how Clearspace won me over:

• People, and the resulting conversations, were at the center of Clearspace’s implementation. Users “got it” after a short period of using it. More “complex options” like tag-clouds often only required quick explanations.

• The wiki app wasn’t even called a wiki, and most definitely did not look like one, even though it was one. Expanding/restricting allowed editors, and commentors, was such a breeze.

• A really neat feature is tight email integration where one can start/continue conversations, and create new content, from email. This allowed people who loathed visiting another website to be part of the community, right from within their email application. With time, some of these same people saw the added value of regularly directly interacting with the web application, because it was worth their time. IM integration was achieved via Openfire, open-sourced by Jive and which has very tight integration with Clearspace. This provided another avenue for people to be part of the community, with the exact same permissions, from their IM clients.

• Clearspace has a great “more like this” widget that displayed people/content that is deemed similar enough to what you are currently interacting with. The search engine behind this is the venerable open-source Apache Lucene.
  • This is one area that is painfully lacking in Elgg, a promising and open-source “social engine” (I’m currently working on a Lucene plugin for Elgg).
  • A “semantic search engine”, for instance OpenCalais, might even work better (I’m also working on an experimental plugin for Clearspace and Elgg).

Better filters, and the “Trust Spectrum”

There is definitely room for improvement in all social platforms/engines/applications that I have come across:

• There is no clear way to quickly determine where the content you are creating will end up in the “trust spectrum” (coined by Gia Lyons: the digital corollary to the observation that we selectively share information in the real world). No application, not even Facebook, has an intuitive and quick indicator of  where, in the trust spectrum, your post will be placed. Remember Twitter’s DM Fail?

• Most social applications use a points system to highlight top/most-helpful/trusted contributers. While a noble attempt at capturing the social nuance that “not all opinions are created equal”, it is not something that comes naturally to most people. What is needed is a better filter, this is what this feature really is, that takes into account how we really determine who to pay attention to in our everyday life. I have some draft ideas on how this may work, perhaps for another post.

What do you think? What has your experience been with regard to Social User Interfaces (SUITs)?

Apache SOLR, an “open source enterprise search server”, demonstrates indecently inconsistent and illogical handling of NOT conditions in it’s queries (like “x:1 AND NOT x:2” producing 0 results, yet “x:1 AND (NOT x:2)” producing correct results etc).

We couldn’t play with braces because our conditions are autogenerated, so we needed a generic solution for this.

Fortunately, workaround has been found for this with simple Googling. Instead of querying for “… AND/OR (NOT [condition])” just use “… AND/OR (*:* NOT [condition])” or “… AND/OR (*:* AND NOT [condition])“.