Figure – Book cover of Toby Segaran’s, “Programming Collective Intelligence – Building Smart Web 2.0 Applications“, O’Reilly Media, 368 pp., August 2007.

{scopus online description} Want to tap the power behind search rankings, product recommendations, social bookmarking, and online matchmaking? This fascinating book demonstrates how you can build Web 2.0 applications to mine the enormous amount of data created by people on the Internet. With the sophisticated algorithms in this book, you can write smart programs to access interesting data-sets from other web sites, collect data from users of your own applications, and analyze and understand the data once you’ve found it. Programming Collective Intelligence takes you into the world of machine learning and statistics, and explains how to draw conclusions about user experience, marketing, personal tastes, and human behavior in general — all from information that you and others collect every day. Each algorithm is described clearly and concisely with code that can immediately be used on your web site, blog, Wiki, or specialized application.

{even if I don’t totally agree, here’s a “over-rated” description – specially on the scientific side, by someone “dwa” – link above} Programming Collective Intelligence is a new book from O’Reilly, which was written by Toby Segaran. The author graduated from MIT and is currently working at Metaweb Technologies. He develops ways to put large public data-sets into Freebase, a free online semantic database. You can find more information about him on his blog:  http://blog.kiwitobes.com/. Web 2.0 cannot exist without Collective Intelligence. The “giants” use it everywhere, YouTube recommends similar movies, Last.fm knows what would you like to listen and Flickr which photos are your favorites etc. This technology empowers intelligent search, clustering, building price models and ranking on the web. I cannot imagine modern service without data analysis. That is the reason why it is worth to start read about it. There are many titles about collective intelligence but recently I have read two, this one and “Collective Intelligence in Action“. Both are very pragmatic, but the O’Reilly’s one is more focused on the merit of the CI. The code listings are much shorter (but examples are written in Python, so that was easy). In general these books comparison is like Java vs. Python. If you would like to build recommendation engine “in Action”/Java way, you would have to read whole book, attach extra jar-s and design dozens of classes. The rapid Python way requires reading only 15 pages and voila, you have got the first recommendations. It is awesome!

So how about rest of the book, there are still 319 pages! Further chapters say about: discovering groups, searching, ranking, optimization, document filtering, decision trees, price models or genetic algorithms. The book explains how to implement Simulated Annealing, k-Nearest Neighbors, Bayesian Classifier and many more. Take a look at the table of contents (here: http://oreilly.com/catalog/9780596529321/preview.html), it does not list all the algorithms but you can find more information there. Each chapter has about 20-30 pages. You do not have to read them all, you can choose the most important and still know what is going on. Every chapter contains minimum amount of theoretical introduction, for total beginners it might be not enough. I recommend this book for students who had statistics course (not only IT or computing science), this book will show you how to use your knowledge in practice _ there are many inspiring examples. For those who do not know Python - do not be afraid _ at the beginning you will find short introduction to language syntax. All listings are very short and well described by the author _ sometimes line by line. The book also contains necessary information about basic standard libraries responsible for xml processing or web pages downloading. If you would like to start learn about collective intelligence I would strongly recommend reading “Programming Collective Intelligence” first, then “Collective Intelligence in Action”. The first one shows how easy it is to implement basic algorithms, the second one would show you how to use existing open source projects related to machine learning.

The life of a webmaster is hard, and web crawlers make it harder

http://www.flickr.com/photos/absolutely_loverly/ / CC BY 2.0

There’s the daily drama of keeping both web site users and web site developers happy. Now mix in the unpredictable side effects of having automated agents hitting the site, and you can see why webmasters might think many web crawlers are evil.

But web crawlers serve a very real, important role in the life of a successful site, and it’s all about traffic. Without search engines like Google and Yahoo/Bing, most sites would be invisible to most users.

Implicit Contracts

An unwritten agreement exists between webmasters and web crawlers, and it reads something like this: you don’t overload my site, and you bring traffic my way. In return, I’ll give you free access lots of valuable content that I host.

And that’s worked reasonably well, for the past 15 years. Yes, there are crawlers that ignore the Robots Exclusion Standard. And there are crawlers that overload the site by hammering it with lots of simultaneous requests for hours on end. And sometimes a crawler goes a little crazy and spends hours trying to fetch non-existent pages using bogus URLs that it incorrectly derived from content on the site’s pages. For the most part, though, web crawlers try to do the Right Thing, and webmasters can always block rogue crawlers by IP address.

Web Mining != Search Index

But now you’ve got web miners – automated agents that collect data which often doesn’t wind up in a search index. And that means no traffic from searches. And thus the implicit contract has been broken.

It hasn’t happened yet, but I can see a day when many sites set up their robots.txt to allow the major search engines access, and then block everybody else.

What does this mean for the web eco-system? Three things, one for each participant:

1. Web miners need to crawl extra-super-politely.
2. Customers need to work with key sites to pick good crawl times.
3. Web sites need to offer for-fee APIs for data mining.

The first point is the easiest one to solve – never hit a site with more than one simultaneous request, never fetch more than a handful of pages a minute, and respect all robots.txt restrictions.

The second is a bit harder, as it currently requires person-to-person contact with the web site in question. It’s possible to derive these “good crawl times” by varying the request rate with the response performance, so there are work-arounds. But eventually I expect to see an extension to robots.txt that lets the site owner provide additional clues to web crawlers about good and bad times for crawling.

The last point, about providing APIs, is the most long-term but also the most powerful. There are many web APIs out there, some of which provide access to valuable web data, but few offer a pay-to-play model. Most are rate limited, where you need to cut special deals if you exceed some relatively low daily threshold. Many have serious terms of use restrictions that limit a caller’s ability to actually mine the response data – often the only option is to republish it, with links/attribution back to the originating site.

What would be great is if everybody had a model like Amazon’s AWIS, where X requests cost N dollars. You can decide how much or how little to spend. There aren’t many restrictions on rate/volume or usage. And as a huge added bonus, the data comes back structured, so you don’t have to waste time hand-crafting some fragile, error-prone HTML scraping code.

And a side-note to companies thinking about the API issue – if you don’t provide one, and you block web miners, then you’ll get crawled anyway, in stealth mode by less scrupulous firms. So then everybody loses, since you’ll still be giving free access while taking a performance hit, while companies that need the data pay more to these “stealth crawlers” and get worse results.

I recently came across MOA (Massive Online Analysis), an environment for what its developers call massive data mining, or data stream mining. This New Zealand-based project is related to Weka, a Java-based framework for machine learning which I’ve used quite a bit over the years. Data stream mining differs from plain old data mining in that the data is assumed to arrive quickly and continuously, as in a stream, and in an unpredictable order. Therefore the full data set will typically be many times larger than your computer’s memory (which already rules out some commonly used algorithms), and each example can only be briefly examined once, after which it is discarded. Therefore the statistical model has to be updated incrementally, and often must be ready to be applied at any point between training examples.

I also came across a press release describing version 2.0 of KnowledgeMiner for Excel, a data mining software apparently used by customers like Pfizer, NASA and Boeing, and which is based on GMDH (Group Method of Data Handling), a paradigm I hadn’t heard about before. I failed to install KnowledgeMiner for Excel for my Mac due to some obscure install error, but from what I gather, the GMDH framework involves a kind of automatic model selection, making it easier to use for non-experts in data mining. (Of course I haven’t tried it, so it’s hard to evaluate the claim.) The example data set provided with the software package has to do with climate data and modeling, so it should be fun to try as soon as I get it working:

The new KnowledgeMiner is now capable of high-dimensional modeling and prediction of climate and has an included example using air and sea surface temperature data. This is a first for a data-mining software package: to offer anyone the ability to see for themselves that global temperatures are rising steadily, using publicly available data. The biggest surprise is seeing that the changes are greatest and accelerating in the northern latitudes. By using data from the past, KnowledgeMiner (yX) can show predictions for future years. Go to this link to see the climate change data displayed graphically in a slideshow through the year 2020:

There’s also an interesting new toolkit for web mining from BixoLabs. They’ve built what they call an elastic web mining platform in Amazon’s Elastic Compute Cloud (on top of Hadoop, Cascading and a web mining framework called Bixo, for those of you who care). The whole thing is pre-configured and scalable, and from the tutorials on the site, it seems pretty easy to set it up to crawl the web to your heart’s content.

Here’s the presentation I gave at the ACM data mining unconference on elastic web mining – how to create scalable, reliable and cost effective web mining solutions using an open source stack (Hadoop, Cascading, Bixo) running in Amazon’s Elastic Compute Cloud (EC2).

But I don’t see my notes showing up, so here’s the PDF version with full notes, which make the resulting slides a lot more meaningful.

I’ve been working on an elastic web mining platform for a few months now, and it was finally time to go public with at least the current state of the union.

I gave a talk at the ACM Data Mining Unconference on Sunday, where I also announced the Public Terabyte Dataset project, so the timing was perfect.

If you want to know what’s been keeping me busy, and looks to be part of my future, check out http://bixolabs.com.

We’re very excited to announce the Public Terabyte Dataset project.

This is a high quality crawl of top web sites, using AWS’s Elastic Map Reduce, Concurrent’s Cascading workflow API, and Bixolab’s elastic web mining platform.

Hosting for the resulting dataset will be provided by Amazon in S3, and freely available to all EC2 users.

In addition, the code used to create and process the dataset will be available for download from http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263

Questions and input on the project can be submitted at http://bixolabs.com/PTD/

This coming Sunday is the big Bay Area data mining “unconference“, and with more than 200 people already signed up, it’s going to be a lot of fun.

I’ll be presenting at some point during the day – since it’s an unconference, you don’t really know who’s going to be talking about what/when. My topic is “Elastic web mining using open source (Hadoop/Cascading/Bixo) in Amazon’s EC2 cloud“.

If you scan the list of attendees (click the “RSVPs” tab near the top of the LinkedIn event page) you’ll see a lot of high powered executives, consultants and researchers, so I’m looking forward to really great lobby conversations.

Many thanks to the San Francisco Bay Area Chapter of the ACM for helping out with the event, which is taking place from noon to 7:30pm at the Hacker’s Dojo in Mountain View. Greg Makowski is the organizer, so he’s probably going a little bit crazy right now

Learning public opinion (or sentiment) about Your brand in traditional way is expensive because surveys or focus groups take much time and human work. Probably in near future alternative solution will gain recognition as some technology vendors launched tools for brand monitoring using text analytics. Initial review of these attempts appeared yesterday on SmartData Collective.

Monitoring brand using discourse analysis differs, to some extent, from the approach based on text analysis. I have very fresh example – a tool for monitoring opinion about retail nets (supermarkets). And now some words how it is made and how it works.

Building Monitor. Analysis of the discourse in the corpus of Internet discussions related to supermarkets gave a collection of subjects interesting for interlocutors, and a collection of expressions of their attitudes.  Using these results the complex queries for semantic search were built for the learning research. It is the crucial stage – we should learn very details of the discourse, and get its math at the same time, as the basis for justification and calibration of the Monitor. The final task is relatively easy – to implement the results and build a “machine” using accessible technology.

How Monitor works. The data for each retail brand is collected using semantic search. Monitor makes all the calculus according to calibrating formulas and provides figures ready for presentation. Please see the pictures made for presentation only (not production version).

First are the “profile” – how the brand is perceived, i. e. how it is distinguished vs the average Internet discourse. The result of such kind is often astonishing because the picture dramatically differs from that of Customer’s (user of monitor) wishes, from official image and marketing buzz. Moreover, the interlocutors’ categories (vertical in the charts) also differ.

Then there is a comparison of the brands monitored. The charts show how people value each brand with regard to the same categories.  2 charts with negative opinions (general index only) are presented as the example.

The third important group of results regards monitoring itself, i. e. presentation of the changes. It depends on the Customer needs. Some customers want to observe the effects of promotional campaigns, and for such purpose day-to-day monitoring is appropriate. Some want to know the general trends… etc.

I have posted a few times on how much I love the ability to access Google Analytics data with Python and bring it into SPSS, the tool I use literally everyday. If I wanted to go the totally free route, I could probably learn how to do this analysis into R, but that is a ways off.

The code below queries a Google Analytics account, filters for a specific set of pages, and uses the results in another block of code that navigates to each page and finds all of the links (I even filtered this set of links too!). I am new to the web mining arena, but just for fun, I want to look at the dynamics of a web site (or in this case, a sub-site) and overlay onto the network actual data on the usage of each page. It might be interesting, it might not be, but at least I will learn a lot about Social Networks, Python, and SPSS along the way!

Anyway, the end product generates two SPSS files; one is simply the results from the query sent to the Google Analytics Data Export API, the other is a file that contains the parent/child links from my web spider. Hopefully you learn something new or it sparks an idea of your own. If you see something in my code and think I could write it better, please let me know. I am learning as I go here and know that I probably could do something better, but hey, at least it works!

One note: I intentionally overwrite the entries in the variable “pages” because I don’t want to navigate to every page returned by GA. I will comment this out at a time later when the web traffic to the site is lighter. I actually haven’t done this yet, but I am assuming it works just fine.

”’
Created on Aug 10, 2009

1. This script connects to Google Analytics and returns a set of pages and some usage data on each
2. Use the set of pages and navigate to each to parse the links from each
3. Save the Google Analytics data to an SPSS dataset “pagedata”
4. Save the spidered links to an SPSS dataset “spider”

REFERENCES:

http://blog.clintecker.com/post/100021441/python-google-analytics-client-how-to-use-it-and-how

http://github.com/clintecker/python-googleanalytics/tree/master

http://code.google.com/apis/analytics/docs/gdata/gdataReferenceDimensionsMetrics.html

http://blog.wellsoliver.com/2009/06/retrosheet/

http://www.spss.com/fusetalk/forum/categories.cfm?catid=9&entercat=y (Jon Peck)

Begginning Python 2nd Edition – Hetland

”’

import urllib
import re
from googleanalytics import Connection
import datetime
import spss
import time

START = time.time()

##############################################################
#Step 1: query GA to get a list of the UGA pages
##############################################################

# setup account and get the external website data
connection = Connection(‘email’, ‘pw’)
account = connection.get_account(‘acct#’)

#define the start and end dates (yyyy,mm,dd)
start_date = datetime.date(2009, 7, 6)
end_date = datetime.date(2009, 8, 9)

#define the dimensions, metrics, and filters
dimensions = ['pagePath']
metrics = ['uniquePageviews', 'entrances', 'exits', 'bounces', 'timeOnPage']
filters = [['pagePath' , '=~', '^/undergraduate']]

#get the data to be processed later
data = account.get_data(start_date = start_date, end_date = end_date, dimensions=dimensions, metrics=metrics, filters=filters)

#create a list of pages that will we navigate to and parse the links from
pages = []

sets = [(pr[0][0]) for pr in data.tuple]
for set in sets:
pages.append(set)

#testing the pages
#for page in pages:
# print page

#this is just a test so I dont run the full spider
pages = ['/undergraduate/']

##############################################################
#Step 2: navigate to each page and parse the links into another tuple
##############################################################

#the tuple that will contain the links
links = []

for page in pages:
url = “http://www.yoursite.edu” + page
#print url
try:
content = urllib.urlopen(url).read()
except:
print “Could not fetch %s” % url
raise SystemExit

#this pattern pulls the page content but excludes header and footer links since they are on every page
pattern = r’

For some months I study the discourse of attitude, and sometimes look back to the nice talk of Shilpa Arora and Mahesh Joshi. In the beginning of this work they consider a comment subjective if it cannot be objectively verified. Such definition seems to be quite reasonable if you are using some arbitrary judgement what can be objectively verified and what cannot. However, arbitrary judgement is subjective.

There is a long tradition in opinion mining to make some arbitrary classifications of the words or expressions. For example, many researchers consider opinions as subjective, and  the statements about facts as objective. Many use external standard dictionary to classify expressions as positive or negative, WordNet to identify semantic similarity etc. It’s no wonder that so many studies suffer problems of identifying ironic expressions, sarcasm etc.

In my studies on expressing attitudes the discourse is taken as is, without arbitrary classification. Instead, I examine in which contexts the expressions appear. And result (for Polish language) is different. In the rough picture expressions of attitude seem to fall into two categories: “at the point” statements, and private, even intimate statements. There is a sharp distinction between these two kinds of expressions, clearly visible in the matrices of context relatedness of the expressions.

Last Sunday I submitted my comment to the people vs machine debate in Research Magazine. Some readers of this comment asked me how I get 97% accuracy of sentiment changes’ measurement in the Web Mining.

Web text analytics is rather new field of research and everybody is using its own approach. So, I would only advice – don’t want to be too quick. If you collect millions of records and focus on thousands of specific sentiment-rich expressions, first look at this data. Make some basic descriptive statistics (Yes!), make some charts of the frequency distributions etc. Try to find proper way of stratification, using your best proven approaches and tools. Don’t avoid this basic examination – I write this because I see many freshmen in analytic business who want to cut corners.

If you find good way of data stratification you will undoubtedly notice, that some expressions occur most frequently in one or two or three specific contexts or specific subject domains. Follow this clue, and limit further research to these expressions. This is the first step to the discourse mining (not simply text mining).

Next steps are obvious. Look for relations between various characteristics of the contexts, subject domains, and these “good” expressions. Make clustering in order to select subjects domains and texts you need. Make the selection from your corpus of texts.

There are a lot of tools to extrude rich and accurate information from data selected in this way.

Limiting the scope of study is the first and very basic way to streamline any research process. It is also a basic step used in Industrial Engineering in streamlining any manufacturing or business process.

Elektronische Geschäftsbeziehungen gewinnen zunehmend an Bedeutung. Interessant ist dabei nicht nur, dass immer mehr Unternehmen über elektronische Kanäle Gewinne erwirtschaften, sondern auch, dass es heute möglich ist, durch Aufzeichnen des Nutzerverhaltens die Basis für umfassende Analysen zu legen.

Diese Möglichkeiten widerspiegeln sich im Marketingtrend der kritischen Bewertung und Maximierung des «Return on Marketing», also dem Wechsel von der reinen Effektivitätsbetrachtung hin zur Effizienzbewertung. Potenziale der Logfiles nutzen…

Web Mining zur Leadgenerierung – Potenziale der Logfiles nutzen.

Das dafür notwendige Data-Mining-Werkzeug ist die innovative Analysemethode «Predictive(Web) Analytics», mit der Kundenwünsche erkannt, Verhalten vorausgesagt und das resultierende Wissen für die Gestaltung von noch profitableren Kundenbeziehungen nutzbar gemacht werden. Predictive Web Analytics verbindet online Daten mit effektiven «Events» wie Bestellungen,Käufen usw. und generiert zuverlässige Rückschlüsse auf zukünftige Ereignisse. Damit wird es möglich, die Affinität von Besuchern vorherzusagen, Cluster von Inhaltsbereichen zu berechnen, Besucher zu profilieren und Sequenzen von Aktivitäten auszuwerten. Predictive Web Analytics leistet somit sehr viel mehr als die reinen Webstatistiken, welche die Anzahl Users, Visits, Page Impressions, Top Pages, Top Referres und Errors eruieren.

Web Mining mit einem Predictive Analytics Modeler

Zur neuen Generation von Predictive Analytics Software (PASW) gehört das Web Mining, das alle genannten Analyseebenen abdeckt. Der Webserver liefert die Rohdaten aus Weblogs und anderen Informationsquellen der Website, aus denen sich ableiten lässt, nach welchen Mustern die Webseiten besucht werden, wo Probleme auftauchen, was den User interessiert, was er unternimmt und wahrscheinlich in Zukunft tun wird. Aufgrund dieser Erkenntnisse lassen sich einerseits die User segmentieren, andererseits kann das Angebot auf die Bedürfnisse der Kunden ausgerichtet und die Website in Echtzeit visuell und inhaltlich angepasst werden – zum Beispiel durch das Integrieren von Kaufanreizen oder Hilfen. Sogar anonyme User lassen sich unter optimalen Umständen nach wenigen Klicks einem Verhaltenssegment zuordnen. Loggen sich die User ein, ist die Identifikation noch einfacher. Das Unternehmen kann sie aufgrund ihres Webverhaltens in Zukunft personalisiert ansprechen.

Von Logzeilen zu relevanten Informationen

Der Wechsel vom technikfokussierten Bild anhand der Klickströme zur besucherzentrierten Sicht der Website erzeugt andere Fragestellungen: Was hat der Besucher gesehen? Was hat er getan? Anhand von Sectionstreams werden die besuchten Bereiche in der Besuchssequenz analysiert. Jeder «Event» in der Ereignissequenz (Eventstream) liefert hierzu präzise Informationen. Eine technische wie auch inhaltliche Herausforderung liegt darin, aus der Flut mehr oder weniger redundanter Log-Inhalte – womöglich noch aus verschiedenen Quellen – relevante Ereignisse, Events oder Aktivitäten zu identifizieren. Die handlungsrelevanten Website-Messwerte lassen sich aufgrund des Geschäftsmodells definieren. Die eigentlichen Analysen erfolgen auf Basis der Events (Bestellungen,Empfehlungen, Offertanfragen usw.). In der Regel geht es um Leadqualifizierung über den Kundenlebenszyklus hinweg.

Automatisierte Angebotsempfehlungen

Professionelle Websites unterstützen ihre User richtungsweisend bei der Auswahl oder beim Erwerb von Produkten. Dafür bedarf es der Resultate aus Empfehlungssystemen (Recommender Systems) bzw. aus Warenkorbanalysen, die Fragen beantworten wie: Welche Produkte oder Themenbereiche werden überdurchschnittlich häufig zusammen erworben bzw. miteinander assoziiert? Welche Produkte werden von Personen mit ähnlichen Interessen empfohlen? Aus beiden Ansätzen resultieren personalisierte (Kauf-)Empfehlungen für den Next Best Buy bis hin zu inhaltlich komplett individualisierten Websites. Individualisierte Empfehlungen basieren auf den persönlichen Präferenzen der Benutzer, dem Verhalten ähnlicher Benutzer (Collaborative Filtering) oder auf Objekteigenschaften(Content Filtering).

In einem hybriden Ansatz, dem Content Boosted Collaborative Filtering, werden die Nachteile der einzelnen Ansätze überwunden. Bei diesem kombinierten Vorgehen werden sparsam besetzte User-Objekt-Bewertungsmatrizen mittels einer inhaltsbasierten Filterung der vorhandenen Präferenzprofile aufgefüllt. Dazu wertet das Empfehlungssystem das Präferenzprofil eines Users aus und erstellt dann auf dessen Basis geschätzte Bewertungen für noch ungesehene Objekte. Hat der Nutzer bereits Objekte bewertet, wird die Schätzung damit ersetzt. Dieser Vorgang wird für alle User durchgeführt. Danach folgt das Collaborative Filtering zur Ermittlung benachbarter, ähnlich gelagerter User. Über Predictive Analytics können Kunden in Interessens- und Bedürfnis-Cluster eingeteilt werden, also in Kundengruppen,die in sich homogen und untereinander abgrenzbar sind. Next-Best-Buy-Empfehlungen stammen hier zum Beispiel aus (segmentspezifischen) Warenkorbanalysen. Die aus Streams vom PASW Modeler generierten Empfehlungen stehen dem User ebenfalls in Realtime zur Verfügung.

Fazit

Neben den Standard-Webstatistik-Tools wie Google Analytics,Webtrends usw., bei welchen Klickraten und Pageviews zählen, ist das Web Mining eine unverzichtbare Methode,um inhaltliche Informationen zu gewinnen – und eine verlässliche Entscheidungshilfe zur Optimierung elektronischer Geschäftsbeziehungen.

Quelle:

* Dr. Christiane Okonek ist Head of Data Intelligence bei

rbc Solutions AG, Meilen.

Kontakt: 044 925 36 36, E-Mail: christiane.okonek@rbc.ch

* Daniel Huber ist Head of Business Development bei

rbc Solutions AG, Meilen. E-Mail: daniel.huber@rbc.ch

Web content mining is an interesting and wide domain. Almost everyone can modify one or several modules from a simple web mining system (like the one we’re building) and create a hole different approach to a more specific problem. There are two thinks that are very often modified (improved):
- the link extractor module
- the document clustering/classification module

The most used techniques for link extraction (the crawling algorithm) are: Breadth-First, Best-First, PageRank, Shark-Search, and InfoSpiders. In the simple focused web spider that I’m building I am using the Best-First technique: best links that match the thesaurus are added to the URL Queue.

The second module that is often improved at a web mining system is the document clustering/classification module. There are differences between clustering and classification. Let’s point the main differences between clustering and classification.

Cluster analysis or clustering is the assignment of objects into groups (called clusters) so that objects from the same cluster are more similar to each other than objects from different clusters. Often similarity is assessed according to a distance measure.

Document classification/categorization is a problem in information science. The task is to assign an electronic document to one or more categories, based on its contents. Document classification tasks can be divided into two sorts: supervised document classification where some external mechanism (such as human feedback) provides information on the correct classification for documents, and unsupervised document classification, where the classification must be done entirely without reference to external information.

Clustering

• Unsupervised
• Input
  • Clustering algorithm
  • Similarity measure
  • Number of clusters
• No specific information for each document

Classification

• Supervised
• Each document is labeled with a class
• Build a classifier that assigns documents to one of the classes

Classification: the task is to learn to assign instances to predefined classes.
Clustering: no predefined classification is required. The task is to learn a classification from the data.
Clustering algorithms divide a data set into natural groups (clusters). Instances in the same cluster are similar to each
other, they share certain properties

Supervised learning: classification requires supervised learning, i.e., the training data has to specify what we are trying to learn (the classes).
Unsupervised learning: clustering is an unsupervised task, i.e., the training data doesn’t specify what we are trying to learn (the clusters).

One of the most used and wide know algorithm for clustering is K-means.

k-means clustering example

Well known document classification techniques are naive Bayes classifier and kNN (K-nearest neighbor).

kNN: k-nearest neighbor

So, to resume, let’s point the following ideas:

• classification and clustering both work with documents
• with classification we try to classify every document by a well established criteria, and we (humans) often supervise the process
• with clustering we let the computer to determine (discover) what are the main classes from some documents and to partition the documents between these classes

I hope you get a good idea about the difference between clustering and classification by reading this article. You can also use the following resources to find out more:

It’s time to make our focused web crawler aware about it’s topic: the thesaurus. The simplest way is to have a flat thesaurus: a list of keywords that are related to our topic. And this is how we’re going to implement it. In the future we could improve it – an hierarchical thesaurus: a tree or even a graph. Take a look at The 1998 ACM Computing Classification System for an hierarchical thesaurus.

Now let’s implement our flat thesaurus:

package ro.teo.ssc.thesaurus;
import java.util.List;

public interface IThesaurus {
	void addKeyword(String keyWord);
	List<String> getKeywords();
}

package ro.teo.ssc.thesaurus;

import java.util.ArrayList;
import java.util.List;

public class SimpleThesaurus implements IThesaurus{
	private static class SingletonHolder {
	     private static final SimpleThesaurus
	     		INSTANCE=new SimpleThesaurus();
	}

	public static synchronized SimpleThesaurus getInstance(){
		return SingletonHolder.INSTANCE;
	}

	private List<String> _keywords;

	private SimpleThesaurus(){
		_keywords=new ArrayList<String>();
	}

	@Override
	public void addKeyword(String keyWord) {
		if (keyWord!=null){
			if (keyWord.length()>0){
				if (!_keywords.contains(keyWord))
					_keywords.add(keyWord);
			}
		}
	}

	@Override
	public List<String> getKeywords() {
		//we don't want the list to be modified by
		//mistake, so we return a copy of it
		List<String> temp=new ArrayList<String>();
		temp.addAll(_keywords);
		return temp;
	}

}

Let’s come back to our simple focused crawler. It’s time to start filtering the links we extract and the pages we download.
We need

1. to know more info about the links – the Link class
2. to know our topic – the Thesaurus class
3. filter extracted links
4. filter downloaded pages

The first step is to define the Link

package ro.teo.ssc.obj;

public class Link {
	private static Link _empty;
	public static Link Empty(){
		if (_empty==null){
			_empty=new Link();
		}
		return _empty;
	}

	/**
	 * The URL to which the link is pointing.
	 */
	private String href;
	/**
	 * The 'title' attribute of the link.
	 */
	private String title;
	/**
	 * Text marked as link.
	 * This is usually the visible text that the user clicks on,
	 * Or the 'alt' attribute for the image marked as link.
	 */
	private String innerText;

	/**
	 * Used for ordering links from a web page.
	 * This has nothing to do with ranking the page
	 * like PageRank or HITS algorithms.
	 */
	private int rank;

	public Link(){
		href="";
		title="";
		innerText="";
		rank=0;
	}

	public Link(String href, String title, String innerHTML) {
		this();
		if (href!=null)
			this.href = href;
		if (title!=null)
			this.title = title;
		if (innerHTML!=null)
			this.innerText = innerHTML;
		rank=0;
	}

	public String getHref() {
		return href;
	}

	public void setHref(String href) {
		this.href = href;
	}

	public String getTitle() {
		return title;
	}

	public void setTitle(String title) {
		this.title = title;
	}

	public String getInnerHTML() {
		return innerText;
	}

	public void setInnerHTML(String innerHTML) {
		this.innerText = innerHTML;
	}

	public int getRank() {
		return rank;
	}

	public void setRank(int rank) {
		this.rank = rank;
	}

	@Override
	public String toString() {
		String s=" href(";

		if (href.length()>0) {s+=href;}
		s+=") ";

		s+=" title(";
		if (title.length()>0) {s+=title;}
		s+=")";

		s+=" innerHTML(";
		if (innerText.length()>0) {s+=innerText;}
		s+=") ";

		return s;
	}

}

Some reports (McClatchy, Washington Technology, Wired)  indicate that Veratect, a web data mining company, beat the CDC by 18 days with an alert on Swine Flu.  Nice job!

Of course, its not all that simple.  Veratect has a lower cost for false positives than the CDC, so one would expect them to sent out earlier, more frequent alerts.  In the IR business its what some would call high recall low precision.  We also don’t know how many false positives Veratect has had.  They also had a team of 35 analysis checking the data.  So don’t be too hard on the CDC.  Nevertheless, its clear that the Web is a great resource for mining information on potential health threats, and that it has a place in early warning systems.  And for such systems, I would imagine that there is value in biasing towards rejecting a null hypothesis.

Let’s see what we’ve covered so far:

as long as there are addresses in URL Queue, repeat:
    - get the first address from the queue
    - download the page from that address and store
       it in a temp location
    - check if that page is relevant to the topic (using
      thesaurus)
    - if the page is irrelevant
        remove it from temporary folder
    - if the page is relevant to the topic
        extract some links/URLs from that page
    - if the URL queue is not full
        and those  links are not visited
            insert extracted URLs into the queue
        extract text content from that page
        save the content in a permanent storage location
    - move page from the New URL Queue to Visited URL Queue

How about extracting only some links from web pages the crawler downloads?

If you’ve heard about Agile Programming, Extreme Programming or you’ve been working on some real projects, you know that specifications change in time.

When I first thought about IURLQueue, I didn’t think I’ll need to know the actual number of items in the queue.
When I actually needed it, I modified the code:

package ro.teo.ssc.urlfrontier;

public interface INewUrlQueue {
	public void enqueue(String address);
	public String dequeue();
	public boolean isEmpty();
	public boolean isFull();

	public int size();
}

package ro.teo.ssc.urlfrontier;
import java.util.ArrayDeque;

public class NewUrlQueue implements INewUrlQueue {
	private static class SingletonHolder {
	     private static final NewUrlQueue
	     		INSTANCE = new NewUrlQueue();
   }
	public static synchronized NewUrlQueue getInstance(){
		return SingletonHolder.INSTANCE;
	}

	private static int MAX_SIZE=2048;
	private ArrayDeque q;

	private NewUrlQueue(){
		q=new ArrayDeque();
	}

	@Override
	public String dequeue() {
		String s="";
		if (!q.isEmpty())
			s=q.remove();
		return s;
	}

	@Override
	public void enqueue(String address) {
		if (q.size()=MAX_SIZE);
	}

	@Override
	public int size(){
		return q.size();
	}
}

How the crawler works

This pseudo algorithm shows how the crawler will work:

as long as there are addresses in URL Queue, repeat:
    - get the first address from the queue
    - download the page from that address and store
       it in a temp location
    - check if that page is relevant to the topic (using
      thesaurus)
    - if the page is irrelevant
        * remove it from temporary folder
    - if the page is relevant to the topic
        * extract some links/URLs from that page
        * if the URL queue is not full and those
           links are not visited
              # insert extracted URLs into the queue
        * extract text content from that page
        * save the content in a permanent storage location
    - move page from the New URL Queue to Visited URL Queue

Initial Setup

1) First we should define our topic.
2) Then we should define a thesaurus for our topic. The simplest way is a list of keywords.
3) Then we need a initial URL seed.
If we don’t know where to start from, the simplest way is to get some links from Google, Yahoo or Wikipedia by searching each term and extracting the addresses. These addresses will be the first URLs in the URL Queue.

Note: Because this project aims to be a very simple, personal web crawler, most of the settings will be hardcoded.
I will not use configuration/properties files or a database for settings and URLs.

Modules composing the simple focused web crawler:

• New URLs Queue
  • a queue of the web addresses that will pe crawled
• Visited URLs List
  • a queue of visited URLs
• Downloader
  • downloads pages from the internet
  • only text pages (text/plain, text/html, etc)
• Topic checker
  • cheks if a downloaded page is relevant to a pre-defined topic
• Link extractor
  • extracts links from downloaded html pages
• Content extractor
  • extracts pure text content from downloaded html pages
  • no tags, just text

Starting with this post I’ll publish a simple tutorial and java code for building a simple serial focused web crawler

Defining the project

In the next post, “A Simple Serial Focused Web Crawler 2″, I’ll present the simple architecture of this focused web crawler.

IMPORTANT:
“While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.” [Shkapenyuk, V. and Suel, T. (2002)]
The purpose of these posts (regarding A Simple Serial Focused Web Crawler) is to present and implement a very simple yet functional approach to a web spider. The result, a minimal focused web spider, could be used as a start point for bigger projects or as a home crawler/documentation downloader.