I recently came across MOA (Massive Online Analysis), an environment for what its developers call massive data mining, or data stream mining. This New Zealand-based project is related to Weka, a Java-based framework for machine learning which I’ve used quite a bit over the years. Data stream mining differs from plain old data mining in that the data is assumed to arrive quickly and continuously, as in a stream, and in an unpredictable order. Therefore the full data set will typically be many times larger than your computer’s memory (which already rules out some commonly used algorithms), and each example can only be briefly examined once, after which it is discarded. Therefore the statistical model has to be updated incrementally, and often must be ready to be applied at any point between training examples.

I also came across a press release describing version 2.0 of KnowledgeMiner for Excel, a data mining software apparently used by customers like Pfizer, NASA and Boeing, and which is based on GMDH (Group Method of Data Handling), a paradigm I hadn’t heard about before. I failed to install KnowledgeMiner for Excel for my Mac due to some obscure install error, but from what I gather, the GMDH framework involves a kind of automatic model selection, making it easier to use for non-experts in data mining. (Of course I haven’t tried it, so it’s hard to evaluate the claim.) The example data set provided with the software package has to do with climate data and modeling, so it should be fun to try as soon as I get it working:

The new KnowledgeMiner is now capable of high-dimensional modeling and prediction of climate and has an included example using air and sea surface temperature data. This is a first for a data-mining software package: to offer anyone the ability to see for themselves that global temperatures are rising steadily, using publicly available data. The biggest surprise is seeing that the changes are greatest and accelerating in the northern latitudes. By using data from the past, KnowledgeMiner (yX) can show predictions for future years. Go to this link to see the climate change data displayed graphically in a slideshow through the year 2020:

There’s also an interesting new toolkit for web mining from BixoLabs. They’ve built what they call an elastic web mining platform in Amazon’s Elastic Compute Cloud (on top of Hadoop, Cascading and a web mining framework called Bixo, for those of you who care). The whole thing is pre-configured and scalable, and from the tutorials on the site, it seems pretty easy to set it up to crawl the web to your heart’s content.

The WEKA (http://www.cs.waikato.ac.nz/ml/weka/) software is very powerful data mining tool that allows you to test your datasets with a broad set of classifiers, and cluster functions. One nice feature is that you can use their JAVA API to develop and incorporate in your own applications and avoiding to write implementations of SVM, K-NN,… algorithms.

Another tool that i used a lot was Matlab – MAthWorks, which in the last version (2008) allows us to write code in a object oriented way. And because is built on top of a Java Virtual Machine, it now allows to use java objects on their .m files.

The code snippets that i will show is to use the WEKA library on Matlab files, but it can be used to link any kind of java library.

file dummy.m:

//include jar libraries to matlab environment
javaaddpath('/opt/weka-3-5-7/weka.jar');
javaaddpath('/opt/weka/libsvm.jar');
//imports java like
import weka.classifiers.*;
import weka.classifiers.Classifier.*;
import weka.classifiers.bayes.BayesNet.*;
import weka.classifiers.Evaluation.*;
//Arff files
ARFFtrainfile = "/path/to/arfftrainfile.arff";
ARFFtestfile = "/path/to/arfftestfile.arff";
//Settings for the classifier
v(1) = java.lang.String('-t');
v(2) = java.lang.String(ARFFtrainfile);
v(3) = java.lang.String('-T');
v(4) = java.lang.String(ARFFtestfile);
v(5) = java.lang.String('-S 0 -K 2 -D 3 -G 0.0010 -R 0.0 -N 0.5 -M 40.0 -C 10.0 -E 0.0010 -P 0.1');
//concatenate the settings in a string
prm = cat(1,v(1:end));
//create classifier instance, and perform the evaluation
classifier = javaObject('weka.classifiers.functions.LibSVM');
weka.classifiers.Evaluation.evaluateModel(classifier,prm)
//open trainset<span style="background-color: #ffffff;"> </span>
reader = javaObject('java.io.FileReader', ARFFTrainfile);
dataset = javaObject('weka.core.Instances', reader);
dataset.setClassIndex(dataset.numAttributes() - 1);
//build classifier model
classifier.buildClassifier(dataset);
...
while (true) {
instance = javaObject('weka.core.Instance', length(attributes_Dataset));
predicted = classifier.classifyInstance(instance);
//predict the class from the instance
class = dataset.attribute(dataset.numAttributes() -1).value(predicted))
}
...

In the code above the part inside the while is for predict the class of each instance; each instance can be any vector but need to have the same number of attributes as the ones in the dataset , in the arff files.

The last couple of days we’ve been tramping New Zealand style.  Although you might draw similarities between our travelling and life on the streets (living out a bag, no fixed abode, smelling a lot, drinking to keep warm at nights) it is not in fact that which we’ve been doing.  Translated ‘tramping’ means ‘hiking’, or it does more or less.

We’ve hiking around the Queen Charlotte track, which is in the Marlborough Sounds area at the top of the South Island.  The first day we drove round the most alpine style road we’ve had to to date and then walked along the end of the track for about three hours – stopping at beaches and the odd lookout.  The next day we did a day walk at the opposite end of the sound by taking a small water taxi to the end of the walk, walking for about five hours then being picked up at a different jetty to be taken back to Picton town.  All of this was really cool, with lovely bush and coast.  The water is really clear and inviting, especially knowing it doesn’t have the usual Australian killer creatures in it!

We did see a little wildlife in the shape of seals, penguins and a weka (http://labarker.com/NZbirds.html#weka).

Later yesterday evening we took the InterIslander ferry across the Cook Strait to the North Island.  All was fairly normal apart from that the military special ops were to perform their annual ‘fast rope’ training half way through the trip!  Basically we were told four helicopters were going to appear and land about a dozen or so soldiers on to the boat by quickly dropping out of the hovering helicopter using a rope – just like in the movies!  It was a totalling amazing sight.  We could hear them coming, but couldn’t see them.  They had no lights on, in the pitch black, and were using night vision goggles.  Then as soon as they arrived men started jumping (you could see them as they had glow sticks on their arms and legs) on to the deck.  Really made the trip that bit more exciting!

Anyway, off to Wellington the capital city this morning…

There’s nothing quite like the double edged blade of progress for producing equal parts spectacular failure and insight. I linked to some of my experiments displaying scatter plot book recommendations. Following feedback I’ve iterated over various displays with less and less “graph” feel. You’ll get a look at that when my first dimension reductions see the light.

On that front I’ve made some progress with Jama, which allows me to represent and manipulate matrices in Java. I ran my lovecraft example data through the paces of SVD, QR, LU, and such, but the real test will be when I try and manipulate the significantly larger amazon Tag matrix.

I DID use the larger data set with Weka, a VERY neat tool that exposes a TON of interesting ways to manipulate large sets of data. This is where I had the largest insights and failures. The failures involved quickly discovering the constraints of my memory as I repeatedly crashed the application with too much data. When I found a subset of the data (and an appropriate heap size) it could handle I actually got some of my first plots of Principal Component data. On one hand I’m giddy and on another I realize that I still have a long way to go. Weka’s Knowledge Flow functionality (complete with flashbacks to MatLab in college) allows for some crazy fast prototyping and experimenting.

I’ve got earlier editions of Numerical Recipes: The Art of Scientific Computing and Data Mining: Practical Machine Learning Tools and Techniques on hold at the library, so it should be a fun weekend.

A new blog is started. God knows its number in the row. Anyhow, I hope that it can deliver solution(s) for my readers.
Who I am? I’m originally a statistician. But I’ve recently been facing with web-based problems, tools and solutions. As I realised that only the prompt answers are adequate even in statistical services, I started to deal with on-line, web-based and one more characteristics, automated reporting.
What did I find? A huge jungle, without specific solutions, where even the directions are questionable sometimes.
I would like to introduce my efforts and solutions in this heroic struggle. The struggle with the statistical tools, like R, SAS, WINBUGS, WEKA, etc., the struggle with web-based environment like PHP, SQL, MYSQL, the struggle with the different CMS’s like WordPress, Textpattern, Typo3 and some connected topics like photo-editing and publishing, archiving, providing and improving security and so on.

Interessante Entwicklung in der WEKA-Affäre. Heute abend veröffentlicht die FDP Iserlohn auf ihrer Homepage zwei Erklärungen. Eine von Westebbe, eine von Stenger und Köpke.

Danach tritt Westebbe mit sofortiger Wirkung aus der FDP aus und steht auch nicht für ein künftiges Ratsmandat bereit. Köpke/Stenger bedankt sich artig.

Damit wollen die Marktradikalen von der FDP den Schaden minimieren, der ihnen ansonsten aus der WEKA-Geschichte erwachsen könnte. Transparenter geht es ja wohl nicht.

Sicher: Westebbe ist erstmal weg. Das allerdings hatte er sich wohl zunächst nicht so vorgestellt, denn sonst hätte er ja gleich seine Ämter niederlegen können. Stattdessen wurde von “ruhen lassen” gesprochen. Vermutlich waren es die heutigen Razzien, die das Fass zum Überlaufen brachten. Gut für den Iserlohner Bürger!

Aber: Eines muss klar sein. Wie bei der Hydra ist es auch bei der FDP. Schlägst Du einen Kopf ab, wachsen zwei nach. Die FDP ist eben einfach und ganz nüchtern betrachtet ein Sammelbecken für die Radikalen. Die Radikalen, die glauben Markt ist alles. Der Trugschluss besteht dabei darin, dass rhetorisch stets davon ausgegangen wird, dass Markt gleichbedeutend mit Chancengleichheit ist.

Das stimmt aber nicht. Heutzutage ist Markt die Dominanz des Kapitals, ein neues Modell von Leibeigenschaft und Fürstentum. Und die FDP ist der Fürsten Lobby!

Wer also bei der FDP das Kreuzchen macht und nicht Unternehmer ist, sollte schnellstens ein EEG anfertigen lassen. Ich vermute einen Befund…

Der deutsch-amerikanische IT-Fachverlag CMP-Weka (“Computer Reseller News”, “Information Week”) fährt seit dem 1. Juli Kurzarbeit. Das bestätigte der Verlag auf kress-Anfrage.

Die Maßnahme laufe bis Ende Januar 2010, sagte CMP-Weka gegenüber dem Branchendienst. Betroffen seien alle Abteilungen. CMP-Weka reagiere auf die “allgemein schlechte Situation im Anzeigenmarkt”, äußert sich ein Verlagssprecher gegenüber kress zu den Hintergründen.

CMP-Weka ist ein Joint-Venture des amerikanischen UBM (United Business Media) mit der deutschen Unternehmensgruppe Weka.

via

IT-Fachverlag: Kurzarbeit bei Weka | NEWS MEDIEN | boersenblatt.net.

Wir erinnern uns: Damals kamen Gerüchte auf, Herbert Goldmann, Leiter einer der überflüssigsten Abteilungen im Iserlohner Rathaus, nämlich der Umweltabteilung, sei ein Freund von IKZ-Möchtegernjournalist Thomas Pütter und umgekehrt. Jedenfalls versuchte Möchtegernjournalist Thomas Pütter dem Grünen-Hobbypolitiker Herbert Goldmann schon damals auf´s Pferd, das da hieß “Übernahme der Abfallentsorgung”, zu helfen. Es klappte nicht.

Wem nun noch Zweifel blieben, Thomas Pütter könnte wirklich irgendwie merkwürdig an der Reputation und Fortentwicklung des Umweltabteilungsleiters interessiert sein, kann nun aufatmen. Denn seit letzten Samstag kann es keinen Zweifel mehr geben. Möchtgernjournalist Thomas Pütter nutzt wiederum seine publizistischen Möglichkeiten, um Hobbypolitiker und Berufsabteilungsleiter Herbert Goldmann vor dem Zorn des Volkes zu schützen.

Wie nämlich bekannt wurde, hat die Umweltabteilung nach der größten Umweltkatastrophe auf Iserlohner Stadtgebiet, dem Großbrand bei WEKA, keinen, nicht einmal den kleinen Finger krumm gemacht. Rund zwei Wochen verharrte die Abteilung, die für sich reklamiert, in Fragen des Umweltschutzes die ausgewiesenen Experten vorzuhalten, in Lethargie und machte … NIX! Erst jüngst entsandte man den einen oder anderen Mitarbeiter an Infostände zur Aufklärung der Bevölkerung im Katastrophengebiet.

Natürlich gerieten der verantwortliche Bereichsleiter, Dr. Peter-Paul Ahrens, wie auch der Leiter der Umweltabteilung entsprechend in die Schusslinie der Kritik. Gar nicht doof versuchten sie sich zu exkulpieren, indem sie frech argumentierten, man habe sie schließlich “nicht hinzu gezogen”.

Man stelle sich folgendes vor: Zwei Männer überfallen eine Spielhalle. Polizisten stehen neben dem Eingang. Die Räuber rennen mit der Beute an den Polizisten vorbei und verschwinden. Die Spielhallenangestellte sieht die Polizisten und fragt aufgeregt: “Warum sind Sie denn nicht eingeschritten?” Antworten die Polizisten: “Naja. Sie haben uns schließlich nicht hinzu gezogen…”

Nicht nur angesichts des Beispiels wird wohl klar, dass diese Argumentation völliger Unsinnn ist. Denn gerade die Mannen des Herbert Goldmann sind bekannt dafür, sich stets und ständig in die Angelegenheiten anderer einzumischen. Das immer unter dem Deckmäntelchen der Umweltrelevanz des jeweiligen Themas, respektive unter Berufung auf die eigene Zuständigkeit. Nur im Falle von WEKA sieht man sich in der Funktion des unbeteiligten Vierten? Lachhaft.

Aber Möchtegernjournalist Thomas Pütter sieht das natürlich so nicht, setzt sich hin und schreibt einen langen Kommentar in sein IKZ-Blättchen. Wortreich regt er sich darüber auf, dass man die Umweltabteilung nicht beteiligt habe, dort hätte man sicherlich Experten gehabt, aber wenn man die nicht abfordere, dann könnten die natürlich auch nicht tätig werden. Mit anderen Worten: “Mein Kumpel Herbert, der konnte da gar nichts für. Der wollte ja, man hat ihn aber nicht gelassen. Doof sind nur die anderen.” Wer´s glaubt, wird vermutlich nicht einmal seelig…

I needed to dump MySQL tables into CSV for use with WEKA. A little google search helped. I found this. I am copying the command here for my reference:

SELECT * INTO OUTFILE ‘result.csv’
FIELDS TERMINATED BY ‘,’ OPTIONALLY ENCLOSED BY ‘”‘
LINES TERMINATED BY ‘\n’
FROM my_table;

Thank you, Terminally Incoherent! I love MySQL too!

Πρόκειται για τη διαδικασία που ακολουθεί ένας στατιστικός ώστε να ανακαλύψει από ένα μεγάλο κομμάτι δεδομένων που έχει την πιο σημαντική πληροφορία, την οποία θα χρησιμοποιήσει αργότερα για τη στατιστική του ανάλυση.

Η διαδικασία της εξόρυξης πληροφοριών από δεδομένα έχει ως στόχο να απλουστεύσει τα δεδομένα ώστε να είναι πιο εύκολη η ανάλυσή τους, να αποκλείσει κάποιες μεταβλητές (ή χαρακτηριστικά, αλλιώς attributes) οι οποίες δεν προσφέρουν επιπλέον πληροφορία για το στατιστικό. Αυτός μπορεί να το καταφέρει αναλύοντας τη σχέση που έχουν οι μεταβλητές μεταξύ τους, η συσχέτιση δηλαδή, ακόμη και να μετασχηματίσει τα δεδομένα για το σκοπό αυτό.

Η διαδικασία της εξόρυξης δεδομένων αποτελεί το 70%-90% του χρόνου για το στατιστικό, ενώ η περαιτέρω στατιστική ανάλυση μόνο το 10%-30%.

Πριν λίγες μέρες ανακάλυψα ένα μάθημα μεταπτυχιακού επιπέδου που προσφέρεται από το Μετσόβιο Πολυτεχνείο το οποίο ασχολείται με την Εξόρυξη πληροφοριών. Υπάρχουν διαφάνειες με τις παρουσιάσεις καθώς και βιντεάκια με τις διαλέξεις. Είναι μία καλή ιδέα για έναν στατιστικό. Ρίξτε μία ματιά.

Το συγκεκριμένο μάθημα περιγράφει κι ένα χρήσιμο πρόγραμμα, open source εφαρμογή, το WEKA.

Πηγή – Χρήσιμοι σύνδεσμοι:
1. Iστοσελίδα του μαθήματος
2. Υπηρεσία εξόρυξης δεδομένων
3. Ιστοσελίδα του weka
4. Τα βίντεο του μαθήματος

The dock where the ferry lands at Stewart Island faces a small half-moon bay full of fishing boats. Though the remote island off the southern tip of New Zealand is increasingly “gentrified” by tourists, it is still far from posh. A ramshackle grocery store, a single hotel/bar, and a top-notch fish and chips restaurant operating out of a modified travel trailer are the highlights of the tiny town of Oban. There are a handful of other attractions: one phone booth, an art gallery that is open for a few hours each day, and a bead-drenched hippie coffee shop offering very expensive internet.

But the main attraction to Stewart Island is the nature. More than three-quarters of the place has been designated as the Rakiura National Park. The endangered, ground-dwelling kiwi bird thrives here, as do many other native birds. Before successive waves of human inhabitants began to arrive, no mammals or predaceous reptiles lived anywhere in New Zealand. Many of the country’s native birds and reptiles – such as the prehistoric, lizardlike tuatara that can live to be over 100 years old – have been driven to the brink of extinction or beyond by a long list of pests featuring rats, cats, possums, mice and pigs. Stewart Island has largely been spared the invasion, by virtue of being so small and so remote. The winters here are harsh; the summer days are long. It is at roughly the same latitude as Seattle but is a microscopic landmass, buffeted by the massive weather patterns that accumulate over the open sea.

My visit to the island was a lazy affair: six of us in a friend’s rustic hut ten minutes outside of town. We drank foliage-flavored water from a tank that funneled rain off the hut’s roof, and watched plump New Zealand pigeons rummage for berries in the bushes outside while muted parrots called kakas swooped overhead. One night I’m pretty sure I heard a kiwi, but we never saw one. During the days we strolled to nearby beaches, and one day hired a local with a little boat to take us over to the neighboring Ulva Island for the afternoon. One friend fished for snapper off the dock, and I got a sunburn while watching one of the kiwi’s un-endangered and unafraid cousins, a weka, try to sample both my beer and my chocolate.

The majority of visitors are more motivated than we were, and come for the hiking. The 18-mile-long Rakiura Track is typically done in three days; hikers sleep in the two huts along the way, which come stocked with firewood provided by park staff. Despite long sections of boardwalks, the track is renowned for its mud: deep, slick and absolutely everywhere. But one hiker told me the beaches and the forests are beautiful, and it is less overrun than the tracks on the mainland.

Part 1 – Python

I downloaded the code for Chapter 2 from the author’s blog. At first I tried to update the code from version 2 to version 3. Some of the main differences had to do with ambiguities in the language with type definitions. Python 3.0 includes some new keywords to clarify issues with exception handling, for example. It also comes with a handy tool called, ‘2t03.py’, which converts the code to work on version 3, but the new generated code had indentation problems. After fixing these, I ran the code from the book:

>>> import pydelicious
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
import pydelicious
File "C:Python30libpydelicious.py", line 89, in <module>
import md5, httplib
ImportError: No module named md5

The md5 encryption library is missing from version 3.0, so I decided that version 3.0 will have to wait.

I installed version 2.6 to find that the md5 library is deprecated, however it only produces a warning. I installed the ‘feedparser.py’ library from feedparser.org. I tried using pydelicious again:

>>> import pydelicious

Warning (from warnings module):
File "C:Python26libpydelicious.py", line 89
import md5, httplib

DeprecationWarning: the md5 module is deprecated; use hashlib instead
>>> py=pydelicious.get_popular(tag='programming')
>>> len(py)
12
>>> py
[{'count': '', 'extended': '', 'hash': '', 'description': u'100 Vim commands every programmer should know', 'tags': u'Vim tips programming editor reference', 'href': u'http://www.catswhocode.com/blog/100-vim-commands-every-programmer-should-know', 'user': u'devabc', 'dt': u'2008-12-08T23:13:36Z'}, ...]

The input continued to print the remaining 11 results in the list. Success! After building the data set, I ran some of the functions from my last blog on delicious users, and found some interesting information about certain users… well, not that interesting.

I also walked through the code for the item-based filtering method that I mentioned in my last blog. Now to test it with data from http://www.grouplens.org/node/73. After setting the path in the ‘recommendations.py’ file, the code ran exactly as printed in the book.

Part 2 – Weka

Following along with the book, I have played around with the formatting of .arff and .csv files. It seems like the only difference between opening a .csv file versus an .arff file in Weka is that in the .arff file you can define numeric attributes with a more specific type, such as ‘real’ instead of ‘numeric’. However it didn’t seem to affect the outcome much in the example I tried.

Looking at the output in figure 10.5, I think I have a good understanding of what the output means.

=== Run Information ===

1. Scheme = the type of classifier used.
2. Relation = the name of the data (as set in the .arff header).
3. Instances = the number of instances in the data (rows in the data section as separated by newlines).
4. Attributes = a section contain the number of attributes followed by their names.
5. Test mode = the way the classifier was tested.

=== Classifier Model (for tree models) ===

1. Decision Tree: the decision tree is displayed here.
   The layout is pretty simple. The rules determine which branch of the tree you traverse until you reach a decision.
2. Number of Leaves = the number of ways a decision can be reached in this model.
3. Size of the tree = the number of nodes.
   This is the number of leaves plus the nodes where the rules are used to determine which path to take.
4. Time Taken to build model = … self-explanatory.

The classifier is then tested by separating the data into 10 sections and training the classifier on 9 of the sections and testing it against the remaining section and cycling the sections. This is called 10-fold cross-validation. The information is compiled, scaled, analyzed, and displayed in the summary section:

=== Summary ===

1. Correctly Classified Instances
2. Incorrectly Classified Instances
3. Error statistics
   I don’t quite understand all the differences between these error values, but it seems like a good section to analyze the ability of this model to predict correctly.

=== Detailed Accuracy By Class ===

This section contains information about how accurate the model was. It contains the precision among other variables.

=== Confusion Matrix ===

The columns of this matrix are the predictions of the model, the rows are the actual results. If the classifier is right, then it will mostly predict the same as the actual data, thus making the number greatest on the main diagonal.

a b <-- classified as
7 2 | a = yes
3 2 | b = no

The confusion matrix is a good way to analyze where the model failed, so as to guess why it might be predicting the data incorrectly.

All-in-all, Weka is a powerful tool for analyzing data. Sweet!

Sweet. As of writing this, halftime of the superbowl, pydelicious does not want to install or play nice. Maybe i’ll figure it out. The good news is that weka is much nicer. It’s a pretty intuitive piece of software too. Everything just fell into place. Running every dataset through every different prediction method seems like it would be pretty fun.

On to J48 and heart disease.

=== Run information ===

Scheme: weka.classifiers.trees.J48 -C 0.25 -M 2
Relation: cleveland-14-heart-disease
Instances: 303
Attributes: 14
age
sex
cp
trestbps
chol
fbs
restecg
thalach
exang
oldpeak
slope
ca
thal
num
Test mode: 10-fold cross-validation

=== Classifier model (full training set) ===

J48 pruned tree
——————

thal = fixed_defect
| ca <= 0
| | exang = no: 50_1 (3.06/1.0)
| ca > 0: >50_1 (10.0)
thal = normal
| ca <= 0: 0
| | cp = typ_angina
| | | trestbps 50_1 (4.0/1.0)
| | | trestbps > 138: 50_1 (20.0/3.0)
| | cp = non_anginal: 50_1 (3.0)
| | | | exang = yes: <50 (2.0)
| | | restecg = normal: <50 (4.0)
| | | restecg = st_t_wave_abnormality: <50 (0.0)
thal = reversable_defect
| cp = typ_angina
| | chol <= 229: 229
| | | age 50_1 (2.0)
| | | age > 48: <50 (3.0/1.0)
| cp = asympt
| | oldpeak 50_1 (8.0/1.0)
| | | restecg = normal
| | | | trestbps <= 136
| | | | | ca <= 0: 0
| | | | | | thalach <= 151: 151: >50_1 (3.0)
| | | | trestbps > 136: >50_1 (4.0)
| | | restecg = st_t_wave_abnormality: >50_1 (0.0)
| | oldpeak > 0.6: >50_1 (57.39)
| cp = non_anginal
| | slope = up: <50 (7.39/1.0)
| | slope = flat
| | | ca <= 0
| | | | trestbps <= 122: 122: >50_1 (3.0)
| | | ca > 0: >50_1 (8.0/1.0)
| | slope = down: <50 (1.0)
| cp = atyp_angina
| | ca <= 0
| | | oldpeak <= 0.1: 0.1: >50_1 (2.75/0.75)
| | ca > 0: >50_1 (2.25/0.25)

Number of Leaves : 30

Size of the tree : 51

Time taken to build model: 0.03 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances 235 77.5578 %
Incorrectly Classified Instances 68 22.4422 %
Kappa statistic 0.5443
Mean absolute error 0.1044
Root mean squared error 0.2725
Relative absolute error 52.0476 %
Root relative squared error 86.5075 %
Total Number of Instances 303

=== Detailed Accuracy By Class ===

TP Rate FP Rate Precision Recall F-Measure ROC Area Class
0.83 0.29 0.774 0.83 0.801 0.809 50_1
0 0 0 0 0 ? >50_2
0 0 0 0 0 ? >50_3
0 0 0 0 0 ? >50_4
Weighted Avg. 0.776 0.235 0.776 0.776 0.774 0.809

=== Confusion Matrix ===

a b c d e <– classified as
137 28 0 0 0 | a = 50_1
0 0 0 0 0 | c = >50_2
0 0 0 0 0 | d = >50_3
0 0 0 0 0 | e = >50_4

There’s the output from running the dataset with the J48 decision tree/classifier with the num class. As you can read above the method was 4 out of 5 times correct in it’s prediction of number?. Thats pretty good. I’m not really sure what the num means in the dataset though. One of J48’s limitations is that it can only predict nominal attributes. The best prediction rate was like 85% right. That’s even better. It was for the fbs category though. I don’t know what that means. But weka can predict t and f’s like no one’s business.

Right now python is my enemy. I’ll post an update later.

Recently, I have finished my bachelor thesis on the topic Machine Learning for Gesture Recognition. The paper is about principles of machine learning using decision trees, rulesets, functions or lazy algorithms, which are already implemented for the machine learning tool WEKA. For this usecase, I’ve developed a paint tool, where a user can record training data in form of geometrical symbols. This symbols can be found in the computer game “Black & White”, where the user can interact by drawing figures with the mouse. Also Opera has such a feature implemented. I used these figures (also complexer too), calculate attributes and evaluated various machine learning strategies, such as a RandomForest or a Multilayer Perceptron. This paper may be used as basics to implement gesture recognition for mobile phones with acceleration sensors.

You will find the paper online in a few days, so if you are interested in it, have a look and tell me what you think about.

*** First written June 16, 2002***

Today is a rather uneventful day. Most of the day is spent working on Midnight Club 2.

The sun came out, however. The rains and winds have past. Sailboats and sea kayaks are out in Oriental Bay in droves. Still chilly though.

Brunch is a hard thing to come by. Most places have stopped serving after 11am. We didn’t wake too late, its just that we pushed eating back a while. We were looking for some pancakes. Evidentally, International House of Pancakes isn’t so international. Everyplace we looked, restaurants offered pancakes for brunch — but every single place had Bacon and Banana Pancakes. We haven’t found out whether or not they mix bacon into the batter or place cooked Bacon on top of the the cooked stack. There just seems to be something wrong about it. Perhaps in a couple months we’ll be ready to venture into the territory of Bacon and Banana Pancakes. Until then, I think they probably have cereal around here someplace.

There is a McDonald’s across the street, so we decide to take that as an alternative. They don’t serve breakfast after 10:30 here (just like home) — but its lunchtime anyway. I look at the menu and decide to go for the Quarter Pounder (not Royale with Cheese – BTW). I could have gone for the MegaFeast burger. It looks like it could be the Big and Tasty. Doesn’t the Big N Tasty have mayo? I’ve never had one before. OR — I could have gone for the Kiwiburger, which has an egg and something else unidentifyable. I ask what it is, but couldn’t understand what the McD employee said. After saying “I’m sorry, I didn’t catch that” and him repeating it and I not understanding it again, I decided to leave it alone.

Its the Queen’s Birthday this weekend. I’m not sure if it was telecast in the States, but there was a huge ordeal in England with Paul McCartny, Rod Stewart, Joe Cocker, Eric Clapton, et al, performing on behalf of the Queen. Most stores are closed or have alternate hours to celebrate the event.  I have to find a technical book on Renderman and bone up before starting work, and now I can’t because a royal figurehead has to have her birthday party.  And everyone is invited.

Also caught the MTV Movie Award. Sigh. It makes me sad. At least Lord of the Rings got best picture. But against Legally Blonde and Fast and the Furious? It just makes me have so much more contempt for the MTV generation.  Is that how our parents felt when we liked movies like Jaws, or Close Encounters or Clash of the Titans?  Did they say “Those darn kids and their Indiana Jones!  Thats not filmmaking!”  Fast and the Furious…REALLY!?  Best picture?  Even if its just a nomination…I have very little hope for the younger generation.

Maori translations of Weta related words.

Weta (as mentioned earlier) – a large indigenous cricket deriving its name from “wetapunga”, which translates to “God of ugly things”

Rongotai = (possibly) “Listen to the Sea”. There are many definitions of both Rongo and Tai — this definition sounds the most poetic.

Weka = Barn Hen. Not really poetic, but the only definition.

Manuka = could be tea-tree — could be anti-inflammatory medicine. Smart money is on tea-tree.

The rest of the day is back to work. Gotta get this thing out of the way before wrangling Gollum.

]]> http://ilmberger.wordpress.com/2008/09/11/aktuelle-online-artikel-bei-wwwmagnusde/ Wed, 10 Sep 2008 23:12:39 +0000 drenegadt http://ilmberger.wordpress.com/2008/09/11/aktuelle-online-artikel-bei-wwwmagnusde/

Ich schreibe gerne und ich hab das Glück, damit sogar Geld zu verdienen. Diese Texte sind allerdings weniger philosophischer Art. Vielmehr handelt es sich um Fachartikel aus der PC- und Home-Entertainment-Branche. Derzeit arbeite ich beim WEKA-Online-Portal. Schau doch einfach mal auf einen Seitensprung vorbei unter www.magnus.de, hier lohnt ein Besuch immer. Und willst Du mal direkt auskundschaften, was ich bei magnus.de so treibe, kannst Du einen der unten angeführten Links als Abkürzung nehmen. 

Mai 2009 

magnus.de History! Erster Computer Zuse Z3 feiert Geburtstag: Konrad Zuses Z3 wurde am 12. Mai 2009 bereits 68 Jahre alt

April 2009

Purpl-ray – Nachfolger von Blu-ray mit 8 Terrabyte erstmals vorgestellt: Mein Beitrag zum 1. April auf magnus.de

März 2009

Krabat auf DVD und Blu-ray: Special und Gewinnspiel zum Film “Krabat”

James Bond 007 – Ein Quantum Trost: Special, Bildergalerie und Gewinnspiel zum neuesten James Bond auf DVD und Blu-ray-Disc

Test – TerraTec Noxon 90elf: Das Internet-Radio für alle Fußball-Bundesliga-Fans im Praxistest

CeBIT-Blog auf magnus.de: Die ungeschminkte Wahrheit zur CeBIT anno 2009

Februar 2009

CeBIT 2009: Die Trends und Highligts auf der weltgrößten Computermesse

Aldi-Notebook Medion Akoya P6613: Vorschau auf ein neues Notebook von unser aller Lieblings-Discounter

Januar 2009 

Magnus-Blog: “Probleme mit Blu-ray” oder “Der Feind in meinem computerfreien Wohnzimmer”

Magnus-Blog: Wohl dem, der noch in die Röhre gucken darf?

Kaufberatung Flachbildfernseher: Die besten TV-Geräte von 32 bis 52 Zoll

Technik-Lexikon HDTV und Flachbildfernseher: Die 50 wichtigsten Begriffe rund um das Thema HDTV und Flachbildfernseher kurz, kompetent und verständlich erkläret

Dezember 2008

Duell der Netbooks: Lidl Targa Traveller 1016 versus Aldi Medion akoya Mini E1210

Sub-Notebook Medion akoya S2210: Swarovski-Notebook reloaded

Notebook Medion Akoya P8610: Desktop-Replacement-Notebook mit 18,4-Zoll-Display bei Aldi

Medion/Tevion life X15002 versus LG 32 PG 6000: Fernseh-Duell zwischen Aldi und Media Markt

Targa-Notebook Traveller 1534: Ausnahmsweise mal wieder ein Mobil-Rechner von Lidl  

November 2008

Medion Navigationssystem P5430: Edles Navi mit 5 Zoll Display und Top-Ausstattung im Experten-Check

Medion-PC Akoya P7300 D: Der neueste Multimedia-PC von Aldi im Experten-Check

Tevion Digitaler Notizblock: Günstiges “Grafiktablett” von Aldi im Experten-Check

Max Payne: Die Film-Kritik und eine Verlosungsaktion zum Action-Streifen mit Mark Wahlberg  

Sub-Notebook Medion akoya S2210: Schmuckes Notebook für Damen bei Aldi

Barbara Schöneberger – Jetzt singt sie auch noch … Live!: Eine kleine DVD-Vorschau

Pearl Navigationssystem NavGear StreetMate GT-43 3D: Vermeintliches Billig-Navi im Praxistest

Oktober 2008

DVD-Tipps zu Halloween: Schaurig schöne Halloween-Filme zum Gruseln

Aldi-Notebook Medion akoya MD 97110 (P6612): Vorschau über das erste 16-Zoll-Notebook beim Lebensmittel-Discounter

Aldi-Navi Medion Design GPS Navigationssystem E4430 (auch E4435 und MD 96960): Einzeltest über den im Oktober 2008 kommenden Wegbegleiter bei Aldi Süd

Aldi Netbook Medion Akoya E1210: Vorschau auf das kommende Aldi-Netbook inklusive kurzem Konkurrenz-Vergleich 

Pearl Somikon Dia- & Foto-Scanner und Universal Foto-Scanner: Praxistests über zwei Bild-Digitalisierer, die leider nicht retten, was zu retten ist…

Pearl auvisio Röhren-Verstärker für iPods: Einzeltest über einen ulkigen Schreibtisch-Beschaller 

September 2008

News über ein Navi (Medion GoPal E3230) und eine externe Festplatte (Medion P83710 / MD90101) bei Aldi

Medion akoya E4320 D: Vorschau auf den neuen Multimedia-PC im September 2008 bei Aldi

Creative ZEN X-Fi vs. Pearl Auvisio WorldBeat WiFi: Die ersten WLAN-MVP-Player für die Hosentasche im Shootout

Top-News Apple iPod: Neuauflage der MP3-Player iPod nano und iPod touch

Chieftec-Gehäuse HTPC H2O: Test eines Desktop-Gehäuses mit integrierter Wasserkühlung

Aldi-Blog: Hightech zwischen Milchtheke und Gemüseregal

Medion Akoya MD 96970: Das Aldi-Notebook im Kurzcheck gegen die Konkurrenz

SAT-Receiver Dreambox DM 8000: IFA-News 2008 zum kommenden HD-Kult-Receiver

August 2008

Notebook Samsung X360: IFA-News 2008 zum Samsung-Notebook X360

In-Ear-Kopfhörer Beyerdynamic DTX 50 Individual: Maßgeschneiderte Nobelhörer im Härtetest

Test Pearl Shiatsu Massageauflage: Draufsetzen, anschalten, wohlfühlen … funktioniert tatsächlich

Test Logitech Squeezbox Duet: Soundübertragung vom PC zur HiFi-Anlage mit kleine Haken…

Blu-ray-Player Sony BDP-S300E & Samsung BD-P1400: Billig-Player müssen nicht immer günstig sein…

Notebook Medion MD 96850 vs. Samsung R60 Aura: Aldi gegen die Media-Markt-Konkurrenz im Vergleich

Ein paar CHIP-Oldies

Test DDR3 versus DDR2-Speicher: Die neue Speichertechnik im Praxisvergleich zur Vorgängertechnik

Test Intel Core-2-Extreme QX9650 Prozessor: Der erste 4-Kern-Penryn-Prozessor im Test

Test PC-Netzteile: Nicht überall wo Power draufsteht, steckt auch wirklich Power drin …

Olá pessoal. Neste fim de semana preparei uma pequena matéria de alguns softwares livres utilizados por quem trabalha com banco de dados. Geralmente estes softwares trabalham com diversos tipos de bancos de dados, livres ou não, e são multiplataforma. Alguns deles seguem o modelo de licença GPL e outros não, mas mesmo assim todos são softwares maduros e muito interessantes para quem trabalha com banco de dados. Esta é apenas uma pequena listagem contendo alguns softwares úteis para o dia a dia.

É importante dizer que não sou especialista nestas ferramentas e que faço uso somente de suas opões básicas. Para mais detalhes, por favor, procurem a documentação da ferramenta, o site do fabricante, etc.. Também é importante dizer que não me responsabilizo por qualquer uso destas ferramentas.

1. Weka

O Weka (Waikato Environment for Knowledge Analisys) é uma excelente ferramenta escrita em Java para mineração de dados. Ele contém um grande conjunto de algoritmos prontos para serem usados. Vale a pena destacar também a grande quantidade de recursos para a limpeza e tratamento de dados, que auxiliam muito o processo de ETL (Extract Transform Load). A figura abaixo apresenta a janela do Weka Knowledge Explorer sendo executada no Mac X, onde podemos perceber que várias abas são disponibilizadas para o tratamento dos dados a serem minerados.

Figura 1. Interface gráfica do Weka.

Link para o download do Weka: http://www.cs.waikato.ac.nz/~ml/weka/index.html

2. TreeMap

O TreeMap é um software que fez a geração de mapas de árvores (TreeMaps). Estes mapas são muito interessantes para a visualização de uma grande quantidade de dados hierárquicos, pois apresentam as informações baseados em retângulos coloridos de acordo com algum atributo das informações. Este tipo de visualização de informação permite uma mineração visual dos dados, tornando mais fácil a descoberta de conhecimento a respeito dos dados. A Figura 2 apresenta o TreeMap com os dados a respeito de uma votação americana. Notem que o estado da Califórnia é apresentado como o maior retângulo e na cor azul. Várias opções permitem customizar a aparência e a navegação dos dados.

Figura 2. Visualizando dados de uma votação americana com o TreeMap.

Link para o download do TreeMap: http://www.cs.umd.edu/hcil/treemap/demos/

3. Mangrove

O Mangrove é um software para Windows apenas que permite a geração e navegação de árvores de decisão. Estas árvores de decisão são úteis para a detecção de padrões e correlações nos atributos dos dados. Vale a pena dar uma navegada pela árvore gerada, e identificar quais são os padrões que os dados apresentam. A Figura 3 apresenta uma árvore de decisão gerada pelo Mangrove, a partir de dados referentes a informações de um zoológico. Além da geração da árvore de decisão, há também como customizar alguns parâmetro do algoritmo e aplicar níveis de Zoom na árvore gerada.

Figura 3. Árvore de decisão gerada pelo Mangrove.

Link para o download do Mangrove: http://www.tetris1d.org/zigah/mangrove/

4. Mondrian

O Mondrian é um excelente servidor OLAP de código livre escrito em Java. Além de permitir a manipulação de cubos de dados através de um browser o Mondrian possui uma extensa API (Application Program Interface) para a manipulação dos seus objetos. Suporte a MDX, geração automática de gráficos e exportação em PDF e arquivo XLS são algumas das suas muitas funcionalidades.

A Figura 4 apresenta o browser FireFox navegando em um cubo montado no Mondrian. Este cubo de dados que aparece na Figura 4 é o mesmo cubo de dados de telefonia que foi apresentado aqui no iMasters por meio da série de  colunas onde expliquei como montar um cubo de dados no Analisys Services.

Figura 4. Cubo de dados de Telefonia implementado no Mondrian.

Link para o download do Mondrian: http://mondrian.sourceforge.net/

5. DBDesigner

O DBDesigner é uma ferramenta de código livre que serve para a modelagem de dados, mais especificamente para a elaboração de diagramas MER (Modelo Entidade Relacionamento). Dentre as suas principais vantagens, podemos citar a fácil geração de código SQL do modelo criado, a separação dos modelos Físico e Lógico, a sua simples interface gráfica e a sua portabilidade. A Figura 5 mostra um modelo multidimensional desenhado no DBDesigner.

Figura 5. Modelo Multidimensional desenha no DBDesigner.

Link para o download do DBDesigner: http://www.fabforce.net/dbdesigner4/

6. SQLUniform

Recentemente, recebi um e-mail solicitando informações sobre uma ferramenta que fosse um front-end no Linux para a administração do SQL Server. Respondi que não conhecia nenhuma, mas o leitor do iMasters e Desenvolvedor Ian Raphael me sugeriu o SQLUniform, um administrador de mais de 20 tipos diferentes de bancos de dados desenvolvido em Java. Para acessar o SQL Server pelo Linux, o SQLUniform utiliza a biblioteca FreeDTS e também pode trabalhar com drivers ODBC ou JDBC.

O SQLUniform também permite a exportação e importação de dados CVS, dentre outras funcionalidades. A Figura 6 mostra o SQLUniform em ação sendo executado em na interface gráfica KDE do Linux e conectado a um servidor MySQL.

Figura 6. SQLUniform administrando uma base de dados do MySQL.

Link para o download do SQLUniform: http://www.sqluniform.com/

7. Pajek

O Pajek é um programa que permite criar, editar e manipular grafos (e não gráficos) de todos os tamanhos. Além disso, ele conta com alguns algoritmos e heurísticas implementadas para facilitar a descoberta de informações. Muito útil para quem trabalha com dados provenientes de redes de relacionamento, como o Orkut, e também com dados hierárquicos. A Figura 7 apresenta um grafo conectado gerado a partir de dados contidos em um arquivo texto.

Figura 7. Grafo conectado gerado pelo Pajek.

Link para o download do Pajek: http://vlado.fmf.uni-lj.si/pub/networks/pajek/

8.  The R Project

O Software chamado ‘The R Project’, ou simplesmente R como é mais conhecido, é um analisador estatístico muito poderoso. Quem precisa trabalhar com dados científicos pode contar com uma ferramenta muito flexível e robusta, a ponto de, em alguns casos, poder substituir o famoso MatLab. Correlação multivariada e testes chi-quadrado são apenas algumas das análises que esta ferramenta para fazer. A Figura 8 apresenta algumas das janelas do R no Mac que podem ser utilizadas durante a manipulação de dados com o R.

Figura 8. Algumas das janelas do R sendo executadas no Mac.

Link para o download do R: http://www.r-project.org/

9. DbGen

O DBGen é um gerador de dados para testes desenvolvido pelo Grupo de Base de Dados e Imagens do ICMC da USP. Ele permite a geração e análise de diversos tipos de dados utilizados para carregar uma base de dados de teste. A Figura 9 mostra a geração de alguns números aleatórios no DbGen.

Figura 9. Geração de dados aleatórios no DBGen.

Para fazer o download do DBGen, antes é necessário o cadastro no site do Grupo de Base de Dados e Imagens.

Link para o download do DBGen: http://gbdi.icmc.usp.br/downloads.php

10. Agata Report

Vários softwares permitem a geração de relatórios customizada pelo usuário. O Agata Report é um projeto de código livre nacional que permite, com uma grande variedade de recursos, a geração de relatórios e gráficos a partir de diferentes bases de dados. Para quem trabalha com o Crystal Reports e o Reporting Sercices, o Agata Report é ótima alternativa de baixo custo. A Figura 10 mostra uma janela de configuração de um relatório no Agata Report.

Figura 10. Janela de configuração de relatórios do Agata Report.

Link para o download do Agata Report: http://www.agata.org.br/

11. iXEdit XML Editor

O iXEdit é um editor de XML desenvolvido em Java bem simples. Ele é útil para a manipulação de arquivos XML com muitas tags e propriedades, além de fornecer suporte para a verificação e validades frente a DTDs e Schemas. A Figura 11 mostra o iXEdit manipulando uma tag de um arquivo XML.

Figura 11. O editor de arquivos XML iXedit manipulando um arquivo XML.

Link para o download do iXEdit XML Editor: http://ixedit.sourceforge.net/

12. Lucene

O Lucene é um biblioteca escrita em Java que contém um mecanismo para se trabalhar com full-text search. De acordo com os desenvolvedores, esta biblioteca apresenta alto-desempenho e possui muitos recursos para a manipulação e pesquisa de diversos tipos de texto. Esta biblioteca deve ser utilizada nas aplicações desenvolvidas em Java para fornecer a funcionalidade de pesquisa full-text.

Link para o download do Lucene: http://lucene.apache.org/java/docs/features.html

Com estas ferramentas livres apresentadas neste post, o desenvolvedor ou DBA tem várias alternativas para tornar o seu trabalha do dia-a-dia mais fácil e produtivo. Vale lembrar que muitos outros tipos de ferramentas livres que auxiliam o trabalho de quem manipula banco de dados ainda existem e estão esperando para serem descobertas e utilizadas.

Por hoje é só pessoal, tenham todos uma maravilhosa semana.

“Porque tu, SENHOR, és a minha lâmpada; e o SENHOR ilumina as minhas trevas.” 2 Samuel 22:29

weka-filters.pdf

weka-select-attributes.pdf

Using Weka in Matlab