Many applications have the primary concern of storing and retrieving data. The raw data by itself is often not very useful, so an additional process is put into place to turn that data into useful information. Many applications generate these reports from the data quickly at the user’s request through either a narrow SQL select statement, in application data processing, or both. However, in larger applications where the data is too large to handle in memory at a time, processing is too heavy, and report customization is too varied, information generation needs to be pushed to its own scalable system.

A lot of data has various pieces of metadata associated with it naturally. Data such as what user added the record, the date it was added or modified (or both), maybe the size of the record, categories, tags, keywords, or other pieces of associate data that is used to break it up into more manageable chunks. This meta data is useful, but only if we can use it as an aggregate to generate specific information relating to the grouping of items based on this data.

Sometimes generating these more specific reports is as easy as adding additional WHERE or GROUP BY clauses in SQL. However, when more advanced business rules are taking place where there isn’t an easy or succinct way of extracting this information via a query, or if the query returns such a large amount of data as to cause memory issues, a different approach can be taken.

For instance; in an application I am currently working on we need to generate reports based on a table with about 5 million rows. The SQL queries we use can limit the amount of rows returned to perhaps a few hundred thousand for some of our larger reports. However, a lot of the data needs to be processed in application code rather than by the database itself due to some special business rules. Because of this, we end up creating a large number of objects in Java to hold these result rows. If multiple users are generating different reports we might end up holding too many of these objects in memory at a time and receive an OOM error. Also, the processing on this data can be intense enough that if the server is slammed with report requests that the entire system slows down, causing difficulties for people wanting to insert or modify data. This is the case I am in while I contemplate offline report generation.

The basic idea is that the main application should be concerned purely with manipulating the data of the system. That is basic CRUD stuff such as creating new records, updating them, the rare deletions, and showing single records to the user (so they can edit or delete it). We want that part of the application to remain fast, and not be effected by the purely read-only needs imposed by report generation. In order to nullify the impact, we move reporting to its own system that reads from a separate read-only replication of our production database.

When a report request comes in to our application, we send a request to the separate reporting system. This can be done either as a web service or maybe an RPC call. The reporting system uses its read-only copy of the data to generate the report numbers and send it back, causing no speed delay for insertion or regular operation of the main application.

This doesn’t solve our OOM issues however, as many drivers for our database (MySQL) return ResultSet objects with the entire contents of the results which might be too large to fit into memory. However, since we’re using a read-only list anyway we can convert the table or tables we use to process our results into flat files that can be read in on a line by line basis, perform some intermediate result processing, deallocate those lines and work on additional lines. Since our reports are mostly generating statistical data over a large data set, we can process results on that data set in parallel using multiple threads or possibly multiple computers using a Hadoop cluster.

By making report generation asynchronous to our applications general work flow we will free up the processing power and the connection pool that’s used to handle requests by asking users to either poll for when the result is finished or to notify the system when a report is finished and thereby avoid the instances where we use all of our available connections or resources processing reports. There is also the added possibility of continuously generating all possible report data on a separate machine or cluster to decrease access time by increasing storage requirements.

I’m currently researching the use of MapReduce in Hadoop for processing this flat file of all data for reports. In addition, I’m researching a few languages that are reported to be good at concurrent processing so that I can benefit from multiple cores when generating reports from our raw data. My current focus is on Scala, Erlang, and Clojure, but I do not have much to report on those areas yet. If anyone has any information those languages as far as report generation based on a largish data set (currently 5 million, but the rate of growth is fairly alarming), let me know.

Um dos maiores problemas da sociedade moderna é a dificuldade de locomoção diária, a maioria das pessoas passa horas em seus carros, ou em meios de transporte públicos para irem de lugar a outro. Há alguns anos atrás quando morava na zona norte de São Paulo e trabalha na zona sul, essa era minha realidade. Uma vez que naquela época passar por isso era inevitável procurei formas de fazer com esse tempo pudesse de alguma forma torna-se produtivo, foi então que comecei a ouvir à podcasts.

iPod FM radio remote por dan taylor

De acordo com a Wikipedia, Podcasting é uma forma de publicação de arquivos de mídia digital (áudio, vídeo, foto, etc.) pela Internet, através de um feed RSS, que permite aos utilizadores acompanhar a sua atualização. Assim, é possível o acompanhamento e/ou download automático do conteúdo de um podcast.

Neste post apresentarei os podcasts aos quais escuto e os episódios principais para que você ouça. Sugiro que você utilize o iTunes para inscrever-se nos podcasts e sincronizar com seu iPod.

Desenvolvimento Ágil

por pcalcado

Podcast da ImproveIt

por Vinícius Teles
http://improveit.com.br/podcast
Português

• Entrevista com Alisson Vale da Phidelis
• Entrevista com Alexandre Magno na Série Experiências Ágeis
• Entrevista com Carlos Barbieri sobre o MPS.BR
• Entrevista com a equipe da Ancar na Série Experiências Ágeis

AgilCast

Por AgilCoop
http://agilcoop.incubadora.fapesp.br/portal/agilcast
Português

• Uma Visão Geral Sobre Scrum
• Testes Automatizados
• Bancos de dados ágeis e refatoração de bancos de dados

Agile Toolkit Podcast
http://agiletoolkit.libsyn.com
Inglês

• Tom Goulet – Cucumber, Ruby and the transition to Generalizing Specialist (2009)
• Jim Miller – The Product Owner Role and Business Alignmnet
• Tips and Advice – Retrospectives

ThoughtWorks Podcast

http://www.thoughtworks.com/what-we-say/podcasts.html
Inglês

Open Source

FLOSS Weekly

por Leo Laport, Jono Bacon e Randal Schwartz
Inglês

• Entrevista com Kent Beck sobre Extreme Programming (XP)
• Entrevista com Linus Torvalds, o criador do Linux e do Git
• Entrevista com David Heinemeier Hansson criador do Ruby On Rails
• Entrevista com Tim O’Reilly, fundador e CEO da  O’Reilly Media
• Entrevista com John Resig criador e líder do Projeto jQuery
• Entrevista com Jan Lehnardt evangelista do projeto CouchDB
• Entrevista com  Jacob Kaplan-Moss criador do Django
• Entrevista com Bruno Souza sobre o OpenJDK
• Entrevista com Ward Cunningham inventor do Wiki e grande Personalidade da Comunidade Ágil
• Entrevista com  D. Richard Hipp criador do SQLite
• Entrevista com Nate Koechley sobre o Yahoo User Interface Library (YUI)
• Entrevista com Junio Hamano, Mantenedor do Git
• Entrevista com Rasmus Lerdorf, criador do PHP
• Entrevista com Guido van Rossum, Criador do Python
• Entrevista com o fundador da Wikipedia, Jimmy Wales

Java

HorecaExpo - Java por bramloquet

JavaPosse

Por Tor Norbye, Carl Quinn, Dick Wall e Joe Nuxoll
Inglês
http://www.javaposse.com

Java Technology Insider

Inglês
http://www.javaworld.com/podcasts/jtech

• Rod Johnson: SpringSource and the future of Spring (2008)

Grails Podcast

Por Glen Smith e Sven Haiges
http://grailspodcast.com

Ruby

Ruby on Rails por Andrew*

Rails Envy

Por Jason Seifer e Gregg Pollack
Inglês
http://railsenvy.com

Rails Podcast

por Geoffrey Grosenbach
Inglês
http://podcast.rubyonrails.com/

• Entrevista com David Heinemeier Hansson (2005)
• Entrevista com Dave Thomas (2005)
• Entrevista com Chad Fowler (2005)
• Entrevista com Obie Fernandez (2006)
• Entrevista com Dave Thomas e Mike Clark (2006)

Rubiverse Podcast

Por Mike Moore
Ingles
http://rubiverse.com

• Dave Hoover on Software Crafsmanship (2009)
• Obie Fernandez on the Rails Maturity Model (2009)
• Ola Bini on Polyglot Programming (2008)

JavaScript

jQuery Podcast

Português
http://blog.jquery.com/2009/11/13/announcing-the-official-jquery-podcast/

Gadgets

GeekBrief TV

por Cali Lewis
Inglês
http://www.geekbrief.tv

Software

Desk por Guillermo Esteves

Pragmatic Podcasts

por Pragmatic Bookshelf
Inglês
http://www.pragprog.com/podcasts

• Chad Fowler on the Passionate Programmer
• Fred Daoud on Stripes
• Chad Fowler Finding the Jagged Edges
• Andy Hunt on Pragmatic Wetware

Software Engineering Radio

por Software Engineering Radio
http://www.se-radio.net
Inglês

• Software Archaelogy with Dame Thomas
• Fearless Change with Linda Rising
• Learning as a Part of Development with Allan Kelly
• SQL with Jim Melton
• Continuous Integration with Chris Read
• Top 10 Architecture Mistakes with Eoin Woods
• Usability with Joachim Machate
• Introduction to AOP with Christa Schwanninger e Iris Groher
• Retrospectives with Linda Rising
• 10 years of Agile Experiences
• Joe Armstrong on Erlang
• Interview Dave Thomas
• Dick Gabriel on Lisp
• Jeff DeLuca on Feature Driven Development
• Interview Erich Gamma
• Gerard Meszaros on XUnit Test Patterns
• Static Code Analysis with Jonathan Aldrich
• Refactoring Pt. 1
• Refactoring Pt. 2
• eXtreme Programming Pt.1
• eXtreme Programming Pt.2
• Agile Documentation
• Interview Jutta Eckstein
• Interview Eric Evans
• Patterns

Elegant Code

por Elegant Code Community
http://elegantcode.com
Inglês

• Agile for Families
• Entrevista com Jim Wierich o Criador do Rake (Ruby)
• David Laribee on Lean / Kanban
• Uncle Bob Martin on Clean Code
• Alan Shalloway on Lean
• Entrevista com Jarod Ferguson
• Entrevista com Darrel Carver
• Entrevista com Scott Nichols
• Entrevista com Scott Schimanski

Google Developer Podcast

http://code.google.com/p/google-developer-podcast/downloads/list
Inglês

Hearding Code

http://herdingcode.com
Inglês

Tecnologia

IT Conversations

http://itc.conversationsnetwork.org
Inglês

net@Night

por Amber MacArthur e Leo Laport
http://www.twit.tv/natn

Twit – This Week in Tech

por  Leo Laporte, Jeff Jarvis, Baratunde Thurston, e John C. Dvorak
http://www.twit.tv/twit

MacBreak Weekly

por Leo Laporte, Don McAllister, Paul Kent, and Andy Ihnatko
http://www.twit.tv/mbw

This Week in Google

por Leo Laporte, Gina Trapani, Jeff Jarvis e Mary Hodder
http://www.twit.tv/twig

SitePoint Podcast

inglês
http://www.sitepoint.com/podcast

Empreendedorismo e Negócios

37 Signals Podcast

por 37 Signals
Inglês
http://37signals.com/podcast

Max Gehringer (CBN)

por Max Gehringer
Português
http://cbn.globoradio.globo.com/servicos/podcast/NOME.htm

Mundo Corporativo (CBN)

por Heródoto Barbeiro
Português em Áudio
http://cbn.globoradio.globo.com/servicos/podcast/NOME.htm

The Startup Success Podcast

http://startuppodcast.wordpress.com
Inglês

TED Talks

por TED Talks
Inglês
http://www.ted.com

Se você quiser incluir algum outro podcast nesta lista, deixe um comentário. Espero que seja Útil!

I seem to be liking Clojure …

Let me backup a bit, in the last couple of weeks I have been debating between picking up Scala or Clojure (don’t get me wrong Ruby is still my favorite).

I always wanted to pick up a functional programming language so I dabbled a bit with Erlang and Haskell, liked Haskell a lot but without much practice it kind of died (sad times)  and Scala seems too much like Java, yeah I know it seems to have a bigger crowd than Clojure and there are a lot of big names behind it.

Maybe that’s exactly why I choose Clojure (since its the underdog), or cause it is different enough from Java or simply cause it has a better syntax and seems more elegant (apparently Clojure has better integration with Java, don’t quote me on it), anyways I decided to learn Clojure.

Peepcode has a nice screen-cast to get you started off on Clojure. If you are on the Mac there is a nice bundle for TextMate and anywhere else Netbeans with the enclojure plugin seems to be the best.

On a side note it seems more and more that Netbeans has the latest and greatest plugins for everything, then comes IntelliJ and finally eclipse, what’s going on with eclipse ? has it reached its peak and now it will start dropping off ? but on the flip side there seems to be more and more apps built on top of the Eclipse RCP like Xmind, so is Eclipse no longer going to be the leader of the IDE and just become a platform for building RCPs. This of course depends  on what Oracle is going to do with NetBeans, I really hope they give the same amount of love to NetBeans as Sun did.

Ok getting back to Clojure, don’t get your panties in a bunch when you see all those parenthesis, it is just the layout that is shocking, indent it well and it is no more than what you are used to.

Here’s an example

(defn fac
“Returns the factorial of n, which must be a positive integer.”
[n]
(if (= n 1)
1
(* n (fac (- n 1)))
)

Is same as

(defn fac [n] (if (= n 1) 1 (* n (fac (- n 1)))))

But the first one is a lot more easier on the eyes (even brain?) than the second one. Most examples that you see look like the second one and it frightens people, don’t let that stop you take my word and go for it.

Clojure seems to be very easy to pick up, things seem very intuitive, like the other day I was wondering, how to return a default value from a map if the key is not found and there is was right there in the api.

(map key default-value)

So simple! I was easily able to extend the examples that came with the peepcode screencast. Anyways I have started on this path, let’s see where it goes.

Erlang Training and Consulting Ltd in conjunction with the University of Kent intends to launch a pilot version of an e-certification platform for Erlang in the near future.

To assist in this process, we have launched a survey on Erlang Certification. The objective of the study is to better understand the expectations of users, in order to ensure a well-focused product.

We would appreciate a few minutes of your time in completing this survey. To take the survey, please go to:

http://surveys.erlang-consulting.com/index.php?sid=21265&lang=en

The results of the survey will be shared with third parties, in publications and as part of research into Certification. When doing so, all results will be completely anonymised and any individual-identifying information removed.

Thank you for your time and cooperation.

I’m probably going to be shot, stabbed, burned drowned and buried because I’m writing this but I hope that I reach out to enough people before I go out!

My friends, watch out! TDD can be a really huge timewaster.

When did you know exactly how you expected your application/component to behave? Probably you didn’t until you tried a few things out, then sketched on some paper then thought some more then tried some other stuff again and then got back to paper etc. After that you had a perhaps clearer idea of what it was that you want to create. This is prototyping and it should be the first stage in any new/enhanced application. Once this has been done and you know the properties of your system then you can commence TDD.

If you didn’t do prototyping and you naively started writing your unit-tests then you have just wasted your time. I really don’t understand how people can argue that they should write tests first before they implement anything. If you write a test and expect it to mysteriously define your application then you are better off just using water-fall. I don’t feel very “agile” when I’m sitting there writing unit tests for something I will develop just to realize that I need to re-write the tests every 10 minutes. ALL the tests that I write before I write code I need to constantly rewrite… This is a black hole for time.

Instead I propose psuedo-tests.

Before I continue some people are probably going to argue that I “didn’t do it right” or that “it works for me you must be doing something wrong” well I don’t really care because you probably wasted your time and you didn’t know about it. I tried the whole lot and it is such an annoying thing that I don’t understand how it has become so popular. TDD should have a big IF in front of it like this: “IF you are going to do TDD, then make sure that you KNOW what it is you are going to develop”.

So what are psuedo-tests?

A psuedo-test is a test that doesn’t test anything it is just used to conceptually affect the design of your code. It is like psuedo-code but instead describes how the tests will run (instead of how your algorithm or code will run). This has the following benefit on your development

1. Psuedo-code is _very_ easy to change. You don’t have to care about the actual interfaces or compilation or running or changing details everytime your code change.
2. Your code takes testing into consideration. Things like get/set functions and various introspection function are taken into account, you seldom have to add these later. In Erlang e.g. it is usual to have a start/0 function instead of start_link/0 for testing purposes.
3. The tests become simpler and smaller and easier to understand. Why? Well because you don’t have to rewrite your tests 600 times, you rewrite less often. Many people forget that tests are code as well and re-writing something and building on top of things in a very fast pace creates bulky code.
4. If you write enough psuedo-tests then the step to property-based testing is not far (using tools like QuickCheck etc).

Example: We want to create a supervisor, then one test could be:

test_sup() ->
  Start supervisor
  Start a child under the supervisor
  Check that the supervisor has 1 child
  Check that the child of that supervisor is alive
  Kill the child
  Check that the supervisor has 1 child
  Check that the child of that supervisor is alive
  Stop supervisor

Then during implementation you realize that it would be a good idea to kill the children if the supervisor was stopped so instead of implementing it in your test code immediately you simply update your psuedo-test first adding

test_sup() ->
  ...
  Stop supervisor
  Check that the child is dead

This is much faster and you don’t have to care about implementation details… you can get on with your coding. The test is your specification and your guide without wasting your time and as you can see… this isn’t very far from a property if you define it well enough. Then just abstract this test to handle N children and you are good to go for a property test. When you have something substantial to test then implement your tests and run it!

NOTE however:
I am NOT saying that you shouldn’t test! That is not at all what I’m saying… you should always properly test your code. I’m just saying don’t spend all your time writing silly tests just because you are doing “TDD” or whatever… have the tests make sense! Test something substantial that make sense!

Just a thought… Now I’m going to bed…

L’altro giorno mi sono messo a fare il cerca bachi per il backend del gioco della scopa…prima di implementarci altre cose sopra vorrei avere un buon livello di qualità per quel codice…ne è uscita una cosa simile

Un'immagine tratta dal libro: "Code clean"

Alla fine il lavoro non è stato di quelli estenuanti…però ci ho messo un pò a scovare tutti i bachi piu nascosti e insidiosi..o almeno un sacco di essi!

A presto posto qualche idea per il gestore dei players……per avere il codice finale dovrete aspettare che sia tutto pronto!

Java. Эта платформа до сих пор пользуется популярностью лишь из-за того, что огромное количество серверных приложений создаются на JavaEE ++ Hibernate ++ Spring ++ whatever. Все не раз читали рекламные брошюры про быстроту и удобство создания таких приложений и про их надежность.

Но все равно большинство компаний, занимающихся “enterprise” разработками будут использовать подобные инструменты, просто потому, что this is the way it is. Почему огромное количество людей программирует на Java? Как правило, еще в учебном заведении люди знакомятся с C/C++. Знакомство это является скоротечным и очень болезненным, большинство людей так и не понимают, как уследить за памятью и за границами объектов на в сотне строк кода. Я уже не говорю про такие вещи, как умышленный выход за границы объектов, это в наше время вообще высший пилотаж программирования на C. Разумеется, познакомившись с Java, эти люди воспринимают ее как манну небесную, избавление от постоянных сегфолтов, утечек памяти и “очень странного” поведения написанного ими кода. Большое число этих людей даже не отдает себе отчета в том, какой кактус они едят, а среди тех, кто это понимает что-либо сказать против использования Java могут лишь единицы.

Данная ситуация достаточно точно описана здесь

А что же можно предложить вместо Java для создания серверных приложений? Каким вообще должно быть серверное приложение?

Да много каким, но во-первых, серверное приложение должно быть многопоточным. Это приводит нас к выводу, что язык и набор библиотек, которые мы хотим использовать должны предоставлять средства для простого и надежного создания многопоточных приложений. Никакой императивный язык не позволит организовать несколько потоков, да еще и взаимодействующих друг с другом, без головной боли. Какими бы не были средства, которые будут для этого использованы, вы упретесь в проблему изменяемого состояния. А здесь семафоры, блокировки, синхронизация, все радости жизни.

В ерланге новый тред порождается функцией spawn(). That’s it. Если вы хотите послать сообщение процессу, вы используете конструкцию вида Pid ! Message, где “!” — оператор посылки сообщения процессу. Если в вашем процессе вы хотите обрабатывать получаемые сообщения вам достаточно использовать конструкцию вида

receive
         Message1 ->
             ....
         MessageN ->
             ....
end

И все! процесс будет обрабатывать входящие сообщения по мере их поступления. Erlang подталкивает программиста к идее “чем больше тредов — тем лучше”.

В типичных серверных приложениях, как правило возникает несколько типовых задач: получение сообщений от клиентов, обработка этих сообщений, взаимодействие с базой данных, отправка сообщений клиентам.

В данной серии примеров описывается процесс создания TCP сервера, получающего информацию от клиентов в виде XML-докуметов. Сервер должен обработать входящие документы, где нужно — обратиться к БД и ответить клиенту.

Ok, let me put the code where my mouth is.

В первую очередь — информация о модуле, экспортируемых им функциях и макросы.

-module(server).
-export([start/1,listen/1]).

-define(TCP_OPTIONS,[binary,{packet,0},{active,false},{reuseaddr,true}]).

Теперь опишем функцию, стартующую наш сервер, слушающий определенный порт. Здесь мы используем модуль gen_tcp. Слушаем сокет, делаем accept.

start(X) ->
        listen(X).
        %listen
listen(Port) ->
        {ok,LSocket} = gen_tcp:listen(Port,?TCP_OPTIONS),
        accept(LSocket).

Accept соединение и сразу же создаем новый тред, работающий с данным соединенением. Рекурсивно вызываем себя же, чтобы слушать другие соединения на сокете.

accept(LSocket) ->
        {ok,Socket} = gen_tcp:accept(LSocket),
        spawn (fun() -> loop(Socket) end),
        accept(LSocket).

Треды в языке Erlang — функции, выполняемые в отдельном потоке. В нашем случае это функция loop. Эта функция читает из сокета поток байт, превращает его в строку (в Erlang строка — это просто список чисел) и вызывает функцию обработки этой строки, определенную в модуле process_msg. Этот модуль в нашем сервере будет разбирать XML сообщения. Он будет описан в одном из следующих постов.

loop(Socket) ->

        case gen_tcp:recv(Socket,0) of
                {ok,Data} ->
                        io:format("~ts~n",[Data]),
                        String = binary_to_list(Data),
                        process_msg:process_msg(String),
                        loop(Socket);
                {error,closed} ->
                        ok
        end.

Как мы видим, данный код занял менее тридцати строк, при этом являясь многопоточным сетевым приложением. А теперь напишите аналогичное приложение на Java. А теперь напишите аналогичное приложение на С.

One of the most interesting things about some of these languages (such as Scala) is their innate support of concurrency.

Concurrency is a huge benefit for making systems work together or for making programming systems that can scale (that is, grow and adapt to heavier usage).

Ted Leung at his blog wrote an article about concurrency called The Cambrian Period of Concurrency (and a follow-up). He also gave a talk at OSCON 2009 titled A Survey of Concurrency Constructs which was well received.

Tim Bray, also from Sun Microsystems, created a series of articles on the same topic entitled Concur.next. He talks about Haskell, Erlang, and Clojure. Erlang and Haskell are fascinating, but to me Clojure seems like it took Lisp and obfuscated it and made it harder to learn and use – and became slower besides.

After reading Tim’s article, one of the Glaskow Haskell Compiler (GHC) folks wrote an article comparing parallelism and concurrency. (By the way, not everyone thinks of Haskell when one talks of GHC…)

What intrigues me most about concurrent systems is the ability of multiple systems to work together – not as a cluster, nor in a failover configuration – but as a coherent set of multiple systems doing what each does best. A compute-farm with dynamically responding discrete and unique systems would be one example. Another example (which intrigues me to no end) is a network of servers that responds as a whole to a threat on any one of the servers individually.

Concurrent programming over the network, when used to its fullest, means that a server no longer needs to be the largest unit of response: it means that a network may respond, not just the server. It means that information can be shared among a group of servers rather than just one. Concurrency opens up a vast amount of fascinating ideas.

Dopo che avete creato un mazzo di carte per un qualsiasi gioco voi stiate implementando, si pone il problema di mescolare tale mazzo, in modo da ottenere una versione dello stesso in cui le carte non siano piu ordinate per valore e seme, ma seguano un ordine casuale.

Dopo essermi sbattuto un po su google, ho trovato un paio di risorse: un algoritmo da seguire in questi casi, il FISHER-YATES (Knuth shuffle) e un po di codice gia implementato dai simpatici ragazzi di TRAP-EXIT e LITERATEPROGRAMS.

Dopo aver letto entrambi gli articoli sono arrivato al seguente codice, ottenuto dal codice di TRAP-EXIT modificando alcuni nomi di variabile:

shuffle(Deck) ->
   randomize(round(math:log(length(Deck)) + 0.5), Deck).

randomize(1, Deck) ->
   randomize(Deck);

randomize(T, Deck) ->
   lists:foldl(fun(_E, Acc) ->
                  randomize(Acc)
               end, randomize(Deck), lists:seq(1, (T - 1))).

randomize(Deck) ->
   D = lists:map(fun(A) ->
                    {random:uniform(), A}
             end, Deck),
   {_, D1} = lists:unzip(lists:keysort(1, D)),
   D1.

La complessita asintotica di tale codice viene quantificata come O(n log n).

A seguire il codice di LITERATE PROGRAMS, come potete vedere questa implementazione appare molto piu compatta…

shuffle(Deck) -> shuffle(Deck, []).

shuffle([], Acc) -> Acc;

shuffle(Deck, Acc) ->
  {Leading, [H | T]} = lists:split(random:uniform(length(Deck)) - 1, Deck),
  shuffle(Leading ++ T, [H | Acc]).

In questo caso il risultato della complessita asintotica appare essere O(n)….
Ora sta a voi decidere quale implementazione usare…o se farvene una vostra…e ricordate di passare nei siti da cui ho tratto i codici listati sopra!

Shuffling a deck (http://i.ehow.com/images/GlobalPhoto/Articles/5062405/Shuffle-Deck_Full.jpg)

One of my tiny projects reads jobs from different’s datasources and push them into a small database which I is running on my computer.

I am sure this idea is quite crazy and useless in other countries but it perfectly make sense in Spain. First, we don’t have a good job website aggregator. Second, almost all the job offers are old although they update their date everyday and they are shown in “today’s job”. And finally, normally there is no way to control how many job offers you can see in a single page, so normally you have to see them in chunks of 30s which is quite annoying.

Anyway what I want to show here is how easy is to get all the job offers from today in a single website, with no ads and not old jobs using hiberl

%% generate modules and header files
4> hiberl:render(hiberl_drv_mysql, "jobs", "DSN=jobs;UID=username", ["job"]).
created file jobs.erl
created file job.erl, got 8 columns []
ok

%%get the fields of a job
5> job:columns().
["id","position","url","city","company","date","created", "new",[]]

%% Start a session
6> hiberl_session:start_register("DSN=jobs;UID=username").
{ok,}

%% get jobs from today
7>Jobs = job:select("date like '27/10' ").
...

%% A function to create a link using the url and the description
8>F = fun(Job) ->
		Job#job{
                     position = ["<A HREF=\"http:",Job#job.url,"\">",Job#job.position,"</A>"],
                     url = ""}
	end.
...

%% generate html
9> hiberl:render_html(lists:map(F,Jobs)).
ok

%% finish session
10>hiberl_session:terminate().
ok

I’ve always been looking for an elegant way of testing custom ejabberd modules.
Tried a couple of ways before but was never convinced. Running tests against a running ejabberd node for example. But it’s not easy, many dependancies, and hard to set up. Mocking modules such as ejabberd_router. But either I hit weird issues, either it’s so cumbersome, I knew I’d never use it again.

But this time, I think I’ve got it.

Check out the cool combination of etap and erl_mock !

It’s on github with more blathering from yours truly.

Recently I’ve been looking for a job involving Haskell or functional programming in general. No luck. I know that there are is page in Haskell wiki about that, but it’s almost empty, if I filter out those which doesn’t match my preferences.

First thing is: most companies are from US. Next came UK finance companies (like Barclays or Credit Suisse), which are way too far for me. There are also a couple of links to now-dead startups that meant to use Haskell as a silver bullet. No luck I suppose. The one interesting company listed there is Starling Software. Too bad they are from Japan. Another notable exception is Erlang Training and Consulting Ltd. Sadly, while they do have an office in Kraków, they don’t seem to willing to expand their staff there.

I also tried several job search engines for a FP job in Poland, without luck.

My research seems to agree with a previous work you can read about here.

It is sad that most companies that use FP seem to be either too far away or long dead.

I would really love to see one of those near Wrocław.

Well I promised a Part 2, and here it is. Only slightly delayed by the madness of school and of prepping for a Midterm in math. I digress, where were we. Ah yes, Erlang.

I have a hunch the direction the world is going to be taking here in the next 5-10-20 years, at least when it comes to computer programming and assuming we don’t blow ourselves up first of course. What it comes down to, really, is that we have nearly hit a limit with Moore’s law. As Stephen Hawking AND Moore have both said, we have limits and in many ways we have already smashed against those concrete laws of the way the universe works. What this means is that we are quickly headed to a place where a single process on a single thread simply can not, and will not, be able to run any faster, short of making the language itself faster. Unfortunately we have been pretty good about optimizing program languages for speed, meaning that is not even a solution we can count on to save the current mindset when it comes to future problem domains. What this means is that we are being forced to scale horizontally…two and four cores have become the norm on most laptops being sold these days. Intel, if I recall correctly, has a 32 core proc they are putting out next year, assuming all goes well. What does this mean? For the programs of the future, single-thread programming is going to be woefully inadequate, and the supremacy of languages like C and C++ which thrive in single-core architecture is going to be over.

I can already hear the detractors out there ridiculing me. I am not the first person to have said things like this, I won’t be the last but most of all I am not even going to be the most articulate, but I will still try to explain my reasoning, so please bear with me. C++, for example, is not going to be going away anytime soon. Too much is written in it and it does have some very good uses as a “lower level” workhorse language in the future. However anyone who has done any sort of multi-thread programming in a language like C++ knows the horrors that await. The pain, the anguish, the multitude of hours spent trying to find elusive bugs…and that is just with programming for TWO threads. Imagine, if you will, trying to find a deadlock case when you are taking 4, or 8, or 32 threads all accessing/changing the same memory. Oh sure you can limit the number of threads that access the same amount of memory with your architecture but then from the outset you are severely limiting what the program can do, based on a language not built from the ground up for multi-core usage.

There are two possible solutions, really, first you could come up with a language built from the ground up to deal with multi-core programming that is able to do everything and has no need for another language to accomplish anything. Now that would be the ideal solution but I dare say one that I doubt will come true anytime soon. The more realistic solution would be to take a language, such as one like Erlang, which is built from the ground up to deal with multi-core architecture in such a way that the details are largely abstracted from the programmer and then use that language for the overall architecture/framework and leave speed critical sections to languages like C++, if the glue-language is not fast enough for you in certain cases. I just don’t see many other directions for the programming industry to go. Modern day programming languages will have their uses for many years to come, but the future is coming far too fast and the types of changes it is brining are far more than the current major programming languages can bear, and so it is time to look for the proper tool to do the job.

For me Erlang is a huge step in the right direction. To be sure it has its flaws, atoms(constant literals with a name) are not garbage collected and are limited to just over 1 million atoms in a table. This is certainly a lot of atoms but for a language which is supposed to be for long running, zero downtime programs it severely limits their usage, meaning you have to program around them by using lists or some other solution. Interfacing with other languages is not as easy as it should be. Could use a better security model that is not all-or-nothing. Those are some of the major points that Armstrong himself has acknowledged need to be spruced up, so at least we can know they are being worked on to some degree. Despite its flaws Erlang holds enough promise for me to want to really dig in and use it. It is a language with a pretty high level abstraction when it comes to multi-core programming, it has a framework designed to allow for faults and recover from them and it is built from the ground up to facilitate EASY hot swapping of code. As someone who wants create games as a career, specifically of the MMO variety, these sorts of “perks” have quite the appeal. So that, in a not-quite-so-small-nutshell, is why I decided to choose Erlang with which to write my experimental MUD with. Hopefully, with enough time stolen from my school studies I will be able to make some decent progress and make the language really work for me.

Now…back to the grind of schoolwork. Never a days rest for the wicked.

Haven’t written in a while… been busy at work and my hobby project. This weekend I realized why I love Erlang… you don’t really understand the strength of Erlang until you have worked on heavy loaded live systems which need immediate attention.

I’m currently involved in a project which has a lot of subscribers. I can’t go into any details (NDA) but I can say this; We had a problem where one of our applications somewhere in the cluster would go down and not bounce back properly (very simple explanation of the problem )

Anyway, we fixed a problem and got clearance to patch it and then went home. During the evening I wanted to check if the system was behaving, fortunately I could access the site from home. 4 logins later and a lot of SSH:ing and RSA keys and Personal codes I logged on to the shell… that’s when I realized that for me to check what I wanted to check on each node would be incredibly tedious since I would have to go through the connection procedure if I wanted to monitor everything at the same time OR I would have to disconnect and connect to another node OR I would have to RPC individually to each node I wanted to check.

Well anyway… after 3 seconds of thinking I just opened my text editor and wrote this:

ChkApp = fun(App) ->
  lists:map(fun(Node) ->
    case rpc:call(Node, application, which_applications, []) of
      {bad_rpc, Reason} ->
        {Node, Reason};
      Apps ->
        {Node, lists:keymember(App, 1, Apps)}
    end
  end,
  Cluster) end.

Copy/Paste into the shell and then just run…

x> ChkApp(my_application).
[{'node1@192.168.1.10', true},
 {'node2@192.168.1.20', true},
 {'node3@192.168.1.30', true}].

After you are done… monitoring/working just f(ChkApp). Done.

This maybe doesn’t look like much, but think about it…

1. You can in a very easy way run a command on another node and get it back in Erlang terms.
2. You can define Funs and run them how ever you want, you can even nest Funs and build up monitor/introspection tools on the fly
3. There is no XML or complicated/tedious setup you have to make to call other nodes… you just… well… rpc:call them

Now a note of caution; Doing these sort of things on a Live system is very sensitive… be careful but I do have a golden rule I have set for myself: As long as you only read and don’t write data, then you are fine.

I was just sitting an imagining trying to do this in Java… and a big relief hit me…

It is these small things that make me love Erlang.

If you’ve been thinking of trying Riak, but hadn’t yet gotten around to downloading and experimenting, there’s one more resource available this morning that might tip you over the edge. Ben Ahlan and I recorded a screencast demonstrating basic setup and usage of Riak. If you can stand watching two guys mumble over a console for 40 minutes, you may find a tip or two to make your experimentation go smoothly.

I also hear that Martin Scholl gave an awesome talk at NoSQL Berlin. I can’t wait to see the video.

On Saturday morning I opened my web browser, built a playlist of a few songs and started to listen to them while I went about my morning computer tasks. Some of the songs in the playlist were on my laptop, while some were on the mac mini in the family room, and some were on a laptop of a friend that was on the other side of the Atlantic ocean. And if my friend in London had closed his laptop before I listened to ‘his’ song on my playlist it could have been replaced by a copy of the song that was on the computer of a friend in Seattle. I had a seamless music listening experience despite the fact that the music was scattered across a handful of computers on two continents. Such is the power of Playdar.

Playdar is a music content resolver. It is designed to solve one problem: given the name of track, find me a way to listen to it right now. You run Playdar on any computer that you own that has music and Playdar will make it easy to listen to all of that music as if it were on your local machine. The Playdar content resolver can also talk to other Playdar resolvers too, so if Playdar can’t find a track on my local network, it can ask my friend if it knows where the track is, extending my listening reach.

Playdar runs as a web service using standard web protocols for communicating with applications.  When Playdar receives a request to resolve a track it runs through a list of prioritized content resolvers looking for the track. First it checks your local machine, then your local network.  If it hasn’t found it there it could, if so configured, try your friends computers, or even a commercial song resolver (One could imagine for example, a music label offering up a portion of their catalog via a content resolver as a way to expose more listeners to their music).  Playdar will do its best to find a copy of a song that you can listen to now. Playdar enables a number of new listening modes:

• Listen to my music anywhere – with Playdar, I don’t have to shoehorn my entire music collection onto every computer that I own just so I can listen to it no matter what computer I’m on.  I can distribute my music collection over all my computers – and no matter what computer I’m on I have all my music available.
• Save money for music streamers – Music streaming services like Last.fm, Spotify and Pandora spend  money for every song that is streamed.  Often times, the listener will already own the song that is being streamed.   Playdar-enabled music streaming services could save streaming costs by playing  a local copy of a song if one is available.
• Share playlists and mixtapes – with Playdar a friend could give me a playlist (perhaps in a XSPF format) and I could listen to the playlist even if I don’t own all of the songs.
• Pool the music – At the Echo Nest, everyone has lots of music in their personal collections.  When we are all in the same room it is fun to be able to sample music from each other.  iTunes lets you do this but  searching through 15 separate collections for music in iTunes is burdensome.  With Playdar, all the music on all of the computers running Playdar on your local lan can be available for you to search and play without any of the iTunes awkwardness.
• Add Play buttons to songs on web pages -  Since Playdar uses standard web protocols, it is possible to query and control Playdar from Javascript – meaning that Playdar functionality can be embedded in any web page.  I could blog about  a song and  sprinkle in a little Javascript to add a ‘play’ button to the song that would use Playdar to find the best way to play the song.  If I write a review about the new Beatles reissue and want the reader to be able to listen to the tracks I’m writing about, I can do that without having to violate Beatles copyrights.  When the reader clicks the play button, Playdar will find the local copy that is already on the reader’s computer.

Playdar’s Marconi Moment

Playdar is the brainchild of RJ, the inventor of the audioscrobbler and one of the founders of Last.fm.  RJ started coding Playdar in March of this year – but a few weeks ago he threw away the 10,000 lines of C++ code and started to rewrite it from scratch in Erlang.  A few days later RJ tweeted:  I should be taken aside and shot for using C++ for Playdar originally. It’s criminal how much more concise Erlang is for this. Less than 3 weeks after starting from a clean sheet of paper, the new Erlang-based Playdar had its first transatlantic track resolution and streaming. The moment occurred on Friday, October 16th.  Here’s the transcript from the IRC channel (tobyp is Toby Padilla, of MusicMobs and Last.fm fame) when London-based RJ first streamed a track from Toby’s Seattle computer:

[15:40:46] <tobyp> http://www.playdar.org/demos/search.html#artist=pantera&album=&track=burnnn
[15:41:06] <RJ2> woo, transatlantic streaming
[15:41:19] <tobyp> hot!
[15:41:35] <RJ2> playdar’s marconi moment
[15:41:42] <tobyp> hahah

An incredible amount of progress has been made in the last two weeks,  a testament to RJ’s skills as much as Erlang’s expressiveness.  Still, Playdar is not ready for the general public.  It requires a bit of work to install and get running – (yep, the erlang runtime is required), but developer Max Howell has been working on making a user-friendly package to make it easy for anyone to install. Hopefully it won’t be too long before Playdar is ready for the masses.

Even though it is new, there’s already some compelling apps that use Playdar.  One is Playlick:

Playlick is a web application, developed by James Wheare that lets you build playlists. It uses Playdar for all music resolution.  Type in the name of an album and Playlick /  Playdar will find the music for you and let you listen to it.  It’s a great way to see/hear the power of Playdar.

Adding custom content resolvers

One of the strengths of Playdar is that it is very easy to add new resolvers.  If you are a music service provider you can create a Playdar content resolver that will serve up your content.    I wrote a content resolver that uses the Echo Nest to resolve tracks using our index of audio that we’ve found on the web.  This resolver can be used as a backstop.  If you can’t find a track on your computer or your friend’s computers the Echo Nest resolver might be able to find a version out there on some music blog.  Of course, the quality and availability of such free-range music is highly variable, so this resolver is a last resort.

Adding a new resolver to Playdar was extremely easy. It took perhaps 30 minutes to write – the hardest part was figuring out git – (thanks to RJ for walking me through the forks, pushes and ssh key settings).    You can see the code here: echonest-resolver.py.  Less than 150 lines of code, half of which is boilerplate.  150 lines and 30 minutes to add a whole new collection of music to the Playdar universe.   Hopefully soon we’ll see resolvers for music streaming services like Napster, Rhapsody and Spotify.

What’s Next for Playdar?

Playdar is new – and the plumbing and wiring are still be worked on – but already it is doing something pretty magical – letting me listen to any track I want to right now.  I can see how Playdar could be extended into acting as my music agent.  Over time, my Playdar servers will get to know quite a bit about my music tastes.  They’ll know what music I like to listen to, and when I like to listen to it.   Perhaps someday, instead of asking Playdar to resolve a specific track by name, I’ll just be able to ask Playdar to give me a playlist of new music that I might like.  Playdar can then use an Echo Nest, Last.fm or an AMG playlister to build a playlist of interesting, relevant new music.  Playdar won’t just be a music resolver, Playdar will be my music agent helping me explore for and discover new music.

I’ve found two more bugs in wxErlang. The library is definitely not done yet but it has come really far. I guess it’s just a matter of people using it and ironing out the bugs. It looks really promising and GUI programming with Erlang is going to be as easy as a snapp (well… ehm…. almost )

The first bug occurs when net_kernel isn’t started before you create and object and then start net_kernel and the object is called (kindof). I had a temporary fix for that but Dan Gudmundsson is going to fix it in better way… my patch was simply a workaround.
details here: http://www.nabble.com/wxe_server-bug–%28and-patch%29-td25890987.html

Second bug is a bit strange and I don’t have it confirmed yet that it is a bug. It happens when you try to destroy an object from an event callback. E.g. if you create a frame, then connect left_down event, handle the event in a callback and destroy the frame then you can not create a new frame until the process has returned from the callback function.
details here: http://www.nabble.com/wx_object-locks-td25925583.html

Peace.

Tornato dalla Finlandia ho iniziato a pensare ad un nuovo progetto da sviluppare in erlang.

Visto che i miei amici italiani sono diventati dei super fanatici del gioco della scopa, ho pensato che sarebbe stata una buona sfida creare un server che gestisse delle partite a 4 giocatori (in coppie di 2) e mi sono scervellato sulla struttura da implementare, partendo dall’idea di usare piu moduli basati sul gen_server behaviour.

La prima cosa che ho iniziato a sviluppare consiste in un gen_server che gestisce il mazzo e il tavolo di gioco, tale server deve creare un mazzo, gestire le richieste di 10 carte da parte di ogni giocatore e gestire le carte giocate e i tentativi di presa.

Ho identificato alcuni records su cui iniziare l’implementazione..

-record(card, {value, seed}).
-record(deck, {cards=[card]}).
-record(state, {deck, table=[]}).

Tutto si basa sull’idea che ogni carta venga rappresentata come un record a due valori: una value che ne indica il numero (da 1 a 10) e un seed, ovvero un seme.
Ho creato anche un record per il deck (ovvero il mazzo): tale record contiene ha solo un valore, che viene identificato come una lista di carte….alla fine ho deciso che lo stato del mio gen_server per il mio momento sia dato da un record contenente un deck e un table (rappresentante le carte sul tavolo) che di default è una lista vuota….ora, stavo effettivamente pensando che deck e table potrebbero essere basati su di un unico record (una lista di carte)…ma come ho detto tutto questo è ancora uno stub…

Nella funzione di init del mio gen_server devo creare un mazzo e mescolarlo…ecco la mia prima implementazione:

init([]) ->
  Coppe = create_cards("Coppe"),
  Denari = create_cards("Denari"),
  Bastoni = create_cards("Bastoni"),
  Spade = create_cards("Spade"),
  ShuffledDeck = #deck{cards=shuffle(lists:append([Coppe, Denari, Bastoni, Spade]))},
  {ok, #state{deck=ShuffledDeck}}.

create_cards(Seed) ->
  lists:foldl(fun(Number, Acc) ->
                                 [#card{value=Number, seed=Seed} | Acc]
              end, [], lists:seq(1, 10)).

La funzione create_cards(Seed) non fa altro che creare dieci carte (da 1 a 10) per un seme dato in ingresso, usando un accumulalore e il metodo lists:seq/2 che dati due numeri crea una lista di interi dal primo al secondo di passo 1.

Tuttavia a vederlo ora, il codice interno a init([]) non mi sembra molto elegante, quindi forse qualcosa del genere sarebbe meglio:

init([]) ->
  Cards = lists:fold(fun(Seed, Acc) ->
                                      [create_cards(Seed) | Acc]
                     end, [], ["Coppe, "Denari", "Spade", "Bastoni"]),
  ShuffledDeck = #deck{cards=shuffle(lists:append(Cards))},
  {ok, #state{deck=ShuffledDeck}}.

L’ultimo codice tuttavia è solo un abbozzo scritto su due piedi qui sul computer della facolta…quindi potrebbe benissimo non compilare ma l’idea mi sembra migliore…se vi va provatelo e ditemi….io intanto continuo nell’implementazione e nel prossimo post spero di potervi spiegare come mescolare il mazzo, ovvero come randomizzare una lista in erlang!