NoSQL est le terme anglais sous lequel sont souvent désignés les bases de données émergentes généralement non relationnelles et donc n’utilisant pas le langage SQL pour faire des requêtes sur la base de donnée.

Les bases de données relationnelles sont aujourd’hui essentiellement utilisées dans les entreprises pour stocker les informations critiques sur le business.

J’ai passé plusieurs années à écrire des logiciels pour permette de faciliter l’accès à ces données par des non experts , des utilisateurs normaux non doués de pouvoirs paranormaux pour intérpréter l’algèbre relationnelle et le SQL.  Je faisais déja du NoSQL, mais uniquement pour les utilisateurs … Sous le capot, il y avait du SQL à gogo.

J’ai eu l’opportunité d’aller à la conférence Open Source OSCON 2009 d’O'Reilly et le sujet des bases de données NoSQL était vraiment un sujet très chaud.

Saviez-vous que Voldemort n’est pas seulement un personnage de Harry Potter ? Que Cassandra ne faisait pas que prédire l’avenir en Grèce ? Que Hadoop (avec HBase) n’est pas le bruit que fait une boisson gazeuse mal digérée ? Que CouchDB permettait de se relaxer au niveau du schéma ?

Au commencement, il y eu le verbe, avec un papier de recherche publié par Google qui a fait date:

Bigtable: A distributed storage system for structured data (2006) par Fay Chang ,  Jeffrey Dean ,  Sanjay Ghemawat ,  Wilson C. Hsieh ,  Deborah A. Wallach ,  Mike Burrows ,  Tushar Chandra ,  Andrew Fikes ,  Robert E. Gruber

Ce papier décrit la base de donnée non relationnelle (NoSQL donc) utilisée par Google pour stocker les données du moteur de recherche, de Google Earth et Google Finance.  L’échelle de grandeur de la quantité de données stockée donne le vertige, on parle en petaoctets (ça fait beaucoup de gigaoctets tout ça!).

Celà a donné plein d’idées à d’autres et une pléthore de bases NoSQL ont vu naissance dans la foulée.  Je tiens de source sûre (un architecte de Yahoo en Californie) que la sortie de Hadoop et HBase en Open Source par Yahoo! a été poussée par l’idée de permettre à tout le monde de faire la même chose que Google avec BigTable et Map/Reduce…

Je ne vous ai pas encore parlé du théorème CAP qui veut qu’entre Cohérence,Disponibilité (Availability en Anglais) et tolérance à la partition/découpage du système, on ne puisse en obtenir que deux.  C’est un peu la théorie des quantas de l’informatique …

Vous pouvez voir une vidéo en Anglais du directeur technique de Amazon qui explique les enjeux dans l’utilisation de ces nouvelles bases de données.

Si vous êtes arrivé jusqu’à la fin de cet article et que vous n’avez pas encore mal à la tête, il est temps de lire l’article sur BigTable

CouchDB does replication, but replication needs to be set up after each server restart. This means you need to ensure that replication is restarted whenever the daemon restarts CouchDB. I have never seen replication stop working without a restart, but I prefer being safe to being sorry about replication. To be perfectly honest, I do not trust that my replication initiation after a soft CouchDB restart works properly either so I prefer to monitor the replication and have a safety mechanism in place to restart replication if needed.

There are several ways to monitor replication. You could fetch the status page of all servers and restart replication on servers with an empty page, but that is a kind of brute force approach in my world. A better solution is to use the replication itself to monitor that it works.

Each server updates their timestamp in CouchDB and this is again replicated to the other servers. This gets us a bit of the way, but not all the way. The server you are checking might have received updates from all the other servers, but you don’t know if it’s pushed out anything to the other servers. To solve this, you can add information about the other servers to the local server as well. This will give you a matrix of server replication status.

For each server, you will see the timestamp replicated from the server and a list of timestamps replicated to that server. The latter often being a generation older than the former. Cron can be used to update this data. The cronjob reads all the server timestamps and updates this servers timestamp followed by a list of the other servers timestamp.

A mapper to get a server id to server status out of the db.

map: function(doc) {
  emit(doc._id, doc);
}

Our monitroing database is called server_status. The design containing the mapper is called collections and the view server_list.

A Ruby database checker that can run on cron.

require 'rubygems'
require 'couchrest'
require 'json'
require 'open-uri'

STATUS_DB = 'http://localhost:5984/server_status'
COLLECTIONS = 'collections'
SERVER_LIST = 'server_list'

hostname = ARGV[0]

status_db = CouchRest.database!(STATUS_DB)
status_view = "#{STATUS_DB}/_design/#{COLLECTIONS}/_view/#{SERVER_LIST}"

# Get the current information about this server if available
server_status = begin
  status_db.get(hostname)
rescue RestClient::ResourceNotFound
  {'_id' => hostname}
end

server_status['time'] = Time.new.to_i
# Get the current times of the other servers and update this server's view of them
JSON(open(status_view).read)['rows'].map do |row|
  {'server' => row['id'], 'status' => row['value']}
end.each do |status|
  server_status['servers'][status['server']] = status['status']['time'] unless status['server'] == hostname
end
status_db.save_doc(server_status)

Now you need to determine when to trigger replication restart. This can be handled in the watchdog cronjob. If the highest timestamp seen for this server at other servers is above a threshold, restart replication.

The final loop triggering when the age is above a threshold. The init_replication method just posts a continuous replication trigger to the db:

JSON(open(status_view).read)['rows'].map do |row|
  {'server' => row['id'], 'status' => row['value']}
end.each do |status|
  init_replication(status['server']) if server_status['time'] - status['status']['time'] > THRESHOLD
  server_status['servers'][status['server']] = status['status']['time'] unless status['server'] == hostname
end

Rudimentary init_replication method.

def init_replication(server)
  target = "http://#{server}:5984"
  databases = ['server_status']
  databases.each do |db|
    config = {
            'source' => "#{db}",
            'target' => "#{target}/#{db}",
            'continuous' => true
    }
    payload = JSON.generate(config)
    result = Net::HTTP.new('127.0.0.1', '5984').post('/_replicate', payload, {'content-type' => 'text/x-json'})
    p "replication to #{target}/#{db} failed with #{result.code}" unless result.code == 200
  end
end

We have a monitoring view of replication ages in our system. It shows the matrix of timestamps as age in seconds rather than the actual timestamp since the age is the important metric.

A bonus of this replication monitoring system is that we can access the status page from a mobil phone and get an accurate picture of the replication status. This doesn’t worry me now, but it did when we first set it up. Now it’s just a part of our general monitoring view.

A second podcast! So, I reckon we’re already in the 75th percentile of podcasts, just by doing more than one — 90% of life is showing up, etc etc.

Recorded on 11/19/2009, this episode features myself, Chris Miles, Joe Brandt, Michael McIntosh, and Michael Klatsky (see links to people’s sites, etc. in the sidebar).  We talked about a smattering of things related to “the cloud”, IT management, Amazon AWS, Rackspace, CouchDB, Redis, and other bits. A couple of topical highlights (mostly in order):

• M. Klatsky was kind enough to bring some of his homemade chocolate coffee stout, which we all enjoyed quite a lot.
• Chas described Snowtide’s in-progress move away from any shade of in-house hosting to using Amazon AWS.
• We talked quite a bit about Amazon’s (relatively) new Relational Database Service, and how the real value of “the cloud” may not be the outsourcing of infrastructure management, but the value-added services like RDS that take a ton of busybody work out of the mix for companies for whom the related “bare metal” services are simply cost sinks.
• There was a fairly extended discussion of the mechanics of managing cloud nodes, specifically Amazon EC2 instances, along with the security issues surrounding AWS’ specific authentication mechanisms (e.g. keys, certs, etc). Update: Here’s a good rundown of EC2 key management strategies, focusing on the security ramifications of each.
• Some talk about cloud federation, cloud service lock-in, and maybe how things like Eucalyptus (an open-source reimplementation of (parts of?) AWS’ cloud APIs) might be a way to mitigate the consequences of cloud vendor lock-in.
• Man, what happened to the Sun and IBM clouds?
• Everyone seems to agree that Rackspace is somewhat out of touch w.r.t. “the cloud”, and have fallen behind Amazon AWS technologically.
• How do you prepare for system failure and disaster recovery in the cloud?  Whoa, it’s a lot easier to test recovery in the cloud than in other environments.  Also, mobile self-contained server rooms, and the pain of depending on unresponsive commercial backup software vendors.
• M. Klatsky reveals the origin of his mapu handle for all to know!
• Miles has been checking out Redis as a zero admin message queue
• Chas likes CouchDB because of its pleasant programming model and rock-solid real-time multi-master replication (in contrast to mysql replication, which is touchy at best)
• We all hate the nosql moniker — catchy, but too easily interpreted as confrontational.
• “Functional programming lends itself for deploying in an elastic, distributed infrastructure…” – Miles
• Single sign on options, including Joe’s shop rolling their own (“Ouch!” – Chas), OAuth, OpenID, and Shibboleth, etc.
• Miles wrote an emacs twitter client (called Parakeet) some time ago.  (TweetDeck, watch out! )
• We’re all REST fanboys, apparently.  Although, if you have a toolchain that deeply supports it, SOAP can be pleasant.
• There appears to be a debate out there between “the purists” that consider REST to be only what was described in the original paper that coined the term, and most of the rest of the world that uses the term “REST” to describe any protocol that is stateless and uses HTTP for transport (e.g. not necessarily including URLs for additional operations on a resource, etc).
• Finally, this episode includes the first Strictly Professional easter egg. Maybe it was funnier in person.

BTW, it turns out that I was totally off-base w.r.t. a couple of facets of CouchDB:

1. We’ve never had to think about CouchDB’s versioning at all, so my description of how it works was simply wrong.  Sorry, folks.  Check out the chapter of the CouchDB book versioning and conflict management for the real skinny.
2. Further, I had some wires crossed in my head when I said that CouchDB had an option for using a binary protocol like stomp.  CouchDB only supports JSON over HTTP, with binary attachments encoded as you might otherwise do for an HTTP request.

I hope people continue to enjoy the material.  If you have any comments, or questions for us, feel free to leave them, and we’ll see about addressing them in future episodes.

Ever since working on IMM I have enjoyed the fact that I haven’t needed to design a database or write any data access code. Having started working on a small side project I was in a situation where I needed to persist data. My first reaction was to use a simple relational database so I started modeling my domain objects in SQL. I soon realized that I was violating DRY. I thought I could try using nHibernate but that would still require building a database model. I instead decided to go with Windows Azure Table Storage but ended up having trouble getting it to work. Instead I started looking for something with a nice REST interface. I came across Amazon SimpleDB but decided against it, for now. Instead I decided to go with CouchDB. The reason I did this is because I don’t need to worry about defining a schema and access is very fast.

Now being completely honest here, I didn’t stay on CouchDB as I was hoping to hand off the project t some other people at Microsoft so I ported it Entity Framework 1.0. After doing this port I found that I had to destroy my domain model and I had to constantly fight for any level of purity. After a while of fighting this I decided to try out nHibernate after watching Ayende teach Rob Conery how to use it. Midway through the first episode I had it working.

Het is feest in open source database land. De laatste tijd ontstaan er steeds meer alternatieven. MySQL, PostgeSQL en FireBird zijn er al een tijdje, maar nieuwe sterren aan het firmament dienen zich aan. Ik noem er maar een paar: MonetDB (de column-based open source database, een mooi Nederlands product), CouchDB (document georienteerd) en LucidDB (speciaal getarget op DWH en BI).

Het interessante is nu dat  de open source markt zelfs winstgevend wordt; er is klaarblijkelijk steeds meer geld mee te verdienen. Althans, dit is de conclusie die ik trek als ik af ga op de bedrijven die professionele ondersteuning bieden op open source databases. Een paar voorbeelden:

MySQL AB doet dit kunstje al sinds jaren. Inmiddels is MySQL van Sun en Sun van Oracle, maar er is een nieuw bedrijf gesignaleerd dat ook professionele ondersteuning gaat bieden: Monty.

Ingres zelf biedt 24×7 ondersteuning op haar eigen open source database (hoewel voormalig closed source).

En vandaag lees ik dat LucidDB ogenschijnlijk ook volwassen wordt, nu DynamoDB een commerciele versie gaat leveren.

Zijn dit incidenten of lijkt het toch structureel te worden? Wie gebruikt er al open source databases voor DWH’s?

I just watched a moving talk, Damien Katz’s CouchDB and Me. It’s about Erlang and financial security, family and distributed databases, being a good programmer and being unemployed, dreams and bureaucrats at IBM.

At the Rubyfringe conference, Katz gave a personal account about how CouchDB came to be. Programming and some technical aspects naturally play a role, but he mainly talks about the personal decisions he had to make. Giving up your steady income and social position to write cool open source software – why would you do that? He did it and made it from feeling like an unemployed loser to being paid to work on his project, on his own terms, by IBM. He comes across as being really open about his experiences and his decisions, and I certainly took away a couple interesting insights. Find out what interests you and just start doing it, learning what you need. It’s ok to be driven by the idea of having an interesting story to tell; after all, the story of your life will be what you did and experienced. These are just two out of many things that stuck.

I came across this talk reading a discussion on HN. Like Katz’s talk, it’s kinda about programming, but really about life. Life as a programmer, maybe; but maybe about everyone. It makes me happy to see these discussions. I’m into programming not because I want to live in a cold world of machines, but because computers challenge your mind, because you can create something with them, even on your own, and maybe even make a living out of it. It’s about you in the end, not about the machine.

I have an idea.  I am going to process the 5 minute aggregates of raw detector data I’ve stored in monthly CouchDB databases using R via Rcurl and RJSONIO.  So, even though my data is split into months physically, I can use Rcurl to pull from each of the databases, and then use RJSONIO to parse the json, then use bootstrap methods to estimate the expected value and confidence bounds, and perhaps more importantly, try to estimate outliers and unusual events.

Update, this works great.  Except it reveals that my JSON structure in CouchDB isn’t so great.  The problem is that I’m dumping JSON objects per line.  For example:

 ["1201044", "00:00:00", "Fri", "12"]:{N:8,O:0.001782, Pct:1, lanes: 5, intrvls: 10}

While that looks great on paper, and logically makes sense if you think about pulling a single record, it doesn’t work so well when you process lots of records. While RJSONIO is pretty darn good, it certainly isn’t a mind reader, and it cannot turn a list of such objects into a matrix or data frame without some help. If you just throw the results of the RCurl fetch at RJSONIO, you get the following:


> demo=fromJSON(data)
> demo$rows[1]
[[1]]
[[1]]$key
[1] "1202024" "17:35:00" "Fri" "12"

[[1]]$value
[[1]]$value$N
[1] 427

[[1]]$value$O
[1] 0.04861833

[[1]]$value$Pct
[1] 1

[[1]]$value$lanes
[1] 6

[[1]]$value$intrvls
[1] 10

In words, what that means is that the CouchDB response of {rows:[...]} is parsed as a labeled list by R, so the response is a list with one element, rows, which contains n elements each with an element key which is a list of character vectors, and another element value, which itself is a list containing several named elements N, O, Pct, lanes, intrvls. I couldn’t figure out a quick way to make R figure out that I wanted a data.frame with named entries for each of the key terms and each of the value terms (9 columns by n rows). Many more gray hairs later, I remembered about unlist and got stuff sorted. Here is my suboptimal R script for the next time I take a long break from using R and can’t remember the syntax anymore.

#parameters: month,id,fivemin
id=1202024  ## randomly chosen
fivemin="17:35"
# get every month in parallel.  RCurl is cool that way
month=c("01","02","03","04","05","06","07","08","09","10","11","12")
couchdb = "http://localhost:5984/"
db = paste("d12_2007_",month,"morehash/_design/summary/_view/fivemin?",sep="")
moreurl = paste("group=true&startkey=[\"",id,"\",\"",fivemin,":00\"]&endkey=[\"",id,"\",\"",fivemin,":01\"]",sep="")
uri=paste(couchdb,db,moreurl,sep="");  ## 12 different URIs to fetch
data = getURL(uri)
## make a list to store data temporarily on the first pass
d1=list()
for(i in 1:length(data)){
  ## parse each month in turn
  jsondata = fromJSON(data[[i]])
  ## unlist flattens the R object
  d1[[i]]=unlist(jsondata$rows)
}
## make the list of flattened R objects into a matrix
## by unlisting again, and specifying that I'm expecting 9 columns
dmatrix = matrix(data=unlist(d1),ncol=9,byrow=TRUE)
## finally, make a dataframe explicitly labeling each column as needed and converting to numeric from text
d2= data.frame(id=dmatrix[,1],
                      tod=dmatrix[,2],
                      dow=dmatrix[,3],
                      dom=as.numeric(dmatrix[,4]),
                      N=as.numeric(dmatrix[,5]),
                      O=as.numeric(dmatrix[,6]),
                      pct=as.numeric(dmatrix[,7]),
                      lanes=as.numeric(dmatrix[,8]),
                      intervals=as.numeric(dmatrix[,9]))

Next up is the actual bootstrapping of interesting statistics.

Well, last Friday I posted “So, slotting 4 months of data away.  I’ll check it again on Monday and see if it worked.”

It didn’t.  Actually I checked later that same day and all of my jobs had died due to recv errors.  I’ve tried lots of hacky things but nothing seems to do the trick.  From some Google searching, it seems that perhaps it is a timeout issue, but I can’t see how to modify the perl library to allow for a longer timeout.

So, I wrote a little hackity hack thing to stop writing for 5 seconds, make a new connection, and go on writing.  Now it only crashes out of the loop if that new connector also fails to write.  And I also don’t crash until I save my place in the CSV file, so I don’t repeat myself.  So I’m not getting a complete failure, but it is still super slow.

While the documentation for Tokyo Tyrant and Tokyo Cabinet is super great, it seems to be thin on documentation and use cases/examples for stuffing a lot of data into the table db at once.

Interesting probably unrelated fact.  The crashing only started when I recomputed my target bnum, and boosted it from 8 million to 480 million.

Anyway, I had time today to tweak the data load script, and also to finalize my CouchDB loading script.  Having started two jobs each, and with tokyo tyrant started first, it looks like couchdb is going to finish first (The January job is running three days completed to every one in Tokyo Tyrant job;  the March jobs are closer together, but that Tyrant job started about an hour before everything else).

I guess there is still a way for Tokyo Tyrant to win this race.  I am planning to set up a map/reduce type of view on my CouchDB datastore to collect hourly summaries of the data.  It might be that computing that view is slow, and that computing similar summaries on the Tokyo Cabinet table is faster.  We’ll see.

Since we launched Sincerial with one connected online store, fundies.no on Thursday, I feel it’s time to go through the system and the choices we made on our way to launch.

Hosting
The obvious and easy choice was Amazon Web Services (AWS) Elastic Compute Cloud (EC2). A hosting service such as EC2 allows a lot of flexibility in server solutions including quick ramp up if the luxury problem of needing more servers occurs. Freedom to choose operating system and providing a virtual server resembling what you get from a traditional hosting services were also important for us. There were alternatives, traditional hosting services where you book physical servers would incur a larger fixed cost than we wanted and at the same time, we would have lost the flexibility. We currently fire up servers and test things in no time just to shut those servers down after the test. With a traditional hosting provider, we would need spare servers for testing or used the live servers. Windows Azure could have been an alternative, but it’s just a .NET Windows hosting environment and not flexible enough. I feel I should mention Google App Engine as well, but we didn’t really consider it since it’s too restricted and even more locked in than Windows Azure.

Operating System
We decided to use CentOS for the servers. The reason for this was favorable experience with Fedora over the last year and Ubuntu getting more painful at the same time. Red Hat Enterprise Linux (RHEL) was a contender, but CentOS provides us with the part of RHEL that we need without all the parts that we would pay for, but not use. Debian would have been our choice a few years back, but the Fedora and Ubuntu experiences over the last year brought us down on the Fedora, RHEL, CentOS side and as mentioned, CentOS was the best fit among those three. Windows wasn’t considered. We had absolutely no need for anything Windows in there and remote management and configuration of Linux systems is so much easier while being extremely stable. The Amazon Machine Image (AMI) we use was created by RightScale.

Web Serving Environment
We chose Apache because that’s what we’ve used in the past and Apache is solid and dependable. The downside of Apache is that it’s a big beast that might be overkill for our need. An alternative that we looked a bit at is nginx, but we didn’t want to spend time on that before launch. We will however look more closely at nginx in the future.

Progamming Language
I am a rubyist and we chose Ruby. The first time I tried Ruby, I wrote a few scripts that I would normally have written in Perl. These weren’t big scripts, but far from one-liners. They downloaded some content and analyzed it. Writing the scripts in Ruby took me a bit longer than it would have taken writing them in Perl, but they worked right away. In Perl there is always something wrong unless you have a one-liner. The strongest alternative was Python and between Ruby and Python it comes down to taste. The first readability I got as a Googler was in Python and I did write a lot of Python, but it still doesn’t feel right. I was very happy when I got a chance to sneak myself to a Ruby readability. Java was sort of not considered, but would have been the choice if the lightweight short time-to-market alternatives hadn’t been available. We use Phusion Passenger to serve a combination of pure Ruby racks and Sinatra apps in Apache.

Storage
CouchDB won this one. Key value storage is what we need so relational databases are a complication that we can happily forget about. Selling points of CouchDB was that it was easy to get started, it has an HTTP RESTful API and views are written as Javascript map-reduce. Map reduces is well suited to our calculation needs so we get a lot more done inside the database than we would do with i.e. SQL. The strongest alternative was definitely MongoDB which is very similar to CouchDB, but uses a more classical query language. We also briefly looked at Voldemort, but our calculation need is not suited to Voldemort’s simple key lookup scheme. SimpleDB ties us to Amazon and Hadoop is a bit too heavy for our needs. We didn’t consider MySQL, but that would have been our choice had we needed a relational database.

Conclusion
Our LARC stack (Linux Apache, Ruby, CouchDB) works very well for us and that combination has enabled us to develop our service very quickly with the result being an easy system to maintain. We haven’t really looked back at those choices except for a few moments after upgrading to CouchDB 0.10.0 and seeing that our map reduces that put a lot of work in the reducers had to be rewritten. One look at MongoDB’s query language stopped those regrets. As mentioned when discussing web serving environments, we do consider switching from Apache to nginx, but it’s not a high priority thing and we are happy with Apache and the consideration comes more from curiosity than need. As for CentOS and EC2, they just work and gets out of the way which is exactly what they should do.

Just to have a recollection of this later, some notes.

setting up tokyo tyrant instances, one per month.  I expect about 4 million records a day, so that is 120 million a month, so I set bnum to 480 million, which seems insane, but worth a shot

One thing I noticed was that in shifting from one day tests to one month populate, and with the bump up of bnum from 8 million (2 times 4 million) to 480 million, I’m noticing a significant speed drop on populating the data from four simultaneous processes (one for each of 4 months).

There is write delay of course, and that may be all of it, since the files are big now.

Perhaps there is a benefit from wider tables, rather than one row per data record?  Like one row per hour of data per sensor, or one row per 5 minutes, etc?

Also, as I wrapped up my initial one-day tests, I got some random crashes on my perl script stuffing data in.  Not sure why.  Could be because I was tweaking parameters and stuff.

One final point, the size of the one day of data in tokyo cabinet is about the same as the size of one day of data in couchdb.  I was hoping to get a much bigger size advantage (smaller file).  The source data is about 100M unzipped csv file, and it balloons to 600 M with bnum set at 8 million in a table database.  Of course, it isn’t strictly the same data… I am splitting the timestamp into parts so I can do more interesting queries without a lot of work (give me an average of data on Mondays in July; Tuesdays all year; 8 am to 9 am last Wednesday, etc.

So, slotting 4 months of data away.  I’ll check it again on Monday and see if it worked.

And by the way, I’m sure I’m not the best at this because I haven’t used it much, but it is orders of magnitude faster to use the COPY command via DBIx::Class to load CSV data into PostgreSQL.  Of course, I don’t want to have all of that data sitting in my relational database, but I’m just saying…

Started testing out TokyoCabinet and TokyoTyrant last Friday, and got my initial test program running this morning.  The documentation is pretty good, but I’m still floundering about a little bit.  Not sure what parameters to pass to the b+ tree database file to make it work well for my data; not sure how to set up multiple databases for sharding; etc etc.  On the plus side, my Perl code that loads the data is running at about 50% CPU, so it is doing something rather than waiting around for writes.  On the down side, now I have to write a small program to check on the progress of those writes to make sure that I am actually writing something!

Update.  I am comparing storing in TokyoTyrant with storing in CouchDB.  CouchDB it turns out is faster for me out of the box because of the way Erlang takes advantage of the multi-core processor.  Tokyo Tyrant server just maxes out one core, and so my loading programs wait around for the server to process the data.  CouchDB, on the other hand, will use up lots more cores (I’ve seen the process go about 400% in top).  So loading a year of data with one data reading process per month simultaneously, TokyoTyrant is only up to day 6 of each month, while my CouchDB loader programs are all up to about day 14 in each month.

I’m sure there is a way to set up TokyoTyrant to use multiple CPUs, but I can’t find it yet.

Esses dias, após o Rails Summit, resolvi testar uma tecnologia que eu vi por lá e achei extremamente interessante: CouchDB.

O CouchDB, projeto hospedado no incubador do Apache, é um banco de dados não-relacional, com algumas características interessantes: ao invés de ordenar os dados por tabela, você ordena por documentos, define um “Tipo” no documento, e todos ficam por lá. Depois, para buscar os documentos que você quer, usa-se um conceito chamado de View – basicamente, uma sequencia de duas funções Javascript que criam um índice para seus documentos, e expoem eles. Seu formato nativo é JSON – ou seja, atributos multivalorados, hashes, dinamicamente tipado e tudo o mais. Um exemplo típico de registro:

{
  tipo: "Pessoa",
  nome: "Maurício Szabo"
  idade: 21
}

Para listar apenas as pessoas (indexando-as por nome, por exemplo), usa-se uma função Javascript semelhante com a seguinte:

function(doc) {
  if(doc.tipo && doc.tipo == "Pessoa") {
    emit(doc.nome, doc)
  }
}

Pensei com meus botões: “Puxa, que maravilha! Eu posso formatar no banco de dados os dados da forma que eu quero! E ele ainda indexa isso para mim!”. Aí, resolvi fazer uma experiência – portar o sistema de matrículas que desenvolvemos para CouchDB (afinal, eu não teria que me preocupar com validações, telas, HTML, etc, porque tudo isso já está pronto).

O primeiro problema foi na hora de modelar os dados – o CouchDB permite que um documento seja relativamente grande, mas no sistema que montamos era possível cadastrar horários das disciplinas em simultâneo! Eu não podia colocar tudo isso num documento Disciplina, porque poderia haver conflitos (duas pessoas editando a mesma disciplina, porém editando horários diferentes). Acabei modelando meio “relacional”, mas vi que essa era uma falha da forma como eu tinha pensado no sistema inicialmente.

O segundo problema foi na hora de buscar um período de matrícula – o CouchDB simplesmente não permite que eu busque usando dois campos! Logo, eu não poderia, por exemplo, buscar um periodo.inicio < Date.today && periodo.fim > Date.today. Perguntei para diversas pessoas uma solução, e todas me mandaram a mesma resposta: “Use Views!”. Como? Até que, finalmente, na lista de mensagens do CouchDB eu consegui uma resposta: “Use o Lucene (uma extensão do Couch) ou use Views, emitindo um registro para cada dia que for ser executado o período de matrícula”. Ou seja, se eu tivesse um período de matrícula que fosse do dia 10/01 até o 15/01, eu precisaria emitir, para cada período disponível no banco de dados, 5 índices. Bom, nem tentei o Lucene – se um banco de dados não faz essa coisa tão básica sozinho, eu realmente nem quero ver o restante dele.

Ah, sim, até tinha pensado em fazer um aplicativo (tipo Wiki) usando CouchDB – para ver se quando você usar algo mais com cara de “Documento” ele se sai melhor. Acabou que, se um artigo tem várias Tags, eu não consigo buscar todos os posts que possuem as tags “Ruby” e “Rails”, por exemplo – eu só consigo buscar uma por vez. Acabei desistindo do CouchDB, e agora estou estudando o MongoDB.

Sim, o MongoDB permite eu fazer essas coisas. O problema dele são os Mapeadores Objeto-Relacionais (que tentam, digamos, deixar o MongoDB com cara de banco de dados relacional). Estou inclusive montando meu próprio ORM para Ruby – MongoParadigm. Novidades em breve.

Just a quick recap of the StackOverflow Dev Days conference that I attended last week…

Overall, I thought it was a good experience. This is a one-day “roadshow” conference, where they are getting local speakers to talk on a group of topics in several different cities around the U.S. and Europe. The basic format was 30-55 minute talks on each topic, with a little time for Q&A at the end. The Austin topics were:

• Keynote by Joel Spolsky
• Python
• iPhone development
• FogBugz 7
• asp.net MVC
• jQuery
• Erlang/CouchDB
• Code Review Doesn’t Have to Suck

I had to leave for a while in the middle of the day and missed ASP.NET (oh well) and jQuery (kind of a bummer, but hopefully not much I didn’t already know).

I enjoyed the rest of the sessions. Here are my brief impressions:

Keynote – Joel gave an entertaining talk about the value of simplicity in developing software. Lots of clever examples of how developers ruin the user experience by presenting the user with too many options (dialogs, wizards, settings panels, etc.).

Python – Eric Jones from enthought gave a good introduction to the power of Python by doing a code walkthrough of Peter Norvig’s 21 line spell-checker. Very cool. He also showed some of the mathematical and scientific visualization packages that his company has developed in Python. Interestingly, since all of our activity around Python at UT involves Django and web applications, there was zero web content to this presentation.

iPhone – Jon Johnson ran through putting together a quick demo iPhone app, as well as discussing the pros and cons of iPhone development. It seems like the level of control that Apple exerts over what goes in to the App Store is unprecedented, and a bit unpredictable/inconsistent. He told a story about submitting several basically identical apps (for different conferences) to the store and having all but one accepted.

FogBugz 7 – Basically a sales demo for the new release of FogCreek’s flagship product. I think we sat through this to subsidize the rest of the day, which was a worthwhile trade. It looks like a great product, and the new hosted Mercurial integration and code review tools look really cool, but we already have Jira, and I don’t see us moving to a commercial hosted product anytime soon.

Erlang/CouchDB – Probably my favorite presentation of the day. I had read a bit about CouchDB (and key-value pair databases, generally), but still wasn’t quite sure how they worked, or what problem they were designed to solve. I’m still not an expert, but Damien Katz was an entertaining and engaging speaker, and I think I understand the product class better now. We didn’t get to see any actual Erlang code, though.

Code Review Doesn’t Have To Suck – Jason Cohen gets two awards for the day — one for the most entertaining presenter, and the second for giving an hour-long presentation about code review without pushing the collaborative code review product that he was there at the conference to sell (as a sponsor). And his talk was chock-full of informative bits of research-backed advice about how to do code review right (ex: Code review sessions should last one hour or less; after that time, the number of bugs found per lines of code reviewed drops off dramatically).

Some of the folks I’ve talked to didn’t actually enjoy the conference that much — they felt that there wasn’t a consistent focus, or that the presentations were basically pitches for particular products. And there’s some truth in both of those statements. But, I’m not going to any other conferences anytime soon, and it was good to experience the wider Austin development community. If they do it again next year, I’ll have to look hard at the topic mix to see if it seems like it will offer any new content, but for $99, I’d probably do it again.

Last year on the PHP Conference in Mainz it was the first time i heard about couchdb. I was quite impressed, but had no use case for this database. In the near future i will need to improve our document handling in the applications. So i want to work with couchdb to improve versioning and searching.

You can find more information about couchdb on http://couchdb.apache.org.

As anybody know, the first step is to install the couchdb on the local system. As i am a apple follower . I looked for a guide to install it on snow leopard.

Original information from http://blog.deadinkvinyl.com/2008/07/12/couchdb-on-macosx-leopard/

I did some corrections for the copy and paste guys like me . But the original post has much more information about the single steps.

# Install needed packages via macports
sudo port install icu erlang spidermonkey

# Download the latest couchdb file
tar xvzf apache-couchdb-0.10.0.tar.gz
./configure
make

sudo make install

# Show all used userId’s
dscl . -list /Users UniqueID | awk ‘{print $2}’ | sort -n

# Show all used groupId’s
dscl . -list /Groups PrimaryGroupID | awk ‘{print $2}’ | sort -n

# We use groupid and userid 103 for the couchdbuser
sudo dseditgroup -o create -i 103 -r “CouchDB Users” couchdb
sudo dscl . -create /Users/couchdb
sudo dscl . -create /Users/couchdb UniqueID 103
sudo dscl . -create /Users/couchdb UserShell /bin/bash
sudo dscl . -create /Users/couchdb RealName “CouchDB Administrator”
sudo dscl . -create /Users/couchdb NFSHomeDirectory \
/usr/local/var/lib/couchdb
sudo dscl . -create /Users/couchdb PrimaryGroupID 103
sudo dscl . -create /Users/couchdb Password *

sudo chown -R couchdb:couchdb /usr/local/var/lib/couchdb
sudo chown -R couchdb:couchdb /usr/local/var/log/couchdb

# Startup couchdb

sudo -u couchdb couchdb

Create a copy of the plist file and edit it.

cp /usr/local/Library/LaunchDaemons/org.apache.couchdb.plist \
/var/tmp/org.apache.couchdb.plist
open /var/tmp/org.apache.couchdb.plist

1. Open Root → EnvironmentVariables
2. Click on Add Child
3. Name: PATH
4. Value: /bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/local/bin:/opt/local/sbin
5. File → Save
6. Quit Property List Editor

sudo cp /var/tmp/org.apache.couchdb.plist \
/usr/local/Library/LaunchDaemons/org.apache.couchdb.plist

# To control the database, simply execute
sudo launchctl load \
/usr/local/Library/LaunchDaemons/org.apache.couchdb.plist
# to start
# or
sudo launchctl unload \
/usr/local/Library/LaunchDaemons/org.apache.couchdb.plist
# to stop the database

# Automatically launch
sudo ln -s /usr/local/Library/LaunchDaemons/org.apache.couchdb.plist \
/Library/LaunchDaemons/org.apache.couchdb.plist

Today it would be nice to hit a few home runs.

• Browsera – Automated browser compatibility testing with a free trial available.
• RubyFIT 1.2 – Port of the FIT testing framework from Java to Ruby.
• The DevIL Image Library For Ruby – Another alternative for loading, saving, thumbnailing, scaling, rotations, and so on.
• Tools of the Trade – Mark Richman chimes in on the theme.
• Gem Bundler is the Future – A rundown on the new plan for handling dependency resolution in Rails 3. I’m not yet convinced.
• CouchDB in Ubuntu – Looks like Ubuntu 10.04 is going to set every user up with CouchDB for cloudish storage.

Ever wanted to have your Rails logs into a different location? Better yet… a database. I am very lucky to play around with ADS(Atlantic Dominion Solutions) opensource project CouchLog. “A Rails plugin that uses CouchDB to store app logs rather than the standard log files.“

If you don’t have CouchDB installed in your system, you can check on my other post for installation.

Here’s how you integrate it to your Rails Application as plugin:

script/plugin install git://github.com/ads/couchlog.git

Upon installation of the plugin it automatically creates the couchlog.rb in your lib folder and a sample couchlog.yml file in your config directory as well. With the couchlog.yml contains the information about your CouchDB server.

Add the following lines in your environment.rb

# config/environment.rb
require 'couch_log'

config.log_level = ENV['RAILS_ENV']=='production' ?
ActiveSupport::BufferedLogger::Severity::INFO :
ActiveSupport::BufferedLogger::Severity::DEBUG

config.logger = CouchLog.new(config.log_path, config.log_level)

CouchLog is still under development, feel free to fork it.

This is a very short guide about installation of Couchdb in Ubuntu. I would suspect that you already updated your system, if not… you can do it first before moving forward with this guide.

sudo apt-get update

After that, we are good to go:

sudo apt-get install couchdb

That’s it. Simple and Relax installation… You can see it by pointing your browser to http://localhost:5984/_utils/

The below post is an edited version of a $work e-mail, re-posted here at request of some colleagues that wanted to forward the story. My apologies if some of the bits are unclear due to lack-of-context. In particular, let me make clear:

• we have had a production CouchDB setup for months that works well
• we are planning to keep that production setup roughly intact for many more months and we are not currently planning to migrate away from CouchDB at all
• overall we are big fans of the CouchDB project and its community and we expect great things to come out of it

Nevertheless using pre-1.0 software based on an archaic language with rather crappy error handling can get frustrating

Subject: [RT] MyCouch
From: Leo Simons
To: Forge Engineering

This particular RT gives one possible answer to the question “what would be a good way to make this KV debugging somewhat less frustrating?” (we have been fighting erratic response times from CouchDB under high load while replicating and compacting)

That answer is “we could probably replace CouchDB with java+mysql, and it might even be easy to do so”. And, then, “if it really is easy, that’s extra cool (and _because of_ CouchDB)”.)

Why replace CouchDB?

Things we really like about CouchDB (as the backend for our KV service):

• The architecture: HTTP/REST all the way down, MVCC, many-to-many replication, scales without bound, neat composable building blocks makes an evolvable platform.
• Working system: Its in production, its running, its running pretty well.
• Community: open source, active project, know the developers, “cool”.
• Integrity: it hasn’t corrupted or lost any data yet, and it probably won’t anytime soon.

Things we like less:

• Debugging: cryptic error messages, erlang stack straces, process deaths.
• Capacity planning: many unknown and changing performance characteristics.
• Immaturity: pre-1.0.
• Humanware: lack of erlang development skills, lack of DBA-like skills, lack of training material (or trainers) to gain skills.
• Tool support: JProfiler for erlang? Eclipse for erlang? Etc.
• Map/Reduce and views: alien concept to most developers, hard to audit and manage free-form javascript from tenants, hard to use for data migrations and aggregations.
• JSON: leads to developers storing JSON which is horribly inefficient.

Those things we don’t like about couch unfortunately aren’t going to change very quickly. For example, the effort required to train up a bunch of DBAs so they can juggle CouchDB namespaces and instances and on-disk data structures is probably rather non-trivial.

The basic idea

It is not easy to see what other document storage system out there would be a particularly good replacement. Tokyo Cabinet, Voldemort, Cassandra, … all of these are also young and immature systems with a variety of quirks. Besides, we really really like the CouchDB architecture.

So why don’t we replace CouchDB with a re-implemented CouchDB? We keep the architecture almost exactly the same, but re-implement the features we care about using technology that we know well and is in many ways much more boring. “HTTP all the way down” should mean this is possible.

We could use mysql underneath (but not use any of its built-in replication features). The java program on top would do the schema and index management, and most importantly implement the CouchDB replication and compaction functionality.

We could even keep the same deployment structure. Assuming one java server is paired with one mysql database instance, we’d end up with 4 tomcat instances on 4 ports (5984-5987) and 4 mysql services on 4
other ports (3306-3309). Use of mysqld_multi probably makes sense. Eventually we could perhaps optimize a bit more by having one tomcat process and one mysql process – it’ll make better use of memory.

Now, what is really really really cool about the CouchDB architecture and its complete HTTP-ness is that we should be able to do any actual migration one node at a time, without downtime. Moving the data across
is as simple as running a replication. Combined with the fact that we’ve been carefully avoiding a lot of its features, CouchDB is probably one of the _easiest_ systems to replace

Database implementation sketch

How would we implement the database? If we think of our KV data as having the form

  ns1:key1 [_rev=1-12345]: { ...}
  ns1:key2 [_rev=2-78901]: { subkey1: ..., }
  ns2:key3 [_rev=1-43210]: { subkey1: ..., subkey2: ...}

where the first integer part of the _rev is dubbed “v” and the remainder part as “src”, then a somewhat obvious database schema looks like (disclaimer: schema has not been tested, do not use ):

CREATE TABLE namespace (
  id varchar(64) NOT NULL PRIMARY KEY
      CHARACTER SET ascii COLLATE ascii_bin,
  state enum('enabled','disabled','deleted') NOT NULL
) ENGINE=InnoDB;

CREATE TABLE {namespace}_key (
  ns varchar(64) NOT NULL
      CHARACTER SET ascii COLLATE ascii_bin,
  key varchar(180) NOT NULL
      CHARACTER SET ascii COLLATE ascii_bin,
  v smallint UNSIGNED NOT NULL,
  src int UNSIGNED NOT NULL,

  PRIMARY KEY (ns, key, v, src),
  FOREIGN KEY (ns) REFERENCES namespace(id)
) ENGINE=InnoDB;

CREATE TABLE {namespace}_value (
  ns varchar(64) NOT NULL
      CHARACTER SET ascii COLLATE ascii_bin,
  key varchar(180) NOT NULL
      CHARACTER SET ascii COLLATE ascii_bin,
  v smallint UNSIGNED NOT NULL,
  src int UNSIGNED NOT NULL,
  subkey varchar(255) NOT NULL
      CHARACTER SET utf8 COLLATE utf8_general_ci,
  small_value varchar(512) DEFAULT NULL
      CHARACTER SET utf8 COLLATE utf8_general_ci
      COMMENT 'will contain the value if it fits',
  large_value mediumtext DEFAULT NULL
      CHARACTER SET utf8 COLLATE utf8_general_ci
      COMMENT 'will contain the value if its big',

  PRIMARY KEY (ns, key, v, src, subkey),
  FOREIGN KEY (ns) REFERENCES namespace(id),
  FOREIGN KEY (ns, key, v, src)
      REFERENCES {namespace}_key(ns, key, v, src)
      ON DELETE CASCADE
) ENGINE=InnoDB;

With obvious queries including

  SELECT id FROM namespace WHERE state = 'enabled';

  SELECT key FROM {namespace}_key WHERE namespace_id = ?;
  SELECT key, v, src FROM {namespace}_key WHERE namespace_id = ?;
  SELECT v, src FROM {namespace}_key WHERE namespace_id = ?
      AND key = ?;
  SELECT v, src FROM {namespace}_key WHERE namespace_id = ?
      AND key = ? ORDER BY version DESC LIMIT 1;
  SELECT subkey, small_value FROM {namespace}_value
      WHERE namespace_id = ? AND key = ? AND v = ? AND src = ?;
  SELECT large_value FROM {namespace}_value
      WHERE namespace_id = ? AND key = ? AND v = ? AND src = ?
      AND subkey = ?;

  BEGIN;
  CREATE TABLE {namespace}_key (...);
  CREATE TABLE {namespace}_value (...);
  INSERT INTO namespace(id) VALUES (?);
  COMMIT;

  UPDATE namespace SET state = 'disabled' WHERE id = ?;
  UPDATE namespace SET state = 'deleted' WHERE id = ?;

  BEGIN;
  DROP TABLE {namespace}_value;
  DROP TABLE {namespace}_key;
  DELETE FROM namespace WHERE id = ?;
  COMMIT;

  INSERT INTO {namespace}_key (ns,key,v,src)
      VALUES (?,?,?,?);
  INSERT INTO {namespace}_value (ns,key,v,src,small_value)
      VALUES (?,?,?,?,?),(?,?,?,?,?),(?,?,?,?,?),(?,?,?,?,?);
  INSERT INTO {namespace}_value (ns,key,v,src,large_value)
      VALUES (?,?,?,?,?);

  DELETE FROM {namespace}_key WHERE ns = ? AND key = ?;
  DELETE FROM {namespace}_key WHERE ns = ? AND key = ?
      AND v < ?;
  DELETE FROM {namespace}_key WHERE ns = ? AND key = ?
      AND v = ? AND src =?;

The usefulness for {namespace}_value is debatable; it helps a lot when implementing CouchDB views or some equivalent functionality (“get my all the documents in this namespace where subkey1=…”), but if we decide not to care, then its redundant and {namespace}_key can grow some additional small_value (which should then be big enough to contain a typical JSON document, i.e. maybe 1k) and large_value columns instead.

Partitioning the tables by {namespace} manually isn’t needed if we use MySQL 5.1 or later; table partitions could be used instead.

I’m not sure if we should have a ’state’ on the keys and do soft-deletes; that might make actual DELETE calls faster; it could also reduce the impact of compactions.

Webapp implementation notes

The java “CouchDB” webapp also does not seem that complicated to build (famous last words?). I would probably build it roughly the same way as [some existing internal webapps].

The basic GET/PUT/DELETE operations are straightforward mappings onto queries that are also rather straightforward.

The POST /_replicate and POST /_compact operations are of course a little bit more involved, but not that much. Assuming some kind of a pool of url fetchers and some periodic executors…

Replication:

1. get last-seen revision number for source
2. get list of updates from source
3. for each update
   • INSERT key
   • if duplicate key error, ignore and don’t update values
   • INSERT OR REPLACE all the values

Compaction:

1. get list of namespaces
2. for each namespace:
   • SELECT key, v, src FROM {namespace}_key WHERE namespace_id = ? ORDER BY key ASC, v DESC, src DESC;
   • skip the first row for each key
   • if the second row for the key is the same v, conflict, don’t compact for this key
   • DELETE IGNORE FROM {namespace}_key WHERE ns = ? AND key = ? AND v = ? AND src =?;

So we need some kind of a replication record; once we have mysql available using “documents” seems awkward; let’s use a database table. We might as well have one more MySQL database on each server with a
full copy of a ‘kvconfig’ database, which is replicated around (using mysql replication) to all the nodes. Might also want to migrate away from NAMESPACE_METADATA documents…though maybe not, it is nice and flexible that way.

Performance notes

In theory, the couchdb on-disk format should be much faster than innodb for writes. In practice, innodb has seen quite a few years of tuning. More importantly, in our tests on our servers raw mysql performance seems to be rather better than couchdb. Some of that is due to the extra fsyncs in couchdb, but not all of it.

In theory, the erlang OTP platform should scale out much better than something java-based. In practice, the http server inside couchdb is pretty much a standard fork design using blocking I/O. More importantly, raw tomcat can take >100k req/s on our hardware, which is much much more than our disks can do.

In theory, having the entire engine inside one process should be more efficient than java talking to mysql over TCP. In practice, I doubt this will really show up if we run java and mysql on the same box. More importantly, if this does become an issue, longer-term we may be able to “flatten the stack” by pushing the java “CouchDB” up into the service layer and merging it with the KV service, at which point java-to-mysql will be rather more efficient than java-to-couch.

In theory and in practice innodb has better indexes for the most common SELECTs/GETs so it should be a bit faster. It also is better at making use of large chunks of memory. I suspect the two most common requests (GET that returns 200, GET that returns 404) will both be faster, which incidentally are the most important for us to optimize, too.

We might worry java is slow. That’s kind-of silly . In theory and in practice garbage collection makes software go faster. We just need to avoid doing those things that make it slow.

The overhead of ACID guarantees might be a concern. Fortunately MySQL is not _really_ a proper relational database if you don’t want it to be. We can probably set the transaction isolation level to READ UNCOMMITTED safely, and the schema design / usage pattern is such that we don’t need transactions in most places. More importantly we are keeping the eventual consistency model, with MVCC and all, on a larger scale. Any over-ACID-ness will be local to the particular node only.

Most importantly, this innodb/mysql thing is mature/boring technology that powers a lot of the biggest websites in the world. As such, you can buy books and consultancy and read countless websites about mysql/innodb/tomcat tuning. Its performance characteristics are pretty well-known and pretty predictable, and lots of people (including here at $work) can make those predictions easily.

So when are we doing this?

No no, we’re not, that’s not the point, this is just a RT! I woke up (rather early) with this idea in my head so I wrote it down to make space for other thoughts. At a minimum, I hope the above helps propagate some ideas:

• just how well we applied REST and service-oriented architecture here and the benefits its giving us
• in particular because we picked the right architecture we are not stuck with / tied to CouchDB, now or later
• we can always re-engineer things (though we should have good enough reasons)
• things like innodb and/or bdb (or any of the old dbs) are actually great tools with some great characteristics

Just like FriendFeed?

Bret Taylor has a good explanation how FriendFeed built a non-relational database on top of a relational one. The approach outlined above reminds rathe a lot of the solution they implemented, though there’s also important differences.

Videos
Google Tech Talk
CouchDB and Me
CouchDB From 10,000 Feet
CouchDB in a Real-World Setting

Documentation
Slides
Book
The RDBMS is not enough

At work we had a lot of headache figuring out how to reshard our CouchDB data. We have 2 data centers with 16 CouchDB instances each. One server holds 4 CouchDB nodes. At the moment each data center has one copy of the data. We want to improve resilience so we are changing this so that each data center has two copies of the data (on different nodes of course). Figuring out how to reshard was not so simple at all.

This inspired some thinking about how we would do the same to our MySQL instances. Its not a challenge yet (we have much more KV data than relational data) but if we’re lucky we’ll get much more data pretty soon, and the issue will pop up in a few months.

I was thinking about what makes this so hard to think about (for someone with a small brain like me at least). It is probably about letting go of symmetry. Do you have the same trouble?

Imagine you have a database with information about users and perhaps data for/by those users. You might use horizontal partitioning with consistent hashing to distribute this data across two machines, which use master/slave replication between them for resilience. You might partition the data into four shards so you can scale out to 4 physical master servers later without repartitioning. It’d look like this:

Now imagine that you add a new data center and you need to split the data between the two data centers. Assuming your database only does master/slave (like MySQL) and you prefer to daisy-chain the replication it might look like this:

Now imagine adding a third data center and an additional machine in each data center to provide extra capacity for reads. Maybe:

You can see that the configuration has suddenly become a bit unbalanced and also somewhat non-obvious. Given this availability of hardware, the most resilient distribution of masters and slaves is not symmetric at all. When you lose symmetry the configuration becomes much more complicated to understand and manage.

Now imagine a bigger database. To highlight how to deal with asymmetry, imagine 4 data centers with 3 nodes per data center. Further imagine having 11 shards to distribute, wanting at least one slave in the same data center as its master. Further imagine wanting 3 slaves for each master. Further imagine wanting to have the data split as evenly as possible across all data centers, so that you can lose up to half of them without losing any data.

Can you work out how to do that configuration? Maybe:

Can you work out how to do it for, oh, 27 data centers, 400 shards with at least 3 copies of each shard, with 7 data centers having 15% beefier boxes and one data center having twice the number of boxes?

As the size of the problem grows, a diagram of a good solution looks more and more like a circle that quickly becomes hard to draw.

When you add in the need to be able to reconfigure server allocations, to fail over masters to one of their slaves, and more, it turns out you eventually need software assistance to solve the mapping of shards to servers. We haven’t written such software yet; I suspect we can steal it from Voldemort by the time we need it

The tipping point is when you lose symmetry (the 3rd data centre, the 5th database node, etc).

Consistent hashing helps make it easier to do resharding and node rewiring, especially when you’re not (only) dependent on something like mysql replication. But figuring out what goes in which bucket is not as easy as figuring out which bucket goes where. Unless you have many hundreds of buckets, then maybe you can assume your distribution of buckets is even enough if you hash the bucket id to find the server id.

Larry Ellison’s is easily one of the greatest strategists in the software industry (ever). Let’s all hope that his role at Oracle comes in handy for MySQL and Solaris to receive continued support and deserved upgrades at their new home.

The NoSQL movement still has a lot more ground to cover before capturing the full attention of IT managers and developers over using RDBMS-based systems (ie: MySQL). Oracle still has time dress MySQL up, or trash it, in favor of a NoSQL-based data management suite.

Michael Stonebraker (co-creator of Ingres and PostGres), in the 70’s, critized the usability, scalability, and performance of RDBMS-based systems. His reasons being that:

In the data warehouse market, a column store beats a row store by approximately a factor of 50 on typical business intelligence queries. The reason is because column stores read only the columns of interest to the query and not all of them. In addition, compression is more effective in a column store. Since the legacy systems are all row stores, they are vulnerable to competition from the newer column stores.

… In the online transaction processing (OLTP) market, a lightweight main memory DBMS beats a row store by a factor of 50. Leveraging main memory and the fact that no DBMS application will send a message to a human user in the middle of a transaction, allows an OLTP DBMS to run transactions to completion with no resource contention or locking overhead.

… In the science DBMS market, users have never liked relational DBMSs and want a non-relational model and query facility.

… Text applications have never used relational DBMSs. This was pointed out to me most clearly by Eric Brewer nearly 15 years ago in the early days of Inktomi. He wanted to use a relational DBMS to store the results of Web crawling, but found RDBMS to be two orders of magnitude slower than a home-brew system. All the major Web-search engines use home-brew text software to serve us search results. None use relational DBMSs.

He has predicted their demise and saw the dawn of a new era decades ago:
http://www.infoq.com/news/2009/08/NoSQL-and-the-End-of-RDBMS-Era

All is not so gravy yet for the NoSQL movement:
http://bjclark.me/2009/08/04/nosql-if-only-it-was-that-easy/

I wouldn’t be surprised if Oracle jumped on the NoSQL train in order for it to compete with open-source packages in the enterprise.

DBA’s need to start planning on migrating and learning architectures of the already (thriving) open-source projects such as:

http://couchdb.apache.org
http://www.mongodb.org
http://project-voldemort.com

Or, a hybrid solutions such as HadoodDB:
http://db.cs.yale.edu/hadoopdb/hadoopdb.html

Switching gears back to Oracle’s acquisition of SUN: Business Week magazine came out with an article clarifying why the European Union will not stop Oracle from taking over SUN:

http://www.businessweek.com/technology/content/sep2009/tc2009093_421812.htm

Let’s all keep our eyes open for January 19, 2010, as the EU’s set date for making a final decision on the case which will determine if Oracle goes forward with MySQL (hopefully) as a hybrid solution — like HadoopDB.

I enjoy using open-source technologies, but Oracle pumps out extraordinary enterprise-ready solutions that far outweigh free alternatives by features, documentation, and security.

Take what I say with a grain of salt; I am no guru, of course. I am simply enthused to see competition which will hopefully benefit programmers and end-users.

Reference

[1] Oracle Poster on SUN Acquisition to Customers: http://cache0.techcrunch.com/wp-content/uploads/2009/09/oracle.jpg

Just tried out swinger.  It is cool.  But I can’t get authorization to work right using the trunk checkout of couch (0.11.blahblah_git).  Something to hack on

But I’m more interested in playing with Sammy.js.  The two application stack figures on the blog page (and in the Swinger slides) are interesting.  Take away the couchdb bit, add Sakai’s K2, and you’ve got a very similar picture.  Sure couchdb can serve the app with attachments to the _design doc, but that’s not the point.  The point is being able to stick documents into a db and then get them out again in interesting ways without having to bend over backwards on the server side.

But again, I have to play with it for a while and see what it can do.

If you haven’t heard of it and are interested about scaling databases of any sort or making fast web applications, I highly suggest you check out this video.  On a high level, it sounds like everything you’ve ever wanted out of a database.  I am absolutely thrilled about it and am excited to start using it.  It also gives me a good reason to actually get down to learning a functional language, erlang.

http://www.youtube.com/watch?v=ESDBM9-U804

Let me know what you think!