http://lyricwiki.org/LyricWiki:Wikia_Migration_FAQ

If you ever looked for song lyrics online, you know how most lyrics sites are. Plenty of ads, popups, silly scrolling flash gadgets, bad punctuation, and no easy way to send in corrections. Why not apply to lyrics the same community-driven editing model that has made Wikipedia so successful?

LyricWiki.org has done exactly that, and more. They have provided an API, making it easy for media players to query the database and fetch the lyrics for a specific song. To get an idea of this project’s success, check out these stats. At the time of this writing, LyricWiki is the fifth largest MediaWiki in existence, and the largest wiki that is not a Wikipedia or Wiktionary. What an inspiring example of a community built around the ideals of improving content and making information available. What could go wrong?

Well, how about this. Who owns the copyright on song lyrics? Right, the artist who wrote them. And who represents the artist? Right, a record company. And what do record companies do to nice sites like LyricWiki?

The following message appeared in the lyrics applet of my beloved Amarok earlier this week:

Unfortunately, due to licensing restrictions from some of the major music publishers we can no longer return lyrics through the LyricWiki API (where this application gets some or all of its lyrics).

My first thought when seeing that was, “Huh?! It used to work yesterday.” Apparently, less than 24 hours have passed between the announcement that developers need to change their applications, and the moment the LyricWiki API actually stopped working for retrieving lyrics. In a way, this is a good thing. I hope a lot of users put the pieces together and figured out who is really to blame.

Now let’s see how Amarok handled the situation. Within 12 hours, the bug report had a patch. In the next two hours, the patch was tested, and a glitch was found and fixed. The next day (still less than 24 hours from the time the bug was reported), the patch was pushed to Git. As an Amarok user observing this process from outside, I thought the turnaround time was pretty darn good!

Unfortunately, getting this fix to the users was not as quick and painless. Since lyrics fetching is done in a script, there is no need to recompile Amarok. Furthermore, we have KNewStuff for installing and updating scripts (among other things). So we are just a step away from allowing users to fix the lyrics problem using Amarok’s Script Manager. Why not take it?

(Besides the fact that the script is not on kde-apps, another minor issue is that it only works for Amarok 2.1.1 and trunk. Because of this, the script needs modifications to work with Amarok 2.1, which is still shipped by Kubuntu. But nothing that can’t be fixed in code in 5 minutes.)

Congrats to the Amarok team for being so close to greatness! Amarok was one of the apps that switched me towards KDE in my early days with Linux (the other one was K3B). So I’m glad to see it rocking even harder today.

Now let me return to LyricWiki for a minute. Can someone enlighten me as to why the music publishers would impose such a restriction on them? If I were an artist and you came asking for my lyrics, I would gladly give them to you and thank you for listening to my stuff. I’ve heard of people selling music, but… selling lyrics?!

I hope that as technology makes recording and distribution cheaper, more artists will take matters into their own hands and release their stuff independently.

PS. If you are reading this on Planet KDE and wondering who I am, Hi! I am doing a GSoC project in Akonadi / KMail, mentored by Thomas McGuire. I may provide an update about that later, if the mood strikes.

We delayed the previous datacenter move and adjusted it to a better time of day: 7am EST on a Sunday.

The outage should be between 30 minutes and an hour if everything goes smoothly and indefinitely longer if it doesn’t go smoothly!

Some additional info that hasn’t been on the blog: we were generously given two more servers by a local company that was upgrading (thanks!) and have ordered gobs of RAM for them. Once that is in, we will be upgrading LyricWiki’s setup once again. We currently have 6 servers, this will bring us up to 8! That’s a significant increase and hopefully will make LyricWiki even faster.

The problems earlier today were apparently caused by an attempted distributed denial of service attack.

The datacenter has got all of this bad traffic blocked now, and the site appears to be back to normal (for several hours in a row now).  The whole problem lasted about 1 to 2 hours but it’s tough to measure since the site was never completely down during that time.

The site is currently behaving very sporadically… there are some problems in the connection between our datacenter (where all of our servers hang out) and the internet.

Not sure whether this is related to the “emergency maintenance” Sprint was doing last night or not.

The team at the datacenter is working on it right now, and hopefully we’ll be back soon.

A post on LifeHacker mentioned that Last.fm has released their lists of the most listened-to songs of 2008.  Since their service “scrobbles” (logs) everything that their users listen to, they have a pretty massive data-set.  They are UK-based, so there is quite a bit of a UK bias in the results, but they are still interesting nonetheless.

There were two somewhat-annoying things about the lists: 1) each entry is on a different page, so you have to go to 30 pages to see all of the results 2) no links to lyrics!

So we’ve taken the liberty of compiling them for you into a nice concise list of links to the lyrics-pages:

http://lyricwiki.org/LyricWiki:Lists/2008/Last.fm

Enjoy!

Gerade Metal-Fans müssen sehr häufig darunter leiden bei härteren Stücken mit gutturalem Gesang praktisch nichts vom Text verstehen zu können. Kein Problem möchte man meinen. Ein Blick ins Booklet und schon wird mitgelesen.

Doch so einfach ist es oftmals nicht. Schon alleine der Trend hin zur digital erworbenen Version eines Stückes setzt dem einen Riegel vor. Und auch bei den altmodischen CDs bleibt man oft erfolglos bei der Suche nach den textlichen Ergüssen der Band: Um die Gefahr der Indizierung ihres aktuellen Albums “Knochenkult” zu minimieren, verzichtete die Band Eisregen um Sänger M. Roth auf den Abdruck der gesanglichen Inhalte.

Was tut nun der interessierte Fan? Es bleibt nichts anderes übrig als zu versuchen aus dem Gekreische etwas sinnvoller herauszuhören. Hier gilt wohl der Grundsatz: Je mehr Ohren, umso besser. Doch dafür muss man organisiert sein. Und ein eben solches Mittel zur Organisation ist die Seite LyricWiki.org.

Ich finde es großartig selbst neue Inhalte hinzufügen oder fehlerhafte Texte flink ausbessern zu können. Die Seite ist gut erreichbar, schnell und übersichtlich. Man findet zu fast allem etwas und falls nicht is es ein Vergnügen selbst die Lücke zu füllen.
Ebenfalls sehr praktisch: Es gibt Plugins für alle großen Audioplayer wie zB foobar2000 oder Winamp um die MP3s direkt und automatisch mit Texten im ID3-Tag zu versehen.

Die Artikel Ewigheim, Eisregen: Knochenkult, Ministry: Cover Up, Pagan Fire, Equilibrium: Sagas und noch einige mehr fanden bereits in mir ihren Schöpfer.

So far, LyricWiki has been running pretty well with the new server (our 5th server, named “Cochise“) set up with the rest of them.  I’ve moved the API completely to that server which takes a good deal of the stress off of the site itself.

The site has been pretty fast since the new server has been up.  However, there have been occasional slow-patches, and looking at the CPU usage on the main webserver for the site – it’s definitely still not “calm”.

Surprise 1

So here’s a fun surprise: today I ordered another server!

For the first time in a looong time, we can hopefully stay ahead of demand instead of suffering for a couple of weeks until it’s unbearable and we’re forced to upgrade.

Surprise 2

W00t! This is an uplifting blogpost… lots of surprises! Anyway: the second surprise is that starting in October, I’m going to be decreasing the time I spend at my day-job by 40% so I’ll only be in the office 3 days per week. That’ll give me two whole days more each week where I can work on Motive Force products like LyricWiki. This should help things become much more stable very quickly.

Well, that was an abnormally enjoyable post for this blog – which usually just announces outages! The site still isn’t totally upgraded (because the weekend ended before I could make all of the extensions work), so I have to run back to that. I took snapshots of a bunch of performance stats before and after, so I’ll post those sometime soon.

Thanks for your patience during those past few weeks. Hopefully we’re in a whole new era for the site!

The new server exists, I’m setting it up and am hoping I can have it all done before I go to sleep.

There is a decent amount of stuff that needs to be done before it’s all working. If you’re technical, or just like watching lists as they are being completed… I’ll be tracking the upgrade here.

I’ll also be making occasional updates to the @lyricwiki twitter account.

The new server will be another Apache server and is named “Cochise“. That is the name of the last great Apache chief, and also the inspriation for the song Cochise by Audioslave.

There will be downtime for an unknown length of time tonight. I’ll try to keep it to a minimum, but the site is so slow it’s practically down anyway.

PS: Special thanks to our awesome webhost for getting the new box here & set up quickly!

And not a moment too soon.  Traffic is down a significant percentage in the last few days.  The site is so slow and bothersome that I even find myself hesitating to load more pages when I go to the site.

Anyway, while I was waiting, I made a list of all of the changes I think are necessary for the upgrade (although there is the potential that I could get surprised).  I plan to update the list during the upgrade.  There will be an outage during which the list will be unavailable, but that’s not a huge deal, hopefully.  Some of the tasks don’t require the new server to be here for them to be started, so I’ve already begun.

Here is the list of upgrade tasks: http://lyricwiki.org/upgradeList.html

ps: If anyone wants to make a style-sheet to make the page less ugly, I’d be more than happy to put it up, but designing one is lower-priority for me than doing the things on the list.

This week, the site has been extremely slow and even gone down and up a couple of times.  I searched for a problem for a while but it appears that we’ve just really hit the wall on how much traffic we can support with our current servers.  That’s fairly good timing since we’d been planning to move to more servers for a little while, so I’d already begun to look into it.

Today I ordered another server with the same specs as the current Apache server.   This will bring us up to 5 total servers running LyricWiki.  For the curious (and tech-savvy): that’s one squid caching server in front of two Apache web servers which talk to one mysql master server and one read-only replica mysql server.

To get the server to be as beefy as we need, I had to ask the hosting company to order extra RAM for it.  So we’re just waiting for that to be delivered (hopefully around this weekend or very soon after) and we’ll be ready to start working to get the new server pulled into our setup.

In addition to just having more man-power machine-power to handle our traffic, this will give two additional benefits immediately.  The first is that we can use the new server to test out the upgrade to the newest version of MediaWiki (the software that runs our site as well as Wikipedia).  The second benefit is that now we’ll have two Apache servers – currently the most overworked part of the system – with one running the API and one running the site itself (lyricwiki.org).  This will let us more quickly identify when something is wrong with one of those two systems and it will make sure that problems with either of them are unlikely to effect the other.

Exciting times… stay tuned!

For the first time all day, the site is moving at what appears to be full-speed.

Also, to answer the earlier-posed question about the Squid having to log a ton of extra space when it restarts, it turns out that is true – apparently something (either Apache restarting or somehow detecting that the Squid just came back) triggers it so that the MediaWiki install sends a <em>ton</em> of “PURGE” requests to the Squid server which basically tell it to forget about a page it may be caching because it’s probably out of date now.  Each request is another line in the log-file, so that’s about 800,000 extra lines in the span of a few minutes.

Apparently the site was down while I slept, but I had emails & comments in my inbox about the outage as soon as I got up so I was able to jump on it right away.

The Squid had run out of memory again.  The access log files for one day were 17 gigs.  That seems awfully high – maybe we’re getting spidered too hard or the logs go through serious stress after a restart?

I’ll be finding a more permanent solution to the issue, but in the meantime the site is “up” but it’s going to be fairly slow while the cache refills… again.

It had run out of harddrive space.   Moved some log files to another disk and we’re back in business.

Finally you can have your watchlist stream you content when the pages you are interested in have been updated!

Thanks to a new extension by Teknomunk, you can now enable the rss feed at http://lyricwiki.org/Special:WatchlistFeed.

Since this is a really great feature to have in any wiki, Teknomunk plans to make this extension public so that all MediaWikis can use this feature (it is very commonly requested). Thanks Teknomunk!

It appears that the problems on the site were due to someone (read: me) messing up when they restarted the database-replication!

Thanks to a bunch of helpful problem-reports from a number of users, I had some good data to look at to figure out what was wrong. It was actually pretty easy to figure out once I had all of those problem-pages to look at (I’m talking about you Brian May!).

Bonus pretzel

While I was waiting for the computers to move some massive files around, I had a couple of minutes here and there to make other tweaks to the site. Two somewhat interesting things that came out of this time are that the 1) “job queue” is getting automatically run every hour now (which keeps things up to date and avoids assigning extra jobs to random users who would get like a 2 minute page-load randomly every 10,000 pages) and 2) the road-block page that shows up when the site is shut-down for maintenance now has an iframe in it which shows a google-search of LyricWiki.org for the same page and suggests that users click the “Cached” link. This will allow people to see a somewhat-recent of most of the pages even while the site is down. I dig it.

Special thanks/shoutouts to Kiefer, Redxx, Senvaikis, Teknomunk and WillMak050389 for their help figuring out what was wrong and testing things to make sure they were fixed!

As always, please let me know if you see something strange on the site. Thanks!

LyricWiki.org is back up. I’m relatively sure I fixed everything. I’ll be back on to check things in a bit & I’ll have more details once I find out if things are actually working.

The site has been behaving strangely lately and I’m not sure how I managed to break it or what is wrong, but I’m looking into it actively.  I’m going to be working on any intensive-changes starting tomorrow (saturday) morning since weekends tend to have less traffic than weekdays.

People have been forwarding me a good bit of info about the problems, and it has been very helpful. If you have seen anything strange (especially if you have noticed a pattern in it), please pass the information along to me!

Hopefully the site will be all patched up by the end of tomorrow. Check the blog for updates as things are happening (brief outages are very likely tomorrow).

I really apologize for the oddities on the site. They are almost certainly my fault (and even if they weren’t… they would be since it’s my responsibility to keep things humming along).

Thanks for your patience,
– Sean

That was faster than I expected

So, in the wake of last week’s problems on LyricWiki, the system was finally purring again at full-steam when this morning at about 1am, our data center had its first-ever full power-outage.  This is the kind of thing that isn’t supposed to happen since good datacenters (like LyricWiki’s) have power from multiple sources and generators that kick on if all of those fail.  But once in a while a human error will mess the whole thing up and briefly cut the power.  The power was only out for about 1 minute which basically just causes the computers to reboot.

This normally would have been fine… the servers are configured in a way that they are all supposed to jump right in where they left off.  Once all of them are up, the site should automatically work again.  “Should”.  For some reason, the master-slave replication is broken again.  The slave is trying to read a position in the log-file that doesn’t exist.  So yet again, I think we’re going to have to start the replication all over.  This means about an hour or more of downtime.

For now, I just disabled the slave (to prevent weird behavior) so the site is going to be working with a lot less resources than it normally is.  I’m planning to do the outage at night when the traffic is lower.  The site will potentially be slower than normal today since it doesn’t have the slave-server helping out.

All-in-all it could have been a <strong>lot</strong> worse since the downtime so far has only been a minute or so, but please be aware that there will be potentailly an hour or two of downtime some time tonight/tomorrow-morning.

Earlier this week we had some really strange errors on the site (detailed description here) that were caused by a mySQL bug which will eventually burn anyone who is running master/slave replication.

I mentioned that you’d need to write a script to prevent this problem from happening to you. The script would delete old, unneeded mySQL bin-log files. In the spirit of stopping this bug from affecting others, I’ve released a generic version of the script… you just have to set up a few variables in the “configuration” section at the top of the script and you should be good to go. For the code and info on how to make it run daily, see the “update” section of the post on my blog. It is, of course, free and open source.

Hope that helps someone!

Das Programm iLyrics hilft allen, die ihre Musik mit iTunes von Apple (unter Windows) verwalten. Auch wenn iTunes bereits hilfreiche Features für komfortable Verwaltung von Musik unterstützt (zum Beispiel intelligente Wiedergabelisten) und vorallem von iPod-besitzern genutzt wird, fehlt meiner Meinung nach ein entscheidendes Feature. Die Songtexte (Lyrics) werden nicht mal beim Kauf im iTunes-Store mitgeliefert.

Für alle die das genauso stört, gibt es iLyrics. Sie markieren einfach ein oder mehrere Songs in Ihrer Bibliothek und starten iLyrics. Auf Knopfdruck werden die Songtexte automatisch hinzugefügt. Auf Wunsch kann man den Text davor noch ergänzen oder ändern.

iLyrics greift hierfür auf LyricWiki zurück, eine Lyric-Datenbank im Wiki(pedia)-Stil. Natürlich kann sich hier auch bedienen, wer kein iTunes und iLyrics nutzt.

I mentioned before that I could explain the weirdness that’s been going on for the last couple of days on the site.  If you’re curious, here are the technical details:

We Use Master-Slave Replication

On LyricWiki, one of the tricks we use for scaling is called master-slave replication (that’s a how-to, not an explanation).  The crux of the concept is that you can have one database (the master) which holds “the truth” and which you can read from and write to.  The other server (the slave) is just for reading – it only knows what’s going on because the master gives it regular (almost instant) updates.  Since the slave knows that it only has to listen to the master for updates, it can hold onto the data in a way that makes it REALLY fast for reading. There are several things you can do on LyricWiki that only require reading. For example, if you’re logged-out and are simply viewing a page, that’s a bunch of read-requests. We send 80% of the reads to the slave, and the master handles all of the writes and the other 20% of reads.

What went wrong – Hard-drive space

The other night, the site started behaving weirdly. We started getting errors saying that there were too many connections. This should be rare at this point, and is generally fixed by restarting the database server (ie: kicking everyone out for a second and only serving those who come back). The errors kept happening though. To make a long story short, it became clear that the reason is that we ran out of hard-drive space on the server (the error messages weren’t too helpful in finding this). The reason we ran out of space is that the master and slave mentioned above communicate through a log-file. Each time a change happens on the master, it writes that change to a log-file which the slave is keeping an eye on.

The problem is that there are a TON of things happening on LyricWiki. Since the last time the logs were cleared (mid-march), we filled up 100gigs of hard-drive space with these log-files! The solution should have been just to move some of the old files to another hard-drive and restart the system. That’s what I did and that’s when things started getting weird.

Enter the mySQL bug

After the fix, things were taking a while to get back to normal. Then it became clear that they weren’t heading back to normal at all. Edits to pages weren’t taking effect, histories and diffs were skipping things… everything was a mess. After some digging, it became clear that the master-slave replication mentioned above was completely split. This means that the slave remembered the site the way it was a couple of days ago and wasn’t getting updates from the master. To make things even more confusing for users, the fact that 80% of the reads go to the master means that users would see the wrong pages 4/5 times and the correct (up-to-date) page 1/5 times. That’s just a really good way to drive someone bonkers .

As it turns out, there is a known bug in mysql which will mess up the log file that the master and slave use to communicate in the event that you run out of hard-drive space. Yikes! This makes your slave completely and hopelessly lost. You have to delete it and start over. This means you need to have at least a brief outage on your site. The more data you have, the longer your outage.

Postmortem

1. mySQL – You’re still my homeboy

That’s a really bad bug for mySQL to have laying around, but I don’t condemn mySQL as a system because of it. I’ve been absolutely abusing that database system for about 6 years and this is the first time I found any errors to be mySQL’s fault. We often hear frustrated programmers saying their must be a bug in their underlying technology, and they are almost always wrong. It is usually the programmer’s fault. I’m as guilty as the next guy (esp. early on in my coding career before I started to realize the embarrassing pattern). Long story short: I still view mySQL as one of the most robust systems I work with. Much love. They need to fix this bug though, it will seriously bite every major site once it has to scale.

2. Keep this from happening to you

If you are using master-slave replication, learn from my mistakes and do a bit of preventative-work, it’ll be faster than mopping up after this mess. Write a daily cron-job to find out what log-file you are currently using on the slave and delete all of the older logs. If you have extra hard-drives, feel free to move the old logs and store them forever, but they’re probably just too big and unnecessary to keep around for very long.

3. I need monitoring!

I’d say this got fixed relatively smoothly considering how messed up things were… I checked disk space really early on just on a hunch, but I really had no hints from error-logs so this could have dragged on much longer. It was still about 2 days of unacceptably poor behavior for a site. About 8 hours of that was wasted before I found out. I’ve got to get a heart-beat script to run on another server and watch to make sure LyricWiki is always working and shoot me an email and/or text-message when it goes down. Then I’ll always be on it. I’ve written about a dewtrillion of these (that’s a lot because it’s caffeinated), I don’t know why I haven’t made one for LyricWiki yet.

Conclusion

Well, there you have it. Way more technical info on the problem than you could have ever wanted! All technical issues aside, I’m sorry it went down again. LyricWiki is not my full-time gig, so admittedly it’s a little sloppy for something used for millions of requests per day. Some day I may be able to devote more time to keep the site humming better, but until then I’ll just have to make better use of my time and be smarter about what I choose to do in the time I have (like write that monitor script!).

Even though it was a mySQL bug that caused it, they’re my servers and there is no excuse. My bad.