A simple, open, server-to-server web-hook-based pubsub (publish/subscribe) protocol as an extension to Atom and RSS.

Parties (servers) speaking the PubSubHubbub protocol can get near-instant notifications (via webhook callbacks) when a topic (feed URL) they’re interested in is updated.

The protocol in a nutshell is as follows:

• An feed URL (a "topic") declares its Hub server(s) in its Atom or RSS XML file, via <link rel="hub" …>. The hub(s) can be run by the publisher of the feed, or can be a community hub that anybody can use. (Atom and RssFeeds are supported)

• A subscriber (a server that’s interested in a topic), initially fetches the Atom URL as normal. If the Atom file declares its hubs, the subscriber can then avoid lame, repeated polling of the URL and can instead register with the feed’s hub(s) and subscribe to updates.

• The subscriber subscribes to the Topic URL from the Topic URL’s declared Hub(s).

• When the Publisher next updates the Topic URL, the publisher software pings the Hub(s) saying that there’s an update.

• The hub efficiently fetches the published feed and multicasts the new/changed content out to all registered subscribers.

The protocol is decentralized and free. No company is at the center of this controlling it. Anybody can run a hub, or anybody can ping (publish) or subscribe using open hubs.

To bootstrap this, we’ve provided an open source reference implementation of the hub (the hard part of the protocol) that runs on Google App Engine, and is open for anybody to use.

<iframe src="http://docs.google.com/present/embed?id=ajd8t6gk4mh2_34dvbpchfs&size=m" frameborder="0" width="555" height="451"></iframe>

I just released the first version of gwt-strophe, GWT bindings for the Strophe XMPP library. Nothing much to say else than it is pretty young, with all that can imply. The project is hosted at https://launchpad.net/gwt-strophe

Google Web Toolkit bindings for Strophe, the JavaScript XMPP library, are available at https://launchpad.net/gwt-strophe. I believe it provides an almost complete API coverage, including the PubSub plugin. However, it is quite young and far from being thoroughly tested, hence comments are more than welcome. No release yet, so to get the code, proceed as follows:

bzr branch lp:gwt-strophe

w00t! New content, see if we get it!

Sometimes RSS isn’t fast enough. We’ve been experimenting with faster blog subscription delivery using Jabber to push the messages. When you use Jabber to subscribe to blogs you get the news as soon as it is published. Now almost every post and comment on WordPress.com blogs is published this way and you can subscribe to these streams using almost any Jabber client. Messages are delivered typically within one second of publication. You can also publish to your blogs by typing instant messages. Soon comments will be appearing in Jabber chat rooms.

Let me break it down. Jabber (XMPP) is an instant messaging protocol. There are dozens of free clients (programs) that can connect to Jabber services, thousands of Jabber servers, and millions of daily Jabber users around the world. Even some phones can connect to it, including the iPhone with an appropriate app. So you can probably use Jabber. The primary exception is people behind company or government firewalls that block XMPP ports, but we’ll have a web-based solution for them soon.

Before I give you a link I have to tell you that this service is experimental. If you use it, you are a tester so please wear your white lab coat. A few dozen people have been using it for several months with very few hiccups, but hiccups are still to be expected. Even so, most of us at Automattic rely on it daily to surface and accelerate the discussions on our private blogs. Finally I must tell you that we have not yet worked out all the business angles, so the feature set and limitations may change to accommodate our inevitable need to feed the monkey.

Now I give you im.wordpress.com, which I demonstrated at the CrunchUp last week. Every WordPress.com account is automatically linked to a Jabber account on im.wordpress.com. We have compiled instructions for setting up some popular Jabber clients.

If you want my personal recommendation for a Jabber client, my choice for Mac OS X is Adium. If you already use iChat, just stick with iChat. Some of my coworkers who run Windows have chosen Pidgin. I also sometimes use Psi, which is available for Mac, Windows, and Linux.

Here is some info for people familiar with XMPP. This service is based on XEP-0060 (Publish-Subscribe) acting as a front-end for WordPress blogs. It started as a simple firehose for our commercial partners and grew from there. People subscribing with Jabber clients don’t need Pubsub. They send simple commands to a chat bot and their items are delivered as XHTML-IM from the blog’s URL. The bot speaks XEP-0060 on their behalf. If you can speak XEP-0060, you can connect to pubsub.im.wordpress.com and subscribe to nodes. The nodes for this blog are /blogs/andy.wordpress.com/ for posts, /blogs/andy.wordpress.com/comments/ for all comments, and /blogs/andy.wordpress.com/2009/07/16/real-time-wordpress-com-subscription/ for comments on this post. Node discovery and item discovery and retrieval are not implemented. Reasonable subscription and traffic limits will be imposed. If you are looking for a complete feed of all our blogs and comments, try the firehose.

Following the last article, i have been experimenting again on laconica and pubsub, this time on the idea of “filtering the real time web“.

Stomp and generally JMS messages offer the ability to specify headers and body of the message to transmit, in a way that resembles http requests.
In fact, Stomp protocol is really similar to Http protocol, at least in the general structure. The difference is that there are different methods instead of GET, POST, etc..

HTTP:

POST /item HTTP/1.0
Header: value

POSTBODY

STOMP (after CONNECT command)

SEND
destination: /item

MESSAGE_BODY

Very readable, like HTTP. However the main difference is that Stomp, like XMPP, is a stateful and bidirectional protocol.

Like XMPP/pubsub, with STOMP and AMQP you can subscribe to a topic (a.k.a. pubsub node). However content-based routing is only done by STOMP and AMQP (in a way specified by the standard).
AMQP is still in its early days. STOMP is a ready-to-use protocol as many of its implementations.

In the Stomp SUBSCRIBE operation it is possible to specify a JMS Selector. It’s a header passed when subscribing and it contains a SQL-92 Statement. SQL attributes to match the conditions against are the other headers.

FILTERING LACONICA PUBLIC TIMELINE BY TAGS
—
Before filtering, the code of the previous post used to push notices needs to be enriched a bit. Particularly, tags present in the notices are now put in a separate header.
Doing this allows to be the target of a Selector.

Push an additional header:

//send tags as headers, so they can be used as JMS selectors
common_log(LOG_DEBUG, 'searching for tags ' . $notice->id);
$tags = array();
$tag = new Notice_tag();
$tag->notice_id = $notice->id;
if ($tag->find()) {
while ($tag->fetch()) {
common_log(LOG_DEBUG, 'tag found = ' . $tag->tag);
array_push($tags,$tag->tag);
}
}
$tag->free();

$con->send('/topic/laconica.allusers',
$notice->content,
array(
'profile_id' => $notice->profile_id,
'tags' => implode($tags,' ')
)
);
common_log(LOG_DEBUG, 'sent to catch-all topic ' . $notice->id);

The Stomp client is really similar to the one of the previous post, but it specifies the additional header “selector” passed when subscribing.

The selector here is “tags LIKE %dent%”. This matches all the posts that contains the tag #dent. Substitute it to match the tag you want…

<?php

require_once "Stomp.php";

$con = new Stomp('tcp://localhost:61613');
if (!$con->connect())
    print 'conn failed';

$what = '/topic/laconica.allusers';

$query = 'tags LIKE \'%dent%\'';

if (!$con->subscribe($what,array("selector" => $query)))
    print "sub failed";
else
    print "sub to ".$what." successful\n";

while (true) {
    $msg = $con->readFrame();
    if ($msg) {
        print $msg->headers['profile_id'].": ".$msg->body. " -- ";
        print "msg_time:".$msg->headers['created']." ";
        print "tags: ".$msg->headers['tags']."\n";
        $con->ack($msg);
    }
}

$con->disconnect();
?

In this blog I publish information and ideally have people that subscribe to it. For the blog I use the WordPress hosting site (service) you may use an RSS aggregator to subscribe (client). In a business application you can achieve this same concept easily with WCF. In this post I will run through an example of the Medical System Client application subscribing to patients of interest. When other clients update those same patients with business critical data the active clients can receive an ‘alert’ immediately at easily.

We begin with the client side setup. First we define the contract interface that contains our operational methods (excluded below, but examples include Save(), Retrieve(), etc). The key things to note on the interface is the ‘CallbackContract’ attribute, and the subscribe/unsubscribe methods. The subscription methods use a Patient Id key but this can be whatever is most suitable in your situation, for example an enum will work fine for more generalised event types.

[ServiceContract(CallbackContract = typeof(IPatientEvents))]
public interface IMyContract
{
   [OperationContract]
   void Subscribe(int patientId);
   [OperationContract]
   void UnSubscribe(int patientId);
}

The next step is to define an events interface. This interface outlines expected methods the client must implement to be able to handle the events raised by the service. Note that the event handler has an attribute defining it as a OneWay call. This event is lab results for a patient have been entered into the system, and are of a critical nature.

public interface IPatientEvents
{
   [OperationContract(IsOneWay = true)]
   void LabResultsUpdated(int id, string text);
}

Then you simply create your client side class against the events interface. In this example the client is a WPF page class.

class PatientDetails : Page, IPatientEvents
{
     MyContractClient proxy;

	 //other methods
}

The client will need to supply a context to the service so it can maintain an association on the channel. This is simply achieved through a constructor call when initialising the proxy. Also at this point we’re subscribing to the events for the given ID. This subscription can be client triggered this way. Alternatively certain actions can have the server subscribe the client server side. An example would be during a data retrieval call for a patient; the server forcing the setup up the subscription.

  InstanceContext context = new InstanceContext(this);

  proxy = new MyContractClient(context);

  proxy.Subscribe(id);

Now on the service we implement the same contract IMyContract we also define the signature of the delegate Action that will be raised as an event. Along with implementing the method that will attach the event handler to the appropriate client. The key thing to note here is the Current.GetCallbackChannel this is linking via the context the client supplied in the proxy initialisation.

[ServiceBehavior(InstanceContextMode
      = InstanceContextMode.PerCall)]
class PatientService : IMyContract
{
    static Action<int, string> m_EventLabs = delegate { };

    public void Subscribe(int patientId)
    {
        IPatientEvents subscriber =
          OperationContext.Current.GetCallbackChannel<IPatientEvents>();

        m_EventLabs += subscriber.LabResultsUpdated;
    }
}

All that’s left now is to write the actual event firing code, that can be triggered after an appropriate event. In this example the firing takes place after the Patients Lab Results have been updated. Thereby allowing any other users currently viewing this patient’s details to be alerted.

public static void FireEvent(LabResult labResult)
{
    m_EventLabs(labResult.PatientId, labResult.ToString());
}

public bool SaveLabResults(LabResult lr)
{
    //save code...

    PatientService.FireEvent(labResult);
}

Back on the client we have defined a method that will be executed upon the event. Remember the service has attached the handler on the Subscribe() method.

public void LabResultsUpdated(int id, string text)
{
    this.displayAlerts.Text = text;
}

That is all that is required to have your client quickly and easily subscribe to and handle events. It’s thread-safe and efficient with the service doing all the hard work.

I will expand on [ServiceContract(CallbackContract ... ] and [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)] in future posts, and I’ll try to remember to update this post with direct links.

Just for some reference sake here are the objects used above that were not defined in the code blocks:

[DataContract]
public class Patient
{    [DataMember]
    public int PatientId { get; set; }
    //more members
}
[DataContract]
public class LabResult
{    [DataMember]
    public int LabResultId { get; set; }
    //more members
}
//displayAlerts is just a page <TextBlock> object

Just a quick post about WebSphere MQ Java Pub/Sub functionality. I have need to write an application that uses the Base Java Classes for WebSphere MQ, making use of some Pub/Sub. There is a fair amount of information on the web on how to subscribe but no information on how to unsubscribve. To subscribe to a topic all you need to do once you have a MQQueueManager object is the following:

int options = CMQC.MQSO_CREATE | CMQC.MQSO_RESUME | CMQC.MQSO_DURABLE | CMQC.MQSO_FAIL_IF_QUIESCING;
MQTopic topic = qmgr.accessTopic("topicName", "topicObject", options, null, "DurabeIdentifier");

Now you have the MQTopic object you can get messages that are published to the topic “topicObject/topicName”. As I mentioned the reason for this post was the lack of any information on the web, including the WebSphere MQ Using Java V7 manual, and the WebSphere MQ V7 infocenter on how to unsubscribe from the topic. Well eventually I gave up on the web, and decided to look at the MQ Test material that we have here at Hursley. The following code will unsubscribe from the topic:

if (topic != null) {
if (topic.isSubscribed()) {
topic.getSubscriptionReference().setCloseOptions(CMQC.MQCO_REMOVE_SUB);
topic.getSubscriptionReference().close();
}
topic.setCloseOptions(CMQC.MQCO_DELETE);
topic.close();
}
That will check if your topic is not null, is subscribed and then set some options to ensure that the subscription is deleted. Job done and only an hour wasted searching the docs!

Update:You do not need to use topic.setCloseOptions(CMQC.MQCO_DELETE);. This would actually throw an MQRC_OPTION_NOT_VALID_FOR_TYPE (mqrc 2045) error!

Lately i have been working on the idea of queues and pubsub mechanisms and how it is possible to integrate them in a microblogging service. I am fascinated by messaging protocols and event pushing!

Let me say that XMPP is in my opinion the protocol of the future, however its pubsub implementation is quite complex if compared to what STOMP offers, and I didn’t find any mature PHP library.

AQMP and STOMP are more suitable for internal processing, while XMPP is the perfect option for exposing data to the outside world and perhaps to connect desktop clients with microblogging services.

Apart from my opinions, STOMP is a mature protocol with a PHP library that works pretty well and Apache ActiveMQ, the message broker, offers access using several protocols.

INTRODUCING A MESSAGE BROKER IN LACONI.CA

If queues are enabled in laconi.ca, it is possible to modify common_enqueue_notice() (called after the storage of the dent) in order to push the dent in the 4 queues that the software uses (OMB, jabber, sms, public), or simply push the dent in one and configure 3 mirror queues.

In this way the internal queue system is not used. Queue_item::top() has potential race conditions and polls the db every 5 seconds. That means that if we’re lucky we get the message quickly, if unlucky in 5 seconds. Latency could even be something secondary, but the system now is pretty inefficient.

If laconi.ca passes the message to an external broker, delivery would be more efficient and it is a piece of code that the community wouldn’t have to maintain.

The downside of this is that another software is on the list of requirements. However, if you have the possibility to run queuehandlers in background, it probably means you own the server and you can afford to install a message broker. Queues can still be disabled and the system still works without queues and message brokers, the only difference is that everything is done synchronously.

WITH A MESSAGE BROKER, PUBSUB COMES FOR FREE

Another feature of ActiveMQ is the use of “topics”. The main differences between queues and topics are that in the queues messages are sent to only one consumer (precisely, round-robin among consumers) and messages are persistent (stored internally while waiting consumers to connect). Topics are the opposite, the same message is sent to all its consumers that are currently connected to the broker.

Topics could be used for introducing a publish-subscribe system in Laconi.ca. In common_enqueue_notice() with only some lines of code it is possible to push each dent to topics such as “/topic/laconica.user.IDUSER”, “/topic/laconica.allusers” or do more complex association like “/topic/laconica.group.IDGROUP” and “/topic/laconica.tag.TAGNAME”.

With a topic dedicated to a group and using the XMPP endpoint of ActiveMQ, you can also gain MUC support in Laconi.ca (only read-only if there isn’t an handler)

Offering pubsub using ActiveMQ and STOMP calls is very easy, however i am not sure how interoperable this solution could be. I did not find any XMPP/Pubsub interface for ActiveMQ, although topics could be easily mapped to XMPP/Pubsub nodes.

If the pubsub notifications have to be XMPP based, i see two solutions: write a wrapper that gets all the messages via an ActiveMQ queue and speaks XMPP with ejabberd, or forget about ActiveMQ and use XMPP requests directly, perhaps using an AtomPub interface.

POSSIBLE FUTURE DIRECTIONS

Integrating queues at the end of the stack is not going to help much in terms of performance or flexibility of the whole system, but it’s a first step and, as they’re used marginally, they’re still an optional requirement.

With all these ideas around, like offering XMPP MUC support, AtomPub interfaces, XMPP/Pubsub output, etc.. in the future the system would have to be “decoupled”.

An input system that pushes dents in a queue as quickly as possible. The main Laconi.ca logic that works in the background and asynchronously. The internal logic will then push the info to queue handlers for real-time delivery AND store the dents in the DB. The main website will still poll data from the DB, with all the necessary caching layers, OR/AND poll directly the message broker for the required type of dent (see the list of topics mentioned before) via AJAX calls (ActiveMQ has support for that).

Any feedback is appreciated, or suggestions on things i might have overlooked! I am not an expert of message queue system so many of the things i have said may seem obvious, but not to someone who has a background of normal website and MVC development so i clearly stated all the steps…

WHAT I DID


function common_enqueue_notice($notice)
{
if (common_config('queue','subsystem') == 'stomp') {
// use an external message queue system via STOMP
require_once("Stomp.php");
$con = new Stomp(common_config('queue','stomp_server'));
if (!$con->connect()) {
common_log(LOG_ERR, 'Failed to connect to queue server');
return false;
}
$queue_basename = common_config('queue','queue_basename');
foreach (array('jabber', 'omb', 'sms', 'public') as $transport) {
if (!$con->send(
'/queue/'.$queue_basename.'-'.$transport, // QUEUE
$notice->id, // BODY of the message
array ( // HEADERS of the msg
'created' => $notice->created
))) {
common_log(LOG_ERR, 'Error sending to '.$transport.' queue');
return false;
}
common_log(LOG_DEBUG, 'complete remote queueing notice ID = ' . $notice->id . ' for ' . $transport);
}
$con->send('/topic/laconica.'.$notice->profile_id,
$notice->content,
array(
'profile_id' => $notice->profile_id,
'created' => $notice->created
)
);
$con->send('/topic/laconica.allusers',
$notice->content,
array(
'profile_id' => $notice->profile_id,
'created' => $notice->created
)
);
$result = true;
}
else {
// in any other case, 'internal'
foreach (array('jabber', 'omb', 'sms', 'public') as $transport) {
$qi = new Queue_item();
#OTHER OLD CODE...


This code sends the dent to 4 queues and 2 topics. Queues are used by the 4 background queuehandlers while 2 topics offer a simple pubsub service to either a single userid or to all of them (public timeline).

As regards queues, queuehandler.php has to be modified:

# called by run()
function stomp_dispatch() {
require("Stomp.php");
$con = new Stomp(common_config('queue','stomp_server'));
if (!$con->connect()) {
$this->log(LOG_ERR, 'Failed to connect to queue server');
return false;
}
$queue_basename = common_config('queue','queue_basename');
// subscribe to the relevant queue (format: basename-transport)
$con->subscribe('/queue/'.$queue_basename.'-'.$this->transport());
do {
$frame = $con->readFrame();
if ($frame) {
$this->log(LOG_INFO, 'Got item enqueued '.common_exact_date($frame->headers['created']));
// XXX: Now the queue handler receives only the ID of the
// notice, and it has to get it from the DB
// A massive improvement would be avoid DB query by transmitting
// all the notice details via queue server...
$notice = Notice::staticGet($frame->body);
if ($notice) {
$this->log(LOG_INFO, 'broadcasting notice ID = ' . $notice->id);
$result = $this->handle_notice($notice);
if ($result) {
// if the msg has been handled positively, ack it
// and the queue server will remove it from the queue
$con->ack($frame);
$this->log(LOG_INFO, 'finished broadcasting notice ID = ' . $notice->id);
}
else {
// no ack
$this->log(LOG_WARNING, 'Failed broadcast for notice ID = ' . $notice->id);
}
$notice->free();
unset($notice);
$notice = null;
} else {
$this->log(LOG_WARNING, 'queue item for notice that does not exist');
}
}
} while (true);
$con->disconnect();
}


Here there’s the code for a simple pubsub consumer

require_once "Stomp.php";
$con = new Stomp('tcp://localhost:61613');
if (!$con->connect())
print 'conn failed';
$what = '/topic/laconica.2';
if (!$con->subscribe($what))
print "sub failed";
else
print "sub to ".$what." successful\n";
while (true) {
$msg = $con->readFrame();
if ($msg) {
print $msg->headers['profile_id'].": ".$msg->body. " -- ";
print "msg_time:".$msg->headers['created']." ";
print "received:".date("Y-m-d H:i:s")."\n";
$con->ack($msg);
}
}
$con->disconnect();


Except the last bit, all this code is on the fmarani-clone of the dev repo on gitorious, branch 0.8.x

Mir ist gerade aufgefallen dass SMS und RSS so ziemlich vergleichbare Technologien sind. Beides ist jeweils eigentlich eine ziemlich beschissene Lösung die sich ziemlich weit verbreitet hat, beides könnte man viel eleganter lösen.. wenn irgendjemand Interesse daran hätte und dennoch sind beides Dinge die ich relativ oft verwende (bei SMS eher weniger, zumindest wenn man “oft” als die Menge festlegt die andere dafür halten). Das natürlich obwohl ich das System für so schlecht wie irgendwie möglich halte.

Bei SMS (oder auch Short Message Service, wehe ich höre noch einmal den Kommentar dass mir jemand eine “SMS-Nachricht” aufs “Handy” schicken will!) handelt es sich eigentlich um die Idee dass man mit nicht-benötigten Sendeblöcken im Netz doch eigentlich auch irgendwelche Daten übertragen könne. Gesagt, getan und schwups haben wir ein Nachrichtensystem das den Provider nichts kostet und Jährlich riesige Gewinne abwirft.

Das beste an diesem System kommt aber noch: 160 Zeichen (Die Menge die sich aus der Blockgröße ergibt) sind gerade so groß, dass permanent Leute diese Grenze überschreiten. Was tut man also? Man hängt mehrere Blöcke hintereinander und bereichnet jeden einzeln. Ein Antwortsystem baut man sich auf die selbe Art, wer auch nur wieder ein Ja/Nein zurückgeben will zahlt gleich noch eine Nachricht. Alternativ kann man natürlich auch Nachrichten an mehrere Leute senden… also mehrmals die Gleiche absenden und für jeden Empfänger erneut bezahlen.
Völlig unbeachtet davon: eine solche Nachricht kostet meistens fast den gleichen Betrag wie eine Minute gespräch via Telefon. Ich möchte jetzt doch schon gerne mal sehen wer es nicht schafft in 60sec den Informationsgehalt von 160 Zeichen per Stimme zu übertragen.

Dazu kommt dann natürlich noch das Eingabeproblem, wodurch man erstmal wieder alle möglichen neuen Ideen für Telefontastaturen oder neue Software verkaufen kann. Natürlich keine brauchbaren, dann könnten die Nutzer ja bei einem System bleiben und bräuchten kein Geld mehr auszugeben.

RSS hingegen (kurzform für Really Stupid Syndication) ist ein gaaaanz ganz tolles Polling-Format auf XML-Basis (und wie wir ja alle wissen ist XML schon rein grundsätzlich toll). Polling an sich sagt einfach schon genug aus und ich werde bestimmt jetzt nicht anfangen wieder alle Nachteile dieses Systemes aufzuführen. Die Twitter-Generation scheint eh nie zu verstehen dass es eben NICHT “echtzeit” bedeutet wenn ich alle 5min nachsehe ob es etwas neues gibt.

Trotz aller Fehler und Idiotien die bereits die Grundideen hinter diesen Systemen beinhalten, nutze ich diesen Mist, genau wie all die anderen Deppen. Warum?
Der Grund ist einfach.. alle anderen tun es auch!!!!!!111111einseinself

Das nervt mich echt.. ich kann meine Termin-Erinnerungen via SMS aufs Handy bekommen, viele Leute in meiner Umgebung senden mir Kurznachrichten um mich schnell was zu fragen, meine Daten zwischen diesen vielen “tollen” Web2.0-Diensten tausche ich via RSS/Atom aus und selbst unser Python-IRC-Bot schreibt und liest diese Feeds (fast schon mehr als er andere Dinge tut).

Es ist irgendwie auch keine Besserung in Aussicht… es gibt ja bereits anerkannte Lösungen die alle Verwenden, warum sollte man dann schwach Verbreitete Sachen wie z.B. Jabber-Pubsub verwenden und quasi das Rad neu erfinden?

Okay, it’s not really a dispute. That was sarcasm in the title. Just to be clear, since there’s always been a tiny bit of useful friction between the idea of XMPP and web hooks, it’s important to remember they are not mutually exclusive. Anybody hyping a battle between the two is trying to create or is imagining a controversy that isn’t real. Many proponents of web hooks are XMPP proponents as well. In fact, the video some are pushing around to promote this supposed throw down is of two XMPP supporters (including djabberd author Brad Fitzpatrick) demoing a pubsub system based on web hook callbacks. And it takes place at an XMPP meetup, so of course there was going to be some proselytizing.

Nevertheless, Jabber/XMPP is a messaging protocol. Web hooks provide a model for functional extensibility, so they are a platform for many different things. A push-based pubsub messaging system is just one use. Even though I originally wrote about web hooks as a notification mechanism, my mind was nowhere near pubsub. I was more focused on the idea of web service integration and orchestration. With the commoditization of CGI-enabled web hosting, I was thinking about how the popularity of web programming combined with the easy invocation of HTTP requests could be used to make a more useful and functionally extensible web. A more programmable web.

With that said, there is nothing wrong with discussing what you can and can’t do with web hooks and XMPP, but there is not some angry fight between camps. Done. Over. Moving on.

As it turns out, the project demoed in that video, which is called pubsubhubbub, is something I had previously stumbled upon while browsing projects a friend of mine was involved in. There wasn’t much of a description, and I didn’t dive too deep into the code, so I bookmarked it to come back to it. After that demo happened, it seemed I misheard (from several people) that it was some XMPP pubsub system, which totally confused me because… it’s not. It’s a neat, distributed pubsub implementation built on web hook callbacks created by Brad Fitzpatrick and Brett Slatkin (a Google App Engine developer). Hopefully they’ll put some more documentation up as it develops, but it’s just really neat to see some XMPP folks build an open pubsub system with web hooks. Cheers to them!

After reading “Beyond REST? Building Data Services with XMPP PubSub” and other articles about PubSub with XMPP, I decided it would be worth to test it. However, I didn’t find any complete step by step guide in how to test it with ejabberd as XMPP server and XMPP4R as XMPP client.

Below are the steps I followed to test a subscriber (user “sub”) waiting for items published to the node (”home/localhost/pub/updates”) for the publisher (user “pub”). Note you can add as many subscribers/publishers as you want.

1. install ejabberd (XMPP server). I followed instructions of this article (spanish) or article (english) without any surprise.
2. create two ejabberd users: “pub” (the publisher) and “sub” (the subscriber).

   sudo ejabberdctl register pub localhost pub
sudo ejabberdctl register sub localhost sub

3. install XMPP4R Ruby gem.
   sudo gem install xmpp4r
4. create file nodecreator.rb. See code below.

   
   #! /usr/bin/ruby
   require "rubygems"
   require "xmpp4r"
   
   require "xmpp4r/pubsub"
   require "xmpp4r/pubsub/helper/servicehelper.rb"
   require "xmpp4r/pubsub/helper/nodebrowser.rb"
   require "xmpp4r/pubsub/helper/nodehelper.rb"
   
   include Jabber
   Jabber::debug = true
   
   service = 'pubsub.localhost'
   jid = 'pub@localhost/laptop'
   
   password = 'pub'
   client = Client.new(JID.new(jid))
   client.connect
   client.auth(password)
   
   client.send(Jabber::Presence.new.set_type(:available))
   pubsub = PubSub::ServiceHelper.new(client, service)
   pubsub.create_node('home/localhost/pub/')
   pubsub.create_node('home/localhost/pub/updates')
   

5. create file publisher.rb. See code below.

#! /usr/bin/ruby
require "rubygems"

require "xmpp4r"
require "xmpp4r/pubsub"
require "xmpp4r/pubsub/helper/servicehelper.rb"
require "xmpp4r/pubsub/helper/nodebrowser.rb"
require "xmpp4r/pubsub/helper/nodehelper.rb"
include Jabber
Jabber::debug = true
jid = 'pub@localhost/laptop'

password = 'pub'
service = 'pubsub.localhost'
node = 'home/localhost/pub/updates'
# connect XMPP client
client = Client.new(JID.new(jid))
# remove "127.0.0.1" if you are not using a local ejabberd
client.connect("127.0.0.1")
client.auth(password)
client.send(Jabber::Presence.new.set_type(:available))
# create item
pubsub = PubSub::ServiceHelper.new(client, service)
item = Jabber::PubSub::Item.new
xml = REXML::Element.new("greeting")
xml.text = 'hello world!'

item.add(xml);
# publish item
pubsub.publish_item_to(node, item)

6. create file subscriber.rb. See code below.

   
   #! /usr/bin/ruby
   require "rubygems"
   require "xmpp4r"
   require "xmpp4r/pubsub"
   require "xmpp4r/pubsub/helper/servicehelper.rb"
   
   require "xmpp4r/pubsub/helper/nodebrowser.rb"
   require "xmpp4r/pubsub/helper/nodehelper.rb"include Jabber
   #Jabber::debug = true
   jid = 'sub@localhost/laptop'
   password = 'sub'
   node = 'home/localhost/pub/updates'
   service = 'pubsub.localhost'
   
   # connect XMPP client
   client = Client.new(JID.new(jid))
   # remove "127.0.0.1" if you are not using a local ejabberd
   client.connect("127.0.0.1")
   client.auth(password)
   client.send(Jabber::Presence.new.set_type(:available))
   sleep(1)
   # subscribe to the node
   pubsub = PubSub::ServiceHelper.new(client, service)
   pubsub.subscribe_to(node)
   subscriptions = pubsub.get_subscriptions_from_all_nodes()
   puts "subscriptions: #{subscriptions}\n\n"
   puts "events:\n"
   
   # set callback for new events
   
   pubsub.add_event_callback do |event|
   begin
   event.payload.each do |e|
   puts e,"----\n"
   end
   rescue
   puts "Error : #{$!} \n #{event}"
   
   end
   # infinite loop
   loop do
   sleep 1
   end
   

7. run nodecreator.rb. It creates the XMPP node “home/localhost/pub/updates”. It creates first the node “home/localhost/pub” and then the “home/localhost/pub/updates”. Seems quite obvious but I spent some hours after I got it.
8. check the nodes have been created. I used the discovery service functionality of Psi client.
9. run subscriber.rb file. The “sub” user subscribes to the node ‘updates’ and waits for items in the “updates” node. Be aware you should close any XMPP connection with users “pub” and “sub” in case you are using any XMPP client such as Pidgin, Psi,… otherwise it won’t work.
10. run publisher.rb file. It will send a message “<greetings>hello world!”</greetings> to the subscriber. Run it as many times as you want.
11. If everything goes well yo will see in the subscriber screen a message like this.

    
    subscriptions: <subscription node='home/localhost/pub/updates' jid='sub@localhost' subscription='subscribed' xmlns='http://jabber.org/protocol/pubsub'/>
    
    events:
    <items node='home/localhost/pub/updates'><item id='3376'><greeting>hello world!</greeting></item></items>
    ----
    

Good Luck!

For this test, I used the following versions:

• Operating System: Ubuntu 8.04
• Ruby: 1.8.6
• XMPP4R: 0.4
• ejabberd: 1.1.4

This looks interesting! I’ll have to follow the next n parts of this!

This system started small but gradually had increased demands in the form of new features clients wanted added. Pretty soon, when you clicked the save button multiple things were happening; a database row would be updated, an email would be sent, a new file would be written, another status file would be updated. These were all executed when the user clicked the button, therefore every action was a blocking action, the server couldn’t move on until it was done servicing that particular request.

The quote above is from the blog post and sort of sums up my experience with a CMS-like application (Synergi), where we once did a Save, Send Mail, Commit we ended up with something like this after a few years; Save, Mail, Mail, Mail, Update Table X, Update Table Y, Mail, Commit. So a solution, like the one proposed in the blog post, could work very well for Synergi. Interesting.

Mass Transit – Part 1 of N – Tim Barcz

Ein sehr großer Anteil meiner Freizeit geht mit dem Lesen von Newsfeeds drauf. Das kann ich mal als einfache Tatsache so stehen lassen. Ich sammle die Feed-Adressen von interessanten Blogs wie andere Briefmarken, sortiere aus was keine interessanten Daten/Artikel/Links mehr liefert, versehe die gelesenen Artikel mit Tags, gebe bestimmte Tags wieder als Newsfeed aus und antworte von Zeit zu Zeit auf Posts. Ich kann wohl kaum noch bestreiten dass in meinem Fall schon ein gewisses Suchtverhalten diesbezüglich vorliegt.

Bei all der Zeit, die ich mit diesen Techniken verbringe, bin ich natürlich immer auf der Suche nach Möglichkeiten wie sich das Ganze bequemer lösen lässt. Im Moment laufen fast alle meine Aktionen über Googles “Reader”, was ein ziemlich guter Webbasierender Feedreader ist. Die Uptime-Statistiken sind natürlich wie bei allen anderen Google-Produkten für ein kostenloses Tool einfach traumhaft, auch treten eigentlich nie irgendwelche Performanceprobleme auf der Google-Seite auf und Funktionen hat das Ding eigentlich auch fast alles was ich mir wünschen kann.

Ab ein paar hundert Newsfeeds jedoch wird Reader immer umständlicher zu verwenden.
Die Seitenleiste mit der Feedübersicht lässt sich auf kleinen Bildschirmen schon lange nicht mehr sinnvoll benutzen und das verwalten der Feeds ist so ziemlich unbrauchbar. (Doppelt eingetragene URLs werden nicht automatisch entdeckt, die Feeds Tags zuzuordnen lässt sich nur sehr umständlich bewerkstelligen und Tags die einem ganzem Feed zugeordnet sind, kann man scheinbar überhaupt nicht von einem einzelnem Item entfernen.)

Auch zeigt sich bei privaten Newsfeeds sehr schnell, dass ein polling der Quelle alle 3 Stunden (wenn man der einzige Leser ist) ziemlich fern von jeder Aktualität ist. Auch 1 Stunde bei allen anderen Feeds ist nicht unbedingt die tollste Sache seit geschnitten Brot.

Lokale Software (Testergebnisse zu einzelnen Clients werden evtl. bald als Extraartikel folgen) hat es aber dennoch schwer mit derartigen Angeboten mitzuhalten. Natürlich würde ich am liebsten einen kleinen, schlanken Konsolen-Client verwenden und den RAM-fressenden Browser ganz deaktivieren. Leider bieten die wenigsten Newsfeeds den kompletten Inhalt gleich per Download an, also hat man dann den Konsolenclient UND den Browser offen. (Von eingebetteten Flash-Videos oder auch nur Bildern will ich lieber garnicht erst anfangen.)

Ein weiteres Problem mit lokaler Software zeigt sich ebenfalls erst wenn man ein paar Newsfeeds mehr als “normal” liest. Es macht verdammt wenig Spaß 1x die Stunde >300 XML-Dateien zu pollen. Dieser Effekt lässt sich verringern indem man die Dateien verteilt nacheinander abruft, mit einem Volumentarif würde ich mir das aber dennoch mehrmals überlegen. Außerdem sind wir bei 1 Abruf pro Stunde gerade erst bei der Aktualität von Google Reader angekommen, vielen wird aber etwas wie “alle 10min” viel eher logisch erscheinen.

Hier sehen wir dann auch bereits das was mich an RSS bzw. Atom am meisten stört.
Egal ob sich überhaupt etwas geändert hat und egal ob wir die letzten 10-50 Posts eh schon auf der Festplatte zwischengespeichert haben, wir laden jedes Mal alles in einer kompletten Datei aufs Neue herunter. Wenn der Newsfeed (wie eigentlich auch wünschenswert) dann auch noch den kompletten Postinhalt enthällt kommen da schnell ein paar hunder KB zusammen.

Besser wäre es z.B. schon eine Indexdatei zu basteln, die dann auf XML-Dateien mit den jeweiligen Inhalten für jeden Post verlinkt. Erstens könnte man so auf viel Overhead seitens der XML-Struktur in der Hauptdatei verzichten (evtl. könnte man sogar auf XML ganz verzichten, ein imho lohnendes Ziel). Zweitens würde es dies auch ermöglichen Änderungen der Quelle und ähnliches zu Indizieren.
Eine CSV-ähnliche Zeile mit <timestamp>,edit|new|deleted|blahfasel,<post-id>,<post-URL> würde pro Eintrag bereits reichen.

Aber warum an einer defekt designten Lösung herumfrickeln wenn es bereits bessere Ideen gibt? Die Einfachste Möglichkeit wäre wohl die Updates eher per per Push-Dienst auf den lokalen PC zu bekommen statt permanent nachsehen zu müssen ob neue Updates vorhanden sind. Eine weitere Möglichkeit sieht man bei diversen Web-Readern, es wird nur noch eine einzige Indexdatei für jeden Nutzer abgerufen, meist per AJAX, die schon sortiert die für den jeweiligen Nutzer interessanten Items enthällt.

Natürlich wäre ersteres eleganter, letzteres jedoch lässt sich offensichtlich einfacher realisieren ohne neben http noch andere Protokolle ins Spiel zu bringen (ansonsten gäbe es wohl auch endlos viele Lösungen, von denen jede besser ist als die aktuelle auf http-Basis).
Allerdings gibt es auch für die erste Version bereits Protokolle über die es sich recht einfach lösen lassen würde.

Meine aktuelle Lieblingsvariante ist wohl der Publish/Subscribe-Mechanismus welcher im Jabber-Protokoll integriert wurde. Dabei handelt es sich um eine Möglichkeit, auf einem beliebigen Pubsub-Server (ist in z.B. ejabberd bereits integriert, ein Beispiel wäre z.B. pubsub.jabber.ccc.de) in einer Struktur seine Updates an eine Node zu senden, von dort aus verteilt der Server dann automatisch die Informationen an alle Clients die sich eingetragen haben weiter.

Leider krankt Pubsub derzeit noch stark daran dass es keine wirklich ausgereifte Implementierung auf Clientseite gibt. Gajim unterstützt das XEP zwar scheinbar zu gewissen Werten (andere Clients scheinen afaik da noch überhaupt keine Anstrengungen zu unternehmen), es gibt aber immer wieder Probleme wenn man versucht es wirklich zu nutzen. In meinen Tests war es mir noch nicht möglich überhaupt eine Meldung von irgendeiner Pubsub-Node zu bekommen.

Auf der Serverseite sieht es theoretisch besser aus, z.B. besteht schon seit längerem ein Plugin für WordPress, dass es recht einfach ermöglichen würde neue Posts oder Kommentare auch per Pubsub auszuliefern. Allerdings hängt dies auch wieder an den Clients etwas, die Serverseitigen Scripte kann man ja atm nur über die XML-Konsole testen, die einige Clients bieten. Hier sollte natürlich die Grundregel gelten dass man NIEMALS einen Menschen dazu zwingen sollte XML-Code zu tippen, in diesem Fall sogar Live mit einem Server zu reden. XML als “menschenlesbares” Datenformat ist wohl einer der Witze, die sich ein denkender Mensch nichtmal absichtlich hätte ausdenken können.

Alle aktuell verwendbaren Lösungen haben ohnehin offensichtlich das XML-Format gemeinsam. Für ein System das immer mehr und mehr Bandbreite schluckt, dessen Bereitstellung und Verarbeitung meiner Meinung nach in Zukunft immer wichtiger wird und dessen Inhalt meist nicht über <timestamp><autor><link><Datenpaket> hinausgeht sollte man sich evtl. doch besser etwas anderes einfallen lassen. Aber das ist mal wieder nur meine Meinung.

Randbemerkung: Während ich diesen Artikel geschrieben hab (dieser stellt mit 4 Tagen als Entwurf einen neuen, privaten Rekord dar) wurde das Thema bei einigen anderen Stellen angeschnitten, vor allem beziehe ich mich damit auf mehrere Diskussionen im IRC (kann ich euch leider nicht zum Nachlesen verlinken) als auch auf einen Beitrag von René, zu dem es teilweise sehr interessante Kommentare gab.

Fellow Vancouver Voip veteran and buddy Ian Bell has reclaimed PubSub and will soon be launching a new FaceBook app. Ian is a very capable fellow and he is in good company with Lance Tracey joining as CEO.

I think Ian is doing the right thing in going after the Facebook crowd, the intelligent search capabilities of PubSub will be put to good use there. Ian is also getting some love from Michael Arrington and Om Malik.

Great stuff Ian, good to see PubSub alive again!

Bob Wyman talks about problem’s he’s noticing with ranking on the long tail:

If Matt had been able to see the full set of millions of ranks that we compute, he would have noticed a second “region” of very stable ranks. That region is at the bottom of the ranking scale — sites that never get linked to by anyone. The harsh reality is that those with the lowest ranks tend to stay that way. Those with the highest rank tend to keep their high rank.

Hm. I don’t see this to be very interesting though. These are probably just new blogs and people aren’t doing much to increase their inbound links. The majority of the thin tail would/should have an indegree of zero.

Matt Hurst has more on this:

It suggests, not surprisingly, that the higher you are ranked, the less likely you are to change rank – or the more stable the rank is.

…

And Bob finishes with a great quote:

The problem is that in almost all rankable systems that deal with information and ideas, the stuff at the top is where you’ll find things that everyone already knows about or, those at the top of the scale tend to reflect points-of-view that are well known, broadly accepted and are generally predictable… Unfortunately, you can’t just invert things and look at the bottom ranks as an alternative. The “signal-to-noise” ratio down there isn’t very good…

The really useful place to look, if new information or new ideas is that you’re interested in, is the middle ranks. As Aristotle said long ago: “Seek the mean.” That’s where the good stuff is.

I couldn’t agree more. This is why I like blogs. The middle ground is very compelling. I can read diverse and interesting topics from passionate people. I don’t care about the short head. I don’t subscribe to anyone on the Technorati Top 100. If there’s something good there I’ll find it on another blog and hit it on a random walk while using my aggregator.

Nice. PubSub increased the space race here and released their PubSub LinkRanks 1000:

The PubSub LinkRanks 1000 is a list of the most consistently influential sites that publish feeds, based on their average LinkRank scores from August 1st to August 30th, 2005. To learn more about LinkRanks, click here.

To create this list, we’ve averaged the daily LinkRanks of more than 15 million sources. We’ve also included a 15-day average as well as daily LinkRank data for August 30th as additional points of comparison.

This is twice as good as the Feedster 500 and 10x as good as the Technorati Top 100! . That said it’s still not long tail enough. There are still 14,999,900 blogs that aren’t ranked here (in PubSub’s index). Still pretty damn cool.

I’m no where to be seen on the list (feedblog.org). Hopefully it’s not personal and just because my blog is so new.

My kingdom for a way to include my PubSub LinkRank directly within my blog. It seems like it would be pretty trivial to implement but they’re using SVG caching right now which means I probably wouldn’t ever see the graph updated if I used their URL.

Here’s my current graph:

Which I’d like to see keep growing but it’s a bit difficult to keep checking back to their site.

Update:

I don’t know why I didn’t think of this before but they’re using a redirect to use a cached copy of graphs. For example this is the link in the HTML which then redirects to this link which is in a cache. I guess I could just include the graphic inline but I don’t want to hurt their servers.

Apparently Zawodny thinks trackback is dead:

I’m convinced now more than ever. But I’ll spare you all long
rant about why it’s dead, since others have already written this for me:

• Trackback is dead. Are Comments dead too?
• Trackback is dead, Use PubSub
• Trackback really is dead – Sorry Elliott et al

I’ve always felt that the case for trackback was a difficult one to make. One problem is that trackbacks are essentially automated comments.

Why not just provide a web comment API? No one has been working on this really and it’s a bit shocking that it’s 2005 and we still don’t have a way to syndicate comments (granted WordPress has a strong implementation of wfw:comments but that’s only one provider). Given a permalink why can’t I get a list of comments? Why can’t I post a comment through NetNewsWire or Ecto? I can send a trackback via Ecto but only with reduced metadata (the extract, not the full text).

I still don’t feel that Pubsub, Technorati, IceRocket, or Feedster are the answer. This just adds a brittle centralized man in the middle which can break when you least expect it.

Neither of these will be fixed anytime soon though. Link tracking services will still be fragile and comment APIs won’t take off (I’m obviously jaded).

Update:

The RSS Weblog has some thoughts:

Personally, I think TrackBacks foster self-promotion more than dialogue. Spam aside, linking offsite to follow the conversation is just too disruptive.