LakTEK A Sri Lankan, A Rubyist and A Web Dude

Implementing Web Socket servers with Node.js

Web Sockets are one of the most interesting features included in HTML5 spec. It would open up a whole different paradigm in web application development by allowing asynchronous, long-lived connections between client and server. As Web Sockets were supported in Google Chrome’s beta release, it signaled now the time to use it in your apps.

However, WebSockets doesn’t really go well with the traditional synchronized web server environments. Use of evented libraries such as Node.js for Web Socket servers seemed more practical and scalable. But the initial versions of Node.js’s didn’t have built-in support for Web Socket connections specifically. There were several Web Socket server implementations based Node.js, which overcame this problem by hijacking its HTTP module.

Node.js still doesn’t include WebSockets in its core modules as in other languages (i.e. Go language). However, the recent overhaul to HTTP module, have made implementation of web sockets with whole lot easy. Now Node.js’s HTTP server module would emit “Upgrade” event, each time a client requests a http upgrade. This event could be trapped when implementing a web socket server.

Here is a simple, low-level example for a Node.js based HTTP server, which supports both common HTTP requests and web socket connections.

var sys = require("sys");
var net = require("net");
var http = require("http");
 
function createTestServer(){
  return new testServer();
};
 
function testServer(){
  var server = this;
  http.Server.call(server, function(){});
 
  server.addListener("connection", function(){
    // requests_recv++;
  });
 
  server.addListener("request", function(req, res){
    res.writeHead(200, {"Content-Type": "text/plain"});
    res.write("okay");
    res.end();
  });
 
  server.addListener("upgrade", function(req, socket, upgradeHead){
    socket.write( "HTTP/1.1 101 Web Socket Protocol Handshake\r\n"
                + "Upgrade: WebSocket\r\n"
                + "Connection: Upgrade\r\n"
                + "WebSocket-Origin: http://localhost:3400\r\n"
                + "WebSocket-Location: ws://localhost:3400/\r\n"
                + "\r\n"
                );
 
    request_upgradeHead = upgradeHead;
 
    socket.ondata = function(d, start, end){
      //var data = d.toString('utf8', start, end);
      var original_data = d.toString('utf8', start, end);
      var data = original_data.split('\ufffd')[0].slice(1);
      if(data == "kill"){
        socket.end();
      } else {
        sys.puts(data);
        socket.write("\u0000", "binary");
        socket.write(data, "utf8");
        socket.write("\uffff", "binary");
      }
    };
  });
};
 
sys.inherits(testServer, http.Server);
 
var server = createTestServer();
server.listen(3400);

There is a more high-level and elegant Web Socket Server library (http://github.com/miksago/node-websocket-server) in development by Micheil Smith. It is built utilizing the new HTTP library and compatible with the draft 76 of the Web Sockets spec (which includes bunch of security improvements).

Here is an example, on how to implement a web socket server with the above mentioned library.

var sys = require("sys");
var ws = require('./vendor/node-websocket-server/lib/ws');
 
function log(data){
  sys.log("\033[0;32m"+data+"\033[0m");
}
 
var server = ws.createServer();
server.listen(3400);
 
server.addListener("request", function(req, res){
  res.writeHead(200, {"Content-Type": "text/plain"});
  res.write("okay");
  res.end();
});
 
server.addListener("client", function(conn){
  log(conn._id + ": new connection");
  conn.addListener("readyStateChange", function(readyState){
    log("stateChanged: "+readyState);
  });
 
  conn.addListener("open", function(){
    log(conn._id + ": onOpen");
    server.clients.forEach(function(client){
      client.write("New Connection: "+conn._id);
    });
  });
 
  conn.addListener("close", function(){
    var c = this;
    log(c._id + ": onClose");
    server.clients.forEach(function(client){
      client.write("Connection Closed: "+c._id);
    });
  });
 
  conn.addListener("message", function(message){
    log(conn._id + ": "+JSON.stringify(message));
 
    server.clients.forEach(function(client){
      client.write(conn._id + ": "+message);
    });
  });
});

I’m with OpenNebula this Summer!

I had the opportunity to get selected for Google Summer of Code on the freshman year itself in my academic life. The experience I gained in that summer working with SilverStripe project boosted my self confidence and helped me immensely to shape up my career.

This year, which happens to be my final year as an undergraduate, I’m going to get yet another stint with Summer of Code. This time, it’s with OpenNebula. OpenNebula is an Open Source toolkit for cloud computing. This project is relatively young and small, but something which could make a great impact for the future. In simple terms, OpenNebula lets you run your own cloud hosting service like Amazon EC2.

As a developer, who makes use of cloud platforms, I really want to see open standards getting adopted among cloud service providers. Vendor lock-in is the biggest threat I see when moving to cloud based platforms. Projects such as OpenNebula are great initiatives to avoid this. OpenNebula is also one of the main supporters for the OCCI standard interface. I have been following their developments closely, from the first day I got to know about them.

When I saw OpenNebula has been selected as a mentoring organization for the first time in this year’s GSOC, I thought this would be a great chance for me to contribute to them. Another interesting thing about OpenNebula is they use Ruby as their main development language.

In this summer, I will work on building a web based administration console for OpenNebula. I believe it will make OpenNebula more usable and increase its adoption. It would be something similar to AWS(Amazon Web Service) management console, but more rich in terms of the capabilities.

Though, we are still in the brain-storming phase of the project, I could give a small hint that we would be using Sinatra and other Rack middleware to build this. So it’s yet another chance to show the power and flexibility of micro-frameworks. I will be sharing my experiences during the project via this blog. I hope it would be really fun and exciting.

Finally, big shout-out goes to my mentor, Jaime Melis, who was very supportive from the time of preparing the proposal. It’s always a pleasure to work with someone like him, who is really passionate and knowledgeable.

If you are interested in reading my full project proposal : http://docs.google.com/Doc?docid=0AeUIyatONYiTZGY3ZnYyZmNfNjljanJyZDNjNg&hl=en


Building Real-time web apps with Rails3

On deciding the web framework to build Realie, one of the main considerations was should I move to a totally asynchronous framework? Most established web frameworks, including my favorite Rails is built in a synchronous manner and follows a call-stack based model. Real-time web apps needs to be asynchronous. Evented programming model ideally suits to this.

Since there were lot of hype around Node.js based async web frameworks in last couple of months, my initial idea was also to use such framework for my project. However, that felt as a totally new learning curve. Apart from grasping how to use JavaScript in server-side, it also meant I need to adopt to a totally new eco-system of templating, routing and etc.

However, when I revisit my requirement it was clear only a part of the web app really needs to be asynchronous. Most parts can still be done with a traditional call stack based web framework. Using a fully async. web framework to build an entire app seems to be useless. Also, it felt an overkill to run two different apps to serve sync and async stuff.

In this context, I came to know about Cramp, a Ruby asynchronous framework written by Pratik Naik. Best thing about Cramp is capability of using Rack middlewares (keep in mind not fully Rack compliant). Then came the option how about using Rails and Cramp together to build a hybrid of real-time web app. With Rails3, it makes it so easy to mix any other Rack endpoint with Rails. So this sounded a perfect solution to my problem.

Since, Cramp follows evented model it needs an evented web server such as Thin or Rainbows!. Further, Cramp has implemented websockets support for these two server backends.

Integrating Rails3 and Cramp

First of all, you will need to bundle Cramp gem, with your Rails app. For this, open the `Gemfile` and add the following:

    gem "cramp", :require => 'cramp/controller'

For my work I only needed the Cramp controller (it has also got an async model) so for now I only required it.

As I mentioned earlier, to support web sockets, cramp needs to extend the web server we use. To specify the web server, I added an initializer (`config/initializers/cramp_server.rb`) with the following line :

    Cramp::Controller::Websocket.backend = :thin

Then, I created a simple Cramp controller, which can respond to web sockets (`app/cramps/communications_controller.rb`)

class CommunicationsController < Cramp::Controller::Websocket
  periodic_timer :send_hello_world, :every => 2
  on_data :received_data
 
  def received_data(data)
    if data =~ /stop/
      render "You stopped the process"
      finish
    else
      render "Got your #{data}"
    end
  end
 
  def send_hello_world
    render "Hello from the Server!"
  end
end

Now the fun part! The new Rails3 router supports pointing to any Rack compatible endpoint, so we can easily hook our cramp controller for public access. In `config/routes.rb` add the following:

  match "/communicate", :to => CommunicationsController

Our Cramp endpoint can co-exist with rest of the Rails controllers without any issues.

Viola!

Another important change with Rails3 is its also a fully compatible Rack app now. This means as any other Rack app, we can also start our Rails app by running `rackup`.

rackup -s thin -p 3000 --env production

This will start our app using Thin server backend on port 3000. Keep in mind, we need to provide an environment other than `development`, to avoid Rack Lint middleware. This is because Cramp is not a fully compatible with Rack SPEC and it will throw exceptions.


Realie Project: Data Structure & Storage

Last couple of days, I found some time to work on my individual research project for the degree course. The topic area I selected for my project was on “Real-time Web”. Real-time Web is just the opposite of the current way we use the web. Rather than we checking (polling) content providers for updates, content providers will feed (push) us with the updates. This concept is getting rapid adoption and I believe it would be the de-facto behaviour of the web in couple of years.

The application I decided to build was a code editor based with real-time collaboration capabilities. Following the hacking traditions and for the ease of remembrance, this project was code named “Realie” (it was the zillionth time the name was changed and hope it would stick this time).

Rather than developing the project behind closed doors and finally presenting a thesis (which would be utterly boring), I thought of building the project in open, by sharing the code and discussing design decisions with the community. I believe what it really matters is the experience and knowledge I could gather from this, rather than the final grade I would get for this.

So lets start by looking at the initial data structure and storage decisions.

What is Realie?

Most of you may have heard or already using Google Wave & Etherpad. Realie also got inspired from those two projects. While, Google Wave & Etherpad are known to be real-time collaborative editors(or canvases) for general public, with Realie we try to cater the niche of hackers & developers. As we know development process is already a collaborative process, which involves lot of real-time communication & decision making. As developers we know this process is not seamless and painstaking. This is the void which Realie tries fill.

Putting it to simple terms, Realie would be like pastie or gist where multiple people could view and edit at the same time. There would be more other jazzy features in the project, but this would be the essence of it. Imagine how such a tool would make your remote pair-programming, code reviewing & brainstorming sessions a breeze?

Starting from the scratch

As I was starting the project, Google acquired Etherpad and that action made Etherpad to release their code as open-source. Though, it sounded a perfect opportunity for me to fork their code and get my project done, I decided to start from the scratch. Etherpad is a grown project and they have already made certain design decisions. Adopting from them, wouldn’t help me to gain any experiences on the implications of designing a real-time system or to explore better ways of doing things.

One of the first challenges I had to face was on deciding the data storage mechanism I’m going to adopt for this project. My initial idea was to create each editing pad as a physical file in the system and track the changes each user would be doing using Git. This sounded very unrealistic as disk IO would be very slow and executing git commands via shell would be even more slower!

Relational databases as storage medium doesn’t seem to be a good for the task, since we should be doing massive writes and continuous querying. The best option in this scenario was to use a key-value based storage.

Choosing Redis

I got convinced to use Redis as the data store, after hearing lot of good about it and further seeing some impressive benchmarks.

Redis is just beyond a normal key-value store, we could have lists, sets or sorted-sets as data structures in Redis and it’s possible to do operations like sorting, taking difference, intersections and unions of the data. Also, one of the most interesting features I saw in Redis was it’s persistence options. You can either use it as only a in-memory storage (which is very fast), or either write the data to disk periodically or even write data to disk only on a change (Append-only file)

Data Structure

The atomic data unit of Realie would be a `Line`. Each pad is composed of bunch of lines. A line would be also equivalent to a single edit a user make on a file (pad). When we are storing the lines, we will need to store the following attributes along with it - user, pad, content, position (line number in the file) and timestamp.

In Redis, we can only store data values as strings. For this, each line will be serialized to JSON before being stored in the data-store. JSON serialization also makes it possible to consume & manipulate line contents in client or server easily.

Since we need to keep references to a line in several places in the data-store, a unique SHA-1 hash based on the contents of the line is calculated and it’s used as the key for that line.

As I mentioned earlier, a Pad made as a collection of line. It’s basically similar to any source files with lines of codes. Beyond that each pad will store the list of users who are working on that pad. Users will have the option to join a pad or leave a pad as they wish.

For a pad, there are two basic views. One is snapshot view, which is the current state of the pad after applying the most recent changes made by the users. The interesting view would be the timeline view - which would have the changes in order of users made. This view can be used to generate the historical versions of the pad (at a given time or checkpoint) or even to create a playback to see how the pad was changed over time.

Source Code

You can checkout the code from following GitHub repository - http://github.com/laktek/Realie. Please note, currently the project source code only contains models & specs for above data structures and it could be changed, as you read.


Understanding Election Results through Economic Theory of Democracy

Though its been two days since the announcement of the results, still there is no end to the spread of speculations & rumors on the concluded Presidential Election in Sri Lanka. I don’t have any strong political bias to either of the main two candidates and didn’t want to accept anything reported in a blind eye.

This morning, I came across an interesting Political Science literature written by Anthony Downs, named “An economic theory of democracy”. I only could read the WikiSummary of it, but while going through it, my mind eventually mapped it to the context of the Sri Lankan Presidential Election. It helped me to dispel some of the doubts that were in my mind and understand how majority of the people would have voted.

I thought of jotting them down here for others who are interested. Also, I hope someone, who is more knowledgeable on this subject would correct me if my line of thoughts was wrong (I haven’t done any formal study on Political Science than casual reading of the stuff here and there).

Downs defines basic logic of voting as follows:

In a world of perfect information, each voter would compare his expected utility of having party A (incumbent) in government (for another term, that is) with the expected utility of having party B (opposition) in government. This utility differential would determine each voter’s choice at the ballot box.

In our case, party A would be current president Hon.Manhinda Rajapakse and party B is opposition’s common candidate Gen.Sarath Fonseka.

Further Downs mentions there are several factors that would matter to a typical voter and would modify the above model. Let’s consider each of those factors.

1. He doesn’t really know what the future holds, so he doesn’t know which party’s rule will give him greater utility (in the future). So instead, he will instead compare the utility he got over the last term from party A with what he thinks party B would have provided under the same circumstances; if he thinks party B would have brought him more utility, he votes for B.

I think from the first Presidential Election what mattered to most of the Sri Lankan average voter was the civil war with LTTE in north & east. Hence, the end of war, the biggest utility he got during the last term of party A (i.e. Mahinda Rajapakse). Though, Gen.Sarath Fonseka played a major role in war victory, I feel due to the other parties involved with him (and their broken promises on ending the war in the past), would have prevented majority from voting to party B over A.

2. He doesn’t just look at raw utility differential, though; he also considers the trend. Is A getting better or worse? If it is getting better, then the voter will forgive A for early failures to deliver utility.

This is the point government (party A) did exploited within the last 2 months after the announcement of the election. Fy-over bridges, International Stadiums, Power Plants and all sorts of the other development efforts started to blossom within this period. Also, fuel prices and price of other essentials were brought down tactfully. Moreover government again used peace as a lucrative belief for future development. Voters would have consider this as a positive trend. While opposition clearly showed the government (party A) has failed to fulfill many of the promises in last election manifesto (Mahinda Chintanaya), voters seems to have forgiven for those failures.

3. If A and B would have provided equal utility, the voter asks himself whether B would have used identical approaches and policies as A, or different ones?

* If B would have been identical, the voter is indifferent and abstains.

Apart from ending corruption and government wastage opposition(party B) didn’t have vastly different approach when it comes to solving other problems (improving agriculture, education, healthcare or on how to create more job opportunities) which would have failed to convert subset of voters from their current stance.

Also, as I see the above point could also explain the reason for the lower voter turnout in North and East. Both A & B didn’t seem to have significantly different or new approaches when it comes to solving the prevailing problems of those areas. This would have led many from those areas to abstain from voting.

* If B would have provided equal utility but by different means, then the voter concludes: “Okay, a vote for B is a vote for something to change, but a vote for A is a vote for no change.” He then must evaluate whether change (generally) is a good thing. To make this evaluation, performance evaluations come into play. Based on the history he has seen of various parties governing in various circumstance, he asks himself, “How much utility would the ideal government have delivered me under the circumstances that A has governed in?” If A stacks up well in comparison, he votes against change (i.e. for A). If not, he votes for change (i.e. for B) and hopes for the best.

Ending the corruption & wastage in government was the popular slogan of the Gen.Fonseka’s campaign (party B). Here I believe again the voters were split into two sides based on the performance evaluations. Some thought Gen.Fonseka, who comes from a totally different background to political arena would surely bring in the change they want by getting rid of all the corrupted politicians. While, there were another bunch, who did look at the past (even considering the cases like Hitler, Idi Amin & etc) feared of he would turn into a dictator and decided to vote against the change. Apparently, based on the results it’s evident the number belong to latter set was high.

Finally, Downs mentions a very important statement in his writing:

These decisions about utility, however, lack perfect information. He must estimate all these questions about utility based on the “few areas of government activity where the difference between parties is great enough to impress him”. In other words, voters use information shortcuts;

As I understand, for voters to use information shortcuts he must be able to get the true picture of the context. This is why existence of free, balanced & impartial media should really matter!


← Before After →