pragmatism: The beauty of node.js
The beauty of node.js
At work I came across a problem which needed ack remote file copying complete event. Most of the data comes on ftp and when the data arrives, some processing has to be done on it. The problem is to when to kick in the processing. If we do a cronjob approach, the cronjob processor can't know for sure if the new file has been pushed completely. inotify posix system call only tell you the status of the file viz. created, modified, read etc.
EDIT: inotify does allow you to listen for the write complete events. The flag to look for is IN_CLOSE_WRITE. I kinda missed this.
The fix we have at work is to change the ftpd itself. The ftpd writes to a unix named pipe when the transfer completer. The listener daemon is watching the named pipe for changes and kicks in the callback when a new entry is made.
This works but has a couple of issues.
- Changing the ftp daemon itself means keeping up with the ftpd changes. If a new release of ftpd is to be deployed, the patch has to be applied.
- The same patching needs to be done if we decide to support another service. If we decide to allow rsync, similar fix has to be done for rsync as well.
How we do at work isn't going to change because our push data is solely ftp. But given a chance to re-design it, I would simply write a port forwarding service. The port forwarding server would listen at a dummy ftp port incoming. When a connection is made to port incoming, it would open a client connection to real ftp server. It would forward the whole traffic and it's only job would be to trigger the callback when the ftp transfer completes.
This solution required writing some code. Out of box port forwarding solutions don't have the customization for listening to end of transfer. So, faced with a problem, I did what everyone does - googled it. I found two acceptable solutions in Python. I prefer to code much of my stuff in Python for some obvious and not-so-obvious reasons(coming soon - topic for some other inflammatory post).
One of the solutions was spawning threads:
Port forwarding using threads
And the other one was using asyncore module.
Port forwarding using asyncore
The solutions worked and looked good for my taste. But something seemed amiss. It looked like just too much work when all I want to do is to know when the communication completes. I looked around a while and coded this in node.js.var tcp = require ( "tcp" );var server = tcp . createServer ( function ( socket ) {var client ;socket . addListener ( "connect" , function () {client = tcp . createConnection ( 22 );client . addListener ( "receive" , function ( data ) {socket . send ( data );});client . addListener ( "eof" , function () {client . close ();});});socket . addListener ( "receive" , function ( data ) {client . send ( data );});socket . addListener ( "eof" , function () {/* Communication complete. */triggerCallback ();socket . close ();});});server . listen ( 7000 , "localhost" );
The solution works and is terse, concise and highly scalable(look at the node.js website for how and why). And above all, it's as simple as it gets.
The code itself is pretty simple. I create a server and install callbacks for it. On connect event, I create the client for real service. Client and servers listen for receive, and the callbacks just sends data from one socket to other. The server socket triggers the callback on receiving the eof.
For the example, I am forwarding port 7000 to port 22. When we run the code and do a ssh on port 7000, the traffic is correctly forwarded to sshd running on port 22.node portfwd.js ssh -p 7000 localhost
This solution can be extended(must be extended) to function as a fully fledged port forwarder. The configurations(which port to forward where) can be passed as JSON. And the services to which we are forwarding should be made inaccessible to the outer world. For example, sshd can be configured to listen only to loopback address. This is important otherwise we run the risk of circumventing port forwarding and not running the callback.
There is much that can be done in node.js. This is just the tip of the iceberg. Looks like Javascript is really on the rise.