Search Results for websockets

How to send the output of Symfony’s process Component to a node.js server in Real Time with Socket.io

Today another crazy idea. Do you know Symfony Process Component? The Process Component is a simple component that executes commands in sub-processes. I like to use it when I need to execute commands in the operating system. The documentation is pretty straightforward. Normally when I want to collect the output of the script (imagine we run those scripts within a crontab) I save the output in a log file and I can check it even in real time with tail -f command.

This approach works, but I want to do it in a browser (call me crazy :)). I’ve written a couple of posts with something similar. What I want to do now? The idea is simple:

First I want to execute custom commands with process. Just follow the process documentation:

<?php
use Symfony\Component\Process\Process;

$process = new Process('ls -lsa');
$process->setTimeout(3600);
$process->run();
if (!$process->isSuccessful()) {
    throw new \RuntimeException($process->getErrorOutput());
}

print $process->getOutput();

Process has one cool feature, we can give feedback in real-time by passing an anonymous function to the run() method:

<?php
use Symfony\Component\Process\Process;

$process = new Process('ls -lsa');
$process->run(function ($type, $buffer) {
    if ('err' === $type) {
        echo 'ERR > '.$buffer;
    } else {
        echo 'OUT > '.$buffer;
    }
});

The idea now is to use this callback to send a TCP socket to one server with node.js

var LOCAL_PORT = 5600;
var server = require('net').createServer(function (socket) {
    socket.on('data', function (msg) {
        console.log(msg);
    });
}).listen(LOCAL_PORT);

server.on('listening', function () {
    console.log("TCP server accepting connection on port: " + LOCAL_PORT);
});

Now we change our php script to

<?php
include __DIR__ . "/../vendor/autoload.php";

use Symfony\Component\Process\Process;

function runProcess($command)
{
    $address = 'localhost';
    $port = 5600;
    $socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
    socket_connect($socket, $address, $port);

    $process = new Process($command);
    $process->setTimeout(3600);
    $process->run(
        function ($type, $buffer) use ($socket) {
            if ('err' === $type) {
                socket_write($socket, "ERROR\n", strlen("ERROR\n"));
                socket_write($socket, $buffer, strlen($buffer));
            } else {

                socket_write($socket, $buffer, strlen($buffer));
            }
        }
    );
    if (!$process->isSuccessful()) {
        throw new \RuntimeException($process->getErrorOutput());
    }
    socket_close($socket);
}

runProcess('ls -latr /');

Now with the node.js started, if we run the php script, we will see the output of the process command in the node’s terminal. But we want to show it in a browser. What can we do? Of course, socket.io. We change the node.js command to:

var LOCAL_PORT = 5600;
var SOKET_IO_PORT = 8000;
var ioClients = [];

var io = require('socket.io').listen(SOKET_IO_PORT);

var server = require('net').createServer(function (socket) {
    socket.on('data', function (msg) {
        ioClients.forEach(function (ioClient) {
            ioClient.emit('log', msg.toString().trim());
        });
    });
}).listen(LOCAL_PORT);

io.sockets.on('connection', function (socketIo) {
    ioClients.push(socketIo);
});

server.on('listening', function () {
    console.log("TCP server accepting connection on port: " + LOCAL_PORT);
});

and finally we create a simple web client:

<script src="http://localhost:8000/socket.io/socket.io.js"></script>
<script>
    var socket = io.connect('http://localhost:8000');
    socket.on('log', function (data) {
        console.log(data);
    });
</script>

Now if we start one browser we will see the output of our command line process within the console tab of the browser.

If you want something like webConsole, I also have created the example, With a Web UI enabling to send custom commands. You can see it in github.

Obviously that’s only one experiment with a lot of security issues that we need to take into account if we want to use it in production. What do you think?

Advertisements

Display errors on screen even with display errors = off with PHP

Last month I was in a coding kata session playing with StingCalculator, and between iteration and iteration we were taking about the problems of shared hostings. Shared hosting are cheap, but normally they don’t allow us the use some kind of features. For example we cannot see the error log. That’s a problem when we need to see what happens within our application. Normally I work with my own servers, and I have got full access to error logs. But if we cannot see the error log and the server is configured with display errors = off in php.ini (typical configuration in shared hosting), we have a problem.

One of my post came to my mind “Real time monitoring PHP applications with websockets and node.js”. Basically we can solve the problem using this technique, but if we are speaking about shared hosting without error log access, it’s very probably that we cannot install a node.js server or even use sockets functions. So it’s not “Real” solution to our problem.

The idea is create a variation of the original script (one kind of script’s Spin-off ;)). In this scprit we will capture errors and exceptions and show them in script at the end of the script. We don’t have access to the error log but we will show it in the browser.

Imagine the following script:

$a = 1/0;
throw new Exception("myException");
echo "hi";

It will show one warning (1/0) and one exception (myException)

  • Warning: Division by zero
  • Fatal error: Uncaught exception ‘Exception’ with message ‘myException’

If we change php.ini show errors = off we will see a nice white screen.

The idea is add to the script:

include('ErrorSniffer.php');
ErrorSniffer::factory('127.0.0.1');

$a = 1/0;
throw new Exception("myException");
echo "hi";

Now with with ErrorSniffer library we will see a nice output, even with display errors = off.
I also add a IP in the class constructor to restrict the output message to one IP. The idea is to place this script at production, so we don’t want to expose error messages to whole users.

If we see the source code of ErrorSniffer class we a constructor like this:

    public function __construct($restingToIp)
    {
        if ($this->getip() == $restingToIp) {
            self::register_exceptionHandler($this);
            self::set_error_handler($this);
            self::register_shutdown_function($this);
        }
    }

    private static function set_error_handler(ErrorSniffer &$that)
    {
        set_error_handler(function ($errno, $errstr, $errfile, $errline) use (&$that) {
                $type = ErrorSniffer::getErrorName($errno);
                $that->registerError(array('type' => $type, 'message' => $errstr, 'file' => $errfile, 'line' => $errline));
                return false;
        });
    }

    private static function register_exceptionHandler(ErrorSniffer &$that)
    {
        set_exception_handler(function($exception) use (&$that) {
                $exceptionName = get_class($exception);
                $message = $exception->getMessage();
                $file  = $exception->getFile();
                $line  = $exception->getLine();
                $trace = $exception->getTrace();
    
                $that->registerError(array('type' => 'EXCEPTION', 'exception' => $exceptionName, 'message' => $message, 'file' => $file, 'line' => $line, 'trace' => $trace));
                return false;
        });
    }

    private static function register_shutdown_function(ErrorSniffer &$that)
    {
        register_shutdown_function(function() use (&$that) {
                $error = error_get_last();

                if ($error['type'] == E_ERROR) {
                    $type = ErrorSniffer::getErrorName($error['type']);
                    $that->registerError(array('type' => $type, 'message' => $error['message'], 'file' => $error['file'], 'line' => $error['line']));
                }

                $that->printErrors();
        });
    }

As we can see we will catch errors and exceptions, we populate the member variable $errors and we also use register_shutdown_function to show information on shutdown.

You can see the full script at github here.

What do you think?

Real time notifications (part II). Now with node.js and socket.io

In one of my previous posts I wrote about Real time notifications with PHP. I wanted to create a simple comet system fully written in PHP and JavaScript. It worked but as Scott Mattocks told me in a comment this implementation was still just doing short polling. The performance with this solution may be bad in a medium/hight traffic site. This days I’m playing with node.js and I want to create a simple test. I want to do exactly the same than the previous post but now with node.js instead of my PHP+js test. Let’s start

Now I want to use socket.io instead of pure web-sockets like my previous posts about node.js. For those who don’t know, socket.io is amazing library that allows us to use real-time technologies within every browsers (yes even with IE6). It uses one technology or another depending on the browser we are using, with the same interface for the developer. That’s means if we’re using Google Chrome we will use websockets, but if our browser does’t support them, socket.io will choose another supported transports. Definitely socket.io is the jQuery of the websockets. The supporter transports are:

  • WebSocket
  • Adobe Flash Socket
  • AJAX long polling
  • AJAX multipart streaming
  • Forever Iframe
  • JSONP Polling

First we create our node.js server. A really simple one.

var http = require('http');
var io = require('socket.io');

server = http.createServer(function(req, res){
});
server.listen(8080);

// socket.io 
var socket = io.listen(server);

socket.on('connection', function(client){
  client.on('message', function(msg){
      socket.broadcast(msg);
  })
}); 

This server will broadcast the message received from the browser to all connected clients.

Our HTML page will look like that:

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Comet Test</title>
    </head>
    <body>
        <p><a id='customAlert' href="#" onclick='socket.send("customAlert")'>publish customAlert</a></p>
        <p><a id='customAlert2' href="#" onclick='socket.send("customAlert2")'>publish customAlert2</a></p>
        <script src="http://localhost:8080/socket.io/socket.io.js" type="text/javascript"></script>
        <script type="text/javascript">

// Start the socket
var socket = new io.Socket(null, {port: 8080});
socket.connect();

socket.on('message', function(msg){
    console.log(msg);
});
        </script>
    </body>
</html>

As we can see we are including the js script called socket.io/socket.io.js. This script is served by our node server.

In fact we can use our node.js to serve everything (HTML, js, CSS) but in our example we will use only node.js for real-time stuff. Apache will serve the rest of the code (only a HTML file in this case).

And that’s all. Those few lines perform the same thing than our PHP and js code in the other post’s example. Our node.js implementation is definitely smarter than the PHP one. The socket.io library also allows us to use the example with all browser. Same code and without any browser nightmare (just like jQuery when we work with DOM).

Here I have a little screencast with the working example. As we will see there We will connect to the node server with firefox and chrome. Firefox will use xhr multipart and Chrome will use Websokets.

Another important issue of socket.io library is that we forget about the reconnection to the web-socket server, if something wrong happens (as we can see in Real time monitoring PHP applications with web-sockets and node.js). If we use raw WebSocket implementations and our connection with the web-socket server crashes or if we stop the server, our application will raise disconnect event and we need to create something to reconnect to the server. socket.io does it for us. With our small piece of JavaScript code we will get a high performance real-time applicatrion. Node is cool. Really cool. Kinda wierd at the beginning but the learning effort will be worthwhile. A few js lines and a real time applications.

I’ve got a problem within our node.js application. If we’ve got some kind of security within our application (imagine for example it’s behind a session based auth form) we need to share this security layer with our node.js server to ensure that non authenticated users aren’t allowed to use our websockets. I don’t know how to do it just now, but I’m investigating. Do you have any idea?

Full code Code at github. Ensure you’re using the stable version of node.js. With the last versión available on github of node.js there’s a bug and server dies when we connect with Google Chrome.

Web console with node.js

Continuing with my experiments of node.js, this time I want to create a Web console. The idea is simple. I want to send a few command to the server and I display the output inside the browser. I can do it entirely with PHP but I want to send the output to the browser as fast as they appear without waiting for the end of the command. OK we can do it flushing the output in the server but this solution normally crashes if we keep the application open for a long time. WebSockets again to the rescue. If we need a cross-browser implementation we need the socket.io library. Let’s start:

The node server is a simple websocket server. In this example we will launch each command with spawn function (require(‘child_process’).spawn) and send the output within the websoket. Simple and pretty straightforward.

var sys   = require('sys'),
http  = require('http'),
url   = require('url'),
spawn = require('child_process').spawn,
ws    = require('./ws.js');

var availableComands = ['ls', 'ps', 'uptime', 'tail', 'cat'];
ws.createServer(function(websocket) {
    websocket.on('connect', function(resource) {
        var parsedUrl = url.parse(resource, true);
        var rawCmd = parsedUrl.query.cmd;
        var cmd = rawCmd.split(" ");
        if (cmd[0] == 'help') {
            websocket.write("Available comands: \n");
            for (i=0;i<availableComands.length;i++) {
                websocket.write(availableComands[i]);
                if (i< availableComands.length - 1) {
                    websocket.write(", ");
                }
            }
            websocket.write("\n");

            websocket.end();
        } else if (availableComands.indexOf(cmd[0]) >= 0) {
            if (cmd.length > 1) {
                options = cmd.slice(1);
            } else {
                options = [];
            }
            
            try {
                var process = spawn(cmd[0], options);
            } catch(err) {
                console.log(err);
                websocket.write("ERROR");
            }

            websocket.on('end', function() {
                process.kill();
            });

            process.stdout.on('data', function(data) {
                websocket.write(data);
            });

            process.stdout.on('end', function() {
                websocket.end();
            });
        } else {
             websocket.write("Comand not available. Type help for available comands\n");
             websocket.end();
        }
    });
  
}).listen(8880, '127.0.0.1');

The web client is similar than the example of my previous post

var timeout = 5000;
var wsServer = '127.0.0.1:8880';

var ws;


function cleanString(string) {
    return string.replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;");
}


function pad(n) {
    return ("0" + n).slice(-2);
}

var cmdHistory = [];
function send(msg) {
    if (msg == 'clear') {
        $('#log').html('');
        return;
    }
    try {
        ws = new WebSocket('ws://' + wsServer + '?cmd=' + msg);
        $('#toolbar').css('background', '#933');
        $('#socketStatus').html("working ... [<a href='#' onClick='quit()'>X</a>]");
        cmdHistory.push(msg);
        $('#log').append("<div class='cmd'>" + msg + "</div>");
        console.log("startWs:");
    } catch (err) {
        console.log(err);
        setTimeout(startWs, timeout);
    }

    ws.onmessage = function(event) {
        $('#log').append(util.toStaticHTML(event.data));
        window.scrollBy(0, 100000000000000000);
    };

    ws.onclose = function(){
        //console.log("ws.onclose");
        $('#toolbar').css('background', '#65A33F');
        $('#socketStatus').html('Type your comand:');
    }
}

function quit() {
    ws.close();
    window.scrollBy(0, 100000000000000000);
}
util = {
  urlRE: /https?:\/\/([-\w\.]+)+(:\d+)?(\/([^\s]*(\?\S+)?)?)?/g, 

  //  html sanitizer 
  toStaticHTML: function(inputHtml) {
    inputHtml = inputHtml.toString();
    return inputHtml.replace(/&/g, "&amp;")
                    .replace(/</g, "&lt;")
                    .replace("/n", "<br/>")
                    .replace(/>/g, "&gt;");
  }, 

  //pads n with zeros on the left,
  //digits is minimum length of output
  //zeroPad(3, 5); returns "005"
  //zeroPad(2, 500); returns "500"
  zeroPad: function (digits, n) {
    n = n.toString();
    while (n.length < digits) 
      n = '0' + n;
    return n;
  },

  //it is almost 8 o'clock PM here
  //timeString(new Date); returns "19:49"
  timeString: function (date) {
    var minutes = date.getMinutes().toString();
    var hours = date.getHours().toString();
    return this.zeroPad(2, hours) + ":" + this.zeroPad(2, minutes);
  },

  //does the argument only contain whitespace?
  isBlank: function(text) {
    var blank = /^\s*$/;
    return (text.match(blank) !== null);
  }
};
$(document).ready(function() {
  //submit new messages when the user hits enter if the message isnt blank
  $("#entry").keypress(function (e) {
    console.log(e);
    if (e.keyCode != 13 /* Return */) return;
    var msg = $("#entry").attr("value").replace("\n", "");
    if (!util.isBlank(msg)) send(msg);
    $("#entry").attr("value", ""); // clear the entry field.
  });
});

And that’s all. In fact we don’t need any line of PHP to perform this web console. Last year I tried to do something similar with PHP but it was a big mess. With node those kind of jobs are trivial. I don’t know if node.js is the future or is just another hype, but it’s easy. And cool. Really cool.

You can see the full code at Github here. Anyway you must take care if you run this application on your host. You are letting user to execute raw unix commands. A bit of security layer would be necessary.

Real time notifications with PHP

Real time communications are cool, isn’t it? Something impossible to do five years ago now (or almost impossible) is already available. Nowadays we have two possible solutions. WebSockets and Comet. WebSockets are probably the best solution but they’ve got two mayor problems:

  • Not all browsers support them.
  • Not all proxy servers allows the communications with websokets.

Because of that I prefer to use comet (at least now). It’s not as good as websockets but pretty straightforward ant it works (even on IE). Now I’m going to explain a little script that I’ve got to perform a comet communications, made with PHP. Probably it’s not a good idea to use it in a high traffic site, but it works like a charm in a small intranet. If you want to use comet in a high traffic site maybe you need have a look to Tornado, twisted, node.js or other comet dedicated servers.

Normally when we are speaking about real-time communications, all the people are thinking about a chat application. I want to build a simpler application. A want to detect when someone clicks on a link. Because of that I will need a combination of HTML, PHP and JavaScript. Let’s start:

For the example I’ll use jquery library, so we need to include the library in our HTML file. It will be a blend of JavaScrip and PHP:

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Comet Test</title>
    </head>
    <body>
        <p><a class='customAlert' href="#">publish customAlert</a></p>
        <p><a class='customAlert2' href="#">publish customAlert2</a></p>
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js" type="text/javascript"></script>
        <script src="NovComet.js" type="text/javascript"></script>
        <script type="text/javascript">
NovComet.subscribe('customAlert', function(data){
    console.log('customAlert');
    //console.log(data);
}).subscribe('customAlert2', function(data){
    console.log('customAlert2');
    //console.log(data);
});

$(document).ready(function() {
    $("a.customAlert").click(function(event) {
        NovComet.publish('customAlert');
    });
    
    $("a.customAlert2").click(function(event) {
        NovComet.publish('customAlert2');
    });
    NovComet.run();
});
        </script>
    </body>
</html>

The client code:

//NovComet.js
NovComet = {
    sleepTime: 1000,
    _subscribed: {},
    _timeout: undefined,
    _baseurl: "comet.php",
    _args: '',
    _urlParam: 'subscribed',

    subscribe: function(id, callback) {
        NovComet._subscribed[id] = {
            cbk: callback,
            timestamp: NovComet._getCurrentTimestamp()
        };
        return NovComet;
    },

    _refresh: function() {
        NovComet._timeout = setTimeout(function() {
            NovComet.run()
        }, NovComet.sleepTime);
    },

    init: function(baseurl) {
        if (baseurl!=undefined) {
            NovComet._baseurl = baseurl;
        }
    },

    _getCurrentTimestamp: function() {
        return Math.round(new Date().getTime() / 1000);
    },

    run: function() {
        var cometCheckUrl = NovComet._baseurl + '?' + NovComet._args;
        for (var id in NovComet._subscribed) {
            var currentTimestamp = NovComet._subscribed[id]['timestamp'];

            cometCheckUrl += '&' + NovComet._urlParam+ '[' + id + ']=' +
               currentTimestamp;
        }
        cometCheckUrl += '&' + NovComet._getCurrentTimestamp();
        $.getJSON(cometCheckUrl, function(data){
            switch(data.s) {
                case 0: // sin cambios
                    NovComet._refresh();
                    break;
                case 1: // trigger
                    for (var id in data['k']) {
                        NovComet._subscribed[id]['timestamp'] = data['k'][id];
                        NovComet._subscribed[id].cbk(data.k);
                    }
                    NovComet._refresh();
                    break;
            }
        });

    },

    publish: function(id) {
        var cometPublishUrl = NovComet._baseurl + '?' + NovComet._args;
        cometPublishUrl += '&publish=' + id;
        $.getJSON(cometPublishUrl);
    }
};

The server-side PHP

// comet.php
include('NovComet.php');

$comet = new NovComet();
$publish = filter_input(INPUT_GET, 'publish', FILTER_SANITIZE_STRING);
if ($publish != '') {
    echo $comet->publish($publish);
} else {
    foreach (filter_var_array($_GET['subscribed'], FILTER_SANITIZE_NUMBER_INT) as $key => $value) {
        $comet->setVar($key, $value);
    }
    echo $comet->run();
}

and my comet library implementation:

// NovComet.php
class NovComet {
    const COMET_OK = 0;
    const COMET_CHANGED = 1;

    private $_tries;
    private $_var;
    private $_sleep;
    private $_ids = array();
    private $_callback = null;

    public function  __construct($tries = 20, $sleep = 2)
    {
        $this->_tries = $tries;
        $this->_sleep = $sleep;
    }

    public function setVar($key, $value)
    {
        $this->_vars[$key] = $value;
    }

    public function setTries($tries)
    {
        $this->_tries = $tries;
    }

    public function setSleepTime($sleep)
    {
        $this->_sleep = $sleep;
    }

    public function setCallbackCheck($callback)
    {
        $this->_callback = $callback;
    }

    const DEFAULT_COMET_PATH = "/dev/shm/%s.comet";

    public function run() {
        if (is_null($this->_callback)) {
            $defaultCometPAth = self::DEFAULT_COMET_PATH;
            $callback = function($id) use ($defaultCometPAth) {
                $cometFile = sprintf($defaultCometPAth, $id);
                return (is_file($cometFile)) ? filemtime($cometFile) : 0;
            };
        } else {
            $callback = $this->_callback;
        }

        for ($i = 0; $i < $this->_tries; $i++) {
            foreach ($this->_vars as $id => $timestamp) {
                if ((integer) $timestamp == 0) {
                    $timestamp = time();
                }
                $fileTimestamp = $callback($id);
                if ($fileTimestamp > $timestamp) {
                    $out[$id] = $fileTimestamp;
                }
                clearstatcache();
            }
            if (count($out) > 0) {
                return json_encode(array('s' => self::COMET_CHANGED, 'k' => $out));
            }
            sleep($this->_sleep);
        }
        return json_encode(array('s' => self::COMET_OK));
    }

    public function publish($id)
    {
        return json_encode(touch(sprintf(self::DEFAULT_COMET_PATH, $id)));
    }
}

As you can see in my example I’ve created a personal protocol for the communications between the client (js at browser), and the server (PHP). It’s a simple one. If you’re looking for a “standard” protocol maybe you need have a look to bayeux protocol from Dojo people.

Let me explain a little bit the usage of the script:

  • In the HTML page we start the listener (NovComet.subscribe).
  • We can subscribe to as many events we want (OK it depends on our resources)
  • When we subscribe to one event we pass a callback function to be triggered.
  • When we subscribe to the event, we pass the current timestamp to the server.
  • Client side script (js with jquery) will call to server-side script (PHP) with the timestamp and will wait until server finish.
  • Server side script will answer when even timestamp changes (someone has published the event)
  • Server side will no keep waiting forever. If nobody publish the event, server will answer after a pre-selected timeout
  • client side script will repeat the process again and again.

There’s something really important with this technique. Our server-side event check need to be as simpler as we can. We cannot execute a SQL query for example (our sysadmin will kill us if we do it). We need to bear in mind that this check will be performed again and again per user, because of that it must be as light as we can. In this example we are checking the last modification date of a file (filemtime). Another good solution is to use a memcached database and check a value.

For the test I’ve also created a publishing script (NovComet.publish). This is the simple part. We only call a server-side script that touch the event file (changing the last modification date), triggering the event.

Now I’m going to explain what we can see on the firebug console:

  1. The first iteration nothing happens. 200 OK Http code after the time-out set in the PHP script
  2. As we can see here the script returns a JSON with s=0 (nothing happens)
  3. Now we publish an event. Script returns a 200 OK but now the JSON is different. s=1 and the time-stamp of the event
  4. Our callback has been triggered
  5. And next iteration waiting

And that’s all. Simple and useful. But remember, you must take care if you are using this solution within a high traffic site. What do you think? Do you use lazy comet with PHP in production servers or would you rather another solution?

You can get the code at github here.

About me

Web architect and web developer from San Sebastián, Spain.

Education

Executive formation

  • IT Project Management at ArcelorMittal University (2010-2011)

Work Experience

  • ArcelorMittal. IT Department. Long Carbon Europe, Spain (from 2002 to now)
  • EON, Sistemas y Tecnologías de procesos, S.L. IT support and consulting. (from 2001 to 2002)
  • Sestra S.L. ERP and MES Developer. (2000)
  • Electrotécnica del Urumea S.A. Technical Office. Automation Control with PLCs. (1997)

For a detailed C.V contact with me at: gonzalo123(at)gmail.com

You also can find me in: GitHubTwitterGoogle+, and Linkedin.