Real time monitoring PHP applications with websockets and node.js


The inspection of the error logs is a common way to detect errors and bugs. We also can show errors on-screen within our developement server, or we even can use great tools like firePHP to show our PHP errors and warnings inside our firebug console. That’s cool, but we only can see our session errors/warnings. If we want to see another’s errors we need to inspect the error log. tail -f is our friend, but we need to surf against all the warnings of all sessions to see our desired ones. Because of that I want to build a tool to monitor my PHP applications in real-time. Let’s start:

What’s the idea? The idea is catch all PHP’s errors and warnings at run time and send them to a node.js HTTP server. This server will work similar than a chat server but our clients will only be able to read the server’s logs. Basically the applications have three parts: the node.js server, the web client (html5) and the server part (PHP). Let me explain a bit each part:

The node Server

Basically it has two parts: a http server to handle the PHP errors/warnings and a websocket server to manage the realtime communications with the browser. When I say that I’m using websockets that’s means the web client will only work with a browser with websocket support like chrome. Anyway it’s pretty straightforward swap from a websocket sever to a socket.io server to use it with every browser. But websockets seems to be the future, so I will use websockets in this example.

The http server:

http.createServer(function (req, res) {
    var remoteAdrress = req.socket.remoteAddress;
    if (allowedIP.indexOf(remoteAdrress) >= 0) {
        res.writeHead(200, {
            'Content-Type': 'text/plain'
        });
        res.end('Ok\n');
        try {
            var parsedUrl = url.parse(req.url, true);
            var type = parsedUrl.query.type;
            var logString = parsedUrl.query.logString;
            var ip = eval(parsedUrl.query.logString)[0];
            if (inspectingUrl == "" ||  inspectingUrl == ip) {
                clients.forEach(function(client) {
                    client.write(logString);
                });
            }
        } catch(err) {
            console.log("500 to " + remoteAdrress);
            res.writeHead(500, {
                'Content-Type': 'text/plain'
            });
            res.end('System Error\n');
        }
    } else {
        console.log("401 to " + remoteAdrress);
        res.writeHead(401, {
            'Content-Type': 'text/plain'
        });
        res.end('Not Authorized\n');
    }
}).listen(httpConf.port, httpConf.host);

and the web socket server:

var inspectingUrl = undefined;

ws.createServer(function(websocket) {
    websocket.on('connect', function(resource) {
        var parsedUrl = url.parse(resource, true);
        inspectingUrl = parsedUrl.query.ip;
        clients.push(websocket);
    });

    websocket.on('close', function() {
        var pos = clients.indexOf(websocket);
        if (pos >= 0) {
            clients.splice(pos, 1);
        }
    });

}).listen(wsConf.port, wsConf.host);

If you want to know more about node.js and see more examples, have a look to the great site: http://nodetuts.com/. In this site Pedro Teixeira will show examples and node.js tutorials. In fact my node.js http + websoket server is a mix of two tutorials from this site.

The web client.

The web client is a simple websockets application. We will handle the websockets connection, reconnect if it dies and a bit more. I’s based on node.js chat demo

<?php $ip = filter_input(INPUT_GET, 'ip', FILTER_SANITIZE_STRING); ?>

        Real time <?= $ip ?> monitor
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.3/jquery.min.js"></script><script type="text/javascript">// <![CDATA[
            selectedIp = '<?= $ip ?>';

// ]]></script>
<script type="text/javascript" src="js.js"></script>
</pre>
<div id="toolbar">
<ul id="status">
	<li>Socket status: <span id="socketStatus">Conecting ...</span></li>
	<li>IP: <!--?= $ip == '' ? 'all' : $ip . " <a href='?ip='-->[all]" ?></li>
	<li>count: <span id="count">0</span></li>
</ul>
</div>
<pre>


And the javascript magic

var timeout = 5000;
var wsServer = '192.168.2.2:8880';
var unread = 0;
var focus = false;

var count = 0;
function updateCount() {
    count++;
    $("#count").text(count);
}

function cleanString(string) {
    return string.replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;");
}

function updateUptime () {
    var now = new Date();
    $("#uptime").text(now.toRelativeTime());
}

function updateTitle(){
    if (unread) {
        document.title = "(" + unread.toString() + ") Real time " + selectedIp + " monitor";
    } else {
        document.title = "Real time " + selectedIp + " monitor";
    }
}

function pad(n) {
    return ("0" + n).slice(-2);
}

function startWs(ip) {
    try {
        ws = new WebSocket("ws://" + wsServer + "?ip=" + ip);
        $('#toolbar').css('background', '#65A33F');
        $('#socketStatus').html('Connected to ' + wsServer);
        //console.log("startWs:" + ip);
        //listen for browser events so we know to update the document title
        $(window).bind("blur", function() {
            focus = false;
            updateTitle();
        });

        $(window).bind("focus", function() {
            focus = true;
            unread = 0;
            updateTitle();
        });
    } catch (err) {
        //console.log(err);
        setTimeout(startWs, timeout);
    }

    ws.onmessage = function(event) {
        unread++;
        updateTitle();
        var now = new Date();
        var hh = pad(now.getHours());
        var mm = pad(now.getMinutes());
        var ss = pad(now.getSeconds());

        var timeMark = '[' + hh + ':' + mm + ':' + ss + '] ';
        logString = eval(event.data);
        var host = logString[0];
        var line = "<table class='message'><tr><td width='1%' class='date'>" + timeMark + "</td><td width='1%' valign='top' class='host'><a href=?ip=" + host + ">" + host + "</a></td>";
        line += "<td class='msg-text' width='98%'>" + logString[1]; + "</td></tr>";
        if (logString[2]) {
            line += "<tr><td>&nbsp;</td><td colspan='3' class='msg-text'>" + logString[2] + "</td></tr>";
        }

        $('#log').append(line);
        updateCount();
        window.scrollBy(0, 100000000000000000);
    };

    ws.onclose = function(){
        //console.log("ws.onclose");
        $('#toolbar').css('background', '#933');
        $('#socketStatus').html('Disconected');
        setTimeout(function() {startWs(selectedIp)}, timeout);
    }
}

$(document).ready(function() {
    startWs(selectedIp);
});

The server part:

The server part will handle silently all PHP warnings and errors and it will send them to the node server. The idea is to place a minimal PHP line of code at the beginning of the application that we want to monitor. Imagine the following piece of PHP code

$a = $var[1];
$a = 1/0;
class Dummy
{
    static function err()
    {
        throw new Exception("error");
    }
}
Dummy1::err();

it will throw:
A notice: Undefined variable: var
A warning: Division by zero
An Uncaught exception ‘Exception’ with message ‘error’

So we will add our small library to catch those errors and send them to the node server

include('client/NodeLog.php');
NodeLog::init('192.168.2.2');

$a = $var[1];
$a = 1/0;
class Dummy
{
    static function err()
    {
        throw new Exception("error");
    }
}
Dummy1::err();

The script will work in the same way than the fist version but if we start our node.js server in a console:

$ node server.js
HTTP server started at 192.168.2.2::5672
Web Socket server started at 192.168.2.2::8880

We will see those errors/warnings in real-time when we start our browser

Here we can see a small screencast with the working application:

This is the server side library:

class NodeLog
{
    const NODE_DEF_HOST = '127.0.0.1';
    const NODE_DEF_PORT = 5672;

    private $_host;
    private $_port;

    /**
     * @param String $host
     * @param Integer $port
     * @return NodeLog
     */
    static function connect($host = null, $port = null)
    {
        return new self(is_null($host) ? self::$_defHost : $host, is_null($port) ? self::$_defPort : $port);
    }

    function __construct($host, $port)
    {
        $this->_host = $host;
        $this->_port = $port;
    }

    /**
     * @param String $log
     * @return Array array($status, $response)
     */
    public function log($log)
    {
        list($status, $response) = $this->send(json_encode($log));
        return array($status, $response);
    }

    private function send($log)
    {
        $url = "http://{$this->_host}:{$this->_port}?logString=" . urlencode($log);
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_NOBODY, true);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        $response = curl_exec($ch);
        $status   = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        return array($status, $response);
    }

    static function getip() {
        $realip = '0.0.0.0';
        if ($_SERVER) {
            if ( isset($_SERVER['HTTP_X_FORWARDED_FOR']) && $_SERVER['HTTP_X_FORWARDED_FOR'] ) {
                $realip = $_SERVER["HTTP_X_FORWARDED_FOR"];
            } elseif ( isset($_SERVER['HTTP_CLIENT_IP']) && $_SERVER["HTTP_CLIENT_IP"] ) {
                $realip = $_SERVER["HTTP_CLIENT_IP"];
            } else {
                $realip = $_SERVER["REMOTE_ADDR"];
            }
        } else {
            if ( getenv('HTTP_X_FORWARDED_FOR') ) {
                $realip = getenv('HTTP_X_FORWARDED_FOR');
            } elseif ( getenv('HTTP_CLIENT_IP') ) {
                $realip = getenv('HTTP_CLIENT_IP');
            } else {
                $realip = getenv('REMOTE_ADDR');
            }
        }
        return $realip;
    }

    public static function getErrorName($err)
    {
        $errors = array(
            E_ERROR             => 'ERROR',
            E_RECOVERABLE_ERROR => 'RECOVERABLE_ERROR',
            E_WARNING           => 'WARNING',
            E_PARSE             => 'PARSE',
            E_NOTICE            => 'NOTICE',
            E_STRICT            => 'STRICT',
            E_DEPRECATED        => 'DEPRECATED',
            E_CORE_ERROR        => 'CORE_ERROR',
            E_CORE_WARNING      => 'CORE_WARNING',
            E_COMPILE_ERROR     => 'COMPILE_ERROR',
            E_COMPILE_WARNING   => 'COMPILE_WARNING',
            E_USER_ERROR        => 'USER_ERROR',
            E_USER_WARNING      => 'USER_WARNING',
            E_USER_NOTICE       => 'USER_NOTICE',
            E_USER_DEPRECATED   => 'USER_DEPRECATED',
        );
        return $errors[$err];
    }

    private static function set_error_handler($nodeHost, $nodePort)
    {
        set_error_handler(function ($errno, $errstr, $errfile, $errline) use($nodeHost, $nodePort) {
            $err = NodeLog::getErrorName($errno);
            /*
            if (!(error_reporting() & $errno)) {
                // This error code is not included in error_reporting
                return;
            }
            */
            $log = array(
                NodeLog::getip(),
                "<strong class="{$err}">{$err}</strong> {$errfile}:{$errline}",
                nl2br($errstr)
            );
            NodeLog::connect($nodeHost, $nodePort)->log($log);
            return false;
        });
    }

    private static function register_exceptionHandler($nodeHost, $nodePort)
    {
        set_exception_handler(function($exception) use($nodeHost, $nodePort) {
            $exceptionName = get_class($exception);
            $message = $exception->getMessage();
            $file = $exception->getFile();
            $line = $exception->getLine();
            $trace = $exception->getTraceAsString();

            $msg = count($trace) > 0 ? "Stack trace:\n{$trace}" : null;
            $log = array(
                NodeLog::getip(),
                nl2br("<strong class="ERROR">Uncaught exception '{$exceptionName}'</strong> with message '{$message}' in {$file}:{$line}"),
                nl2br($msg)
            );
            NodeLog::connect($nodeHost, $nodePort)->log($log);
            return false;
        });
    }

    private static function register_shutdown_function($nodeHost, $nodePort)
    {
        register_shutdown_function(function() use($nodeHost, $nodePort) {
            $error = error_get_last();

            if ($error['type'] == E_ERROR) {
                $err = NodeLog::getErrorName($error['type']);
                $log = array(
                    NodeLog::getip(),
                    "<strong class="{$err}">{$err}</strong> {$error['file']}:{$error['line']}",
                    nl2br($error['message'])
                );
                NodeLog::connect($nodeHost, $nodePort)->log($log);
            }
            echo NodeLog::connect($nodeHost, $nodePort)->end();
        });
    }

    private static $_defHost = self::NODE_DEF_HOST;
    private static $_defPort = self::NODE_DEF_PORT;

    /**
     * @param String $host
     * @param Integer $port
     * @return NodeLog
     */
    public static function init($host = self::NODE_DEF_HOST, $port = self::NODE_DEF_PORT)
    {
        self::$_defHost = $host;
        self::$_defPort = $port;

        self::register_exceptionHandler($host, $port);
        self::set_error_handler($host, $port);
        self::register_shutdown_function($host, $port);

        $node = self::connect($host, $port);
        $node->start();
        return $node;
    }

    private static $time;
    private static $mem;

    public function start()
    {
        self::$time = microtime(TRUE);
        self::$mem = memory_get_usage();
        $log = array(NodeLog::getip(), "<strong class="OK">Start</strong> >>>> {$_SERVER['REQUEST_URI']}");
        $this->log($log);
    }

    public function end()
    {
        $mem = (memory_get_usage() - self::$mem) / (1024 * 1024);
        $time = microtime(TRUE) - self::$time;
        $log = array(NodeLog::getip(), "<strong class="OK">End</strong> <<<< mem: {$mem} time {$time}");         $this->log($log);
    }
}

And of course the full code on gitHub: RealTimeMonitor

About these ads

About Gonzalo Ayuso

Web Architect specialized in Open Source technologies. PHP, Python, JQuery, Dojo, PostgreSQL, CouchDB and node.js but always learning.

Posted on May 9, 2011, in HTML5, node.js, php, Technology, Websockets and tagged , , , , . Bookmark the permalink. 38 Comments.

  1. Wow, this is a nice idea and looks to work well too, there is so much now people are using node.js to do, I need to move it up my list of things to learn asap!

    • Gonzalo Ayuso

      I’m not sure if node is another hype, or if it’s the future. But it’s cool and easy. Really dificult things to develop with pure PHP become trivial with node.js. We need to change our mind a little bit to javascript and its functional programming but it’s not as different as Earlang.

  2. Great job! This would be very useful. As someone who wants to do as much as possible in the browser, something like this seems pretty cool. Nice color scheme on the log output too :)

  3. Gonzalo Ayuso

    txs. If you want more in the browser, node is your friend. It’s only js and really powerfull (few lines of code and you can do amazing things). JS is something familiar for all web developers, but we must remember this great post: http://goo.gl/LSEYV, Js is not as easy as we think. We need to study something new (but new things are cool ;)).

    btw the colour scheme is just a copy from the nodejs.org chat demo, but txs anyway, ;)

  4. Looks like very nice. And it temps to do something with PHP and node.js
    Well done! Cheer

    Cany**on

  5. Gonzalo Ayuso

    Many txs.

  6. That’s really cool post, thanks!

    However, I have one question, is there particular reason you’ve used HTTP as a protocol for submitting the logs? TCP would make more sense.

    • Gonzalo Ayuso

      Really good annotation. I will tell you the true: When I started with the library, I wanted a TCP server. But I didn’t worked correctly. Because of that I swapped to a HTTP server. At the end I realized TCP server didn’t work because of my local node.js instalation. TCP server seems to be a better solution (or even UDP one). The change is trivial, but all worked correctly (lazy mode on ;) )

      • Ah I see, thanks for explaining, I’m still learning nodejs. On small scale HTTP is fine.

        At the moment I’m using Shanty-Mongo and MongoDB to store all logs for my project (up to 40 writes per page request during extensive logging), and was thinking about putting logging behind nodejs. You are right, UDP is probably the best option for logging.

  7. Gonzalo your blog is great..!!!
    I have ubuntu installed in my laptop and i want to install sqlrelay to avoid load data in BD.Do you any suggestion of using slqrelay?

    • Gonzalo Ayuso

      I don’t understand what you want to say with “avoid load data in BD”.

      There’s something I dont like of sqlrelay. I’m not sure if it’s an alive project. The Database driver is an impotant part of the application and I don’t like to use very exotic things here (even mine ones). I normally use PDO (+ pgpool2 at the database server). Two years ago I test sqlrelay within my applications and the performance was similar than PDO without connection pooling. A bit better in some cases but I preffer the “standard”.

  8. Hi There,

    thanks for your Scripts, i have tried to set it up on my servers and tried twice on different machines. But on both i just see an error, everytime a php error occoures, logged to the terminal with this message: “This type of response MUST NOT have a body. Ignoring data passed to end().”

    I dont know what to change to get it working, it would be great to use the monitoring tool.

    Thx for Help

    • Ensure you are using the stable version of node.js. I’m not sure but I remember problems like that when I started to play with node. If you have the stable version of node try to run the http and tcp server examples (the first two examples of nodejs project page). They must work.

  9. Hi,
    First, big THANKS for this tutorial.
    Second, I got a problem, the connection is on, Socket status: Connected to 192.168.2.2:888, but when I’ve run the test1.php and test2.php I got the following error:
    Call to undefined function curl_init() in NodLog.php line 41.
    I’ve looked for a solution, I’ve found one, it was to uncomment this:
    ;extension=php_curl.dll on php.ini page, and then, restart the server (WAMP in my case).
    After doing so, I got another problem, the page loads very slowly and ends with these errors:
    Fatal error: Maximum execution time of 30 seconds exceeded in C:\wamp\www\gonzalo123-RealTimeMonitor-3e10c56\client\NodeLog.php on line 44
    and:

    • Fatal error: Maximum execution time of 30 seconds exceeded in C:\wamp\www\gonzalo123-RealTimeMonitor-3e10c56\client\NodeLog.php on line 137

    • Awaiting your help Gonzalo,
      I’m getting this err when I launch the server.js from command prompt:
      How to fix this: The “sys” module is now called “util”. It should have a similar interface.
      I’ve Googled the issue and the solution was to change every instance of
      var sys = require(‘sys’),
      to
      var sys = require (‘util’),
      But I’m still getting the same error message.
      Here is a screenshot: http://www.4shared.com/photo/vGlPfrJa/nodeErr.html
      Any help will be greatly appreciated.

      • It’s looks like node libraries has been changed and script is out of date. I need to adapt it to the new version. Added to my (huge ;) ) todo list

  10. Can’t get this working :( I want to monitor my remote app, so I run server.js on my server with IP set to server’s IP. I run client from my laptop with IP set also to server’s IP. But connection is aborted. After client try to connect, WS disconnects.

    This is probbably not firewall issue, because I can connect on IP:PORT from telnet if server.js is running.

    Any idea?

  11. Hi Gonzalo,

    Suppose i have a login system and i kept the user session in the server, now so i want to
    retrieve the session(the logged in user information) form the server so is it possible and if yes can you tell me the way to do in php.

    reply me by my mail id .

    • It’s possible (all is possible), but it’s out of the scope of this post. I don’t really know why you need the session in the client. session is a way to maintain state between requests in the server, but in the client you don’t have this problem (you can use a simple js variable or even localstorage)

      And sorry, I don’t answer post comments by email.

  12. hello, I am very curios about this real time monitoring for PHP logs, how can I configure this to work on windows? please help

    • I’m not sure if node.js works properly on windows. AFAIK it works, but I’m not windows user, so I cannot help you. Anyway it should work.

  13. Hallo, recently I got a interest in sockets but I’m still doubting to use nodejs or native PHP sockets. Wouldn’t it make more sense to use default PHP instead of nodejs? As far I know is nodejs limited and barely has support for a good MYSQL connection, isn’t PHP more optimized for this?

    I know that compared to apache is slower than NodeJS, but for what kind of applications is it better to use NodeJS?

    Sorry, I do not have the idea figured out yet, of nodejs advantages over PHP Socket, perhaps the speed, but I will be using NGinx or something familiar.

    • Websockets with Rachet (PHP) works fine. I started to play with socket.io (node.js) time ago and now I’m very confortable with it. I haven’t done any in-deep benchmark to choose one or another, indeed. Socket.io for example has a very fail-over system to work with old browsers (it works even with IE6), but Rachet works fine with new ones.

      As you said to perform traditional SQL operations is a bit nightmare with node. Mainly because to the functional nature of js (OK js it’s not fully functional programming, but it’s very close). I don’t have this problem because I always rely on backend server (PHP for me) and I only use node.js to the strictly real-time operations. My node.js servers for websockets are just a proxy and all the logic are bypass to the backend.

      Anyway if you choose PHP or node.js for websockets you must take into account that you will need to handle with two different servers. That’s means share authorization layer and things like that.

      • Thank you, I will look into it. I recently got interested, So I’m basically exploring the possibilities.

        I think I will stick for a little while with PHP, Since it will be a bit easier to get used. Afterwards I think it will be better to make a switch to NodeJS. One of the reasons I got interested in NodeJS is because PHP sockets does not have multi-threading, it’s rather unfortunate for that :(

        I’m still a bit wondering about the Authication you are talking about, is there any way to access the session of PHP?

        Or can we read the header for session information and use ‘session_start(newsession);’ to keep changing?

      • Also A question I wanted to ask. At the moment I’m thinking about using it like this:

        //php
        $bytes = @socket_recv($socket, $buffer, 2048, 0);
        $action = $this->unmask($buffer);
        $data = $this->unmask(socket_read($socket, 2048, 0));

        //js
        socket.send(text); // action
        socket.send(“extra data”); // Extra data

        This is just a plain example, how I’m thinking about using this, Do you think this could cause a problem?

        First one is to get a action for the method, the second wil be used for data that will be send to a method.
        ———–
        I was thinking that It might cause this problem:
        With another socket read, it can happen that it will receive only one instead of the suspected two. The problem might be that It will wait for the second data to arrive and halt everything.

      • It’s look like you are blocking socket server. I strongly recommend not use raw sockets to handle websockets. Use a PHP library Rachet (http://socketo.me/) or node.js one (http://socket.io/). Raw sockets are good to understand in deep the low level process (that’s perfect), but if you want to use it in production you will need to face to a lot of problems solved within library. PHP library use libevent to handle requests. It isn’t multithread (but node.js it isn’t too). It’s possible to run multithread with PHP (pthreads for example), but it’s a bit nighmare (as allways in multithread world)

        I’ve written a bit about websockets and security:

        http://gonzalo123.com/2014/08/25/playing-with-websockets-angularjs-and-socket-io/

        http://gonzalo123.com/2013/12/24/integrating-websockets-with-php-applications-silex-and-socket-io-playing-together/

        http://gonzalo123.com/2011/05/09/real-time-monitoring-php-applications-with-websockets-and-node-js/

        Basically I use the approach explained here:

        http://socketo.me/docs/push

        In the beginning I tried to share session database between PHP server and websocket server. I wrote an experiment

        http://gonzalo123.com/2011/07/25/using-node-js-to-store-php-sessions/

        But it’s a very exotic idea.

  1. Pingback: Real time monitoring PHP applications with websockets and node.js … | Internet blog

  2. Pingback: Gonzalo Ayuso’s Blog: Real time monitoring PHP applications with websockets and node.js | Scripting4You Blog

  3. Pingback: 5/10/2011 Blogs Update « Go Code

  4. Pingback: Web console with node.js « Gonzalo Ayuso | Web Architect

  5. Pingback: Real time notifications (part II). Now with node.js and socket.io « Gonzalo Ayuso | Web Architect

  6. Pingback: Display errors on screen even with display errors = off in PHP « Gonzalo Ayuso | Web Architect

  7. Pingback: How to send the output of Symfony’s process Component to a node.js server in Real Time with Socket.io « Gonzalo Ayuso | Web Architect

  8. Pingback: Playing with websockets, angularjs and socket.io | Gonzalo Ayuso | Web Architect

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,003 other followers

%d bloggers like this: