Blog Archives

Playing with websockets, angularjs and socket.io

I’m a big fan of websockets. I’ve got various post about them (here, here). Last months I’m working with angularjs projects and because of that I wanna play a little bit with websockets (with socket.io) and angularjs.

I want to build one angular service.

angular.module('io.service', []).
    factory('io', function ($http) {
        var socket,
            apiServer,
            ioEvent,
            watches = {};

        return {
            init: function (conf) {
                apiServer = conf.apiServer;
                ioEvent = conf.ioEvent;

                socket = io.connect(conf.ioServer);
                socket.on(ioEvent, function (data) {
                    return watches.hasOwnProperty(data.item) ? watches[data.item](data) : null;
                });
            },

            emit: function (arguments) {
                return $http.get(apiServer + '/request', {params: arguments});
            },

            watch: function (item, closure) {
                watches[item] = closure;
            },

            unWatch: function (item) {
                delete watches[item];
            }
        };
    });

And now we can build the application

angular.module('myApp', ['io.service']).

    run(function (io) {
        io.init({
            ioServer: 'http://localhost:3000',
            apiServer: 'http://localhost:8080/api',
            ioEvent: 'io.response'
        });
    }).

    controller('MainController', function ($scope, io) {
        $scope.$watch('question', function (newValue, oldValue) {
            if (newValue != oldValue) {
                io.emit({item: 'question', newValue: newValue, oldValue: oldValue});
            }
        });

        io.watch('answer', function (data) {
            $scope.answer = data.value;
            $scope.$apply();
        });
    });

And this’s the html

<!doctype html>
<html>

<head>
    <title>ws experiment</title>
</head>

<body ng-app="myApp">

<div ng-controller="MainController">

    <input type="text" ng-model="question">
    <hr>
    <h1>Hello {{answer}}!</h1>
</div>

<script src="assets/angular/angular.min.js"></script>
<script src="//localhost:3000/socket.io/socket.io.js"></script>

<script src="js/io.js"></script>
<script src="js/app.js"></script>

</body>
</html>

The idea of the application is to watch one model’s variable (‘question’ in this example) and each time it changes we will send the value to the websocket server and we’ll so something (we will convert the string to upper case in our example)

As you can read in one of my previous post I don’t like to send messages from the web browser to the websocket server directly (due do to authentication issues commented here). I prefer to use one server (a Silex server in this example)

include __DIR__ . '/../../vendor/autoload.php';

use Silex\Application;
use Symfony\Component\HttpFoundation\Request;

$app = new Application(['debug' => true]);
$app->register(new G\Io\EmitterServiceProvider());

$app->get('/request', function (Application $app, Request $request) {

    $params = [
        'item'     => $request->get('item'),
        'newValue' => strtoupper($request->get('newValue')),
        'oldValue' => $request->get('oldValue'),
    ];

    try {
        $app['io.emit']($params);
        $params['status'] = true;
    } catch (\Exception $e) {
        $params['status'] = false;
    }

    return $app->json($params);
});

$app->run();

You can see the code within my github account.

Debugging android cordova/phonegap apps with Chrome

Maybe this post can be obvious but I’ve spoken about it with various developers who don’t know it. It really improves the developing process of cordova/phonegap apps with android at least for me.

With android we can see the log with “adb logcat” but it’s a nightmare. Huge amount of information about our app and also about the operating system. If we’re grep ninjas we can handle it, but as well as I’m not a ninja I prefer another solution. Do you know “chrome://inspect/”? If not, have a look as soon as possible to this tool. We can see the browser’s console of our android in our desktop browser. We only need to enable “usb remote debugger” within our android device and plug with a USB cable. Chrome will detect the remote browser and we can see the console in the same way than we see it when we use Chrome locally.

But we’re speaking about cordova/phonegap apps here so, what we need to do to use chrome://inspect with our hybrid apps? The answer is simple: we don’t need to do anything. Cordova applications is nothing than a Webkit browser inside a native app. Chrome es Webkit too so chrome://inspect will detect our remote device app and we will open console.

Inspect_with_Chrome_Developer_Tools_and_bad_religion-the_gray_race

This small trick in addition to the last post really marks a before and an after at least in my developing process.

If our app crashes in the device we only need to see the console’s log within our browser and see what happens. We also can add functionality, change variables, and override functions in the same way than we do it with our local browser.

Upgrading Cordova-Android apps outside Google Play Store with angularjs

Recent months I’ve working with enterprise mobile applications. This apps are’t distributed using any marketplace, so I need to handle the distributions process. With Android you can compile your apps, create your APK files and distribute them. You can send the files by email, use a download link, send the file with bluetooth, or whatever. With iOS is a bit different. You need to purchase one Enterprise license, compile the app and distribute your IPA files using Apple’s standards.

OK, but this post is not about how to distribute apps outside the markets. This post is about one big problem that appears when we need to upgrade our apps. How do the user knows that there’s a new version of the application and he needs to upgrade? When we work inside Google Play Store we don’t need to worry about it, but if we distribute our apps manually we need do something. We can send push notifications or email to the user to inform about the new version. Let me show you how I’m doing it.

My problem isn’t only to let know to the user about a new version. Sometimes I also need to ensure that the user runs the last version of the app. Imagine a critical bug (solved in the last release) but the user don’t upgrade.

First we need to create a static html page where the user can download the APK file. Imagine that this is the url where the user can download the last version of the app:


http://192.168.1.1:8888/app.apk

We can check the version of the app against the server each time the user opens the application, but this check means network communication and it’s slow. We need to reduce the communication between client and server to the smallest expression and only when it’s strictly necessary. Check the version each time can be good in a desktop application, but it reduces the user experience with mobile apps. My approach is slightly different. Normally we use token based authentication within mobile apps. That’s means we need to send our token with all request. If we send the token, we also can send the version.

In a angular app we can define the version and the path of our apk using a key-value store.

.value('config', {
        version: 4,
        androidAPK: "http://192.168.1.1:8888/app.apk"
    })

Now we need to add version parameter to each request (we can easily create a custom http service to append this parameter to each request automatically, indeed)

$http.get('http://192.168.1.1:8888/api/doSomething', {params: {_version: config.version}})
    .success(function (data) {
        alert("OK");
    })
    .error(function (err, status) {
        switch (status) {
            case 410:
                $state.go('upgrade');
                break;
        }
    });

We can create a simple backend to take care of the version and throws an HTTP exception (one 410 HTTP error for example) if versions doesn’t match. Here you can see a simple Silex example:

<?php

include __DIR__ . "/../vendor/autoload.php";

use Silex\Application;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\HttpException;

$app = new Application([
    'debug'   => true,
    'version' => 4,
]);

$app->after(function (Request $request, Response $response) {
    $response->headers->set('Access-Control-Allow-Origin', '*');
});

$app->get('/api/doSomething', function (Request $request, Application $app) {
    if ($request->get('_version') != $app['version']) {
        throw new HttpException(410, "Wrong version");
    } else {
        return $app->json('hello');
    }
});

$app->run();

As you can see we need to take care about CORS

With this simple example we can realize if user has a wrong version within each server request. If version don’t match we can, for example redirect to an specific route to inform that the user needs to upgrade the app and provide a link to perform the action.

With Android we cannot create a link to APK file. It doesn’t work. We need to download the APK (using FileTransfer plugin) and open the file using webintent plugin.

The code is very simple:

var fileTransfer = new FileTransfer();
fileTransfer.download(encodeURI(androidUrl), 
    "cdvfile://localhost/temporary/app.apk",
    function (entry) {
        window.plugins.webintent.startActivity({
            action: window.plugins.webintent.ACTION_VIEW,
            url: entry.toURL(),
            type: 'application/vnd.android.package-archive'
        }, function () {
        }, function () {
            alert('Failed to open URL via Android Intent.');
            console.log("Failed to open URL via Android Intent. URL: " + entry.fullPath);
        });
    }, function (error) {
        console.log("download error source " + error.source);
        console.log("download error target " + error.target);
        console.log("upload error code" + error.code);
    }, true);

And basically that’s all. When user self-upgrade the app it closes automatically and he needs to open it again, but now with the correct version.

Testing Phonegap/Cordova applications fast as hell in the device (with ionic framework)

Normally when we work with Phonegap/Cordova applications we work in two phases. First we develop the application locally using our browser. That’s “fast” phase. We change something within our code, then we reload our browser and we see the outcome. It isn’t different from a “traditional” web developing process. Normally I use the ionic framework. Ionic is great and it also provides us a good tool to run a local server. We just type:

ionic serve

And ionic starts a local server on port 8100 with our Cordova application, ready to test with the browser (it also opens the browser). That’s not the cool part. Ionic also starts a live reload server at http://0.0.0.0:35729 and adds the following snippet at the end of our index.html

<script type="text/javascript">//<![CDATA[
document.write('<script src="' + (location.protocol || 'http:') + '//' + (location.hostname || 'localhost') + ':35729/livereload.js?snipver=1" type="text/javascript"><\/script>')
//]]></script>

With this snippet our application will be reloaded when we add/remove something in our file tree (it runs a filesystem watcher in background).

But as I said before it’s the “fast” phase and sooner or later we will need to run the application in the real device. OK we’ve got emulators, but they are horrible. Android emulator is incredible slow. IOS one is faster but we need to redeploy the application again and again with each change. For example when we correct a silly bug we need to run the following command to see the application running on the device:

cordova run android --device

And it takes time (around 10 seconds). We’ve gone from the “fast” phase to the “slooooow” one. That means that I tried to avoid this phase until no remedy.

If you don’t use plugins you can let this “slow” phase to the end, only to see the behaviour in the device and fix customizations, but ir we use plugins (camera plugin, push notifications or things like that) we really need to test on the real device. Those kind of things doesn’t work in the browser or even with the emulator.

This “slow” phase droves me crazy, so I started to think a little bit about it. One Cordova app has two parts. The native one (java code in android and objective-c in ios) and the html/js part. We need to tell to our Cordova application where is the initial index.html. We usually do it in config.xml

<content src="index.html" />

But we can change this initial file and use a remote one. That’s the way to create a “native” app from and existing web application.

<content src="http://gonzalo123.com" />

According to this we can start a local server in our host and use this local web server. Even in our LAN (if our android/ios device is the LAN of course)

<content src="http://192.168.1.1:8100/index.html" />

But, what happens with the plugins? Plugins needs cordova.js file and this file isn’t in www folder. This file is generated when we build the application to a specific platform

platforms/android/assets/www/cordova.js

So, what’s the idea. The idea is:

  • Run a local server with (inoic serve for example)
  • Enable the fs watcher to restart the application when we change one file in the filesystem (inonic serve do it by default)
  • Build the application and install it in the real device
  • Use our local server to serve static files instead of build again and again the application with each change.

With this approach we only need to deploy the application to the real device when we want to add/remove one plugin. If we change anything in the static files (html, js, css) our app will be reloaded automatically. The “slow” phase turns into a “fast” phase.

How can we do it? It’s easy. In this example I suppose that we’re using one android device. If we use on iPhone we only need to change “adroid” to “ios”.

First of all we need to prepare our index.html to enable auto-reload. “ionic serve” do it automatically but it thinks that we’re going to use it with your host browser. Not with the “real” device. We can change it manually adding to our index.html (this snippet suppose that your host is 192.168.1.1 if it’s a different one use your local IP address):

    <script type="text/javascript">//<![CDATA[
    document.write('<script src="http://192.168.1.1:35729/livereload.js?snipver=1" type="text/javascript"><\/script>')
    //]]></script>

Now we change our config.xml to use our local server instead of device’s files:

<!--<content src="index.html" />-->
<content src="http://192.168.1.105:8100/index.html" />

Now we need to deploy the application to our device:

cordova run android --device

Each time we add/remove one plugin we need to redeploy to the device. But we need to keep in mind that our device will use the cordova.js from our local server, and not from its filesystem. “cordova run android –device” will generate the file to the platform and deploy them to the real device, but as well as we’re going to use this file from our local server (in www), we need to create a set of symlinks in our www folder.

(I’ve got one setUp.sh file with this commands)

cd www
ln -s ../platforms/android/assets/www/cordova.js
ln -s ../platforms/android/assets/www/cordova_plugins.js
ln -s ../platforms/android/assets/www/plugins
cd ..

Now can start the application’s server in our host with:

ionic serve --nobrowser

notice that we’re using –nobrowser. We’re using this parameter to not to open our local browser. We’re going to use de device’s Cordova’s Webkit one, and also if we open our browser it will crash because cordova.js is present now and our local host isn’t a real device.

Each time we need to redeploy the application to the device (new plugin for example) we need to remember to quit the symlinks, and redeploy.

(I’ve got one tearDown.sh file with this commands)

rm www/cordova.js
rm www/cordova_plugins.js
rm -Rf www/plugins 

And that’s all. I now that this little hack may looks like something difficult but we need less than a minute to set up the environment and we will save thousand of seconds in the development process. I we work a little bit we can automate this process and turn it into a trivial opperation, but at least now I feel very comfortable.

Of course you need to remember to clean the project when you finish and use the device’s files. So we need to remove the auto-reload snippet in the index.html, remove symlinks and restore config.xml.

Setting up states from a json file in angularjs applications

Imagine a this simple angularjs application using angular-ui-router:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Example</title>
    <script src="bower_components/angular/angular.js"></script>
    <script src="bower_components/angular-ui-router/release/angular-ui-router.js"></script>
    <script src="js/app.js"></script>

</head>
<body ng-app="App" ng-controller="MainController">

<div ui-view></div>
</body>
</html>

angular.module('App', ['ui.router'])

    .config(function ($stateProvider, $urlRouterProvider, routerProvider) {
        $stateProvider
            .state('home', {
                url: '/home',
                templateUrl: 'templates/home.html'
            });

        $urlRouterProvider.otherwise('/home');
    })

    .controller('MainController', function ($scope, router) {
        $scope.reload = function() {
            router.setUpRoutes();
        };
    })
;

We’ve defined only one state called “home”. If we need more states we just add more within config() function. In this post we’re going to try to add more states from a json file instead of hardcode the states within the code.

Let’s create our json file with the states definitions:

{
    "xxx": {
        "url": "/xxx",
        "templateUrl": "templates/xxx.html"
    },

    "yyy": {
        "url": "/yyy",
        "templateUrl": "templates/yyy.html"
    },

    "zzz": {
        "url": "/zzz",
        "templateUrl": "templates/zzz.html"
    }
}

Now our application looks like this:

angular.module('App', ['ui.router', 'Routing'])

    .config(function ($stateProvider, $urlRouterProvider, routerProvider) {
        $stateProvider
            .state('home', {
                url: '/home',
                templateUrl: 'templates/home.html'
            });

        $urlRouterProvider.otherwise('/home');

        routerProvider.setCollectionUrl('js/routeCollection.json');
    })

    .controller('MainController', function ($scope, router) {
        $scope.reload = function() {
            router.setUpRoutes();
        };
    })
;

As we can see now we’re using ‘Routing’

angular.module('Routing', ['ui.router'])
    .provider('router', function ($stateProvider) {

        var urlCollection;

        this.$get = function ($http, $state) {
            return {
                setUpRoutes: function () {
                    $http.get(urlCollection).success(function (collection) {
                        for (var routeName in collection) {
                            if (!$state.get(routeName)) {
                                $stateProvider.state(routeName, collection[routeName]);
                            }
                        }
                    });
                }
            }
        };

        this.setCollectionUrl = function (url) {
            urlCollection = url;
        }
    })

    .run(function (router) {
        router.setUpRoutes();
    });

‘Routing’ provides us a provider called ‘router’ that fetch the json file and build the states.

That’s a proof of concept.
There’s a couple of problems (please tell me if you know how to solve them):

  • As far as we’re loading states from a http connection, angular application don’t have all the states when it starts, so we need to create at least the first state with the “old style”
  • We can reload states with the application running. We also can add new states, but we cannot modify the existing ones.

you can see the one example project within my github account.

Token based authentication with Silex and AngularJS

According to my last post today we’re going to create a AngularJS application that uses the Silex Backend that we create previously. The idea of this application is to use it within a Phonegap/Cordova application running in a mobile device.

The application will be show a login form if device haven’t a correct token.

Gonzalo_Login_Example_and_LoginServiceProvider_php_-_token_-____work_projects_token_

And whit a correct token:

Gonzalo_Login_Example

Nothing new under the sun, isn’t it?

Our front-end application will use AngularJS and Topcoat.

<!DOCTYPE html>
<html xmlns:ng="http://angularjs.org" lang="es" ng-app="G">
<head>
    <meta charset="utf-8"/>
    <meta name="format-detection" content="telephone=no"/>
    <!-- WARNING: for iOS 7, remove the width=device-width and height=device-height attributes. See https://issues.apache.org/jira/browse/CB-4323 -->
    <meta name="viewport"
          content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width, height=device-height, target-densitydpi=device-dpi"/>
    <link rel="stylesheet" type="text/css" href="/bower_components/topcoat/css/topcoat-mobile-light.min.css">
    <title>Gonzalo Login Example</title>
</head>
<body ng-controller="MainController">

<div ng-view class="main-content"></div>

<script src="/bower_components/angular/angular.min.js"></script>
<script src="/bower_components/angular-route/angular-route.min.js"></script>

<script src="js/app.js"></script>
<script src="js/services.js"></script>

</body>
</html>

And our AngularJS application:

'use strict';
var appControllers, G;
var host = 'http://localhost:8080'; // server API url

appControllers = angular.module('appControllers', []);
G = angular.module('G', ['ngRoute', 'appControllers']);

G.run(function (httpG) {
    httpG.setHost(host);
});

G.config(['$routeProvider', function ($routeProvider) {
    $routeProvider.
        when('/login', {templateUrl: 'partials/login.html', controller: 'LoginController'}).
        when('/home', {templateUrl: 'partials/home.html', controller: 'HomeController'});
}]);

appControllers.controller('HomeController', ['$scope', 'httpG', '$location', function ($scope, httpG, $location) {
    $scope.hello = function () {
        httpG.get('/api/info').success(function (data) {
            if (data.status) {
                alert("Hello " + data.info.name + " " + data.info.surname);
            }
        });
    };

    $scope.logOut = function () {
        alert("Good bye!");
        httpG.removeToken();
        $scope.isAuthenticated = false;
        $location.path('login');
    };
}]);

appControllers.controller('MainController', ['$scope', '$location', 'httpG', function ($scope, $location, httpG) {
    $scope.isAuthenticated = false;

    if (httpG.getToken()) {
        $scope.isAuthenticated = true;
        $location.path('home');
    } else {
        $location.path('login');
    }
}]);


appControllers.controller('LoginController', ['$scope', '$location', 'httpG', function ($scope, $location, httpG) {
    $scope.user = {};

    $scope.doLogIn = function () {
        httpG.get('/auth/validateCredentials', {user: $scope.user.username, pass: $scope.user.password}).success(function (data) {
            if (data.status) {
                httpG.setToken(data.info.token);
                $scope.isAuthenticated = true;
                $location.path('home');
            } else {
                alert("login error");
            }
        }).error(function (error) {
            alert("Login Error!");
        });
    };

    $scope.doLogOut = function () {
        httpG.removeToken();
    };
}]);

In this example I’m using angular-route to handle the application’s routes. Nowadays I’m swaping to angular-ui-router, but this example I’m still using “old-style” routes. We define two partials:

partial/home.html

<div class="topcoat-button-bar full" style="position: fixed; bottom: 0px;">
    <label class="topcoat-button-bar__item">
        <button class="topcoat-button full" ng-click="logOut()">
            <span class="">Logout</span>
        </button>
    </label>
    <label class="topcoat-button-bar__item">
        <button class="topcoat-button--cta full" ng-click="hello()">
            <span class="">Hello</span>
        </button>
    </label>
</div>

partial/login.html

<div class="topcoat-navigation-bar">
    <div class="topcoat-navigation-bar__item center full">
        <h1 class="topcoat-navigation-bar__title">Login</h1>
    </div>
</div>

<ul class="topcoat-list__container">
    <li class="topcoat-list__item center">
        <input ng-model="user.username" class="topcoat-text-input--large" type="text" name="user"
               placeholder="Username"/>
    </li>
    <li class="topcoat-list__item center">
        <input ng-model="user.password" class="topcoat-text-input--large" type="password" name="pass"
               placeholder="Password"/>
    </li>
</ul>

<div class="topcoat-button-bar full" style="position: fixed; bottom: 0px;">
    <label class="topcoat-button-bar__item">
        <button class="topcoat-button--cta full" ng-click="doLogIn()">
            <span class="">Login</span>
        </button>
    </label>
</div>

As we can see in the application we’re using a service to handle Http connections with the token information.

'use strict';

G.factory('httpG', ['$http', '$window', function ($http, $window) {
    var serviceToken, serviceHost, tokenKey;
    tokenKey = 'token';
    if (localStorage.getItem(tokenKey)) {
        serviceToken = $window.localStorage.getItem(tokenKey);
    }

    $http.defaults.headers.post["Content-Type"] = "application/x-www-form-urlencoded";

    return {
        setHost: function (host) {
            serviceHost = host;
        },

        setToken: function (token) {
            serviceToken = token;
            $window.localStorage.setItem(tokenKey, token);
        },

        getToken: function () {
            return serviceToken;
        },

        removeToken: function() {
            serviceToken = undefined;
            $window.localStorage.removeItem(tokenKey);
        },

        get: function (uri, params) {
            params = params || {};
            params['_token'] = serviceToken;
            return $http.get(serviceHost + uri, {params: params});
        },

        post: function (uri, params) {
            params = params || {};
            params['_token'] = serviceToken;

            return $http.post(serviceHost + uri, params);
        }
    };
}]);

And that’s all. You can see the full example in my github account.

Real time notifications with PHP

Real time communications are cool, isn’t it? Something impossible to do five years ago now (or almost impossible) is already available. Nowadays we have two possible solutions. WebSockets and Comet. WebSockets are probably the best solution but they’ve got two mayor problems:

  • Not all browsers support them.
  • Not all proxy servers allows the communications with websokets.

Because of that I prefer to use comet (at least now). It’s not as good as websockets but pretty straightforward ant it works (even on IE). Now I’m going to explain a little script that I’ve got to perform a comet communications, made with PHP. Probably it’s not a good idea to use it in a high traffic site, but it works like a charm in a small intranet. If you want to use comet in a high traffic site maybe you need have a look to Tornado, twisted, node.js or other comet dedicated servers.

Normally when we are speaking about real-time communications, all the people are thinking about a chat application. I want to build a simpler application. A want to detect when someone clicks on a link. Because of that I will need a combination of HTML, PHP and JavaScript. Let’s start:

For the example I’ll use jquery library, so we need to include the library in our HTML file. It will be a blend of JavaScrip and PHP:

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Comet Test</title>
    </head>
    <body>
        <p><a class='customAlert' href="#">publish customAlert</a></p>
        <p><a class='customAlert2' href="#">publish customAlert2</a></p>
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js" type="text/javascript"></script>
        <script src="NovComet.js" type="text/javascript"></script>
        <script type="text/javascript">
NovComet.subscribe('customAlert', function(data){
    console.log('customAlert');
    //console.log(data);
}).subscribe('customAlert2', function(data){
    console.log('customAlert2');
    //console.log(data);
});

$(document).ready(function() {
    $("a.customAlert").click(function(event) {
        NovComet.publish('customAlert');
    });
    
    $("a.customAlert2").click(function(event) {
        NovComet.publish('customAlert2');
    });
    NovComet.run();
});
        </script>
    </body>
</html>

The client code:

//NovComet.js
NovComet = {
    sleepTime: 1000,
    _subscribed: {},
    _timeout: undefined,
    _baseurl: "comet.php",
    _args: '',
    _urlParam: 'subscribed',

    subscribe: function(id, callback) {
        NovComet._subscribed[id] = {
            cbk: callback,
            timestamp: NovComet._getCurrentTimestamp()
        };
        return NovComet;
    },

    _refresh: function() {
        NovComet._timeout = setTimeout(function() {
            NovComet.run()
        }, NovComet.sleepTime);
    },

    init: function(baseurl) {
        if (baseurl!=undefined) {
            NovComet._baseurl = baseurl;
        }
    },

    _getCurrentTimestamp: function() {
        return Math.round(new Date().getTime() / 1000);
    },

    run: function() {
        var cometCheckUrl = NovComet._baseurl + '?' + NovComet._args;
        for (var id in NovComet._subscribed) {
            var currentTimestamp = NovComet._subscribed[id]['timestamp'];

            cometCheckUrl += '&' + NovComet._urlParam+ '[' + id + ']=' +
               currentTimestamp;
        }
        cometCheckUrl += '&' + NovComet._getCurrentTimestamp();
        $.getJSON(cometCheckUrl, function(data){
            switch(data.s) {
                case 0: // sin cambios
                    NovComet._refresh();
                    break;
                case 1: // trigger
                    for (var id in data['k']) {
                        NovComet._subscribed[id]['timestamp'] = data['k'][id];
                        NovComet._subscribed[id].cbk(data.k);
                    }
                    NovComet._refresh();
                    break;
            }
        });

    },

    publish: function(id) {
        var cometPublishUrl = NovComet._baseurl + '?' + NovComet._args;
        cometPublishUrl += '&publish=' + id;
        $.getJSON(cometPublishUrl);
    }
};

The server-side PHP

// comet.php
include('NovComet.php');

$comet = new NovComet();
$publish = filter_input(INPUT_GET, 'publish', FILTER_SANITIZE_STRING);
if ($publish != '') {
    echo $comet->publish($publish);
} else {
    foreach (filter_var_array($_GET['subscribed'], FILTER_SANITIZE_NUMBER_INT) as $key => $value) {
        $comet->setVar($key, $value);
    }
    echo $comet->run();
}

and my comet library implementation:

// NovComet.php
class NovComet {
    const COMET_OK = 0;
    const COMET_CHANGED = 1;

    private $_tries;
    private $_var;
    private $_sleep;
    private $_ids = array();
    private $_callback = null;

    public function  __construct($tries = 20, $sleep = 2)
    {
        $this->_tries = $tries;
        $this->_sleep = $sleep;
    }

    public function setVar($key, $value)
    {
        $this->_vars[$key] = $value;
    }

    public function setTries($tries)
    {
        $this->_tries = $tries;
    }

    public function setSleepTime($sleep)
    {
        $this->_sleep = $sleep;
    }

    public function setCallbackCheck($callback)
    {
        $this->_callback = $callback;
    }

    const DEFAULT_COMET_PATH = "/dev/shm/%s.comet";

    public function run() {
        if (is_null($this->_callback)) {
            $defaultCometPAth = self::DEFAULT_COMET_PATH;
            $callback = function($id) use ($defaultCometPAth) {
                $cometFile = sprintf($defaultCometPAth, $id);
                return (is_file($cometFile)) ? filemtime($cometFile) : 0;
            };
        } else {
            $callback = $this->_callback;
        }

        for ($i = 0; $i < $this->_tries; $i++) {
            foreach ($this->_vars as $id => $timestamp) {
                if ((integer) $timestamp == 0) {
                    $timestamp = time();
                }
                $fileTimestamp = $callback($id);
                if ($fileTimestamp > $timestamp) {
                    $out[$id] = $fileTimestamp;
                }
                clearstatcache();
            }
            if (count($out) > 0) {
                return json_encode(array('s' => self::COMET_CHANGED, 'k' => $out));
            }
            sleep($this->_sleep);
        }
        return json_encode(array('s' => self::COMET_OK));
    }

    public function publish($id)
    {
        return json_encode(touch(sprintf(self::DEFAULT_COMET_PATH, $id)));
    }
}

As you can see in my example I’ve created a personal protocol for the communications between the client (js at browser), and the server (PHP). It’s a simple one. If you’re looking for a “standard” protocol maybe you need have a look to bayeux protocol from Dojo people.

Let me explain a little bit the usage of the script:

  • In the HTML page we start the listener (NovComet.subscribe).
  • We can subscribe to as many events we want (OK it depends on our resources)
  • When we subscribe to one event we pass a callback function to be triggered.
  • When we subscribe to the event, we pass the current timestamp to the server.
  • Client side script (js with jquery) will call to server-side script (PHP) with the timestamp and will wait until server finish.
  • Server side script will answer when even timestamp changes (someone has published the event)
  • Server side will no keep waiting forever. If nobody publish the event, server will answer after a pre-selected timeout
  • client side script will repeat the process again and again.

There’s something really important with this technique. Our server-side event check need to be as simpler as we can. We cannot execute a SQL query for example (our sysadmin will kill us if we do it). We need to bear in mind that this check will be performed again and again per user, because of that it must be as light as we can. In this example we are checking the last modification date of a file (filemtime). Another good solution is to use a memcached database and check a value.

For the test I’ve also created a publishing script (NovComet.publish). This is the simple part. We only call a server-side script that touch the event file (changing the last modification date), triggering the event.

Now I’m going to explain what we can see on the firebug console:

  1. The first iteration nothing happens. 200 OK Http code after the time-out set in the PHP script
  2. As we can see here the script returns a JSON with s=0 (nothing happens)
  3. Now we publish an event. Script returns a 200 OK but now the JSON is different. s=1 and the time-stamp of the event
  4. Our callback has been triggered
  5. And next iteration waiting

And that’s all. Simple and useful. But remember, you must take care if you are using this solution within a high traffic site. What do you think? Do you use lazy comet with PHP in production servers or would you rather another solution?

You can get the code at github here.

Speed up page page load combining javascript files with PHP

One of the golden rules when we want a high performance web site is minimize the HTTP requests. Normally we have several JavaScript files within our projects. It’s a very good practice to combine all our JavaScript files into an only one file. We can do it manually. Not a hard work. We only need to copy and paste the source files into our single js file. There’s even tools to do it. You can have look to Yslow (if you don’t know it yet). That’s a good solution if your project is finished. But if your project is alive and you are changing it, it’s helpful to spare your JavaScript files between several files. It’s good to organize them (at least for me). So we need to choose between high performance and development comfort. Because of that I like to use the simple script I’m going to show now. Let’s start.

The idea is the following one. Normally I have all js files into a folder called js (original isn’t?). I also have a development server and a production one (really original again, isn’t it?). When I’m developing my application I like to have my js files separated and in a human readable way (I’m human), but in production I want to combine them and even minimized and gziped to improve the performance.

The script I have is a simple script that combines all the JavaScript files, minimizes and gzips.

//js.php
require 'jsmin.php';

function checkCanGzip(){
    if (array_key_exists('HTTP_ACCEPT_ENCODING', $_SERVER)) {
        if (strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip') !== false) return "gzip";
        if (strpos($_SERVER['HTTP_ACCEPT_ENCODING'], 'x-gzip') !== false) return "x-gzip";
    }
    return false;
}

function gzDocOut($contents, $level=6){
    $return = array();
    $return[] = "\x1f\x8b\x08\x00\x00\x00\x00\x00";
    $size = strlen($contents);
    $crc = crc32($contents);
    $contents = gzcompress($contents,$level);
    $contents = substr($contents, 0, strlen($contents) - 4);
    $return[] = $contents;
    $return[] = pack('V',$crc);
    $return[] = pack('V',$size);
    return implode(null, $return);
}

$ite = new RecursiveDirectoryIterator(dirname(__FILE__));
foreach(new RecursiveIteratorIterator($ite) as $file => $fileInfo) {
    $extension = strtolower(pathinfo($file, PATHINFO_EXTENSION));
    if ($extension == 'js') {
        $f = $fileInfo->openFile('r');
        $fdata = "";
        while ( ! $f->eof()) {
            $fdata .= $f->fgets();
        }
        $buffer[] = $fdata;
    }
}

$output = JSMin::minify(implode(";\n", $buffer));

header("Content-type: application/x-javascript; charset: UTF-8");
$forceGz    = filter_input(INPUT_GET, 'gz', FILTER_SANITIZE_STRING);
$forcePlain = filter_input(INPUT_GET, 'plain', FILTER_SANITIZE_STRING);

$encoding = checkCanGzip();
if ($forceGz) {
    header("Content-Encoding: {$encoding}");
    echo gzDocOut($output);
} elseif ($forcePlain) {
    echo $output;
} else {
    if ($encoding){
        header("Content-Encoding: {$encoding}");
        echo GzDocOut($output);
    } else {
        echo $output;
    }
}

As you can see the script checks recursively all the js files inside one folder, combine them and also use jsmin library for PHP to improve the download time in the browser.

It’s very easy now when we’re building our HTML file switch from one js file to another. Here you can see an example with Smarty template engine:

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title></title>
    </head>
    <body>
        Hello World
{if $dev}
        <script src="js1.js" type="text/javascript"></script>
        <script src="js2.js" type="text/javascript"></script>
        <script src="xxx/js1.js" type="text/javascript"></script>
{else}
        <script src="js.php" type="text/javascript"></script>
{/if}
    </body>
</html>

Yes. I know. There’s a problem with this solution. Maybe we’ve improved the client side performance reducing the number of HTTP requests but, what about our server side performance? We’ve changed from serving static js files to dinamic PHP file. Now our server’s CPU will work more. Another great hight performance golden rule is to place static files into a server dedicated to serve static files (without PHP support). Whit this golden rule we help to the browser to perform multiple downloads and also we reduce the use of CPU (static server will not instance any PHP session).

So a better solution is the offline generation of the static js file when we deploy the application. I do it with a simple curl at command line.

curl http://nov/js/js.php -o jsfull.minified.js

So the smarty template will change to

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title></title>
    </head>
    <body>
        Hello World
{if $dev}
        <script src="js1.js" type="text/javascript"></script>
        <script src="js2.js" type="text/javascript"></script>
        <script src="xxx/js1.js" type="text/javascript"></script>
{else}
        <script src="jsfull.minified.js" type="text/javascript"></script>
{/if}
    </body>
</html>

It’s also a good solution put a prefix in our JavaScript file and change it each time we build it (easy to automate) to ensure the cache renew.

<script src="jsfull.minified.20110216.js" type="text/javascript"></script>

And yes we can use the same trick with our css files.

Follow

Get every new post delivered to your Inbox.

Join 972 other followers