Monthly Archives: March 2011

Performance analysis fetching data with PDO and PHP.

Fetching data from databases is a common operation in our work as developers. There are many drivers (normally I use PDO), but the usage of all of them are similar and switch from one to another is not difficult (they almost share the same interface). In this post I will focus on fetching data. Basically we’ve got two functions: fetch and fetchAll. I’ve created two examples. One with fetch and another one with fetchAll:

// Example with fetch
error_reporting(-1);
$time = microtime(TRUE);
$mem = memory_get_usage();

$dbh = new PDO('pgsql:dbname=mydb;host=localhost', 'username', 'password');
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$stmt = $dbh->prepare('SELECT * FROM tableName limit 10000');
$stmt->execute();

$i=0;
while ($row = $stmt->fetch()) {
	$i++;
}
echo '
<h1>fetch()</h1>
';
echo '
<strong>{$i} </strong>

';
print_r(array('memory' => (memory_get_usage() - $mem) / (1024 * 1024), 'seconds' => microtime(TRUE) - $time));
// Example with fetchAll
error_reporting(-1);
$time = microtime(TRUE);
$mem = memory_get_usage();

$dbh = new PDO('pgsql:dbname=mydb;host=localhost', 'username', 'password');
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$stmt = $dbh->prepare('SELECT * FROM tableName limit 10000');
$stmt->execute();

$i=0;
$data = $stmt->fetchAll();
foreach ($data as $row) {
	$i++;
}

echo '
<h1>fetchAll()</h1>
';
echo '
<strong>{$i}</strong>

';
print_r(array('memory' => (memory_get_usage() - $mem) / (1024 * 1024), 'seconds' => microtime(TRUE) - $time));

if we execute the test we obtain:

fetchAll: [memory] => 31.305999755859
fetch: [memory] => 0.002532958984375

OK. It’s obvious. If we approach to the data extraction with fetchAll method we will use more memory. That’s because we’re mapping the whole recorded to a variable ($data) at once. With the fetch loop we are mapping only on row per iteration. By the way if we change the fetch loop to:

$data = array();
while ($row = $stmt->fetch()) {
	$i++;
	$data[] = $row;
}

We will use almost the same amount of memory than the fetchAll method
[memory] => 31.267543792725

Conclusion:
Is it better fetch than fetchAll? The answer is simple: No. We only need to take care what are we doing and use the best solution that fix to our need. If we’re handling small recordset, they’re similar, but if we work with big ones we need to realize that the memory usage we are using changes drastically if we use one method or another.

Howto Send/Read SMSs using a GSM modem, AT+ commands and PHP

I need to send/read SMS with PHP. To do that, I need a GSM modem. There are few of them if we google a little bit. GSM modems are similar than normal modems. They’ve got a SIM card and we can do the same things we can do with a mobile phone, but using AT and AT+ commands programmatically. That’s means we can send (and read) SMSs and create scripts to perform those operations. Normally those kind of devices uses a serial interface. So we need to connect our PC/server with a serial cable to the device. That’s a problem. Modern PCs sometimes dont’t have serial ports. Modern GSM modems have a USB port, and even we can use serial/USB converters. I don’t like to connect directly those devices to the PC/server. They must be close. Serial/USB cables cannot be very long (2 or 3 meters). I prefer to connect those devices using serial/ethernet converters. Because of that I will create a dual library. The idea is enable the operations when device is connected via serial cable and also when it’s connected thought a serial/ethernet converter.

The idea is the following one: We are going to create a main class called Sms. It takes in the constructor (via dependency injection) the HTTP wrapper or the serial one (both with the same interface). That means our Sms class will work exactly in the same way with one interface or another.

Let’s start. First we’re going to create a Dummy Mock object, sharing the same interface than the others. The purpose of that is to test the main class (Sms.php)

<?php
class Sms_Dummy implements Sms_Interface
{
    public function deviceOpen()
    {
    }

    public function deviceClose()
    {
    }

    public function sendMessage($msg)
    {
    }

    public function readPort()
    {
        return array("OK", array());
    }

    private $_validOutputs = array();

    public function setValidOutputs($validOutputs)
    {
        $this->_validOutputs = $validOutputs;
    }
}

And we test the code:

<?php

require_once('Sms.php');
require_once('Sms/Interface.php');
require_once('Sms/Dummy.php');

$pin = 1234;

$serial = new Sms_Dummy;

if (Sms::factory($serial)->insertPin($pin)
                ->sendSMS(555987654, "test Hi")) {
    echo "SMS sent\n";
} else {
    echo "SMS not Sent\n";
}

It works. Our Sms class sends a fake sms using Sms_Dummy

Now we only need to create Sms_Serial and Sms_Http. For Sms_Serial I’ve use the great library (with a slight modifications) of Rémy Sanchez to read/write serial port in PHP (you can find it here). We can run it with a script similar than the Dummy one, thanks to dependency injection:

require_once('Sms.php');
require_once('Sms/Interface.php');
require_once('Sms/Serial.php');

$pin = 1234;

try {
    $serial = new Sms_Serial;
    $serial->deviceSet("/dev/ttyS0");
    $serial->confBaudRate(9600);
    $serial->confParity('none');
    $serial->confCharacterLength(8);

    $sms = Sms::factory($serial)->insertPin($pin);

    if ($sms->sendSMS(555987654, "test Hi")) {
        echo "SMS sent\n";
    } else {
        echo "Sent Error\n";
    }

    // Now read inbox
    foreach ($sms->readInbox() as $in) {
        echo"tlfn: {$in['tlfn']} date: {$in['date']} {$in['hour']}\n{$in['msg']}\n";

        // now delete sms
        if ($sms->deleteSms($in['id'])) {
            echo "SMS Deleted\n";
        }
    }
} catch (Exception $e) {
    switch ($e->getCode()) {
        case Sms::EXCEPTION_NO_PIN:
            echo "PIN Not set\n";
            break;
        case Sms::EXCEPTION_PIN_ERROR:
            echo "PIN Incorrect\n";
            break;
        case Sms::EXCEPTION_SERVICE_NOT_IMPLEMENTED:
            echo "Service Not implemented\n";
            break;
        default:
            echo $e->getMessage();
    }
}

And finaly the Http one. As you can see the the script is the same than the Serial one. the only difference is the class passed to the constructor of the Sms class:

<?php 
require_once('Sms.php'); 
require_once('Sms/Interface.php'); 
require_once('Sms/Http.php'); 

$serialEternetConverterIP = '192.168.1.10'; 
$serialEternetConverterPort = 1113; 
$pin = 1234; 
try {
     $sms = Sms::factory(new Sms_Http($serialEternetConverterIP, $serialEternetConverterPort));
     $sms->insertPin($pin);

    if ($sms->sendSMS(555987654, "test Hi")) {
        echo "SMS Sent\n";
    } else {
        echo "Sent Error\n";
    }

    // Now read inbox
    foreach ($sms->readInbox() as $in) {
        echo"tlfn: {$in['tlfn']} date: {$in['date']} {$in['hour']}\n{$in['msg']}\n";

        // now delete sms
        if ($sms->deleteSms($in['id'])) {
            echo "SMS Deleted\n";
        }
    }
} catch (Exception $e) {
    switch ($e->getCode()) {
        case Sms::EXCEPTION_NO_PIN:
            echo "PIN Not set\n";
            break;
        case Sms::EXCEPTION_PIN_ERROR:
            echo "PIN Incorrect\n";
            break;
        case Sms::EXCEPTION_SERVICE_NOT_IMPLEMENTED:
            echo "Service Not implemented\n";
            break;
        default:
            echo $e->getMessage();
    }
}

Serial/Ethernet converters normally have different operational modes. They allow you to create a fake serial port on your PC and work exactly in the same way than if you have your device connected with a serial cable. I don’t like this operation mode. I prefer to use the TCP server mode. With the TCP server mode I can read/write from the serial device with the standard socket functions. So if you dive into Sms_Http class you will find TCP socket’s functions.

And finally I will do a brief excerpt of AT+ commands used to send /read SMSs:

AT+CPIN?\r : checks if SIM has the pin code. It answers +CPIN: READY or +CPIN: SIM PIN if we need to insert the pin number
AT+CMGS=”[number]”\r[text]Control-Z : to send a SMS (in php we have Control-Z with chr(26)). It returns OK or ERROR
AT+CMGF=1\r: set the device to operate in SMS text mode (0=PDU mode and 1=text mode). Returns OK.
AT+CMGL=\”ALL”\r read all the sms stored in the device. We also can use “REC UNREAD” Instead of “ALL”.
AT+CMGD=[ID]\r: Deletes a SMS from the device

You can download the source code from github here

Real time notifications with PHP

Real time communications are cool, isn’t it? Something impossible to do five years ago now (or almost impossible) is already available. Nowadays we have two possible solutions. WebSockets and Comet. WebSockets are probably the best solution but they’ve got two mayor problems:

  • Not all browsers support them.
  • Not all proxy servers allows the communications with websokets.

Because of that I prefer to use comet (at least now). It’s not as good as websockets but pretty straightforward ant it works (even on IE). Now I’m going to explain a little script that I’ve got to perform a comet communications, made with PHP. Probably it’s not a good idea to use it in a high traffic site, but it works like a charm in a small intranet. If you want to use comet in a high traffic site maybe you need have a look to Tornado, twisted, node.js or other comet dedicated servers.

Normally when we are speaking about real-time communications, all the people are thinking about a chat application. I want to build a simpler application. A want to detect when someone clicks on a link. Because of that I will need a combination of HTML, PHP and JavaScript. Let’s start:

For the example I’ll use jquery library, so we need to include the library in our HTML file. It will be a blend of JavaScrip and PHP:

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Comet Test</title>
    </head>
    <body>
        <p><a class='customAlert' href="#">publish customAlert</a></p>
        <p><a class='customAlert2' href="#">publish customAlert2</a></p>
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js" type="text/javascript"></script>
        <script src="NovComet.js" type="text/javascript"></script>
        <script type="text/javascript">
NovComet.subscribe('customAlert', function(data){
    console.log('customAlert');
    //console.log(data);
}).subscribe('customAlert2', function(data){
    console.log('customAlert2');
    //console.log(data);
});

$(document).ready(function() {
    $("a.customAlert").click(function(event) {
        NovComet.publish('customAlert');
    });
    
    $("a.customAlert2").click(function(event) {
        NovComet.publish('customAlert2');
    });
    NovComet.run();
});
        </script>
    </body>
</html>

The client code:

//NovComet.js
NovComet = {
    sleepTime: 1000,
    _subscribed: {},
    _timeout: undefined,
    _baseurl: "comet.php",
    _args: '',
    _urlParam: 'subscribed',

    subscribe: function(id, callback) {
        NovComet._subscribed[id] = {
            cbk: callback,
            timestamp: NovComet._getCurrentTimestamp()
        };
        return NovComet;
    },

    _refresh: function() {
        NovComet._timeout = setTimeout(function() {
            NovComet.run()
        }, NovComet.sleepTime);
    },

    init: function(baseurl) {
        if (baseurl!=undefined) {
            NovComet._baseurl = baseurl;
        }
    },

    _getCurrentTimestamp: function() {
        return Math.round(new Date().getTime() / 1000);
    },

    run: function() {
        var cometCheckUrl = NovComet._baseurl + '?' + NovComet._args;
        for (var id in NovComet._subscribed) {
            var currentTimestamp = NovComet._subscribed[id]['timestamp'];

            cometCheckUrl += '&' + NovComet._urlParam+ '[' + id + ']=' +
               currentTimestamp;
        }
        cometCheckUrl += '&' + NovComet._getCurrentTimestamp();
        $.getJSON(cometCheckUrl, function(data){
            switch(data.s) {
                case 0: // sin cambios
                    NovComet._refresh();
                    break;
                case 1: // trigger
                    for (var id in data['k']) {
                        NovComet._subscribed[id]['timestamp'] = data['k'][id];
                        NovComet._subscribed[id].cbk(data.k);
                    }
                    NovComet._refresh();
                    break;
            }
        });

    },

    publish: function(id) {
        var cometPublishUrl = NovComet._baseurl + '?' + NovComet._args;
        cometPublishUrl += '&publish=' + id;
        $.getJSON(cometPublishUrl);
    }
};

The server-side PHP

// comet.php
include('NovComet.php');

$comet = new NovComet();
$publish = filter_input(INPUT_GET, 'publish', FILTER_SANITIZE_STRING);
if ($publish != '') {
    echo $comet->publish($publish);
} else {
    foreach (filter_var_array($_GET['subscribed'], FILTER_SANITIZE_NUMBER_INT) as $key => $value) {
        $comet->setVar($key, $value);
    }
    echo $comet->run();
}

and my comet library implementation:

// NovComet.php
class NovComet {
    const COMET_OK = 0;
    const COMET_CHANGED = 1;

    private $_tries;
    private $_var;
    private $_sleep;
    private $_ids = array();
    private $_callback = null;

    public function  __construct($tries = 20, $sleep = 2)
    {
        $this->_tries = $tries;
        $this->_sleep = $sleep;
    }

    public function setVar($key, $value)
    {
        $this->_vars[$key] = $value;
    }

    public function setTries($tries)
    {
        $this->_tries = $tries;
    }

    public function setSleepTime($sleep)
    {
        $this->_sleep = $sleep;
    }

    public function setCallbackCheck($callback)
    {
        $this->_callback = $callback;
    }

    const DEFAULT_COMET_PATH = "/dev/shm/%s.comet";

    public function run() {
        if (is_null($this->_callback)) {
            $defaultCometPAth = self::DEFAULT_COMET_PATH;
            $callback = function($id) use ($defaultCometPAth) {
                $cometFile = sprintf($defaultCometPAth, $id);
                return (is_file($cometFile)) ? filemtime($cometFile) : 0;
            };
        } else {
            $callback = $this->_callback;
        }

        for ($i = 0; $i < $this->_tries; $i++) {
            foreach ($this->_vars as $id => $timestamp) {
                if ((integer) $timestamp == 0) {
                    $timestamp = time();
                }
                $fileTimestamp = $callback($id);
                if ($fileTimestamp > $timestamp) {
                    $out[$id] = $fileTimestamp;
                }
                clearstatcache();
            }
            if (count($out) > 0) {
                return json_encode(array('s' => self::COMET_CHANGED, 'k' => $out));
            }
            sleep($this->_sleep);
        }
        return json_encode(array('s' => self::COMET_OK));
    }

    public function publish($id)
    {
        return json_encode(touch(sprintf(self::DEFAULT_COMET_PATH, $id)));
    }
}

As you can see in my example I’ve created a personal protocol for the communications between the client (js at browser), and the server (PHP). It’s a simple one. If you’re looking for a “standard” protocol maybe you need have a look to bayeux protocol from Dojo people.

Let me explain a little bit the usage of the script:

  • In the HTML page we start the listener (NovComet.subscribe).
  • We can subscribe to as many events we want (OK it depends on our resources)
  • When we subscribe to one event we pass a callback function to be triggered.
  • When we subscribe to the event, we pass the current timestamp to the server.
  • Client side script (js with jquery) will call to server-side script (PHP) with the timestamp and will wait until server finish.
  • Server side script will answer when even timestamp changes (someone has published the event)
  • Server side will no keep waiting forever. If nobody publish the event, server will answer after a pre-selected timeout
  • client side script will repeat the process again and again.

There’s something really important with this technique. Our server-side event check need to be as simpler as we can. We cannot execute a SQL query for example (our sysadmin will kill us if we do it). We need to bear in mind that this check will be performed again and again per user, because of that it must be as light as we can. In this example we are checking the last modification date of a file (filemtime). Another good solution is to use a memcached database and check a value.

For the test I’ve also created a publishing script (NovComet.publish). This is the simple part. We only call a server-side script that touch the event file (changing the last modification date), triggering the event.

Now I’m going to explain what we can see on the firebug console:

  1. The first iteration nothing happens. 200 OK Http code after the time-out set in the PHP script
  2. As we can see here the script returns a JSON with s=0 (nothing happens)
  3. Now we publish an event. Script returns a 200 OK but now the JSON is different. s=1 and the time-stamp of the event
  4. Our callback has been triggered
  5. And next iteration waiting

And that’s all. Simple and useful. But remember, you must take care if you are using this solution within a high traffic site. What do you think? Do you use lazy comet with PHP in production servers or would you rather another solution?

You can get the code at github here.

Watermarks in our images with PHP and Gearman

In my last post I’ve tried to explain how to add a dynamic watermarks in our images using PHP and the GD library. Someone told me in a comment that he has used something similar, but he had to disable it because of the use of huge resources. Probably the use of the solution of the previous post doesn’t scale well. If you have a hight traffic site your memory and CPU usage will increase a lot because of the image transformation (especially if you work with big images). It’d be better if you can generate the watermarks offline, but if is mandatory to create them dynamically (for example we need to place the current timestamp), there are other solutions.

In this second solution I will use a gearman worker to generate the watermarks. The benefits of gearman is the possibility of use a pool of workers. We can add/remove workers if our application scales. Those workers can be placed even at different hosts, and we can swap easily from one configuration to another. Imagine we have an application that uses a single worker at the same host of the webserver. Maybe it’s enough for a small site, but suddenly we increase our users. We can add new workers to our host. But if our single host is not enough, we can rent new host/hosts (with amazon for example) and our application will adapt easily to the new scenario. Gearman allows an easy way to scale out our applications.

Let’s start:

Now our main script instead of doing the hard work, will be a gearman client

<?php 
$filename = "/path/to/img.jpg";
$footer = date('d/m/Y H:i:s');

$gmclient = new GearmanClient();
$gmclient->addServer();

$handle = $gmclient->do("watermark", json_encode(array($filename, $footer)));

if ($gmclient->returnCode() != GEARMAN_SUCCESS){
    echo "Ups something wrong happen";
} else {
    header( 'Content-Type: image/jpeg' );
    echo $handle;
}

And our worker will do the hard work.

<?php 
$gmw = new GearmanWorker(); 
$gmw->addServer();
$gmw->addFunction("watermark", function($job) {

    $workload = $job->workload();
    $workload_size = $job->workloadSize();

    list($filename, $footer) = json_decode($workload);

    $footerSize = 15;
    list($width, $height, $image_type) = getimagesize($filename);

    $im = imagecreatefromjpeg($filename);

    imagefilledrectangle (
            $im,
            0,
            $height,
            $width,
            $height - $footerSize, imagecolorallocate($im, 49, 49, 156));

    imagestring($im,
            $stringSize,
            $width-(imagefontwidth($stringSize)*strlen($footer)) - 2,
            $height-$footerSize,
            $footer,
            imagecolorallocate($im, 255, 255, 255));

    ob_start();
    ob_implicit_flush(0);
    imagepng($im);
    $img=ob_get_contents();
    ob_end_clean();

    return $img;
});
while(1) {
  $gmw->work();
}

We must remember to start our worker/workers within our server.

php /path/to/worker/worker.watermark.php

What do you think with this solution?

Follow

Get every new post delivered to your Inbox.

Join 1,003 other followers