Building a REST client with asynchronous calls using PHP and curl


One month ago I posted an article called Building a simple HTTP client with PHP. A REST client. In this post I tried to create a simple fluid interface to call REST webservices in PHP. Some days ago I read an article and one light switch on on my mind. I can use curl’s “multi” functions to improve my library and perform simultaneous calls very easily.

I’ve got a project that needs to call to different webservices. Those webservices sometimes are slow (2-3 seconds). If I need to call to, for example, three webservices my script will use the add of every single call’s time. With this improve to the library I will use only the time of the slowest webservice. 2 seconds instead of 2+2+2 seconds. Great.

For the example I’ve created a really complex php script that sleeps x seconds depend on an input param:

sleep((integer) $_REQUEST['sleep']);
echo $_REQUEST['sleep'];

With synchronous calls:

echo Http::connect('localhost', 8082)
    ->doGet('/tests/gam_http/sleep.php', array('sleep' => 3));
echo Http::connect('localhost', 8082)
    ->doPost('/tests/gam_http/sleep.php', array('sleep' => 2));
echo Http::connect('localhost', 8082)
    ->doGet('/tests/gam_http/sleep.php', array('sleep' => 1));

This script takes more or less 6 seconds (3+2+1)

But If I switch it to:

$out = Http::connect('localhost', 8082)
    ->get('/tests/gam_http/sleep.php', array('sleep' => 3))
    ->post('/tests/gam_http/sleep.php', array('sleep' => 2))
    ->get('/tests/gam_http/sleep.php', array('sleep' => 1))
    ->run();
print_r($out);

The script only uses 3 seconds (the slowest process)

I’ve got a project that uses it. But I have a problem. I have webservices in different hosts so I’ve done a bit change to the library:

$out = Http::multiConnect()
    ->add(Http::connect('localhost', 8082)->get('/tests/gam_http/sleep.php', array('sleep' => 3)))
    ->add(Http::connect('localhost', 8082)->post('/tests/gam_http/sleep.php', array('sleep' => 2)))
    ->add(Http::connect('localhost', 8082)->get('/tests/gam_http/sleep.php', array('sleep' => 1)))
    ->run();

With a single connection, the exceptions are easy to implement. If curl_getinfo() returns an error message I throw an exception, but now with a multiple interface how I can do it? I throw an exception if one call fail, or not? I have decided not to use exceptions in multiple interface. I always return an array with all the output of every webservice’s call and if something wrong happens instead of the output I will return an instance of Http_Multiple_Error class. Why I use a class instead of a error message? The answer is easy. If I want to check all the answers I can check if any of them is an instanceof Http_Multiple_Error. Also I don’t want to check anything I put a silentMode() function to switch off all error messages.

$out = Http::multiConnect()
    ->silentMode()
    ->add(Http::connect('localhost', 8082)->get('/tests/gam_http/sleep.php', array('sleep' => 3)))
    ->add(Http::connect('localhost', 8082)->post('/tests/gam_http/sleep.php', array('sleep' => 2)))
    ->add(Http::connect('localhost', 8082)->get('/tests/gam_http/sleep.php', array('sleep' => 1)))
    ->run();

The full code is available on google code but the main function is the following one:

    ...
    private function _run()
    {
        $headers = $this->_headers;
        $curly = $result = array();

        $mh = curl_multi_init();
        foreach ($this->_requests as $id => $reg) {
            $curly[$id] = curl_init();

            $type     = $reg[0];
            $url       = $reg[1];
            $params = $reg[2];

            if(!is_null($this->_user)){
               curl_setopt($curly[$id], CURLOPT_USERPWD, $this->_user.':'.$this->_pass);
            }

            switch ($type) {
                case self::DELETE:
                    curl_setopt($curly[$id], CURLOPT_URL, $url . '?' . http_build_query($params));
                    curl_setopt($curly[$id], CURLOPT_CUSTOMREQUEST, self::DELETE);
                    break;
                case self::POST:
                    curl_setopt($curly[$id], CURLOPT_URL, $url);
                    curl_setopt($curly[$id], CURLOPT_POST, true);
                    curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $params);
                    break;
                case self::GET:
                    curl_setopt($curly[$id], CURLOPT_URL, $url . '?' . http_build_query($params));
                    break;
            }
            curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, true);
            curl_setopt($curly[$id], CURLOPT_HTTPHEADER, $headers);

            curl_multi_add_handle($mh, $curly[$id]);
        }

        $running = null;
        do {
            curl_multi_exec($mh, $running);
            sleep(0.2);
        } while($running > 0);

        foreach($curly as $id => $c) {
            $status = curl_getinfo($c, CURLINFO_HTTP_CODE);
            switch ($status) {
                case self::HTTP_OK:
                case self::HTTP_CREATED:
                case self::HTTP_ACEPTED:
                    $result[$id] = curl_multi_getcontent($c);
                    break;
                default:
                    if (!$this->_silentMode) {
                        $result[$id] = new Http_Multiple_Error($status, $type, $url, $params);
                    }
            }
            curl_multi_remove_handle($mh, $c);
        }

        curl_multi_close($mh);
        return $result;
About these ads

About Gonzalo Ayuso

Web Architect specialized in Open Source technologies. PHP, Python, JQuery, Dojo, PostgreSQL, CouchDB and node.js but always learning.

Posted on February 6, 2010, in php, Technology, Web Development. Bookmark the permalink. 23 Comments.

  1. Hi

    This is very very useful. I just want to ask one more question. What about the port 8082. I did not get connection on where did you use the port number. Please reply

    Thanks

    • Gonzalo Ayuso

      8082 was the port number I’ve got the server at localhost. By default is 80. Depends on your web server configurartion. In fact I’ve used a port different than 80 to ensure the library works with non-standard ports. First you need to ensure http://host:port/path/to/sleep.php works standalone, where host is your webserver (for example localhost) and port your webserver port (for example 80)

  2. I really enjoyed your tutorial. But can you please tell me how to get a meaningful error message out?

    Thank you

  3. Gonzalo Ayuso

    $out var has the error code of every assinc executions (it’s an array) check it out

  4. Hi,
    Thanks for the good tutorial. I tried to download it from the google code URL(http://code.google.com/p/gam-http/) provided but i got this on the download page
    “This project currently has no downloads.”

    Where can i download the full code?

  5. Thanks Gonzalo, i already downloaded it from github before your reply. i found my way there.Still have questions as i’m very new in this RESTFul thing. I can see that in order to use your code, one has to put the domain and the port number.

    Is the port detail required because most web services offering REST API doesn’t tell you their server port number. Is the port number a required option that the code need?

    Also can this code work for RESTFul XML?

  6. heh i like this i user oauth2 rest api service and connect with deamon pipline how can manage each request in pipline.

  7. Thanks for this snippet, Gonzalo, it saved me a lot of time.
    I however have the following issue:
    If i POST data, my custom headers aren’t being set (application/json), GET is working fine. Any idea? I went through your Class Code but didn’t get it.

    $Response = json_decode(Http::connect($this->hostname, 443, Http::HTTPS)->setHeaders(array(“Content-type” => “application/json”))->doGet($url, $parms));

    $Response = json_decode(Http::connect($this->hostname, 443, Http::HTTPS)->setHeaders(array(“Content-type” => “application/json”))->doPost($url, $parms));

  8. Just in case, anybody has the same problem ..
    I stuck in the Header Array assignment, so do it the right way!

    WRONG
    setHeaders(array(“Content-type” => “application/json”))

    RIGHT
    setHeaders(array(“Content-type: application/json”))

  9. What if each webservices return XML but with different nodes and DTD? Is there responses ok to be process or all the data are merged?

    • curl only gives you asyncronous calls. You need to merge the outcomes. If it’s XML, you need to create a function to merge.

  10. Hello :)

    First, thank you for sharing all your stuff!

    I’m trying to develop a REST client based on your class, but I’m totally newbie in OOP.
    My need looks pretty simple, but I can’t find a way due to my lacks.

    I need to call different urls at once, but I don’t know them in advance. So I want to generate the calls within a foreach loop.

    Here is what I thought I could do :

    $call = Http::connect($server, 8080)
    foreach ($array AS $key=>$value) {
    ->get(‘test/’.$value, array())
    }

    Obviously, it doesn’t work (syntax error).

    So I tried something like this :

    $call= Http::connect($server, 8080);
    foreach ($array AS $key=>$value) {
    $call->get(‘test/’.$value, array());
    }

    But then, when I print_r($call), it gives me the details of the object created by your class, not the result of the call.

    If you could help me, I would me more than grateful!

    • Gonzalo Ayuso

      This post is about how to call different curl sentences at the same time. But maybe you don’t need to do it, and you can execute them synchronously. I recommend first learn how to do it with curl, instead of using a library. BTW The code of your example looks l you forget to execute run()

      • Thanks for your quick reply Gonzalo :)

        In fact, I do need to execute them synchronously. I first tried with one at a time but regarding how the REST server is made, it took to much time (hundreds of calls to the server to gather the data I need for a single page on a client side).

  11. Mistery of OOP for a newbie …

    $call= Http::connect($conf_server_url, 8080);
    $call->get(‘test/’.$value, array());
    $call->run();
    print_r($call);
    ==> it prints the whole HTTP object

    $call= Http::connect($conf_server_url, 8080);
    $call->get(‘test/’.$value, array());
    print_r($call->run);
    ==> it prints the result I’m looking for.

    Why that!?

    Anyway, I’m not stuck anymore, and the conclusion is that I need to learn a bit more of OOP ;)

    • Gonzalo Ayuso

      In all cases learning OOP is good :)

      Your problem is simple here call is the instance of the object and with $call->run() we get the output.

      Anyway the naming that I use within this library is let’s say “improvable”. It’s a bit old library. Nowadays I’m a big fan of SOLID principles and good naming. Because of that if I need to rewrite the library I will use different names for functions. For example instead of get() I’d use registerGet() because the function doesn’t execute any get request (the verb is important in function names. It must show us what the function does). It only register the request for being executed later.

      Also the name run() is ambiguous. It does two things: run and return the outcome. This violates the single responsibility rule. So a best solution is use one function to execute the requests and another to fetch the outcome.

  12. Hi Gonza!

    What is the purpose of the `sleep(0.2)` after `curl_multi_exec($mh, $running);`?

    • There is a little bug there. I must use usleep instead sleep. sleep take seconds (integer) as argument. Anyway the idea is sleep the process to avoid hight loads. In this kind of iterations If we iterate without sleep our microprocessor will use all of its resources, but if we sleep the process our server load will decrease.

  1. Pingback: Building a simple API proxy server with PHP « Gonzalo Ayuso | Web Architect

  2. Pingback: How to configure Symfony’s Service Container to use Twitter API « Gonzalo Ayuso | Web Architect

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 869 other followers

%d bloggers like this: