Using cache buster with OpenUI5 outside SCP

When we work with SPAs and web applications we need to handle with the browser’s cache. Sometimes we change our static files but the client’s browser uses a cached version of the file instead of the new one. We can tell the user: Please empty your cache to use the new version. But most of the times the user don’t know what we’re speaking about, and we have a problem. There’s a technique called cache buster used to bypass this issue. It consists on to change the name of the file (or adding an extra parameter), basically to ensure that the browser will send a different request to the server to prevent the browser from reusing the cached version of the file.

When we work with sapui5 application over SCP, we only need to use the cachebuster version of sap-ui-core

<script id="sap-ui-bootstrap"
      src="https://sapui5.hana.ondemand.com/resources/sap-ui-cachebuster/sap-ui-core.js"
      data-sap-ui-libs="sap.m"
      data-sap-ui-theme="sap_belize"
      data-sap-ui-compatVersion="edge"
      data-sap-ui-appCacheBuster=""
      data-sap-ui-preload="async"
      data-sap-ui-resourceroots='{"app": ""}'
      data-sap-ui-frameOptions="trusted">
</script>

With this configuration, our framework will use a “cache buster friendly” version of our files and SCP will serve them properly.

For example, when our framework wants the /dist/Component.js file, the browser will request /dist/~1541685070813~/Component.js to the server. And the server will server the file /dist/Component.js. As I said before when we work with SCP, our standard build process automatically takes care about it. It creates a file called sap-ui-cachebuster-info.json where we can find all our files with one kind of hash that our build process changes each time our file is changed.

{
  "Component-dbg.js": 1545316733136,
  "Component-preload.js": 1545316733226,
  "Component.js": 1541685070813,
  ...
}

It works like a charm but I not always use SCP. Sometimes I use OpenUI5 in one nginx server, for example. So cache buster “doesn’t work”. That’s a problem because I need to handle with browser caches again each time we deploy the new version of the application. I wanted to solve the issue. Let me explain how I did it.

Since I was using one Lumen/PHP server to the backend, my first idea was to create a dynamic route in Lumen to handle cache-buster urls. With this approach I know I can solve the problem but there’s something that I did not like: I’ll use a dynamic server to serve static content. I don’t have a huge traffic. I can use this approach but it isn’t beautiful.

My second approach was: Ok I’ve got a sap-ui-cachebuster-info.json file where I can see all the files that cache buster will use and their hashes. So, Why not I create those files in my build script. With this approach I will create the full static structure each time I deploy de application, without needing any server side scripting language to generate dynamic content. OpenUI5 uses grunt so I can create a simple grunt task to create my files.

'use strict';

var fs = require('fs');
var path = require('path');
var chalk = require('chalk');

module.exports = function(grunt) {
  var name = 'cacheBuster';
  var info = 'Generates Cache buster files';

  var cacheBuster = function() {
    var config = grunt.config.get(name);
    var data, t, src, dest, dir, prop;

    data = grunt.file.readJSON(config.src + '/sap-ui-cachebuster-info.json');
    for (prop in data) {
      if (data.hasOwnProperty(prop)) {
        t = data[prop];
        src = config.src + '/' + prop;
        dest = config.src + '/~' + t + '~/' + prop;
        grunt.verbose.writeln(
            name + ': ' + chalk.cyan(path.basename(src)) + ' to ' +
            chalk.cyan(dest) + '.');
        dir = path.dirname(dest);
        grunt.file.mkdir(dir);
        fs.copyFileSync(src, dest);
      }
    }
  };

  grunt.registerMultiTask(name, info, cacheBuster);
};

I deploy my grunt task to npm so when I need to use it I only need to:

Install the task

npm install gonzalo123-cachebuster

Add the task to my gruntfile

require('gonzalo123-cachebuster')(grunt);

and set up the path where ui5 task generates our dist files

  grunt.config.merge({
    pkg: grunt.file.readJSON('package.json'),
    ...
    cacheBuster: {
      src: 'dist'
    }
  });

And that’s all. My users with enjoy (or suffer) the new versions of my applications without having problems with cached files.

Grunt task available in my github

Advertisement

Working with SAPUI5 locally (part 3). Adding more services in Docker

In the previous project we moved one project to docker. The idea was to move exactly the same functionality (even without touching anything within the source code). Now we’re going to add more services. Yes, I know, it looks like overenginering (it’s exactly overenginering, indeed), but I want to build something with different services working together. Let start.

We’re going to change a little bit our original project. Now our frontend will only have one button. This button will increment the number of clicks but we’re going to persists this information in a PostgreSQL database. Also, instead of incrementing the counter in the backend, our backend will emit one event to a RabbitMQ message broker. We’ll have one worker service listening to this event and this worker will persist the information. The communication between the worker and the frontend (to show the incremented value), will be via websockets.

With those premises we are going to need:

  • Frontend: UI5 application
  • Backend: PHP/lumen application
  • Worker: nodejs application which is listening to a RabbitMQ event and serving the websocket server (using socket.io)
  • Nginx server
  • PosgreSQL database.
  • RabbitMQ message broker.

As the previous examples, our PHP backend will be server via Nginx and PHP-FPM.

Here we can see to docker-compose file to set up all the services

version: '3.4'

services:
  nginx:
    image: gonzalo123.nginx
    restart: always
    ports:
    - "8080:80"
    build:
      context: ./src
      dockerfile: .docker/Dockerfile-nginx
    volumes:
    - ./src/backend:/code/src
    - ./src/.docker/web/site.conf:/etc/nginx/conf.d/default.conf
    networks:
    - app-network
  api:
    image: gonzalo123.api
    restart: always
    build:
      context: ./src
      dockerfile: .docker/Dockerfile-lumen-dev
    environment:
      XDEBUG_CONFIG: remote_host=${MY_IP}
    volumes:
    - ./src/backend:/code/src
    networks:
    - app-network
  ui5:
    image: gonzalo123.ui5
    ports:
    - "8000:8000"
    restart: always
    volumes:
    - ./src/frontend:/code/src
    build:
      context: ./src
      dockerfile: .docker/Dockerfile-ui5
    networks:
    - app-network
  io:
    image: gonzalo123.io
    ports:
    - "9999:9999"
    restart: always
    volumes:
    - ./src/io:/code/src
    build:
      context: ./src
      dockerfile: .docker/Dockerfile-io
    networks:
    - app-network
  pg:
    image: gonzalo123.pg
    restart: always
    ports:
    - "5432:5432"
    build:
      context: ./src
      dockerfile: .docker/Dockerfile-pg
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_DB: ${POSTGRES_DB}
      PGDATA: /var/lib/postgresql/data/pgdata
    networks:
    - app-network
  rabbit:
    image: rabbitmq:3-management
    container_name: gonzalo123.rabbit
    restart: always
    ports:
    - "15672:15672"
    - "5672:5672"
    environment:
      RABBITMQ_ERLANG_COOKIE:
      RABBITMQ_DEFAULT_VHOST: /
      RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
    networks:
    - app-network
networks:
  app-network:
    driver: bridge

We’re going to use the same docker files than in the previous post but we also need new ones for worker, database server and message queue:

Worker:

FROM node:alpine

EXPOSE 8000

WORKDIR /code/src
COPY ./io .
RUN npm install
ENTRYPOINT ["npm", "run", "serve"]

The worker script is simple script that serves the socket.io server and emits a websocket within every message to the RabbitMQ queue.

var amqp = require('amqp'),
  httpServer = require('http').createServer(),
  io = require('socket.io')(httpServer, {
    origins: '*:*',
  }),
  pg = require('pg')
;

require('dotenv').config();
var pgClient = new pg.Client(process.env.DB_DSN);

rabbitMq = amqp.createConnection({
  host: process.env.RABBIT_HOST,
  port: process.env.RABBIT_PORT,
  login: process.env.RABBIT_USER,
  password: process.env.RABBIT_PASS,
});

var sql = 'SELECT clickCount FROM docker.clicks';

// Please don't do this. Use lazy connections
// I'm 'lazy' to do it in this POC 🙂
pgClient.connect(function(err) {
  io.on('connection', function() {
    pgClient.query(sql, function(err, result) {
      var count = result.rows[0]['clickcount'];
      io.emit('click', {count: count});
    });

  });

  rabbitMq.on('ready', function() {
    var queue = rabbitMq.queue('ui5');
    queue.bind('#');

    queue.subscribe(function(message) {
      pgClient.query(sql, function(err, result) {
        var count = parseInt(result.rows[0]['clickcount']);
        count = count + parseInt(message.data.toString('utf8'));
        pgClient.query('UPDATE docker.clicks SET clickCount = $1', [count],
          function(err) {
            io.emit('click', {count: count});
          });
      });
    });
  });
});

httpServer.listen(process.env.IO_PORT);

Database server:

FROM postgres:9.6-alpine
COPY pg/init.sql /docker-entrypoint-initdb.d/

As we can see we’re going to generate the database estructure in the first build

CREATE SCHEMA docker;

CREATE TABLE docker.clicks (
clickCount numeric(8) NOT NULL
);

ALTER TABLE docker.clicks
OWNER TO username;

INSERT INTO docker.clicks(clickCount) values (0);

With the RabbitMQ server we’re going to use the official docker image so we don’t need to create one Dockerfile

We also have changed a little bit our Nginx configuration. We want to use Nginx to serve backend and also socket.io server. That’s because we don’t want to expose different ports to internet.

server {
    listen 80;
    index index.php index.html;
    server_name localhost;
    error_log  /var/log/nginx/error.log;
    access_log /var/log/nginx/access.log;
    root /code/src/www;

    location /socket.io/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass "http://io:9999";
    }

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass api:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }
}

To avoid CORS issues we can also use SCP destination (the localneo proxy in this example), to serve socket.io also. So we need to:

  • change our neo-app.json file
  • "routes": [
        ...
        {
          "path": "/socket.io",
          "target": {
            "type": "destination",
            "name": "SOCKETIO"
          },
          "description": "SOCKETIO"
        }
      ],
    

    And basically that’s all. Here also we can use a “production” docker-copose file without exposing all ports and mapping the filesystem to our local machine (useful when we’re developing)

    version: '3.4'
    
    services:
      nginx:
        image: gonzalo123.nginx
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-nginx
        networks:
        - app-network
      api:
        image: gonzalo123.api
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-lumen
        networks:
        - app-network
      ui5:
        image: gonzalo123.ui5
        ports:
        - "80:8000"
        restart: always
        volumes:
        - ./src/frontend:/code/src
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-ui5
        networks:
        - app-network
      io:
        image: gonzalo123.io
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-io
        networks:
        - app-network
      pg:
        image: gonzalo123.pg
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-pg
        environment:
          POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
          POSTGRES_USER: ${POSTGRES_USER}
          POSTGRES_DB: ${POSTGRES_DB}
          PGDATA: /var/lib/postgresql/data/pgdata
        networks:
        - app-network
      rabbit:
        image: rabbitmq:3-management
        restart: always
        environment:
          RABBITMQ_ERLANG_COOKIE:
          RABBITMQ_DEFAULT_VHOST: /
          RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
          RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
        networks:
        - app-network
    networks:
      app-network:
        driver: bridge
    

    And that’s all. The full project is available in my github account

    Working with SAPUI5 locally (part 2). Now with docker

    In the first part I spoke about how to build our working environment to work with UI5 locally instead of using WebIDE. Now, in this second part of the post, we’ll see how to do it using docker to set up our environment.

    I’ll use docker-compose to set up the project. Basically, as I explain in the first part, the project has two parts. One backend and one frontned. We’re going to use exactly the same code for the frontend and for the backend.

    The frontend is build over a localneo. As it’s a node application we’ll use a node:alpine base host

    FROM node:alpine
    
    EXPOSE 8000
    
    WORKDIR /code/src
    COPY ./frontend .
    RUN npm install
    ENTRYPOINT ["npm", "run", "serve"]
    

    In docker-compose we only need to map the port that we´ll expose in our host and since we want this project in our depelopemet process, we also will map the volume to avoid to re-generate our container each time we change the code.

    ...
      ui5:
        image: gonzalo123.ui5
        ports:
        - "8000:8000"
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-ui5
        volumes:
        - ./src/frontend:/code/src
        networks:
        - api-network
    

    The backend is a PHP application. We can set up a PHP application using different architectures. In this project we’ll use nginx and PHP-FPM.

    for nginx we’ll use the following Dockerfile

    FROM  nginx:1.13-alpine
    
    EXPOSE 80
    
    COPY ./.docker/web/site.conf /etc/nginx/conf.d/default.conf
    COPY ./backend /code/src
    

    And for the PHP host the following one (with xdebug to enable debugging and breakpoints):

    FROM php:7.1-fpm
    
    ENV PHP_XDEBUG_REMOTE_ENABLE 1
    
    RUN apt-get update && apt-get install -my \
        git \
        libghc-zlib-dev && \
        apt-get clean
    
    RUN apt-get install -y libpq-dev \
        && docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql \
        && docker-php-ext-install pdo pdo_pgsql pgsql opcache zip
    
    RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
    
    RUN composer global require "laravel/lumen-installer"
    ENV PATH ~/.composer/vendor/bin:$PATH
    
    COPY ./backend /code/src
    

    And basically that’s all. Here the full docker-compose file

    version: '3.4'
    
    services:
      nginx:
        image: gonzalo123.nginx
        restart: always
        ports:
        - "8080:80"
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-nginx
        volumes:
        - ./src/backend:/code/src
        - ./src/.docker/web/site.conf:/etc/nginx/conf.d/default.conf
        networks:
        - api-network
      api:
        image: gonzalo123.api
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-lumen-dev
        environment:
          XDEBUG_CONFIG: remote_host=${MY_IP}
        volumes:
        - ./src/backend:/code/src
        networks:
        - api-network
      ui5:
        image: gonzalo123.ui5
        ports:
        - "8000:8000"
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-ui5
        networks:
        - api-network
    
    networks:
      api-network:
        driver: bridge
    

    If we want to use this project you only need to:

    • clone the repo fron github
    • run ./ui5 up

    With this configuration we’re exposing two ports 8080 for the frontend and 8000 for the backend. We also are mapping our local filesystem to containers to avoid to regenerate our containers each time we change the code.

    We also can have a variation. A “production” version of our docker-compose file. I put production between quotation marks because normally we aren’t going to use localneo as a production server (please don’t do it). We’ll use SCP to host the frontend.

    This configuration is just an example without filesystem mapping, without xdebug in the backend and without exposing the backend externally (Only the frontend can use it)

    version: '3.4'
    
    services:
      nginx:
        image: gonzalo123.nginx
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-nginx
        networks:
        - api-network
      api:
        image: gonzalo123.api
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-lumen
        networks:
        - api-network
      ui5:
        image: gonzalo123.ui5
        ports:
        - "8000:8000"
        restart: always
        build:
          context: ./src
          dockerfile: .docker/Dockerfile-ui5
        networks:
        - api-network
    
    networks:
      api-network:
        driver: bridge
    

    And that’s all. You can see the all the source code in my github account

    Working with SAPUI5 locally and deploying in SCP

    When I work with SAPUI5 projects I normally use WebIDE. WebIDE is a great tool but I’m more confortable working locally with my local IDE.
    I’ve this idea in my mind but I never find the time slot to work on it. Finally, after finding this project from Holger Schäfer in github, I realized how easy it’s and I started to work with this project and adapt it to my needs.

    The base of this project is localneo. Localneo starts a http server based on neo-app.json file. That means we’re going to use the same configuration than we’ve in production (in SCP). Of course we’ll need destinations. We only need one extra file called destination.json where we’ll set up our destinations (it creates one http proxy, nothing else).

    In this project I’ll create a simple example application that works with one API server.

    The backend

    I’ll use in this example one PHP/Lumen application:

    $app->router->group(['prefix' => '/api', 'middleware' => Middleware\AuthMiddleware::NAME], function (Router $route) {
        $route->get('/', Handlers\HomeHandler::class);
        $route->post('/', Handlers\HomeHandler::class);
    });
    

    Basically it has two routes. In fact both routes are the same. One accept POST request and another one GET requests.
    They’ll answer with the current date in a json file

    namespace App\Http\Handlers;
    
    class HomeHandler
    {
        public function __invoke()
        {
            return ['date' => (new \DateTime())->format('c')];
        }
    }
    

    Both routes are under one middleware to provide the authentication.

    namespace App\Http\Middleware;
    
    use Closure;
    use Illuminate\Http\Request;
    
    class AuthMiddleware
    {
        public const NAME = 'auth';
    
        public function handle(Request $request, Closure $next)
        {
            $user = $request->getUser();
            $pass = $request->getPassword();
    
            if (!$this->validateDestinationCredentials($user, $pass)) {
                $headers = ['WWW-Authenticate' => 'Basic'];
    
                return response('Backend Login', 401, $headers);
            }
    
            $authorizationHeader = $request->header('Authorization2');
            if (!$this->validateApplicationToken($authorizationHeader)) {
                return response('Invalid token ', 403);
            }
    
            return $next($request);
    
        }
    
        private function validateApplicationToken($authorizationHeader)
        {
            $token = str_replace('Bearer ', null, $authorizationHeader);
    
            return $token === getenv('APP_TOKEN');
        }
    
        private function validateDestinationCredentials($user, $pass)
        {
            if (!($user === getenv('DESTINATION_USER') && $pass === getenv('DESTINATION_PASS'))) {
                return false;
            }
    
            return true;
        }
    }
    

    That means our service will need Basic Authentication and also one Token based authentication.

    The frontend

    Our ui5 application will use one destination called BACKEND. We’ll configure it in our neo-app.json file

        ...
        {
          "path": "/backend",
          "target": {
            "type": "destination",
            "name": "BACKEND"
          },
          "description": "BACKEND"
        }
        ...
    

    Now we’ll create our extra file called destinations.json. Localneo will use this file to create a web server to serve our frontend locally (using the destination).

    As I said before our backend will need a Basic Authentication. This Authentication will be set up in the destination configuration

    {
      "server": {
        "port": "8080",
        "path": "/webapp/index.html",
        "open": true
      },
      "service": {
        "sapui5": {
          "useSAPUI5": true,
          "version": "1.54.8"
        }
      },
      "destinations": {
        "BACKEND": {
          "url": "http://localhost:8888",
          "auth": "superSecretUser:superSecretPassword"
        }
      }
    }
    

    Our application will be a simple list of items

    <mvc:View controllerName="gonzalo123.controller.App" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:mvc="sap.ui.core.mvc" displayBlock="true" xmlns="sap.m">
      <App id="idAppControl">
        <pages>
          <Page title="{i18n>appTitle}">
            <content>
              <List>
                <items>
                  <ObjectListItem id="GET" title="{i18n>get}"
                                  type="Active"
                                  press="getPressHandle">
                    <attributes>
                      <ObjectAttribute id="getCount" text="{/Data/get/count}"/>
                    </attributes>
                  </ObjectListItem>
                  <ObjectListItem id="POST" title="{i18n>post}"
                                  type="Active"
                                  press="postPressHandle">
                    <attributes>
                      <ObjectAttribute id="postCount" text="{/Data/post/count}"/>
                    </attributes>
                  </ObjectListItem>
                </items>
              </List>
            </content>
          </Page>
        </pages>
      </App>
    </mvc:View>
    

    When we click on GET we’ll perform a GET request to the backend and we’ll increment the counter. The same with POST.
    We’ll also show de date provided by the backend in a MessageToast.

    sap.ui.define([
      "sap/ui/core/mvc/Controller",
      "sap/ui/model/json/JSONModel",
      'sap/m/MessageToast',
      "gonzalo123/model/api"
    ], function (Controller, JSONModel, MessageToast, api) {
      "use strict";
    
      return Controller.extend("gonzalo123.controller.App", {
        model: new JSONModel({
          Data: {get: {count: 0}, post: {count: 0}}
        }),
    
        onInit: function () {
          this.getView().setModel(this.model);
        },
    
        getPressHandle: function () {
          api.get("/", {}).then(function (data) {
            var count = this.model.getProperty('/Data/get/count');
            MessageToast.show("Pressed : " + data.date);
            this.model.setProperty('/Data/get/count', ++count);
          }.bind(this));
        },
    
        postPressHandle: function () {
          var count = this.model.getProperty('/Data/post/count');
          api.post("/", {}).then(function (data) {
            MessageToast.show("Pressed : " + data.date);
            this.model.setProperty('/Data/post/count', ++count);
          }.bind(this));
        }
      });
    });
    

    Start our application locally

    Now we only need to start the backend

    php -S 0.0.0.0:8888 -t www

    And the frontend
    localneo

    Debugging locally

    As we’re working locally we can use local debugger in the backend and we can use breakpoints, inspect variables, etc.

    We also can debug the frontend using Chrome developer tools. We can also map our local filesystem in the browser and we can save files directly in Chrome.

    Testing

    We can test the backend using phpunit and run our tests with
    composer run test

    Here we can see a simple test of the backend

        public function testAuthorizedRequest()
        {
            $headers = [
                'Authorization2' => 'Bearer superSecretToken',
                'Content-Type'   => 'application/json',
                'Authorization'  => 'Basic ' . base64_encode('superSecretUser:superSecretPassword'),
            ];
    
            $this->json('GET', '/api', [], $headers)
                ->assertResponseStatus(200);
            $this->json('POST', '/api', [], $headers)
                ->assertResponseStatus(200);
        }
    
    
        public function testRequests()
        {
    
            $headers = [
                'Authorization2' => 'Bearer superSecretToken',
                'Content-Type'   => 'application/json',
                'Authorization'  => 'Basic ' . base64_encode('superSecretUser:superSecretPassword'),
            ];
    
            $this->json('GET', '/api', [], $headers)
                ->seeJsonStructure(['date']);
            $this->json('POST', '/api', [], $headers)
                ->seeJsonStructure(['date']);
        }
    

    We also can test the frontend using OPA5.

    As Backend is already tested we’ll mock the backend here using sinon (https://sinonjs.org/) server

    ...
        opaTest("When I click on GET the GET counter should increment by one", function (Given, When, Then) {
          Given.iStartMyApp("./integration/Test1/index.html");
          When.iClickOnGET();
          Then.getCounterShouldBeIncrementedByOne().and.iTeardownMyAppFrame();
        });
    
        opaTest("When I click on POST the POST counter should increment by one", function (Given, When, Then) {
          Given.iStartMyApp("./integration/Test1/index.html");
          When.iClickOnPOST();
          Then.postCounterShouldBeIncrementedByOne().and.iTeardownMyAppFrame();
        });
    ...
    

    The configuration of our sinon server:

    sap.ui.define(
      ["test/server"],
      function (server) {
        "use strict";
    
        return {
          init: function () {
            var oServer = server.initServer("/backend/api");
    
            oServer.respondWith("GET", /backend\/api/, [200, {
              "Content-Type": "application/json"
            }, JSON.stringify({
              "date": "2018-07-29T18:44:57+02:00"
            })]);
    
            oServer.respondWith("POST", /backend\/api/, [200, {
              "Content-Type": "application/json"
            }, JSON.stringify({
              "date": "2018-07-29T18:44:57+02:00"
            })]);
          }
        };
      }
    );
    

    The build process

    Before uploading the application to SCP we need to build it. The build process optimizes the files and creates Component-preload.js and sap-ui-cachebuster-info.json file (to ensure our users aren’t using a cached version of our application)
    We’ll use grunt to build our application. Here we can see our Gruntfile.js

    module.exports = function (grunt) {
      "use strict";
    
      require('load-grunt-tasks')(grunt);
      require('time-grunt')(grunt);
    
      grunt.config.merge({
        pkg: grunt.file.readJSON('package.json'),
        watch: {
          js: {
            files: ['Gruntfile.js', 'webapp/**/*.js', 'webapp/**/*.properties'],
            tasks: ['jshint'],
            options: {
              livereload: true
            }
          },
    
          livereload: {
            options: {
              livereload: true
            },
            files: [
              'webapp/**/*.html',
              'webapp/**/*.js',
              'webapp/**/*.css'
            ]
          }
        }
      });
    
      grunt.registerTask("default", [
        "clean",
        "lint",
        "build"
      ]);
    };
    

    In our Gruntfile I’ve also configure a watcher to build the application automatically and triggering the live reload (to reload my browser every time I change the frontend)

    Now I can build the dist folder with the command:

    grunt

    Deploy to SCP

    The deploy process is very well explained in the Holger’s repository
    Basically we need to download MTA Archive builder and extract it to ./ci/tools/mta.jar.
    Also we need SAP Cloud Platform Neo Environment SDK (./ci/tools/neo-java-web-sdk/)
    We can download those binaries from here

    Then we need to fulfill our scp credentials in ./ci/deploy-mta.properties and configure our application in ./ci/mta.yaml
    Finally we will run ./ci/deploy-mta.sh (here we can set up our scp password in order to input it within each deploy)

    Full code (frontend and backend) in my github account

    Playing with IoT, MQTT, Arduino and Raspberry Pi. Building a dashboard with OpenUI5

    I’ve been playing with MQTT in previous posts. Today I want to build a simple dashboard. Basically because I’ve got a 3.5inch display for my Raspberry Py and I want to use it. The idea is set up my Rasperry Pi as a web kiosk and display the MQTT variables in real time using websockets. Let’s start.

    Set up Raspberry Pi as a web kiosk is pretty straightforward. You only need to follow instructions detailed here. Now we will prepare the MQTT inputs. Today we’re going to reuse one example of previous post. A potentiometer controlled by a nodemcu microcontroller connected to our MQTT server via Wifi.

    We also will build another circuit using a Arduino board and a ethernet Shield.

    With this circuit we’ll register the temperature (using a LM35 temperature sensor), a photo resistor (CDS) to show the light level and a relay to switch on/off a light bulb. The Idea of the circuit is emit the temperature and light level to mosquitto mqtt server and listen to switch status form mqtt server to fire the relay. That’s the arduino code

    #include <SPI.h>
    #include <Ethernet.h>
    #include <PubSubClient.h>
    
    const int photocellPin = 1;
    const int tempPin = 0;
    const int relayPin = 9;
    bool lightStatus = false;
    
    const byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
    
    // mqtt configuration
    const char* mqttServer = "192.168.1.104";
    const int mqttPort = 1883;
    const String topicLightChange = "sensors/arduino/light/change";
    const String topicLightStatus = "sensors/arduino/light/status";
    const String topicTemp = "sensors/arduino/temperature/room1";
    const String topicLight = "sensors/arduino/light/room1";
    const char* clientName = "com.gonzalo123.arduino";
    
    EthernetClient ethClient;
    PubSubClient client(ethClient);
    
    void mqttReConnect() {
      while (!client.connected()) {
        Serial.print("Attempting MQTT connection...");
        if (client.connect(clientName)) {
          Serial.println("connected");
          client.subscribe(topicLightChange.c_str());
        } else {
          Serial.print("failed, rc=");
          Serial.print(client.state());
          Serial.println(" try again in 5 seconds");
          delay(5000);
        }
      }
    }
    
    void mqttEmit(String topic, String value) {
      if (client.publish((char*) topic.c_str(), (char*) value.c_str())) {
        //Serial.print("Publish ok (topic: ");
        //Serial.print(topic);
        //Serial.print(", value: ");
        //Serial.print(value);
        //Serial.println(")");
      } else {
        Serial.println("Publish failed");
      }
    }
    
    void callback(char* topic, byte* payload, unsigned int length) {
      Serial.print("Message arrived [");
      Serial.print(topic);
      Serial.print("] payload: ");
      String data;
      for (int i = 0; i < length; i++) {
        data += (char)payload[i];
      }
    
      if (strcmp(topic, topicLightChange.c_str()) == 0) {
        lightStatus = (data == "1") ? true : false;
        Serial.print(data);
      }
      
      Serial.println("");
    }
    
    void setup()
    {
      Serial.begin(9600);
      pinMode(relayPin, OUTPUT);
      digitalWrite(relayPin, LOW);
      
      client.setServer(mqttServer, mqttPort);
      client.setCallback(callback);
      if (Ethernet.begin(mac) == 0) {
        Serial.println("Failed to configure Ethernet using DHCP");
      }
    
      delay(1500);
    }
    
    void loop()
    {
      if (!client.connected()) {
        mqttReConnect();
      }
    
      client.loop();
    
      if (lightStatus == 1) {
        digitalWrite(relayPin, HIGH);
      } else {
        digitalWrite(relayPin, LOW);
      }
      mqttEmit(topicLightStatus, lightStatus ? "1" : "0");
      mqttEmit(topicLight, (String) analogRead(photocellPin));
      mqttEmit(topicTemp, (String) ((5.0 * analogRead(tempPin) * 100.0) / 1024.0));
    
      delay(500);
    }
    

    Now we’re going to work with dashboard. This days I’m working with OpenUI5 within various projects and because of that we’ll use this library to build the dashboard. we’ll build something like this:

    Basically it’s a view

    <mvc:View
            controllerName="gonzalo123.controller.Controller"
            height="100%"
            width="100%"
            xmlns="sap.m"
            xmlns:mvc="sap.ui.core.mvc"
            xmlns:app="http://schemas.sap.com/sapui5/extension/sap.ui.core.CustomData/1"
    >
        <IconTabBar expandable="false"
                    stretchContentHeight="true"
                    class="sapUiResponsiveContentPadding">
            <items>
                <IconTabFilter icon="sap-icon://bbyd-dashboard">
                    <TileContainer>
                        <StandardTile
                                icon="sap-icon://explorer"
                                number="{/potentiometer}"
                                numberUnit="%"
                                title="{i18n>potentiometer}"/>
                        <StandardTile
                                icon="sap-icon://temperature"
                                number="{/temperature}"
                                numberUnit="ºC"
                                title="{i18n>temperature}"/>
                        <StandardTile
                                icon="sap-icon://lightbulb"
                                number="{/light/level}"
                                title="{i18n>light}"/>
                    </TileContainer>
                </IconTabFilter>
                <IconTabFilter icon="sap-icon://lightbulb">
                    <Page showHeader="false"
                          enableScrolling="true">
                        <List>
                            <InputListItem label="{i18n>light}">
                                <Switch state="{/light/status}"
                                        change="onStatusChange"/>
                            </InputListItem>
                        </List>
                    </Page>
                </IconTabFilter>
            </items>
        </IconTabBar>
    </mvc:View>
    

    And a controller:

    sap.ui.define([
            'jquery.sap.global',
            'sap/ui/core/mvc/Controller',
            'sap/ui/model/json/JSONModel',
            "sap/ui/model/resource/ResourceModel",
            'gonzalo123/model/io'
        ],
    
        function (jQuery, Controller, JSONModel, ResourceModel, io) {
            "use strict";
    
            io.connect("//192.168.1.104:3000/");
    
            return Controller.extend("gonzalo123.controller.Controller", {
                model: new JSONModel({
                    light: {
                        status: false,
                        level: undefined
                    },
                    potentiometer: undefined,
                    temperature: undefined
                }),
    
                onInit: function () {
                    var model = this.model;
                    io.on('mqtt', function (data) {
                        switch (data.topic) {
                            case 'sensors/arduino/temperature/room1':
                                model.setProperty("/temperature", data.payload);
                                break;
                            case 'sensors/arduino/light/room1':
                                model.setProperty("/light/level", data.payload);
                                break;
                            case 'sensors/nodemcu/potentiometer/room1':
                                model.setProperty("/potentiometer", data.payload);
                                break;
                            case 'sensors/arduino/light/status':
                                model.setProperty("/light/status", data.payload == "1");
                                break;
                        }
                    });
    
                    this.getView().setModel(this.model);
    
                    var i18nModel = new ResourceModel({
                        bundleName: "gonzalo123.i18n.i18n"
                    });
    
                    this.getView().setModel(i18nModel, "i18n");
                },
    
                onStatusChange: function () {
                    io.emit('mqtt', {
                        topic: 'sensors/arduino/light/change',
                        payload: (this.getView().getModel().oData.light.status ? "1" : "0")
                    });
                }
            });
        }
    );
    

    The real time part we need a gateway between websockets and mqtt data. We’ll use socket.io. Here is the server:

    var mqtt = require('mqtt');
    var mqttClient = mqtt.connect('mqtt://192.168.1.104');
    var httpServer = require('http').createServer();
    io = require('socket.io')(httpServer, {origins: '*:*'});
    
    io.on('connection', function(client){
        client.on('mqtt', function(msg){
            console.log("ws", msg);
            mqttClient.publish(msg.topic, msg.payload.toString());
        })
    });
    
    mqttClient.on('connect', function () {
        mqttClient.subscribe('sensors/#');
    });
    
    mqttClient.on('message', function (topic, message) {
        console.log("mqtt", topic, message.toString());
        io.sockets.emit('mqtt', {
            topic: topic,
            payload: message.toString()
        });
    });
    
    httpServer.listen(3000, '0.0.0.0');
    

    Hardware

    • 1 Arduino Uno
    • 1 NodeMCU (V3)
    • 1 potentiometer
    • 1 Servo (SG90)
    • 1 Raspberry Pi 3
    • 3.5inch Display Hat for Raspberry Pi
    • LM35
    • CDS
    • pull down resistor

    Source code available in my github account

    PHP application in SAP Cloud Platform. With PostgreSQL, Redis and Cloud Foundry

    Keeping on with my study of SAP’s cloud platform (SCP) and Cloud Foundry today I’m going to build a simple PHP application. This application serves a simple Bootstrap landing page. The application uses a HTTP basic authentication. The credentials are validated against a PostgreSQL database. It also has a API to retrieve the localtimestamp from database server (just for play with a database server). I also want to play with Redis in the cloud too, so the API request will have a Time To Live (ttl) of 5 seconds. I will use a Redis service to do it.

    First we create our services in cloud foundry. I’m using the free layer of SAP cloud foundry for this example. I’m not going to explain here how to do that. It’s pretty straightforward within SAP’s coopkit. Time ago I played with IBM’s cloud foundry too. I remember that it was also very simple too.

    Then we create our application (.bp-config/options.json)

    {
    "WEBDIR": "www",
    "LIBDIR": "lib",
    "PHP_VERSION": "{PHP_70_LATEST}",
    "PHP_MODULES": ["cli"],
    "WEB_SERVER": "nginx"
    }

    If we want to use our PostgreSQL and Redis services with our PHP Appliacation we need to connect those services to our application. This operation can be done also with SAP’s Cockpit.

    Now is the turn of PHP application. I normally use Silex framework within my backends, but now there’s a problem: Silex is dead. I feel a little bit sad but I’m not going to cry. It’s just a tool and there’re another ones. I’ve got my example with Silex but, as an exercise, I will also do it with Lumen.

    Let’s start with Silex. If you’re familiar with Silex micro framework (or another microframework, indeed) you can see that there isn’t anything especial.

    use Symfony\Component\HttpKernel\Exception\HttpException;
    use Symfony\Component\HttpFoundation\Request;
    use Silex\Provider\TwigServiceProvider;
    use Silex\Application;
    use Predis\Client;
    
    if (php_sapi_name() == "cli-server") {
        // when I start the server my local machine vendors are in a different path
        require __DIR__ . '/../vendor/autoload.php';
        // and also I mock VCAP_SERVICES env
        $env   = file_get_contents(__DIR__ . "/../conf/vcap_services.json");
        $debug = true;
    } else {
        require 'vendor/autoload.php';
        $env   = $_ENV["VCAP_SERVICES"];
        $debug = false;
    }
    
    $vcapServices = json_decode($env, true);
    
    $app = new Application(['debug' => $debug, 'ttl' => 5]);
    
    $app->register(new TwigServiceProvider(), [
        'twig.path' => __DIR__ . '/../views',
    ]);
    
    $app['db'] = function () use ($vcapServices) {
        $dbConf = $vcapServices['postgresql'][0]['credentials'];
        $dsn    = "pgsql:dbname={$dbConf['dbname']};host={$dbConf['hostname']};port={$dbConf['port']}";
        $dbh    = new PDO($dsn, $dbConf['username'], $dbConf['password']);
        $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
        $dbh->setAttribute(PDO::ATTR_CASE, PDO::CASE_UPPER);
        $dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
    
        return $dbh;
    };
    
    $app['redis'] = function () use ($vcapServices) {
        $redisConf = $vcapServices['redis'][0]['credentials'];
    
        return new Client([
            'scheme'   => 'tcp',
            'host'     => $redisConf['hostname'],
            'port'     => $redisConf['port'],
            'password' => $redisConf['password'],
        ]);
    };
    
    $app->get("/", function (Application $app) {
        return $app['twig']->render('index.html.twig', [
            'user' => $app['user'],
            'ttl'  => $app['ttl'],
        ]);
    });
    
    $app->get("/timestamp", function (Application $app) {
        if (!$app['redis']->exists('timestamp')) {
            $stmt = $app['db']->prepare('SELECT localtimestamp');
            $stmt->execute();
            $app['redis']->set('timestamp', $stmt->fetch()['TIMESTAMP'], 'EX', $app['ttl']);
        }
    
        return $app->json($app['redis']->get('timestamp'));
    });
    
    $app->before(function (Request $request) use ($app) {
        $username = $request->server->get('PHP_AUTH_USER', false);
        $password = $request->server->get('PHP_AUTH_PW');
    
        $stmt = $app['db']->prepare('SELECT name, surname FROM public.user WHERE username=:USER AND pass=:PASS');
        $stmt->execute(['USER' => $username, 'PASS' => md5($password)]);
        $row = $stmt->fetch();
        if ($row !== false) {
            $app['user'] = $row;
        } else {
            header("WWW-Authenticate: Basic realm='RIS'");
            throw new HttpException(401, 'Please sign in.');
        }
    }, 0);
    
    $app->run();
    

    Maybe the only especial thing is the way that autoloader is done. We are initializing autoloader in two different ways. One way when the application is run in the cloud and another one when the application is run locally with PHP’s built-in server. That’s because vendors are located in different paths depending on which environment the application lives in. When Cloud Foundry connect services to appliations it injects environment variables with the service configuration (credentials, host, …). It uses VCAP_SERVICES env var.

    I use the built-in server to run the application locally. When I’m doing that I don’t have VCAP_SERVICES variable. And also my services information are different than when I’m running the application in the cloud. Maybe it’s better with an environment variable but I’m using this trick:

    if (php_sapi_name() == "cli-server") {
        // I'm runing the application locally
    } else {
        // I'm in the cloud
    }
    

    So when I’m locally I mock VCAP_SERVICES with my local values and also, for example, configure Silex application in debug mode.

    Sometimes I want to run my application locally but I want to use the cloud services. I cannot connect directly to those services, but we can do it over ssh through our connected application. For example If our PostgreSQL application is running on 10.11.241.0:48825 we can map this remote port (in a private network) to our local port with this command.

    cf ssh -N -T -L 48825:10.11.241.0:48825 silex
    

    You can see more information about this command here.

    Now we can use pgAdmin, for example, in our local machine to connect to cloud server.

    We can do the same with Redis

    cf ssh -N -T -L 54266:10.11.241.9:54266 silex
    

    And basically that’s all. Now we’ll do the same with Lumen. The idea is create the same application with Lumen instead of Silex. It’s a dummy application but it cover task that I normally use. I also will re-use the Redis and PostgreSQL services from the previous project.

    use App\Http\Middleware;
    use Laravel\Lumen\Application;
    use Laravel\Lumen\Routing\Router;
    use Predis\Client;
    
    if (php_sapi_name() == "cli-server") {
        require __DIR__ . '/../vendor/autoload.php';
        $env = 'dev';
    } else {
        require 'vendor/autoload.php';
        $env = 'prod';
    }
    
    (new Dotenv\Dotenv(__DIR__ . "/../env/{$env}"))->load();
    
    $app = new Application();
    
    $app->routeMiddleware([
        'auth' => Middleware\AuthMiddleware::class,
    ]);
    
    $app->register(App\Providers\VcapServiceProvider::class);
    $app->register(App\Providers\StdoutLogServiceProvider::class);
    $app->register(App\Providers\DbServiceProvider::class);
    $app->register(App\Providers\RedisServiceProvider::class);
    
    $app->router->group(['middleware' => 'auth'], function (Router $router) {
        $router->get("/", function () {
            return view("index", [
                'user' => config("user"),
                'ttl'  => getenv('TTL'),
            ]);
        });
    
        $router->get("/timestamp", function (Client $redis, PDO $conn) {
            if (!$redis->exists('timestamp')) {
                $stmt = $conn->prepare('SELECT localtimestamp');
                $stmt->execute();
                $redis->set('timestamp', $stmt->fetch()['TIMESTAMP'], 'EX', getenv('TTL'));
            }
    
            return response()->json($redis->get('timestamp'));
        });
    });
    
    $app->run();
    

    I’ve created four servicer providers. One for handle Database connections (I don’t like ORMs)

    namespace App\Providers;
    
    use Illuminate\Support\ServiceProvider;
    use PDO;
    
    class DbServiceProvider extends ServiceProvider
    {
        public function register()
        {
        }
    
        public function boot()
        {
            $vcapServices = app('vcap_services');
    
            $dbConf = $vcapServices['postgresql'][0]['credentials'];
            $dsn    = "pgsql:dbname={$dbConf['dbname']};host={$dbConf['hostname']};port={$dbConf['port']}";
            $dbh    = new PDO($dsn, $dbConf['username'], $dbConf['password']);
            $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
            $dbh->setAttribute(PDO::ATTR_CASE, PDO::CASE_UPPER);
            $dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
    
            $this->app->bind(PDO::class, function ($app) use ($dbh) {
                return $dbh;
            });
        }
    }
    

    Another one for Redis. I need to study a little bit more Lumen. I know that Lumen has a built-in tool to work with Redis.

    namespace App\Providers;
    
    use Illuminate\Support\ServiceProvider;
    use Predis\Client;
    
    class RedisServiceProvider extends ServiceProvider
    {
        public function register()
        {
        }
    
        public function boot()
        {
            $vcapServices = app('vcap_services');
            $redisConf    = $vcapServices['redis'][0]['credentials'];
    
            $redis = new Client([
                'scheme'   => 'tcp',
                'host'     => $redisConf['hostname'],
                'port'     => $redisConf['port'],
                'password' => $redisConf['password'],
            ]);
    
            $this->app->bind(Client::class, function ($app) use ($redis) {
                return $redis;
            });
        }
    }
    

    Another one to tell monolog to send logs to Stdout

    namespace App\Providers;
    
    use Illuminate\Support\ServiceProvider;
    use Monolog;
    
    class StdoutLogServiceProvider extends ServiceProvider
    {
        public function register()
        {
            app()->configureMonologUsing(function (Monolog\Logger $monolog) {
                return $monolog->pushHandler(new \Monolog\Handler\ErrorLogHandler());
            });
        }
    }
    

    And the last one to work with Vcap environment variables. Probably I need to integrate it with dotenv files

    namespace App\Providers;
    
    use Illuminate\Support\ServiceProvider;
    
    class VcapServiceProvider extends ServiceProvider
    {
        public function register()
        {
            if (php_sapi_name() == "cli-server") {
                $env = file_get_contents(__DIR__ . "/../../conf/vcap_services.json");
            } else {
                $env = $_ENV["VCAP_SERVICES"];
            }
    
            $vcapServices = json_decode($env, true);
    
            $this->app->bind('vcap_services', function ($app) use ($vcapServices) {
                return $vcapServices;
            });
        }
    }
    

    We also need to handle authentication (http basic auth in this case) so we’ll create a simple middleware

    namespace App\Http\Middleware;
    
    use Closure;
    use Illuminate\Http\Request;
    use PDO;
    
    class AuthMiddleware
    {
        public function handle(Request $request, Closure $next)
        {
            $user = $request->getUser();
            $pass = $request->getPassword();
    
            $db = app(PDO::class);
            $stmt = $db->prepare('SELECT name, surname FROM public.user WHERE username=:USER AND pass=:PASS');
            $stmt->execute(['USER' => $user, 'PASS' => md5($pass)]);
            $row = $stmt->fetch();
            if ($row !== false) {
                config(['user' => $row]);
            } else {
                $headers = ['WWW-Authenticate' => 'Basic'];
                return response('Admin Login', 401, $headers);
            }
    
            return $next($request);
        }
    }
    

    In summary: Lumen is cool. The interface is very similar to Silex. I can swap my mind from thinking in Silex to thinking in Lumen easily. Blade instead Twig: no problem. Service providers are very similar. Routing is almost the same and Middlewares are much better. Nowadays backend is a commodity for me so I don’t want to spend to much time working on it. I want something that just work. Lumen looks like that.

    Both projects: Silex and Lumen are available in my github

    Real Time IoT in the cloud with SAP’s SCP, Cloud Foundry and WebSockets

    Nowadays I’m involved with a cloud project based on SAP Cloud Platform (SCP). Side projects are the best way to mastering new technologies (at least for me) so I want to build something with SCP and my Arduino stuff. SCP comes whit one IoT module. In fact every cloud platforms have, in one way or another, one IoT module (Amazon, Azure, …). With SCP the IoT module it’s just a Hana Database where we can push our IoT values and we’re able to retrieve information via oData (the common way in SAP world).

    It’s pretty straightforward to configure the IoT module with the SAP Cloud Platform Cockpit (Every thing can be done with a hana trial account).

    NodeMcu

    First I’m going to use a simple circuit with my NodeMcu connected to my wifi network. The prototype is a potentiometer connected to the analog input. I normally use this this circuit because I can change the value just changing the potentiometer wheel. I know it’s not very usefull, but we can easily change it and use a sensor (temperature, humidity, light, …)

    It will send the percentage (from 0 to 100) of the position of the potentiometer directly to the cloud.

    #include <ESP8266WiFi.h>
    
    const int potentiometerPin = 0;
    
    // Wifi configuration
    const char* ssid = "my-wifi-ssid";
    const char* password = "my-wifi-password";
    
    // SAP SCP specific configuration
    const char* host = "mytenant.hanatrial.ondemand.com";
    String device_id = "my-device-ide";
    String message_type_id = "my-device-type-id";
    String oauth_token = "my-oauth-token";
    
    String url = "https://[mytenant].hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/data/" + device_id;
    
    const int httpsPort = 443;
    
    WiFiClientSecure clientTLS;
    
    void wifiConnect() {
      Serial.println();
      Serial.print("Connecting to ");
      Serial.println(ssid);
    
      WiFi.begin(ssid, password);
    
      while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
      }
      Serial.println("");
      Serial.print("WiFi connected.");
      Serial.print("IP address: ");
      Serial.println(WiFi.localIP());
    }
    
    void sendMessage(int value) {
      String payload = "{\"mode\":\"async\", \"messageType\":\"" + message_type_id + "\", \"messages\":[{\"value\": " + (String) value + "}]}";
      Serial.print("connecting to ");
      Serial.println(host);
      if (!clientTLS.connect(host, httpsPort)) {
        Serial.println("connection failed");
        return;
      }
    
      Serial.print("requesting payload: ");
      Serial.println(url);
    
      clientTLS.print(String("POST ") + url + " HTTP/1.0\r\n" +
                   "Host: " + host + "\r\n" +
                   "Content-Type: application/json;charset=utf-8\r\n" +
                   "Authorization: Bearer " + oauth_token + "\r\n" +
                   "Content-Length: " + payload.length() + "\r\n\r\n" +
                   payload + "\r\n\r\n");
    
      Serial.println("request sent");
    
      Serial.println("reply was:");
      while (clientTLS.connected()) {
        String line = clientTLS.readStringUntil('\n');
        Serial.println(line);
      }
    }
    
    void setup() {
      Serial.begin(9600);
      wifiConnect();
    
      delay(10);
    }
    
    int mem;
    void loop() {
    
      int value = ((analogRead(potentiometerPin) * 100) / 1010);
      if (value < (mem - 1) or value > (mem + 1)) {
        sendMessage(value);
        Serial.println(value);
        mem = value;
      }
    
      delay(200);
    }
    

    SCP

    SAP Cloud Platform allows us to create web applications using SAPUI5 framework easily. It also allows us to create a destination (the way that SAP’s cloud uses to connect different modules) to our IoT module. Also every Hana table can be accessed via oData so and we can retrieve the information easily within SAPIUI5.

    onAfterRendering: function () {
        var model = this.model;
    
        this.getView().getModel().read("/my-hana-table-odata-uri", {
            urlParameters: {
                $top: 1,
                $orderby: "G_CREATED desc"
            },
            success: function (oData) {
                model.setProperty("/value", oData.results[0].C_VALUE);
            }
        });
    }
    

    and display in a view

    <mvc:View controllerName="gonzalo123.iot.controller.Main" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:mvc="sap.ui.core.mvc"
              displayBlock="true" xmlns="sap.m">
        <App>
            <pages>
                <Page title="{i18n>title}">
                    <content>
                        <GenericTile class="sapUiTinyMarginBegin sapUiTinyMarginTop tileLayout" header="nodemcu" frameType="OneByOne">
                            <tileContent>
                                <TileContent unit="%">
                                    <content>
                                        <NumericContent value="{view>/value}" icon="sap-icon://line-charts"/>
                                    </content>
                                </TileContent>
                            </tileContent>
                        </GenericTile>
                    </content>
                </Page>
            </pages>
        </App>
    </mvc:View>
    

    Cloud Foundry

    The web application (with SCP and SAPUI5) can access to IoT values via oData. We can fetch data again and again, but that’s not cool. We want real time updates in the web application. So we need WebSockets. SCP IoT module allows us to use WebSockets to put information, but not get updates (afaik. Let me know if I’m wrong). We also can connect our IoT to an existing MQTT server, but in this prototype I only want to use websockets. So we’re going to create a simple WebSocket server with node and socket.io. This server will be listening to devices updates (again and again with a setInterval function via oData) and when it detects a change it will emit a broadcast to the WebSocket.

    SAP’s SCP also allows us to create services with Cloud Foundry. So we’ll create our nodejs server there.

    var http = require('http'),
        io = require('socket.io'),
        request = require('request'),
        auth = "Basic " + new Buffer(process.env.USER + ":" + process.env.PASS).toString("base64"),
        url = process.env.IOT_ODATA,
        INTERVAL = process.env.INTERVAL,
        socket,
        value;
    
    server = http.createServer();
    server.listen(process.env.PORT || 3000);
    
    socket = io.listen(server);
    
    setInterval(function () {
        request.get({
            url: url,
            headers: {
                "Authorization": auth,
                "Accept": "application/json"
            }
        }, function (error, response, body) {
            var newValue = JSON.parse(body).d.results[0].C_VALUE;
            if (value !== newValue) {
                value = newValue;
                socket.sockets.emit('value', value);
            }
        });
    }, INTERVAL);
    

    And that’s all. My NodeMcu device connected to the cloud.

    Full project available in my github

    Taking photos with an ionic2 application and upload them to S3 Bucket with SAP’s Cloud Foundry using Silex and Lumen

    Today I want to play with an experiment. When I work with mobile applications, I normally use ionic and on-premise backends. Today I want play with cloud based backends. In this small experiment I want to use an ionic2 application to take pictures and upload them to an S3 bucket. Let’s start.

    First I’ve created a simple ionic2 application. It’s a very simple application. Only one page with a button to trigger the device’s camera.

    <ion-header>
        <ion-navbar>
            <ion-title>
                Photo
            </ion-title>
        </ion-navbar>
    </ion-header>
    
    <ion-content padding>
        <ion-fab bottom right>
            <button ion-fab (click)="takePicture()">
                <ion-icon  name="camera"></ion-icon>
            </button>
        </ion-fab>
    </ion-content>
    

    The controller uses @ionic-native/camera to take photos and later we use @ionic-native/transfer to upload them to the backend.

    import {Component} from '@angular/core';
    import {Camera, CameraOptions} from '@ionic-native/camera';
    import {Transfer, FileUploadOptions, TransferObject} from '@ionic-native/transfer';
    import {ToastController} from 'ionic-angular';
    import {LoadingController} from 'ionic-angular';
    
    @Component({
        selector: 'page-home',
        templateUrl: 'home.html'
    })
    export class HomePage {
        constructor(private transfer: Transfer,
                    private camera: Camera,
                    public toastCtrl: ToastController,
                    public loading: LoadingController) {
        }
    
        takePicture() {
            const options: CameraOptions = {
                quality: 100,
                destinationType: this.camera.DestinationType.FILE_URI,
                sourceType: this.camera.PictureSourceType.CAMERA,
                encodingType: this.camera.EncodingType.JPEG,
                targetWidth: 1000,
                targetHeight: 1000,
                saveToPhotoAlbum: false,
                correctOrientation: true
            };
    
            this.camera.getPicture(options).then((uri) => {
                const fileTransfer: TransferObject = this.transfer.create();
    
                let options: FileUploadOptions = {
                    fileKey: 'file',
                    fileName: uri.substr(uri.lastIndexOf('/') + 1),
                    chunkedMode: true,
                    headers: {
                        Connection: "close"
                    },
                    params: {
                        metadata: {foo: 'bar'},
                        token: 'mySuperSecretToken'
                    }
                };
    
                let loader = this.loading.create({
                    content: 'Uploading ...',
                });
    
                loader.present().then(() => {
                    let s3UploadUri = 'https://myApp.cfapps.eu10.hana.ondemand.com/upload';
                    fileTransfer.upload(uri, s3UploadUri, options).then((data) => {
                        let message;
                        let response = JSON.parse(data.response);
                        if (response['status']) {
                            message = 'Picture uploaded to S3: ' + response['key']
                        } else {
                            message = 'Error Uploading to S3: ' + response['error']
                        }
                        loader.dismiss();
                        let toast = this.toastCtrl.create({
                            message: message,
                            duration: 3000
                        });
                        toast.present();
                    }, (err) => {
                        loader.dismiss();
                        let toast = this.toastCtrl.create({
                            message: "Error",
                            duration: 3000
                        });
                        toast.present();
                    });
                });
            });
        }
    }
    

    Now let’s work with the backend. Next time I’ll use JavaScript AWS SDK to upload pictures directly from mobile application (without backend), but today We’ll use a backend. Nowadays I’m involved with SAP Cloud platform projects, so we’ll use SAP’s Cloud Foundry tenant (using a free account). In this tenant we’ll create a PHP application using the PHP buildpack with nginx

    applications:
    - name:    myApp
      path: .
      memory:  128MB
      buildpack: php_buildpack
    

    The PHP application is a simple Silex application to handle the file uploads and post the pictures to S3 using the official AWS SDK for PHP (based on Guzzle)

    use Symfony\Component\HttpFoundation\Request;
    use Silex\Application;
    use Aws\S3\S3Client;
    
    require 'vendor/autoload.php';
    
    $app = new Application([
        'debug'        => false,
        'aws.config'   => [
            'debug'       => false,
            'version'     => 'latest',
            'region'      => 'eu-west-1',
            'credentials' => [
                'key'    => $_ENV['s3key'],
                'secret' => $_ENV['s3secret'],
            ],
        ],
    ]);
    
    $app['aws'] = function () use ($app) {
        return new S3Client($app['aws.config']);
    };
    
    $app->post('/upload', function (Request $request, Application $app) {
        $metadata = json_decode($request->get('metadata'), true);
        $token    = $request->get('token');
    
        if ($token === $_ENV['token']) {
            $fileName = $_FILES['file']['name'];
            $fileType = $_FILES['file']['type'];
            $tmpName  = $_FILES['file']['tmp_name'];
    
            /** @var \Aws\S3\S3Client $s3 */
            $s3 = $app['aws'];
            try {
                $key = date('YmdHis') . "_" . $fileName;
                $s3->putObject([
                    'Bucket'      => $_ENV['s3bucket'],
                    'Key'         => $key,
                    'SourceFile'  => $tmpName,
                    'ContentType' => $fileType,
                    'Metadata'    => $metadata,
                ]);
                unlink($tmpName);
    
                return $app->json([
                    'status' => true,
                    'key'    => $key,
                ]);
            } catch (Aws\S3\Exception\S3Exception $e) {
                return $app->json([
                    'status' => false,
                    'error'  => $e->getMessage(),
                ]);
            }
        } else {
            return $app->json([
                'status' => false,
                'error'  => "Token error",
            ]);
        }
    });
    
    $app->run();
    

    I just wanted a simple prototype (a working one). Enough for a Sunday morning hacking.

    UPDATE

    I had this post ready weeks ago but something has changed. Silex is dead. So, as an exercise I’ll migrate current Silex application to Lumen (a quick prototype).

    That’s the main application.

    use App\Http\Middleware;
    use Aws\S3\S3Client;
    use Illuminate\Http\Request;
    use Laravel\Lumen\Application;
    
    require 'vendor/autoload.php';
    
    (new Dotenv\Dotenv(__DIR__ . "/../env"))->load();
    
    $app = new Application();
    
    $app->routeMiddleware([
        'auth' => Middleware\AuthMiddleware::class,
    ]);
    
    $app->register(App\Providers\S3ServiceProvider::class);
    
    $app->group(['middleware' => 'auth'], function (Application $app) {
        $app->post('/upload', function (Request $request, Application $app, S3Client $s3) {
            $metadata = json_decode($request->get('metadata'), true);
            $fileName = $_FILES['file']['name'];
            $fileType = $_FILES['file']['type'];
            $tmpName  = $_FILES['file']['tmp_name'];
    
            try {
                $key = date('YmdHis') . "_" . $fileName;
                $s3->putObject([
                    'Bucket'      => getenv('s3bucket'),
                    'Key'         => $key,
                    'SourceFile'  => $tmpName,
                    'ContentType' => $fileType,
                    'Metadata'    => $metadata,
                ]);
                unlink($tmpName);
    
                return response()->json([
                    'status' => true,
                    'key'    => $key,
                ]);
            } catch (Aws\S3\Exception\S3Exception $e) {
                return response()->json([
                    'status' => false,
                    'error'  => $e->getMessage(),
                ]);
            }
        });
    });
    
    $app->run();
    

    Probably we can find a S3 Service provider, but I’ve built a simple one for this example.

    namespace App\Providers;
    
    use Illuminate\Support\ServiceProvider;
    use Aws\S3\S3Client;
    
    class S3ServiceProvider extends ServiceProvider
    {
        public function register()
        {
            $this->app->bind(S3Client::class, function ($app) {
                $conf = [
                    'debug'       => false,
                    'version'     => getenv('AWS_VERSION'),
                    'region'      => getenv('AWS_REGION'),
                    'credentials' => [
                        'key'    => getenv('s3key'),
                        'secret' => getenv('s3secret'),
                    ],
                ];
    
                return new S3Client($conf);
            });
        }
    }
    

    And also I’m using a middleware for the authentication

    namespace App\Http\Middleware;
    
    use Closure;
    use Illuminate\Http\Request;
    
    class AuthMiddleware
    {
        public function handle(Request $request, Closure $next)
        {
            $token = $request->get('token');
            if ($token === getenv('token')) {
                return response('Admin Login', 401);
            }
    
            return $next($request);
        }
    }
    

    Ok. I’ll post this article soon. At least before Lumen will be dead also, and I need to update this post again 🙂

    Full project (mobile application and both backends) in my githubgithub

    Using OpenUI5 table and Angularjs

    Last days I’ve been playing with OpenUI5. OpenUI5 is a web toolkit that SAP people has released as an open source project. I’ve read several good reviews about this framework, and because of that I started to hack a little bit with it. OpenUI5 came with a very complete set of controls. In this small example I want to use the “table” control. It’s just a datagrid. This days I playing a lot with Angular.js so I wanted to use together OpenUI5’s table control and Angularjs.

    I’m not quite sure if it’s a good decision to use together both frameworks. In fact we don’t need Angular.js to create web applications using OpenUI5. OpenUI5 uses internally jQuery, but I wanted to hack a little bit and create one Angularjs directive to enclose one OpenUI5 datagrid.

    First of all, we create one index.html. It’s just a boilerplate with angular + ui-router + ui-bootstrap. We also start our OpenUi5 environment with de default theme and including the table library

    <!doctype html>
    <html ng-app="G">
    <head>
        <meta name="viewport" content="width=device-width, initial-scale=1">
    
        <link rel="stylesheet" href="assets/bootstrap/dist/css/bootstrap.min.css">
    
        <script src="assets/angular/angular.js"></script>
        <script src="assets/angular-ui-router/release/angular-ui-router.js"></script>
        <script src="assets/angular-bootstrap/ui-bootstrap-tpls.js"></script>
    
        <script id='sap-ui-bootstrap' type='text/javascript'
                src="https://openui5.hana.ondemand.com/resources/sap-ui-core.js"
                data-sap-ui-theme='sap_bluecrystal'
                data-sap-ui-libs='sap.ui.commons, sap.ui.table'></script>
    
        <script src="js/ngOpenUI5.js"></script>
    
        <script src="js/app.js"></script>
        <link rel="stylesheet" href="css/app.css">
    </head>
    <body class="ng-cloak">
    
    <div class="container">
    
        <div class="starter-template">
            <div ui-view></div>
        </div>
    </div>
    
    <script src="assets/html5shiv/dist/html5shiv.js"></script>
    <script src="assets/respond/dest/respond.src.js"></script>
    
    </body>
    </html>
    

    Then we create a directive enclosing the OpenUI5 needed code within a Angular module

    (function () {
        'use strict';
    
        angular.module('ng.openui5', [])
            .directive('openui5Table', function () {
    
                function renderColumns(columns, oTable) {
                    for (var i = 0; i <= columns.length; i++) {
                        oTable.addColumn(new sap.ui.table.Column(columns[i]));
                    }
                }
    
                var link = function (scope, element) {
    
                    var oData = scope.model.data,
                        oTable = new sap.ui.table.Table(scope.model.conf),
                        oModel = new sap.ui.model.json.JSONModel();
    
                    oModel.setData({modelData: oData});
                    renderColumns(scope.model.columns, oTable);
    
                    oTable.setModel(oModel);
                    oTable.bindRows("/modelData");
                    oTable.sort(oTable.getColumns()[0]);
    
                    oTable.placeAt(element);
    
                    scope.$watch('model.data', function (data) {
                        if (data) {
                            oModel.setData({modelData: data});
                            oModel.refresh();
                        }
                    }, true);
    
                };
    
                return {
                    restrict: 'E',
                    scope: {model: '=ngModel'},
                    link: link
                };
            });
    }());
    

    And now we can create a simple Angular.js using the ng.openui5 module. In this application we configure the table and fetch the data from an externar API server

    (function () {
        'use strict';
    
        angular.module('G', ['ui.bootstrap', 'ui.router', 'ng.openui5'])
    
            .value('config', {
                apiUrl: '/api'
            })
    
            .config(function ($stateProvider, $urlRouterProvider) {
                $urlRouterProvider.otherwise("/");
                $stateProvider
                    .state('home', {
                        url: "/",
                        templateUrl: "partials/home.html",
                        controller: 'HomeController'
                    });
            })
    
            .controller('HomeController', function ($scope, $http, $log, config) {
                $scope.refresh = function () {
                    $http.get(config.apiUrl + '/gridData').success(function (data) {
                        $scope.datagrid.data = data;
                    });
                };
    
                $scope.datagrid = {
                    conf: {
                        title: "Table example",
                        navigationMode: sap.ui.table.NavigationMode.Paginator
                    },
                    columns: [
                        {
                            label: new sap.ui.commons.Label({text: "Last Name"}),
                            template: new sap.ui.commons.TextView().bindProperty("text", "lastName"),
                            sortProperty: "lastName",
                            filterProperty: "lastName",
                            width: "200px"
                        }, {
                            label: new sap.ui.commons.Label({text: "First Name"}),
                            template: new sap.ui.commons.TextField().bindProperty("value", "name"),
                            sortProperty: "name",
                            filterProperty: "name",
                            width: "100px"
                        }, {
                            label: new sap.ui.commons.Label({text: "Checked"}),
                            template: new sap.ui.commons.CheckBox().bindProperty("checked", "checked"),
                            sortProperty: "checked",
                            filterProperty: "checked",
                            width: "75px",
                            hAlign: "Center"
                        }, {
                            label: new sap.ui.commons.Label({text: "Web Site"}),
                            template: new sap.ui.commons.Link().bindProperty("text", "linkText").bindProperty("href", "href"),
                            sortProperty: "linkText",
                            filterProperty: "linkText"
                        }, {
                            label: new sap.ui.commons.Label({text: "Image"}),
                            template: new sap.ui.commons.Image().bindProperty("src", "src"),
                            width: "75px",
                            hAlign: "Center"
                        }, {
                            label: new sap.ui.commons.Label({text: "Gender"}),
                            template: new sap.ui.commons.ComboBox({
                                items: [
                                    new sap.ui.core.ListItem({text: "female"}),
                                    new sap.ui.core.ListItem({text: "male"})
                                ]
                            }).bindProperty("value", "gender"),
                            sortProperty: "gender",
                            filterProperty: "gender"
                        }, {
                            label: new sap.ui.commons.Label({text: "Rating"}),
                            template: new sap.ui.commons.RatingIndicator().bindProperty("value", "rating"),
                            sortProperty: "rating",
                            filterProperty: "rating"
                        }
    
                    ]
                };
            })
        ;
    }());
    

    The API server is a simple Silex server

    <?php
    include __DIR__ . '/../../vendor/autoload.php';
    use Silex\Application;
    
    $app = new Application();
    $app->get("/", function (Application $app) {
    
    $app->get('gridData', function (Application $app) {
        return $app->json([
            ['lastName' => uniqid(), 'name' => "Al", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 4, 'src' => "images/person1.gif"],
            ['lastName' => "Friese", 'name' => "Andy", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 2, 'src' => "images/person1.gif"],
            ['lastName' => "Mann", 'name' => "Anita", 'checked' => false, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 3, 'src' => "images/person1.gif"],
            ['lastName' => "Schutt", 'name' => "Doris", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 4, 'src' => "images/person1.gif"],
            ['lastName' => "Open", 'name' => "Doris", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 2, 'src' => "images/person1.gif"],
            ['lastName' => "Dewit", 'name' => "Kenya", 'checked' => false, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 3, 'src' => "images/person1.gif"],
            ['lastName' => "Zar", 'name' => "Lou", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 1, 'src' => "images/person1.gif"],
            ['lastName' => "Burr", 'name' => "Tim", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 2, 'src' => "images/person1.gif"],
            ['lastName' => "Hughes", 'name' => "Tish", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 5, 'src' => "images/person1.gif"],
            ['lastName' => "Lester", 'name' => "Mo", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 3, 'src' => "images/person1.gif"],
            ['lastName' => "Case", 'name' => "Justin", 'checked' => false, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 3, 'src' => "images/person1.gif"],
            ['lastName' => "Time", 'name' => "Justin", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 4, 'src' => "images/person1.gif"],
            ['lastName' => "Barr", 'name' => "Gaye", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 2, 'src' => "images/person1.gif"],
            ['lastName' => "Poole", 'name' => "Gene", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 1, 'src' => "images/person1.gif"],
            ['lastName' => "Ander", 'name' => "Corey", 'checked' => false, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 5, 'src' => "images/person1.gif"],
            ['lastName' => "Early", 'name' => "Brighton", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 3, 'src' => "images/person1.gif"],
            ['lastName' => "Noring", 'name' => "Constance", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 4, 'src' => "images/person1.gif"],
            ['lastName' => "Haas", 'name' => "Jack", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 2, 'src' => "images/person1.gif"],
            ['lastName' => "Tress", 'name' => "Matt", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "male", 'rating' => 4, 'src' => "images/person1.gif"],
            ['lastName' => "Turner", 'name' => "Paige", 'checked' => true, 'linkText' => "www.sap.com", 'href' => "http://www.sap.com", 'gender' => "female", 'rating' => 3, 'src' => "images/person1.gif"]
        ]);
    });
    $app->run();
    

    And basically that’s all. You can see the whole project within my github account.

    working example
    working example