Category Archives: Technology
PHP/Lumen data source for Grafana
Today I need to integrate a third party service into Grafana. I cannot access directly to the service’s database, so I will integrate via JSON datasource. Grafana allows us to build custom data sources but in this case I don’t need to create a new one. I can use the simple JSON datasource
grafana-cli plugins install grafana-simple-json-datasource
Now I need to create one REST server to serve the data that our JSON datasource needs. According to the documentation we need three routes:
- GET / should return 200 ok.
- POST /search used by the find metric options on the query tab in panels.
- POST /query should return metrics based on input.
- POST /annotations should return annotations.
We’re going to create a PHP/Lumen server. Basically the routes of the application are those ones:
<?php
use Laravel\Lumen\Routing\Router;
use App\Http\Middleware;
use Laravel\Lumen\Application;
use Dotenv\Dotenv;
use App\Http\Handlers;
require_once __DIR__ . '/../vendor/autoload.php';
(Dotenv::create(__DIR__ . '/../env/local'))->load();
$app = new Application(dirname(__DIR__));
$app->middleware([
Middleware\CorsMiddleware::class,
]);
$app->router->group(['middleware' => Middleware\AuthMiddleware::class], function (Router $router) {
$router->get('/', Handlers\HelloHandler::class);
$router->post('/search', Handlers\SearchHandler::class);
$router->post('/query', Handlers\QueryHandler::class);
$router->post('/annotations', Handlers\AnnotationHandler::class);
});
return $app;
We need to take care with CORS. I will use the Middleware that I normally use in those cases
<?php
namespace App\Http\Middleware;
use Closure;
class CorsMiddleware
{
public function handle($request, Closure $next)
{
$headers = [
'Access-Control-Allow-Origin' => 'http://localhost:3000',
'Access-Control-Allow-Methods' => 'POST, GET, OPTIONS, PUT, DELETE',
'Access-Control-Allow-Credentials' => 'true',
'Access-Control-Max-Age' => '86400',
'Access-Control-Allow-Headers' => 'accept, content-type, Content-Type, Authorization, X-Requested-With',
];
if ($request->isMethod('OPTIONS')) {
return response()->json('{"method":"OPTIONS"}', 200, $headers);
}
$response = $next($request);
foreach ($headers as $key => $value) {
$response->header($key, $value);
}
return $response;
}
}
I’ll use also a basic authentication so we’ll use a simple Http Basic Authentication middleware
<?php
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
class AuthMiddleware
{
const NAME = 'auth.web';
public function handle(Request $request, Closure $next)
{
if ($request->getUser() != env('HTTP_USER') || $request->getPassword() != env('HTTP_PASS')) {
$headers = ['WWW-Authenticate' => 'Basic'];
return response('Unauthorized', 401, $headers);
}
return $next($request);
}
}
HelloHandler is a dummy route that the datasource needs to check the connection. We only need to answer with a 200-OK
<?php
namespace App\Http\Handlers;
class HelloHandler
{
public function __invoke()
{
return "Ok";
}
}
SearchHandler will return the list of available metrics that we´ll use within our grafana panels. They aren’t strictly necessary. We can return an empty array and use later one metric that it isn’t defined here (it’s only to fill the combo that grafana shows us)
<?php
namespace App\Http\Handlers;
class SearchHandler
{
public function __invoke()
{
return [25, 50, 100];
}
}
```
QueryHandler is an important one. Here we'll return the datapoints that we´ll show in grafana. For testing purposes I've created one handler that read the metric, and the date from and date to that grafana sends to the backend and return a random values for several metrics and fixed ones to the rest. It's basically to see something in grafana. Later, in the real life project, I'll query the database and return real data.
<?php
namespace App\Http\Handlers;
use Illuminate\Http\Request;
class QueryHandler
{
public function __invoke(Request $request)
{
$json = $request->json();
$range = $json->get('range');
$target = $json->get('targets')[0]['target'];
$tz = new \DateTimeZone('Europe/Madrid');
$from = \DateTimeImmutable::createFromFormat("Y-m-d\TH:i:s.uP", $range['from'], $tz);
$to = \DateTimeImmutable::createFromFormat("Y-m-d\TH:i:s.uP", $range['to'], $tz);
return ['target' => $target, 'datapoints' => $this->getDataPoints($from, $to, $target)];
}
private function getDataPoints($from, $to, $target)
{
$interval = new \DateInterval('PT1H');
$period = new \DatePeriod($from, $interval, $to->add($interval));
$dataPoints = [];
foreach ($period as $date) {
$value = $target > 50 ? rand(0, 100) : $target;
$dataPoints[] = [$value, strtotime($date->format('Y-m-d H:i:sP')) * 1000];
}
return $dataPoints;
}
}
Also I’ll like to use annotations. It’s something similar. AnnotationHandler will handle this request. For this test I’ve created two types of annotations: One each hour and another one each 6 hours
<?php
namespace App\Http\Handlers;
use Illuminate\Http\Request;
class AnnotationHandler
{
public function __invoke(Request $request)
{
$json = $request->json();
$annotation = $json->get('annotation');
$range = $json->get('range');
return $this->getAnnotations($annotation, $range);
}
private function getAnnotations($annotation, $range)
{
return $this->getValues($range, 'PT' . $annotation['query'] . 'H');
}
private function getValues($range, $int)
{
$tz = new \DateTimeZone('Europe/Madrid');
$from = \DateTimeImmutable::createFromFormat("Y-m-d\TH:i:s.uP", $range['from'], $tz);
$to = \DateTimeImmutable::createFromFormat("Y-m-d\TH:i:s.uP", $range['to'], $tz);
$annotation = [
'name' => $int,
'enabled' => true,
'datasource' => "gonzalo datasource",
'showLine' => true,
];
$interval = new \DateInterval($int);
$period = new \DatePeriod($from, $interval, $to->add($interval));
$annotations = [];
foreach ($period as $date) {
$annotations[] = ['annotation' => $annotation, "title" => "H " . $date->format('H'), "time" => strtotime($date->format('Y-m-d H:i:sP')) * 1000, 'text' => "teeext"];
}
return $annotations;
}
}
And that’s all. I’ve also put the whole example in a docker-compose file to test it
version: '2'
services:
nginx:
image: gonzalo123.nginx
restart: always
ports:
- "80:80"
build:
context: ./src
dockerfile: .docker/Dockerfile-nginx
volumes:
- ./src/api:/code/src
api:
image: gonzalo123.api
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-lumen-dev
environment:
XDEBUG_CONFIG: remote_host=${MY_IP}
volumes:
- ./src/api:/code/src
grafana:
image: gonzalo123.grafana
build:
context: ./src
dockerfile: .docker/Dockerfile-grafana
restart: always
environment:
- GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
- GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
- GF_USERS_DEFAULT_THEME=${GF_USERS_DEFAULT_THEME}
- GF_USERS_ALLOW_SIGN_UP=${GF_USERS_ALLOW_SIGN_UP}
- GF_USERS_ALLOW_ORG_CREATE=${GF_USERS_ALLOW_ORG_CREATE}
- GF_AUTH_ANONYMOUS_ENABLED=${GF_AUTH_ANONYMOUS_ENABLED}
- GF_INSTALL_PLUGINS=${GF_INSTALL_PLUGINS}
ports:
- "3000:3000"
volumes:
- grafana-db:/var/lib/grafana
- grafana-log:/var/log/grafana
- grafana-conf:/etc/grafana
volumes:
grafana-db:
driver: local
grafana-log:
driver: local
grafana-conf:
driver: local
Here you can see the example in action:
Full code in my github
Playing with lambda, serverless and Python
Couple of weeks ago I attended to serverless course. I’ve played with lambdas from time to time (basically when AWS forced me to use them) but without knowing exactly what I was doing. After this course I know how to work with the serverless framework and I understand better lambda world. Today I want to hack a little bit and create a simple Python service to obtain random numbers. Let’s start
We don’t need Flask to create lambdas but as I’m very comfortable with it so we’ll use it here.
Basically I follow the steps that I’ve read here.
from flask import Flask
app = Flask(__name__)
@app.route("/", methods=["GET"])
def hello():
return "Hello from lambda"
if __name__ == '__main__':
app.run()
And serverless yaml to configure the service
service: random
plugins:
- serverless-wsgi
- serverless-python-requirements
- serverless-pseudo-parameters
custom:
defaultRegion: eu-central-1
defaultStage: dev
wsgi:
app: app.app
packRequirements: false
pythonRequirements:
dockerizePip: non-linux
provider:
name: aws
runtime: python3.7
region: ${opt:region, self:custom.defaultRegion}
stage: ${opt:stage, self:custom.defaultStage}
functions:
home:
handler: wsgi_handler.handler
events:
- http: GET /
We’re going to use serverless plugins. We need to install them:
npx serverless plugin install -n serverless-wsgi npx serverless plugin install -n serverless-python-requirements npx serverless plugin install -n serverless-pseudo-parameters
And that’s all. Our “Hello world” lambda service with Python and Flask is up and running.
Now We’re going to create a “more complex” service. We’re going to return a random number with random.randint function.
randint requires two parameters: start, end. We’re going to pass the end parameter to our service. The start value will be parameterized. I’ll parameterize it only because I want to play with AWS’s Parameter Store (SSM). It’s just an excuse.
Let’s start with the service:
from random import randint
from flask import Flask, jsonify
import boto3
from ssm_parameter_store import SSMParameterStore
import os
from dotenv import load_dotenv
current_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(dotenv_path="{}/.env".format(current_dir))
app = Flask(__name__)
app.config.update(
STORE=SSMParameterStore(
prefix="{}/{}".format(os.environ.get('ssm_prefix'), os.environ.get('stage')),
ssm_client=boto3.client('ssm', region_name=os.environ.get('region')),
ttl=int(os.environ.get('ssm_ttl'))
)
)
@app.route("/", methods=["GET"])
def hello():
return "Hello from lambda"
@app.route("/random/<int:to_int>", methods=["GET"])
def get_random_quote(to_int):
from_int = app.config['STORE']['from_int']
return jsonify(randint(from_int, to_int))
if __name__ == '__main__':
app.run()
Now the serverless configuration. I can use only one function, handling all routes and letting Flask do the job.
functions:
app:
handler: wsgi_handler.handler
events:
- http: ANY /
- http: 'ANY {proxy+}'
But in this example I want to create two different functions. Only for fun (and to use different role statements and different logs in cloudwatch).
service: random
plugins:
- serverless-wsgi
- serverless-python-requirements
- serverless-pseudo-parameters
- serverless-iam-roles-per-function
custom:
defaultRegion: eu-central-1
defaultStage: dev
wsgi:
app: app.app
packRequirements: false
pythonRequirements:
dockerizePip: non-linux
provider:
name: aws
runtime: python3.7
region: ${opt:region, self:custom.defaultRegion}
stage: ${opt:stage, self:custom.defaultStage}
memorySize: 128
environment:
region: ${self:provider.region}
stage: ${self:provider.stage}
functions:
app:
handler: wsgi_handler.handler
events:
- http: ANY /
- http: 'ANY {proxy+}'
iamRoleStatements:
- Effect: Allow
Action: ssm:DescribeParameters
Resource: arn:aws:ssm:${self:provider.region}:#{AWS::AccountId}:*
- Effect: Allow
Action: ssm:GetParameter
Resource: arn:aws:ssm:${self:provider.region}:#{AWS::AccountId}:parameter/random/*
home:
handler: wsgi_handler.handler
events:
- http: GET /
And that’s all. “npx serverless deploy” and my random generator is running.
Data Analysis with Python. Pivot tables with Pandas
One of the first post in my blog was about Pivot tables. I’d created a library to pivot tables in my PHP scripts. The library is not very beautiful (it throws a lot of warnings), but it works. These days I’m playing with Python Data Analysis and I’m using Pandas. The purpose of this post is something that I like a lot: Learn by doing. So I want to do the same operations that I did eight years ago in the post but now with Pandas. Let’s start.
I’ll start with the same datasource that I used almost ten years ago. One simple recordset with cliks and number of users
I create a dataframe with this data
import numpy as np
import pandas as pd
data = pd.DataFrame([
{'host': 1, 'country': 'fr', 'year': 2010, 'month': 1, 'clicks': 123, 'users': 4},
{'host': 1, 'country': 'fr', 'year': 2010, 'month': 2, 'clicks': 134, 'users': 5},
{'host': 1, 'country': 'fr', 'year': 2010, 'month': 3, 'clicks': 341, 'users': 2},
{'host': 1, 'country': 'es', 'year': 2010, 'month': 1, 'clicks': 113, 'users': 4},
{'host': 1, 'country': 'es', 'year': 2010, 'month': 2, 'clicks': 234, 'users': 5},
{'host': 1, 'country': 'es', 'year': 2010, 'month': 3, 'clicks': 421, 'users': 2},
{'host': 1, 'country': 'es', 'year': 2010, 'month': 4, 'clicks': 22, 'users': 3},
{'host': 2, 'country': 'es', 'year': 2010, 'month': 1, 'clicks': 111, 'users': 2},
{'host': 2, 'country': 'es', 'year': 2010, 'month': 2, 'clicks': 2, 'users': 4},
{'host': 3, 'country': 'es', 'year': 2010, 'month': 3, 'clicks': 34, 'users': 2},
{'host': 3, 'country': 'es', 'year': 2010, 'month': 4, 'clicks': 1, 'users': 1}
])
Now we want to do a simple pivot operation. We want to pivot on host
pd.pivot_table(data, index=['host'], values=['users', 'clicks'], columns=['year', 'month'], fill_value='' )
We can add totals
pd.pivot_table(data,
index=['host'],
values=['users', 'clicks'],
columns=['year', 'month'],
fill_value='',
aggfunc=np.sum,
margins=True,
margins_name='Total'
)
We can also pivot on more than one column. For example host and country
pd.pivot_table(data,
index=['host', 'country'],
values=['users', 'clicks'],
columns=['year', 'month'],
fill_value=''
)
and also with totals
pd.pivot_table(data,
index=['host', 'country'],
values=['users', 'clicks'],
columns=['year', 'month'],
aggfunc=np.sum,
fill_value='',
margins=True,
margins_name='Total'
)
We can group by dataframe and calculate subtotals
data.groupby(['host', 'country'])[('clicks', 'users')].sum()
data.groupby(['host', 'country'])[('clicks', 'users')].mean()
And finally we can mix totals and subtotals.
out = data.groupby('host').apply(lambda sub: sub.pivot_table(
index=['host', 'country'],
values=['users', 'clicks'],
columns=['year', 'month'],
aggfunc=np.sum,
margins=True,
margins_name='SubTotal',
))
out.loc[('', 'Max', '')] = out.max()
out.loc[('', 'Min', '')] = out.min()
out.loc[('', 'Total', '')] = out.sum()
out.index = out.index.droplevel(0)
out.fillna('', inplace=True)
And that’s all. A lot of to learn yet about data analysis, but Pandas will be definitely a good friend of mine.
You can see the Jupiter notebook in my github account
Monitoring the bandwidth (part 2) now with Python Nameko microservice
This days I’ve been playing with Nameko. The Python framework for building microservices. Today I want to upgrade one small pet project that I’ve got in my house to monitor the bandwidth of my internet connection. I want to use one nameko microservice using the Timer entrypoint.
That’s the worker:
from nameko.timer import timer
import datetime
import logging
import os
import speedtest
from dotenv import load_dotenv
from influxdb import InfluxDBClient
current_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(dotenv_path="{}/.env".format(current_dir))
class SpeedService:
name = "speed"
def __init__(self):
self.influx_client = InfluxDBClient(
host=os.getenv("INFLUXDB_HOST"),
port=os.getenv("INFLUXDB_PORT"),
database=os.getenv("INFLUXDB_DATABASE")
)
@timer(interval=3600)
def speedTest(self):
logging.info("speedTest")
current_time = datetime.datetime.utcnow().isoformat()
speed = self.get_speed()
self.persists(measurement='download', fields={"value": speed['download']}, time=current_time)
self.persists(measurement='upload', fields={"value": speed['upload']}, time=current_time)
self.persists(measurement='ping', fields={"value": speed['ping']}, time=current_time)
def persists(self, measurement, fields, time):
logging.info("{} {} {}".format(time, measurement, fields))
self.influx_client.write_points([{
"measurement": measurement,
"time": time,
"fields": fields
}])
@staticmethod
def get_speed():
logging.info("Calculating speed ...")
s = speedtest.Speedtest()
s.get_best_server()
s.download()
s.upload()
return s.results.dict()
I need to adapt my docker-compose file to include the RabbitMQ server (Nameko needs a RabbitMQ message broker)
version: '3'
services:
speed.worker:
container_name: speed.worker
image: speed/worker
restart: always
build:
context: ./src/speed.worker
dockerfile: .docker/Dockerfile-worker
command: /bin/bash run.sh
rabbit:
container_name: speed.rabbit
image: rabbitmq:3-management
restart: always
ports:
- "15672:15672"
- "5672:5672"
environment:
RABBITMQ_ERLANG_COOKIE:
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
influxdb:
container_name: speed.influxdb
image: influxdb:latest
restart: always
environment:
- INFLUXDB_DB=${INFLUXDB_DB}
- INFLUXDB_ADMIN_USER=${INFLUXDB_ADMIN_USER}
- INFLUXDB_ADMIN_PASSWORD=${INFLUXDB_ADMIN_PASSWORD}
- INFLUXDB_HTTP_AUTH_ENABLED=${INFLUXDB_HTTP_AUTH_ENABLED}
volumes:
- influxdb-data:/data
grafana:
container_name: speed.grafana
build:
context: ./src/grafana
dockerfile: .docker/Dockerfile-grafana
restart: always
environment:
- GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
- GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
- GF_USERS_DEFAULT_THEME=${GF_USERS_DEFAULT_THEME}
- GF_USERS_ALLOW_SIGN_UP=${GF_USERS_ALLOW_SIGN_UP}
- GF_USERS_ALLOW_ORG_CREATE=${GF_USERS_ALLOW_ORG_CREATE}
- GF_AUTH_ANONYMOUS_ENABLED=${GF_AUTH_ANONYMOUS_ENABLED}
ports:
- "3000:3000"
depends_on:
- influxdb
volumes:
influxdb-data:
driver: local
And that’s all. Over engineering to control my Internet Connection? Maybe, but that’s the way I learn new stuff 🙂
Source code available in my github.
Using cache buster with OpenUI5 outside SCP
When we work with SPAs and web applications we need to handle with the browser’s cache. Sometimes we change our static files but the client’s browser uses a cached version of the file instead of the new one. We can tell the user: Please empty your cache to use the new version. But most of the times the user don’t know what we’re speaking about, and we have a problem. There’s a technique called cache buster used to bypass this issue. It consists on to change the name of the file (or adding an extra parameter), basically to ensure that the browser will send a different request to the server to prevent the browser from reusing the cached version of the file.
When we work with sapui5 application over SCP, we only need to use the cachebuster version of sap-ui-core
<script id="sap-ui-bootstrap"
src="https://sapui5.hana.ondemand.com/resources/sap-ui-cachebuster/sap-ui-core.js"
data-sap-ui-libs="sap.m"
data-sap-ui-theme="sap_belize"
data-sap-ui-compatVersion="edge"
data-sap-ui-appCacheBuster=""
data-sap-ui-preload="async"
data-sap-ui-resourceroots='{"app": ""}'
data-sap-ui-frameOptions="trusted">
</script>
With this configuration, our framework will use a “cache buster friendly” version of our files and SCP will serve them properly.
For example, when our framework wants the /dist/Component.js file, the browser will request /dist/~1541685070813~/Component.js to the server. And the server will server the file /dist/Component.js. As I said before when we work with SCP, our standard build process automatically takes care about it. It creates a file called sap-ui-cachebuster-info.json where we can find all our files with one kind of hash that our build process changes each time our file is changed.
{
"Component-dbg.js": 1545316733136,
"Component-preload.js": 1545316733226,
"Component.js": 1541685070813,
...
}
It works like a charm but I not always use SCP. Sometimes I use OpenUI5 in one nginx server, for example. So cache buster “doesn’t work”. That’s a problem because I need to handle with browser caches again each time we deploy the new version of the application. I wanted to solve the issue. Let me explain how I did it.
Since I was using one Lumen/PHP server to the backend, my first idea was to create a dynamic route in Lumen to handle cache-buster urls. With this approach I know I can solve the problem but there’s something that I did not like: I’ll use a dynamic server to serve static content. I don’t have a huge traffic. I can use this approach but it isn’t beautiful.
My second approach was: Ok I’ve got a sap-ui-cachebuster-info.json file where I can see all the files that cache buster will use and their hashes. So, Why not I create those files in my build script. With this approach I will create the full static structure each time I deploy de application, without needing any server side scripting language to generate dynamic content. OpenUI5 uses grunt so I can create a simple grunt task to create my files.
'use strict';
var fs = require('fs');
var path = require('path');
var chalk = require('chalk');
module.exports = function(grunt) {
var name = 'cacheBuster';
var info = 'Generates Cache buster files';
var cacheBuster = function() {
var config = grunt.config.get(name);
var data, t, src, dest, dir, prop;
data = grunt.file.readJSON(config.src + '/sap-ui-cachebuster-info.json');
for (prop in data) {
if (data.hasOwnProperty(prop)) {
t = data[prop];
src = config.src + '/' + prop;
dest = config.src + '/~' + t + '~/' + prop;
grunt.verbose.writeln(
name + ': ' + chalk.cyan(path.basename(src)) + ' to ' +
chalk.cyan(dest) + '.');
dir = path.dirname(dest);
grunt.file.mkdir(dir);
fs.copyFileSync(src, dest);
}
}
};
grunt.registerMultiTask(name, info, cacheBuster);
};
I deploy my grunt task to npm so when I need to use it I only need to:
Install the task
npm install gonzalo123-cachebuster
Add the task to my gruntfile
require('gonzalo123-cachebuster')(grunt);
and set up the path where ui5 task generates our dist files
grunt.config.merge({
pkg: grunt.file.readJSON('package.json'),
...
cacheBuster: {
src: 'dist'
}
});
And that’s all. My users with enjoy (or suffer) the new versions of my applications without having problems with cached files.
Grunt task available in my github
Playing with microservices, Docker, Python an Nameko
In the last projects that I’ve been involved with I’ve playing, in one way or another, with microservices, queues and things like that. I’m always facing the same tasks: Building RPCs, Workers, API gateways, … Because of that I’ve searching one framework to help me with those kind of stuff. Finally I discover Nameko. Basically Nameko is the Python tool that I’ve been looking for. In this post I will create a simple proof of concept to learn how to integrate Nameko within my projects. Let start.
The POC is a simple API gateway that gives me the localtime in iso format. I can create a simple Python script to do it
import datetime import time print(datetime.datetime.fromtimestamp(time()).isoformat())
We also can create a simple Flask API server to consume this information. The idea is create a rpc worker to generate this information and also generate another worker to send the localtime, but taken from a PostgreSQL database (yes I know it not very useful but it’s just an excuse to use a PG database in the microservice)
We’re going to create two rpc workers. One giving the local time:
from nameko.rpc import rpc
from time import time
import datetime
class TimeService:
name = "local_time_service"
@rpc
def local(self):
return datetime.datetime.fromtimestamp(time()).isoformat()
And another one with the date from PostgreSQL:
from nameko.rpc import rpc
from dotenv import load_dotenv
import os
from ext.pg import PgService
current_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(dotenv_path="{}/.env".format(current_dir))
class TimeService:
name = "db_time_service"
conn = PgService(os.getenv('DSN'))
@rpc
def db(self):
with self.conn:
with self.conn.cursor() as cur:
cur.execute("select localtimestamp")
timestamp = cur.fetchone()
return timestamp[0]
I’ve created a service called PgService only to learn how to create dependency providers in nameko
from nameko.extensions import DependencyProvider
import psycopg2
class PgService(DependencyProvider):
def __init__(self, dsn):
self.dsn = dsn
def get_dependency(self, worker_ctx):
return psycopg2.connect(self.dsn)
Now we only need to setup the api gateway. With Nameko we can create http entrypoint also (in the same way than we create rpc) but I want to use it with Flask
from flask import Flask
from nameko.standalone.rpc import ServiceRpcProxy
from dotenv import load_dotenv
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
load_dotenv(dotenv_path="{}/.env".format(current_dir))
app = Flask(__name__)
def rpc_proxy(service):
config = {'AMQP_URI': os.getenv('AMQP_URI')}
return ServiceRpcProxy(service, config)
@app.route('/')
def hello():
return "Hello"
@app.route('/local')
def local_time():
with rpc_proxy('local_time_service') as rpc:
time = rpc.local()
return time
@app.route('/db')
def db_time():
with rpc_proxy('db_time_service') as rpc:
time = rpc.db()
return time
if __name__ == '__main__':
app.run()
As well as I wanna run my POC with docker, here the docker-compose file to set up the project
version: '3.4'
services:
api:
image: nameko/api
container_name: nameko.api
hostname: api
ports:
- "8080:8080"
restart: always
links:
- rabbit
- db.worker
- local.worker
environment:
- ENV=1
- FLASK_APP=app.py
- FLASK_DEBUG=1
build:
context: ./api
dockerfile: .docker/Dockerfile-api
#volumes:
#- ./api:/usr/src/app:ro
command: flask run --host=0.0.0.0 --port 8080
db.worker:
container_name: nameko.db.worker
image: nameko/db.worker
restart: always
build:
context: ./workers/db.worker
dockerfile: .docker/Dockerfile-worker
command: /bin/bash run.sh
local.worker:
container_name: nameko.local.worker
image: nameko/local.worker
restart: always
build:
context: ./workers/local.worker
dockerfile: .docker/Dockerfile-worker
command: /bin/bash run.sh
rabbit:
container_name: nameko.rabbit
image: rabbitmq:3-management
restart: always
ports:
- "15672:15672"
- "5672:5672"
environment:
RABBITMQ_ERLANG_COOKIE:
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
pg:
container_name: nameko.pg
image: nameko/pg
restart: always
build:
context: ./pg
dockerfile: .docker/Dockerfile-pg
#ports:
#- "5432:5432"
environment:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
PGDATA: /var/lib/postgresql/data/pgdata
And that’s all. Two nameko rpc services working together behind a api gateway
Code available in my github
Monitoring the bandwidth with Grafana, InfluxDB and Docker
Time ago, when I was an ADSL user in my house I had a lot problems with my internet connection. I was a bit lazy to switch to a fiber connection. Finally I changed it, but meanwhile the my Internet company was solving one incident, I started to hack a little bit a simple and dirty script that monitors my connection speed (just for fun and to practise with InfluxDB and Grafana).
Today I’ve lost my quick and dirty script (please Gonzalo keep a working backup the SD card of your Raspberry Pi Server always updated! Sometimes it crashes. It’s simple: “dd if=/dev/disk3 of=pi3.img” 🙂 and I want to rebuild it. This time I want to use Docker (just for fun). Let’s start.
To monitor the bandwidth we only need to use the speedtest-cli api. We can use this api from command line and, as it’s a python library, we can create one python script that uses it.
import datetime import logging import os import speedtest import time from dotenv import load_dotenv from influxdb import InfluxDBClient logging.basicConfig(level=logging.INFO) current_dir = os.path.dirname(os.path.abspath(__file__)) load_dotenv(dotenv_path="{}/.env".format(current_dir)) influxdb_host = os.getenv("INFLUXDB_HOST") influxdb_port = os.getenv("INFLUXDB_PORT") influxdb_database = os.getenv("INFLUXDB_DATABASE") def persists(measurement, fields, time): logging.info("{} {} {}".format(time, measurement, fields)) influx_client.write_points([{ "measurement": measurement, "time": time, "fields": fields }]) influx_client = InfluxDBClient(host=influxdb_host, port=influxdb_port, database=influxdb_database) def get_speed(): logging.info("Calculating speed ...") s = speedtest.Speedtest() s.get_best_server() s.download() s.upload() return s.results.dict() def loop(sleep): current_time = datetime.datetime.utcnow().isoformat() speed = get_speed() persists(measurement='download', fields={"value": speed['download']}, time=current_time) persists(measurement='upload', fields={"value": speed['upload']}, time=current_time) persists(measurement='ping', fields={"value": speed['ping']}, time=current_time) time.sleep(sleep) while True: loop(sleep=60 * 60) # each hourNow we need to create the docker-compose file to orchestrate the infrastructure. The most complicate thing here is, maybe, to configure grafana within docker files instead of opening browser, create datasoruce and build dashboard by hand. After a couple of hours navigating into github repositories finally I created exactly what I needed for this post. Basically is a custom entry point for my grafana host that creates the datasource and dashboard (via Grafana’s API)
version: '3' services: check: image: gonzalo123.check restart: always volumes: - ./src/beat:/code/src depends_on: - influxdb build: context: ./src dockerfile: .docker/Dockerfile-check networks: - app-network command: /bin/sh start.sh influxdb: image: influxdb:latest restart: always environment: - INFLUXDB_INIT_PWD="${INFLUXDB_PASS}" - PRE_CREATE_DB="${INFLUXDB_DB}" volumes: - influxdb-data:/data networks: - app-network grafana: image: grafana/grafana:latest restart: always ports: - "3000:3000" depends_on: - influxdb volumes: - grafana-db:/var/lib/grafana - grafana-log:/var/log/grafana - grafana-conf:/etc/grafana networks: - app-network networks: app-network: driver: bridge volumes: grafana-db: driver: local grafana-log: driver: local grafana-conf: driver: local influxdb-data: driver: localAnd that’s all. My Internet connection supervised again.
Project available in my github.
Working with SAPUI5 locally (part 3). Adding more services in Docker
In the previous project we moved one project to docker. The idea was to move exactly the same functionality (even without touching anything within the source code). Now we’re going to add more services. Yes, I know, it looks like overenginering (it’s exactly overenginering, indeed), but I want to build something with different services working together. Let start.
We’re going to change a little bit our original project. Now our frontend will only have one button. This button will increment the number of clicks but we’re going to persists this information in a PostgreSQL database. Also, instead of incrementing the counter in the backend, our backend will emit one event to a RabbitMQ message broker. We’ll have one worker service listening to this event and this worker will persist the information. The communication between the worker and the frontend (to show the incremented value), will be via websockets.
With those premises we are going to need:
- Frontend: UI5 application
- Backend: PHP/lumen application
- Worker: nodejs application which is listening to a RabbitMQ event and serving the websocket server (using socket.io)
- Nginx server
- PosgreSQL database.
- RabbitMQ message broker.
As the previous examples, our PHP backend will be server via Nginx and PHP-FPM.
Here we can see to docker-compose file to set up all the services
version: '3.4'
services:
nginx:
image: gonzalo123.nginx
restart: always
ports:
- "8080:80"
build:
context: ./src
dockerfile: .docker/Dockerfile-nginx
volumes:
- ./src/backend:/code/src
- ./src/.docker/web/site.conf:/etc/nginx/conf.d/default.conf
networks:
- app-network
api:
image: gonzalo123.api
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-lumen-dev
environment:
XDEBUG_CONFIG: remote_host=${MY_IP}
volumes:
- ./src/backend:/code/src
networks:
- app-network
ui5:
image: gonzalo123.ui5
ports:
- "8000:8000"
restart: always
volumes:
- ./src/frontend:/code/src
build:
context: ./src
dockerfile: .docker/Dockerfile-ui5
networks:
- app-network
io:
image: gonzalo123.io
ports:
- "9999:9999"
restart: always
volumes:
- ./src/io:/code/src
build:
context: ./src
dockerfile: .docker/Dockerfile-io
networks:
- app-network
pg:
image: gonzalo123.pg
restart: always
ports:
- "5432:5432"
build:
context: ./src
dockerfile: .docker/Dockerfile-pg
environment:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
PGDATA: /var/lib/postgresql/data/pgdata
networks:
- app-network
rabbit:
image: rabbitmq:3-management
container_name: gonzalo123.rabbit
restart: always
ports:
- "15672:15672"
- "5672:5672"
environment:
RABBITMQ_ERLANG_COOKIE:
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
networks:
- app-network
networks:
app-network:
driver: bridge
We’re going to use the same docker files than in the previous post but we also need new ones for worker, database server and message queue:
Worker:
FROM node:alpine EXPOSE 8000 WORKDIR /code/src COPY ./io . RUN npm install ENTRYPOINT ["npm", "run", "serve"]
The worker script is simple script that serves the socket.io server and emits a websocket within every message to the RabbitMQ queue.
var amqp = require('amqp'),
httpServer = require('http').createServer(),
io = require('socket.io')(httpServer, {
origins: '*:*',
}),
pg = require('pg')
;
require('dotenv').config();
var pgClient = new pg.Client(process.env.DB_DSN);
rabbitMq = amqp.createConnection({
host: process.env.RABBIT_HOST,
port: process.env.RABBIT_PORT,
login: process.env.RABBIT_USER,
password: process.env.RABBIT_PASS,
});
var sql = 'SELECT clickCount FROM docker.clicks';
// Please don't do this. Use lazy connections
// I'm 'lazy' to do it in this POC 🙂
pgClient.connect(function(err) {
io.on('connection', function() {
pgClient.query(sql, function(err, result) {
var count = result.rows[0]['clickcount'];
io.emit('click', {count: count});
});
});
rabbitMq.on('ready', function() {
var queue = rabbitMq.queue('ui5');
queue.bind('#');
queue.subscribe(function(message) {
pgClient.query(sql, function(err, result) {
var count = parseInt(result.rows[0]['clickcount']);
count = count + parseInt(message.data.toString('utf8'));
pgClient.query('UPDATE docker.clicks SET clickCount = $1', [count],
function(err) {
io.emit('click', {count: count});
});
});
});
});
});
httpServer.listen(process.env.IO_PORT);
Database server:
FROM postgres:9.6-alpine COPY pg/init.sql /docker-entrypoint-initdb.d/
As we can see we’re going to generate the database estructure in the first build
CREATE SCHEMA docker; CREATE TABLE docker.clicks ( clickCount numeric(8) NOT NULL ); ALTER TABLE docker.clicks OWNER TO username; INSERT INTO docker.clicks(clickCount) values (0);
With the RabbitMQ server we’re going to use the official docker image so we don’t need to create one Dockerfile
We also have changed a little bit our Nginx configuration. We want to use Nginx to serve backend and also socket.io server. That’s because we don’t want to expose different ports to internet.
server {
listen 80;
index index.php index.html;
server_name localhost;
error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;
root /code/src/www;
location /socket.io/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass "http://io:9999";
}
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass api:9000;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
}
}
To avoid CORS issues we can also use SCP destination (the localneo proxy in this example), to serve socket.io also. So we need to:
"routes": [
...
{
"path": "/socket.io",
"target": {
"type": "destination",
"name": "SOCKETIO"
},
"description": "SOCKETIO"
}
],
And basically that’s all. Here also we can use a “production” docker-copose file without exposing all ports and mapping the filesystem to our local machine (useful when we’re developing)
version: '3.4'
services:
nginx:
image: gonzalo123.nginx
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-nginx
networks:
- app-network
api:
image: gonzalo123.api
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-lumen
networks:
- app-network
ui5:
image: gonzalo123.ui5
ports:
- "80:8000"
restart: always
volumes:
- ./src/frontend:/code/src
build:
context: ./src
dockerfile: .docker/Dockerfile-ui5
networks:
- app-network
io:
image: gonzalo123.io
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-io
networks:
- app-network
pg:
image: gonzalo123.pg
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-pg
environment:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
PGDATA: /var/lib/postgresql/data/pgdata
networks:
- app-network
rabbit:
image: rabbitmq:3-management
restart: always
environment:
RABBITMQ_ERLANG_COOKIE:
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
networks:
- app-network
networks:
app-network:
driver: bridge
And that’s all. The full project is available in my github account
Working with SAPUI5 locally (part 2). Now with docker
In the first part I spoke about how to build our working environment to work with UI5 locally instead of using WebIDE. Now, in this second part of the post, we’ll see how to do it using docker to set up our environment.
I’ll use docker-compose to set up the project. Basically, as I explain in the first part, the project has two parts. One backend and one frontned. We’re going to use exactly the same code for the frontend and for the backend.
The frontend is build over a localneo. As it’s a node application we’ll use a node:alpine base host
FROM node:alpine EXPOSE 8000 WORKDIR /code/src COPY ./frontend . RUN npm install ENTRYPOINT ["npm", "run", "serve"]
In docker-compose we only need to map the port that we´ll expose in our host and since we want this project in our depelopemet process, we also will map the volume to avoid to re-generate our container each time we change the code.
...
ui5:
image: gonzalo123.ui5
ports:
- "8000:8000"
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-ui5
volumes:
- ./src/frontend:/code/src
networks:
- api-network
The backend is a PHP application. We can set up a PHP application using different architectures. In this project we’ll use nginx and PHP-FPM.
for nginx we’ll use the following Dockerfile
FROM nginx:1.13-alpine EXPOSE 80 COPY ./.docker/web/site.conf /etc/nginx/conf.d/default.conf COPY ./backend /code/src
And for the PHP host the following one (with xdebug to enable debugging and breakpoints):
FROM php:7.1-fpm
ENV PHP_XDEBUG_REMOTE_ENABLE 1
RUN apt-get update && apt-get install -my \
git \
libghc-zlib-dev && \
apt-get clean
RUN apt-get install -y libpq-dev \
&& docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql \
&& docker-php-ext-install pdo pdo_pgsql pgsql opcache zip
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
RUN composer global require "laravel/lumen-installer"
ENV PATH ~/.composer/vendor/bin:$PATH
COPY ./backend /code/src
And basically that’s all. Here the full docker-compose file
version: '3.4'
services:
nginx:
image: gonzalo123.nginx
restart: always
ports:
- "8080:80"
build:
context: ./src
dockerfile: .docker/Dockerfile-nginx
volumes:
- ./src/backend:/code/src
- ./src/.docker/web/site.conf:/etc/nginx/conf.d/default.conf
networks:
- api-network
api:
image: gonzalo123.api
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-lumen-dev
environment:
XDEBUG_CONFIG: remote_host=${MY_IP}
volumes:
- ./src/backend:/code/src
networks:
- api-network
ui5:
image: gonzalo123.ui5
ports:
- "8000:8000"
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-ui5
networks:
- api-network
networks:
api-network:
driver: bridge
If we want to use this project you only need to:
- clone the repo fron github
- run ./ui5 up
With this configuration we’re exposing two ports 8080 for the frontend and 8000 for the backend. We also are mapping our local filesystem to containers to avoid to regenerate our containers each time we change the code.
We also can have a variation. A “production” version of our docker-compose file. I put production between quotation marks because normally we aren’t going to use localneo as a production server (please don’t do it). We’ll use SCP to host the frontend.
This configuration is just an example without filesystem mapping, without xdebug in the backend and without exposing the backend externally (Only the frontend can use it)
version: '3.4'
services:
nginx:
image: gonzalo123.nginx
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-nginx
networks:
- api-network
api:
image: gonzalo123.api
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-lumen
networks:
- api-network
ui5:
image: gonzalo123.ui5
ports:
- "8000:8000"
restart: always
build:
context: ./src
dockerfile: .docker/Dockerfile-ui5
networks:
- api-network
networks:
api-network:
driver: bridge
And that’s all. You can see the all the source code in my github account
Working with SAPUI5 locally and deploying in SCP
When I work with SAPUI5 projects I normally use WebIDE. WebIDE is a great tool but I’m more confortable working locally with my local IDE.
I’ve this idea in my mind but I never find the time slot to work on it. Finally, after finding this project from Holger Schäfer in github, I realized how easy it’s and I started to work with this project and adapt it to my needs.
The base of this project is localneo. Localneo starts a http server based on neo-app.json file. That means we’re going to use the same configuration than we’ve in production (in SCP). Of course we’ll need destinations. We only need one extra file called destination.json where we’ll set up our destinations (it creates one http proxy, nothing else).
In this project I’ll create a simple example application that works with one API server.
The backend
I’ll use in this example one PHP/Lumen application:
$app->router->group(['prefix' => '/api', 'middleware' => Middleware\AuthMiddleware::NAME], function (Router $route) {
$route->get('/', Handlers\HomeHandler::class);
$route->post('/', Handlers\HomeHandler::class);
});
Basically it has two routes. In fact both routes are the same. One accept POST request and another one GET requests.
They’ll answer with the current date in a json file
namespace App\Http\Handlers;
class HomeHandler
{
public function __invoke()
{
return ['date' => (new \DateTime())->format('c')];
}
}
Both routes are under one middleware to provide the authentication.
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
class AuthMiddleware
{
public const NAME = 'auth';
public function handle(Request $request, Closure $next)
{
$user = $request->getUser();
$pass = $request->getPassword();
if (!$this->validateDestinationCredentials($user, $pass)) {
$headers = ['WWW-Authenticate' => 'Basic'];
return response('Backend Login', 401, $headers);
}
$authorizationHeader = $request->header('Authorization2');
if (!$this->validateApplicationToken($authorizationHeader)) {
return response('Invalid token ', 403);
}
return $next($request);
}
private function validateApplicationToken($authorizationHeader)
{
$token = str_replace('Bearer ', null, $authorizationHeader);
return $token === getenv('APP_TOKEN');
}
private function validateDestinationCredentials($user, $pass)
{
if (!($user === getenv('DESTINATION_USER') && $pass === getenv('DESTINATION_PASS'))) {
return false;
}
return true;
}
}
That means our service will need Basic Authentication and also one Token based authentication.
The frontend
Our ui5 application will use one destination called BACKEND. We’ll configure it in our neo-app.json file
...
{
"path": "/backend",
"target": {
"type": "destination",
"name": "BACKEND"
},
"description": "BACKEND"
}
...
Now we’ll create our extra file called destinations.json. Localneo will use this file to create a web server to serve our frontend locally (using the destination).
As I said before our backend will need a Basic Authentication. This Authentication will be set up in the destination configuration
{
"server": {
"port": "8080",
"path": "/webapp/index.html",
"open": true
},
"service": {
"sapui5": {
"useSAPUI5": true,
"version": "1.54.8"
}
},
"destinations": {
"BACKEND": {
"url": "http://localhost:8888",
"auth": "superSecretUser:superSecretPassword"
}
}
}
Our application will be a simple list of items
<mvc:View controllerName="gonzalo123.controller.App" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:mvc="sap.ui.core.mvc" displayBlock="true" xmlns="sap.m">
<App id="idAppControl">
<pages>
<Page title="{i18n>appTitle}">
<content>
<List>
<items>
<ObjectListItem id="GET" title="{i18n>get}"
type="Active"
press="getPressHandle">
<attributes>
<ObjectAttribute id="getCount" text="{/Data/get/count}"/>
</attributes>
</ObjectListItem>
<ObjectListItem id="POST" title="{i18n>post}"
type="Active"
press="postPressHandle">
<attributes>
<ObjectAttribute id="postCount" text="{/Data/post/count}"/>
</attributes>
</ObjectListItem>
</items>
</List>
</content>
</Page>
</pages>
</App>
</mvc:View>
When we click on GET we’ll perform a GET request to the backend and we’ll increment the counter. The same with POST.
We’ll also show de date provided by the backend in a MessageToast.
sap.ui.define([
"sap/ui/core/mvc/Controller",
"sap/ui/model/json/JSONModel",
'sap/m/MessageToast',
"gonzalo123/model/api"
], function (Controller, JSONModel, MessageToast, api) {
"use strict";
return Controller.extend("gonzalo123.controller.App", {
model: new JSONModel({
Data: {get: {count: 0}, post: {count: 0}}
}),
onInit: function () {
this.getView().setModel(this.model);
},
getPressHandle: function () {
api.get("/", {}).then(function (data) {
var count = this.model.getProperty('/Data/get/count');
MessageToast.show("Pressed : " + data.date);
this.model.setProperty('/Data/get/count', ++count);
}.bind(this));
},
postPressHandle: function () {
var count = this.model.getProperty('/Data/post/count');
api.post("/", {}).then(function (data) {
MessageToast.show("Pressed : " + data.date);
this.model.setProperty('/Data/post/count', ++count);
}.bind(this));
}
});
});
Start our application locally
Now we only need to start the backend
php -S 0.0.0.0:8888 -t www
And the frontend
localneo
Debugging locally
As we’re working locally we can use local debugger in the backend and we can use breakpoints, inspect variables, etc.
We also can debug the frontend using Chrome developer tools. We can also map our local filesystem in the browser and we can save files directly in Chrome.
Testing
We can test the backend using phpunit and run our tests with
composer run test
Here we can see a simple test of the backend
public function testAuthorizedRequest()
{
$headers = [
'Authorization2' => 'Bearer superSecretToken',
'Content-Type' => 'application/json',
'Authorization' => 'Basic ' . base64_encode('superSecretUser:superSecretPassword'),
];
$this->json('GET', '/api', [], $headers)
->assertResponseStatus(200);
$this->json('POST', '/api', [], $headers)
->assertResponseStatus(200);
}
public function testRequests()
{
$headers = [
'Authorization2' => 'Bearer superSecretToken',
'Content-Type' => 'application/json',
'Authorization' => 'Basic ' . base64_encode('superSecretUser:superSecretPassword'),
];
$this->json('GET', '/api', [], $headers)
->seeJsonStructure(['date']);
$this->json('POST', '/api', [], $headers)
->seeJsonStructure(['date']);
}
We also can test the frontend using OPA5.
As Backend is already tested we’ll mock the backend here using sinon (https://sinonjs.org/) server
...
opaTest("When I click on GET the GET counter should increment by one", function (Given, When, Then) {
Given.iStartMyApp("./integration/Test1/index.html");
When.iClickOnGET();
Then.getCounterShouldBeIncrementedByOne().and.iTeardownMyAppFrame();
});
opaTest("When I click on POST the POST counter should increment by one", function (Given, When, Then) {
Given.iStartMyApp("./integration/Test1/index.html");
When.iClickOnPOST();
Then.postCounterShouldBeIncrementedByOne().and.iTeardownMyAppFrame();
});
...
The configuration of our sinon server:
sap.ui.define(
["test/server"],
function (server) {
"use strict";
return {
init: function () {
var oServer = server.initServer("/backend/api");
oServer.respondWith("GET", /backend\/api/, [200, {
"Content-Type": "application/json"
}, JSON.stringify({
"date": "2018-07-29T18:44:57+02:00"
})]);
oServer.respondWith("POST", /backend\/api/, [200, {
"Content-Type": "application/json"
}, JSON.stringify({
"date": "2018-07-29T18:44:57+02:00"
})]);
}
};
}
);
The build process
Before uploading the application to SCP we need to build it. The build process optimizes the files and creates Component-preload.js and sap-ui-cachebuster-info.json file (to ensure our users aren’t using a cached version of our application)
We’ll use grunt to build our application. Here we can see our Gruntfile.js
module.exports = function (grunt) {
"use strict";
require('load-grunt-tasks')(grunt);
require('time-grunt')(grunt);
grunt.config.merge({
pkg: grunt.file.readJSON('package.json'),
watch: {
js: {
files: ['Gruntfile.js', 'webapp/**/*.js', 'webapp/**/*.properties'],
tasks: ['jshint'],
options: {
livereload: true
}
},
livereload: {
options: {
livereload: true
},
files: [
'webapp/**/*.html',
'webapp/**/*.js',
'webapp/**/*.css'
]
}
}
});
grunt.registerTask("default", [
"clean",
"lint",
"build"
]);
};
In our Gruntfile I’ve also configure a watcher to build the application automatically and triggering the live reload (to reload my browser every time I change the frontend)
Now I can build the dist folder with the command:
grunt
Deploy to SCP
The deploy process is very well explained in the Holger’s repository
Basically we need to download MTA Archive builder and extract it to ./ci/tools/mta.jar.
Also we need SAP Cloud Platform Neo Environment SDK (./ci/tools/neo-java-web-sdk/)
We can download those binaries from here
Then we need to fulfill our scp credentials in ./ci/deploy-mta.properties and configure our application in ./ci/mta.yaml
Finally we will run ./ci/deploy-mta.sh (here we can set up our scp password in order to input it within each deploy)
Full code (frontend and backend) in my github account
