Login forms are bored. In this example we’re going to create an especial login form. Only for happy users. Happiness is something complicated, but at least, one smile is more easy to obtain, and all is better with one smile :). Our login form will only appear if the user smiles. Let’s start.
I must admit that this project is just an excuse to play with different technologies that I wanted to play. Weeks ago I discovered one library called face_classification. With this library I can perform emotion classification from a picture. The idea is simple. We create RabbitMQ RPC server script that answers with the emotion of the face within a picture. Then we obtain on frame from the video stream of the webcam (with HTML5) and we send this frame using websocket to a socket.io server. This websocket server (node) ask to the RabbitMQ RPC the emotion and it sends back to the browser the emotion and a the original picture with a rectangle over the face.
Frontend
As well as we’re going to use socket.io for websockets we will use the same script to serve the frontend (the login and the HTML5 video capture)
<!doctype html>
<html>
<head>
<title>Happy login</title>
<link rel="stylesheet" href="css/app.css">
</head>
<body>
<div id="login-page" class="login-page">
<div class="form">
<h1 id="nonHappy" style="display: block;">Only the happy user will pass</h1>
<form id="happyForm" class="login-form" style="display: none" onsubmit="return false;">
<input id="user" type="text" placeholder="username"/>
<input id="pass" type="password" placeholder="password"/>
<button id="login">login</button>
<p></p>
<img id="smile" width="426" height="320" src=""/>
</form>
<div id="video">
<video style="display:none;"></video>
<canvas id="canvas" style="display:none"></canvas>
<canvas id="canvas-face" width="426" height="320"></canvas>
</div>
</div>
</div>
<div id="private" style="display: none;">
<h1>Private page</h1>
</div>
<script src="https://code.jquery.com/jquery-3.2.1.min.js" integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4=" crossorigin="anonymous"></script>
<script src="https://unpkg.com/sweetalert/dist/sweetalert.min.js"></script>
<script type="text/javascript" src="/socket.io/socket.io.js"></script>
<script type="text/javascript" src="/js/app.js"></script>
</body>
</html>
Here we’ll connect to the websocket and we’ll emit the webcam frame to the server. We´ll also be listening to one event called ‘response’ where server will notify us when one emotion has been detected.
let socket = io.connect(location.origin),
img = new Image(),
canvasFace = document.getElementById('canvas-face'),
context = canvasFace.getContext('2d'),
canvas = document.getElementById('canvas'),
width = 640,
height = 480,
delay = 1000,
jpgQuality = 0.6,
isHappy = false;
socket.on('response', function (r) {
let data = JSON.parse(r);
if (data.length > 0 && data[0].hasOwnProperty('emotion')) {
if (isHappy === false && data[0]['emotion'] === 'happy') {
isHappy = true;
swal({
title: "Good!",
text: "All is better with one smile!",
icon: "success",
buttons: false,
timer: 2000,
});
$('#nonHappy').hide();
$('#video').hide();
$('#happyForm').show();
$('#smile')[0].src = 'data:image/png;base64,' + data[0].image;
}
img.onload = function () {
context.drawImage(this, 0, 0, canvasFace.width, canvasFace.height);
};
img.src = 'data:image/png;base64,' + data[0].image;
}
});
navigator.getMedia = (navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia);
navigator.getMedia({video: true, audio: false}, (mediaStream) => {
let video = document.getElementsByTagName('video')[0];
video.src = window.URL.createObjectURL(mediaStream);
video.play();
setInterval(((video) => {
return function () {
let context = canvas.getContext('2d');
canvas.width = width;
canvas.height = height;
context.drawImage(video, 0, 0, width, height);
socket.emit('img', canvas.toDataURL('image/jpeg', jpgQuality));
}
})(video), delay)
}, error => console.log(error));
$(() => {
$('#login').click(() => {
$('#login-page').hide();
$('#private').show();
})
});
Backend
Finally we’ll work in the backend. Basically I’ve check the examples that we can see in face_classification project and tune it a bit according to my needs.
from rabbit import builder
import logging
import numpy as np
from keras.models import load_model
from utils.datasets import get_labels
from utils.inference import detect_faces
from utils.inference import draw_text
from utils.inference import draw_bounding_box
from utils.inference import apply_offsets
from utils.inference import load_detection_model
from utils.inference import load_image
from utils.preprocessor import preprocess_input
import cv2
import json
import base64
detection_model_path = 'trained_models/detection_models/haarcascade_frontalface_default.xml'
emotion_model_path = 'trained_models/emotion_models/fer2013_mini_XCEPTION.102-0.66.hdf5'
emotion_labels = get_labels('fer2013')
font = cv2.FONT_HERSHEY_SIMPLEX
# hyper-parameters for bounding boxes shape
emotion_offsets = (20, 40)
# loading models
face_detection = load_detection_model(detection_model_path)
emotion_classifier = load_model(emotion_model_path, compile=False)
# getting input model shapes for inference
emotion_target_size = emotion_classifier.input_shape[1:3]
def format_response(response):
decoded_json = json.loads(response)
return "Hello {}".format(decoded_json['name'])
def on_data(data):
f = open('current.jpg', 'wb')
f.write(base64.decodebytes(data))
f.close()
image_path = "current.jpg"
out = []
# loading images
rgb_image = load_image(image_path, grayscale=False)
gray_image = load_image(image_path, grayscale=True)
gray_image = np.squeeze(gray_image)
gray_image = gray_image.astype('uint8')
faces = detect_faces(face_detection, gray_image)
for face_coordinates in faces:
x1, x2, y1, y2 = apply_offsets(face_coordinates, emotion_offsets)
gray_face = gray_image[y1:y2, x1:x2]
try:
gray_face = cv2.resize(gray_face, (emotion_target_size))
except:
continue
gray_face = preprocess_input(gray_face, True)
gray_face = np.expand_dims(gray_face, 0)
gray_face = np.expand_dims(gray_face, -1)
emotion_label_arg = np.argmax(emotion_classifier.predict(gray_face))
emotion_text = emotion_labels[emotion_label_arg]
color = (0, 0, 255)
draw_bounding_box(face_coordinates, rgb_image, color)
draw_text(face_coordinates, rgb_image, emotion_text, color, 0, -50, 1, 2)
bgr_image = cv2.cvtColor(rgb_image, cv2.COLOR_RGB2BGR)
cv2.imwrite('predicted.png', bgr_image)
data = open('predicted.png', 'rb').read()
encoded = base64.encodebytes(data).decode('utf-8')
out.append({
'image': encoded,
'emotion': emotion_text,
})
return out
logging.basicConfig(level=logging.WARN)
rpc = builder.rpc("image.check", {'host': 'localhost', 'port': 5672})
rpc.server(on_data)
Here you can see in action the working prototype
Maybe we can do the same with another tools and even more simple but as I said before this example is just an excuse to play with those technologies:
- Send webcam frames via websockets
- Connect one web application to a Pyhon application via RabbitMQ RPC
- Play with face classification script
Please don’t use this script in production. It’s just a proof of concepts. With smiles but a proof of concepts 🙂
You can see the project in my github account
Gonzalo Ayuso - Web Architect 