How we improved our WebSockets by removing all database access

Twake is a real-time application which use WebSockets in order to work. We use the Symfony framework for our backend (PHP), but what I describe here exist in any other languages or framework, it’s about how we designed our app. We use a module called GOS WebSockets (https://github.com/GeniusesOfSymfony/WebSocketBundle) which adapt the Ratchet WebSocket library (http://socketo.me/) for Symfony. Once enabled this module is very easy to use and I think it’s also easy to make mistakes.

WebSockets is a web protocol that allows bidirectional communication between a server and a browser. Usually, the server cannot send any message directly to your computer because you do not have static IP or anything like that, and because you don’t have any server to receive incoming requests. Server side, usual web servers are made to receive HTTP requests which are not bidirectional. What does WebSocket protocol is establishing a permanent connection between you and the server, doing so your browser is “waiting” for messages from the server. This allow you to receive data in real-time. Server side you must run a WebSocket server which manage connected clients, this WebSocket server is different than your Apache server if you use it. Finally this server is not really multi thread, if a client start some heavy computation on the WebSocket server every other users will have to wait : you start to see the problem now.

We’ll take the messaging app case to explain what we did and why we changed it. With GOS WebSocket we use something called “PubSub” it’s a concaténation of the words “publish” and “subscribe”. To make your real-time messaging app, you just need users to “subscribe” to their current channel using an unique id called “route” (then you will be notified each time something happen on the channel) and each time and user send a message you just have to “publish” it (and everybody receive the new data). WebSockets do have their own “HTTPS” it’s called “WSS” and you can install it on your server easily and transparently. But beside of this WebSocket does not come with any security layer, for instance with what I described above, anybody can “subscribe” or “publish” to any channel if the route to it is known (and in most cases the route is something like channels/[channel_id]).

The first idea is to check user access rights on each subscribe and publish in the WebSockets server but it cause at least four issues :

• it slows the WebSockets server,
• it increases crashes (because unlike http, if the WebSockets server goes down because of any random bug, it hangs for everyone),
• you need to sync user sessions to know who is connected to the WebSockets (remind that your WebSockets server is different than your apache or fpm server, so PHP Sessions are not shared, instead you need to store sessions on database to share them across servers),
• you need to maintain a permanent connexion with your database and mysql don’t like permanent connexion in my experience (I didn’t try with Cassandra or Scylladb).

It would be so much simpler if the WebSockets server was autonomous and only used to share data in realtime don’t you think ?

Compute over HTTP and share result over WebSockets

When we started using WebSockets with Symfony and when we followed the docs we saw it was possible to use Doctrine (Symfony ORM used to manipulate database) and to share user session, so in the first place our vision was biased. We started to do a lot of things over WebSockets, writing new messages on database, editing and deleting messages, verifying user rights etc. But what is the point to use WebSocket for that when you can use classic and reliable HTTP ? At first I thought is was a bad idea to use HTTP too much, to call HTTP each time I wanted to send a message, reorder tasks or edit an event. But I looked other web apps like Notion.io and I saw that it was not an issue for them.

So I changed our design to use HTTP for every modifications and computation, and WebSockets only for sharing theses modifications. The process is really simple, I first subscribe to the WebSocket route, then when I want to send a message I send it over HTTP and when I receive the response (with a brand new created unique id and all computation done) I can broadcast it to everyone with a WebSockets publish. This way, a lot of thing was better :

• never need to worry about database and sessions in the WebSocket server,
• WebSockets server can use any implementation for single node on our GOS WebSockets, or multiple scalable node using Redis,
• after all you can wait the server response before to share the data with everyone, if they aren’t beside you they’ll never see the 200ms delay,
• you can rely on HTTP and Ajax APIs to detect errors, timeout, slow network, etc.

Not using WebSockets for object storage and any other computation.

Secure your routes

Ok, now we have working scalable simple WebSockets, but they are absolutly not secure. Anybody can subscribe or publish to any channel and even if they cannot write in the database, they can receive sensitive data and send fake messages. The only thing they need is the route name and it’s not difficult to find it when the route name looks like «channel/[some id]» especially if you use auto-incremented ids (but even with unique random uuid, it’s not perfect).

The first thing you can do is to not use human comprehensive routes but random ones ! Instead of directly subscribe to your channel using the route «channel/42 » you can call a HTTP route which take as a parameter the requested route, and give you a random route as a response (after verifying your identity and your access rights), this route can look like «collection/randomToken». It doesn’t change anything for the WebSockets server itself but nobody can know the route you use now. In order to do that you need to manage a new table in your database to match the requested human readable route to the random generated one and also to the rights needed to access this route. I thought of using a hash function without any database table but I was concerned by hash collision which is critical in this situation.

Random token instead of human readable routes for WebSockets.

More security with data encryption

On Twake we decided to secure even more our communications with an extra layer of encryption. We know that WSS exists (equivalent of HTTPS for WebSocket protocol), but with our current implementation if an user achieve to know the route token for a sensitive channel, it can do whatever he want like described at the end of the part “Compute over HTTP and share result over WebSockets”. The idea is now to generate a new random encryption key with our HTTP server each time a user want to join our WebSockets channel. This random encryption key will be used by each member of the channel to exchange end-to-end encrypted messages. We have to resolve some point to make this to work :

• Encryption in JavaScript (hoping this will not break our performances),
• Encryption in JavaScript and in PHP working together (if the HTTP server want to send messages to the WebSockets Server),
• Update key each time a user enter the channel (without giving the key to everyone in clear of course).

There is the idea to keep updated keys for everyone :

The idea for end-to-end encrypted WebSockets.

With this schema, nobody can never know the current key shared between users and so can neither send or receive messages.

Implementing end-to-end encryption

To implement this, we used the CryptoJS library client side (https://github.com/brix/crypto-js) with the following simple code to generate key AB from keys A and B :

//JavaScript
kAB = CryptoJS.sha256(kA + kB).toString();

//PHP
$kAB = hash('sha256', $kA . $kB);

It was a bit difficult to find how to have a working JS encryption and PHP decryption (or vice-versa) so there is our code in the both languages :

//JavaScript decrypt

var salt = CryptoJS.enc.Hex.parse(encrypted_message.salt);
var prepared_key = CryptoJS.PBKDF2(current_key, salt, { hasher: CryptoJS.algo.SHA512, keySize: 64/8, iterations: 9});

var iv = CryptoJS.enc.Hex.parse(encrypted_message.iv);
var bytes  = CryptoJS.AES.decrypt(encrypted_message.data, prepared_key, {iv: iv});

var result = JSON.parse(bytes.toString(CryptoJS.enc.Utf8));

//PHP encrypt

$salt = openssl_random_pseudo_bytes(256);
$iv = openssl_random_pseudo_bytes(16);
$iterations = 9;
$prepared_key = hash_pbkdf2("sha512", $key, $salt, $iterations, 64);
$string = json_encode($message_to_encrypt);
$encrypted = trim(
    base64_encode(
        openssl_encrypt(
            $string,
            'aes-256-cbc',
            hex2bin($prepared_key),
            OPENSSL_RAW_DATA,
            $iv
        )
    )
);

$this->pusher->push(Array(
    "data" => $encrypted,
    "iv" => bin2hex($iv),
    "salt" => bin2hex($salt)
), "collections/" . $route_endpoint);

⚠️ When I first used this code, I was dealing with serious performance issues ! Using Chrome Developers Tools, I found that my encryption and decryption functions was very slow (browser freeze during more than 500ms). I changed some parameters and I found that the “iteration” parameter was really important for this issue. At first we had a value of 999 iterations and we changed it to only 9 iteration reducing the computation time lower than 10ms. Of course I looked what was this number and how it will affect the security of Twake. First of all, you can see that this parameter is only used to prepare the key before any encryption or decryption. Actually, this parameter is useful only in the case where a hacker achieve to decode your message and found the encryption key. If this encryption key is a personal password or a credit card identifier it could be very bad news for you because the hacker can use this key to access some other account of yours ! But in our case the key is a long random generated key used nowhere but in this specific channel during a specific period of time. So no need to worry, you can event reduce this parameter to 1.

⚠️ If you change the encryption key each time a new user enter your channel, you might experience some issues. First of all use some versioning with your encryption key and keep old version for some time in case somebody did not receive the new key at the time he sent the message. Then if 10 users try to connect at the same time you will have concurrency and useless key updates (it is not necessary to update your key 10 time within a second) so you can add some code to update the key only if it was not updated for more than a minute for instance. And finally client side if you cannot receive any message you can try again all the procedure to get the latest encryption key.

Conclusion

Of course you can do everything with your WebSocket server, but I found much more elegant to not mix WebSockets usage and HTTP server usage. Here WebSockets are only used for exchanging data and nothing else. It allows us to use any stack we want for it like Redis or a NodeJS server and it make scalability simpler. I hope you enjoyed reading this article, don’t hesitate to comment if I said anything wrong or if you want to share you experience, I would love to improve this article 😇