Achieve 600k Concurrent WebSocket Connections.

Recently we have done a performance test for WebSocket connections using AWS infrastructure how much concurrent connections can be handled by a single server.

Technology Used:

• 1 Machine of M3.xlarge. It has 4 CPUs and 15Gb of memory for the server.
• 10 Machine of c5.large It has 2CPUs and 4Gb of memory for client
• Spring Webflux
• Netty Server: An Event-driven, Non-blocking Server
• Tsung: Performance Testing tool

Server Side:

It just contains simple one controller which handles WebSocket connections

and log the message received.

public class ReactiveWebSocketHandler implements WebSocketHandler {

@Override

    public Mono<Void> handle(WebSocketSession webSocketSession) {

        return webSocketSession.send(intervalFlux

                   .map(webSocketSession::textMessage))

                   .and(webSocketSession.receive()

                   .map(WebSocketMessage::getPayloadAsText)

                   .log());

      }

}

Configurations:

EC2 Configurations:

• Set the Soft and Hard nofile limit to 1000000.

File : /etc/security/limits.d/custom.conf

append in the above file.

root soft nofile 1000000
root hard nofile 1000000
* soft nofile 1000000
* hard nofile 1000000

• Now set

File: /etc/sysctl.conf

append in the above file.

fs.file-max = 1000000
fs.nr_open = 1000000
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.ipv4.ip_local_port_range= 1024 65535
net.ipv4.tcp_mem='10000000 10000000 10000000
net.ipv4.tcp_rmem='1024 4096 16384'
net.ipv4.tcp_wmem='1024 4096 16384'
net.core.rmem_max=16384
net.core.wmem_max=16384

fs.file-max: The maximum file handles that can be allocated.

fs.nr_open: Max amount of file handles that can be opened.

net.ipv4.netfilter.ip_conntrack_max: Specifies how many connections the NAT can keep track of in the tracking table before it starts to drop packets and just break connections.

net.ipv4.ip_local_port_range:[ Link for more explaination](https://medium.com/@sauravomar01/how-to-increase-the-port-range-in-linux-b7df2f0b0b58).

Tsung :

Tsung is an open-source multi-protocol distributed load testing tool. It’s written in Erlang. Tsung also supports benchmarking WebSocket protocol. Tsung configuration and scenarios are written in xml.

Install:

sudo apt-get update
sudo apt-get install tsung
tsung -v // to validate tsung installation

Configurations:

We have used 10 Tsung server because maximum we can create 64K connection from a single server.

• We have also tried Apache Thor for creating connections its taking lot of time to create connections.
• Tsung has very good UI and graphs so it easy for us to get the stats.

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/user/share/tsung/tsung-1.0.dtd">
<tsung loglevel="notice" version="1.0">
  <clients>
    <client host="tsung-machine-1" use_controller_vm="false" maxusers="64000" />
    <client host="tsung-machine-2" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-3" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-4" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-5" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-6" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-7" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-8" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-9" use_controller_vm="false" maxusers="64000" />

<client host="tsung-machine-10" use_controller_vm="false" maxusers="64000" />  

</clients>

  <servers>
    <server host="${SERVER IP}" port="3000" type="tcp" />
  </servers>

  <load>
    <arrivalphase phase="1" duration="100" unit="second">
      <users maxnumber="100000" arrivalrate="1000" unit="second" />
    </arrivalphase>
  </load>

  <sessions>
    <session name="websocket" probability="100" type="ts_websocket">
        <request>
             <websocket type="connect" path="/"></websocket>
        </request>

        <request subst="true">
            <websocket type="message">{"name":"Websocket"}</websocket>
        </request>

        <for var="i" from="1" to="100" incr="1">
          <thinktime value="10"/>
        </for>
    </session>
  </sessions>
</tsung>

Start:

To start Tsung below is the usage

tsung -f config.xml start

Dashboard:

to see the dashboard in Tsung use URL http://${server_url}:8091/

For example:

for more info : http://tsung.erlang-projects.org/user_manual/reports.html#fig-graph.

Thats it.

Happy Learning

Thanks