Question

I’m currently testing extreme condition on a piece of code written with Erlang.

I have implemented learnyousomeerlang.com‘s technique of supervisor to have multiple accept capability.

Here the code slightly modified to handle SSL connections of the supervisor:

-module(mymodule).

-behaviour(supervisor).

-export([start/0, start_socket/0]).
-define(SSL_OPTIONS, [{active, true},
              {mode, list},
              {reuseaddr, true},
              {cacertfile, "./ssl_key/server/gd_bundle.crt"},
              {certfile, "./ssl_key/server/cert.pem"},
              {keyfile, "./ssl_key/server/key.pem"},
              {password, "********"}
             ]).

-export([init/1]).

start_link() ->
    application:start(crypto),
    crypto:start(),
    application:start(public_key),
    application:start(ssl),
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    {ok, LSocket} = ssl:listen(4242, ?SSL_OPTIONS),
    spawn_link(fun empty_listeners/0),
    {ok, {{simple_one_for_one, 60, 3600},
      [{socket,
        {mymodule_serv, start_link, [LSocket]}, % pass the socket!
        temporary, 1000, worker, [mymodule_serv]}
      ]}}.

empty_listeners() ->
    [start_socket() || _ <- lists:seq(1,100)],
    ok.

start_socket() ->
    supervisor:start_child(?MODULE, []).

Here’s the code for gen_server which will represent every client connecting :

-module(mymodule_serv).
-behaviour(gen_server).

-export([start_link/1]).
-export([init/1, handle_call/3, handle_cast/2, terminate/2, code_change/3, handle_info/2]).

start_link(Socket) ->
    gen_server:start_link(?MODULE, Socket, []).

init(Socket) ->
    gen_server:cast(self(), accept),
    {ok, #client{socket=Socket, pid=self()}}.

handle_call(_E, _From, Client) ->
    {noreply, Client}.

handle_cast(accept, C = #client{socket=ListenSocket}) ->
    {ok, AcceptSocket} = ssl:transport_accept(ListenSocket),
    mymodule:start_socket(),
    ssl:ssl_accept(AcceptSocket),
    ssl:setopts(AcceptSocket, [{active, true}, {mode, list}]),
    {noreply, C#client{socket=AcceptSocket, state=connecting}}.

[...]

I have the ability to launch close to 10.000 connections at once from multiple server.
While it will take 10 second to a ssl accepting bit of C++ code to accept all of them (which don’t even have multiple accept pending), in Erlang this is quite different. It will accept at most 20 connections a second (according to netstat info, whilst C++ accept more like 1K connection per seconds)

While the 10K connections are awaiting for acceptance, I’m manually trying to connect as well.

openssl s_client -ssl3 -ign_eof -connect myserver.com:4242

3 cases happen when I do :

• Connection simply timeout
• Connection will connect after waiting for it 30 sec. at least
• Connection will occur almost directly

When I try connecting manually with 2 consoles, the first done handshaking will not always be the first which tried to connect… Which I found particular.

The server configuration is :

• 2 x Intel® Xeon® E5620
• 8x 2.4GHz
• 24 Go RAM

I’m starting the Erlang shell with :

$erl +S 8:8

EDIT 1:

I have even tried to accept the connection with gen_tcp, and upgrading afterwards the connection to a SSL one. Still the same issue, it won’t accept more than 10 connections a second… Is ssl:ssl_accept is doing this ? does it lock anything that would prevent Erlang to scale this ?

EDIT 2:

After looking around on other SSL server created in erlang, it seems that they use some kind of driver for SSL/TLS connection, my examples are RabbitMQ and EjabberD.
Nowhere there is ssl:ssl_accept in their Erlang code, I haven’t investigate a lot, but it seems they have created their own driver in order to upgrade the TCP Socket to a SSL/TLS one.
Is that because there is an issue with Erlang’s module SSL ? Does anyone know why they are using custom driver for SSL/TLS ?

Any thoughts on this ?

Answer 1

Actually it was not the SSL accept or handshake that was slowing the whole thing.

We found on the erlang question list that it was the backlog.

Backlog is set to 5 by default. I have set it to SOMAXCONN and everything works fine now !