listen socket more frequent service
From an idea by Edwin van den Oetelaar <oetelaar.automatisering@gmail.com>
When testing libwebsockets with ab, Edwin found an unexpected bump in
the distribution of latencies, some connections were held back almost
the whole test duration.
http://ml.libwebsockets.org/pipermail/libwebsockets/2013-January/000006.html
Studying the problem revealed that when there are mass pending connections
amongst many active connections, we do not service the listen socket often
enough to clear the backlog, some seem to get stale violating FIFO ordering.
This patch introduces listen socket service "piggybacking", where every n
normal socket service actions we also check the listen socket and deal with
pending connections there.
Normally, it checks the listen socket gratuitously every 10 normal socket
services. However, if it finds something waiting, it forces a check on the
next normal socket service too by keeping stats on how often something was
waiting. If the probability of something waiting each time becomes high,
it will allow up to two waiting connections to be serviced for each normal
socket service.
In that way it has low burden in the normal case, but rapidly adapts by
detecting mass connection loads as found in ab.
Signed-off-by: Andy Green <andy.green@linaro.org>
diff --git a/lib/libwebsockets.c b/lib/libwebsockets.c
index 8bc91de..bbbbe49 100644
--- a/lib/libwebsockets.c
+++ b/lib/libwebsockets.c
@@ -1534,6 +1534,41 @@
/* no, here to service a socket descriptor */
+ /*
+ * deal with listen service piggybacking
+ * every listen_service_modulo services of other fds, we
+ * sneak one in to service the listen socket if there's anything waiting
+ *
+ * To handle connection storms, as found in ab, if we previously saw a
+ * pending connection here, it causes us to check again next time.
+ */
+
+ if (context->listen_service_fd && pollfd->fd != context->listen_service_fd) {
+ context->listen_service_count++;
+ if (context->listen_service_extraseen ||
+ context->listen_service_count == context->listen_service_modulo) {
+ context->listen_service_count = 0;
+ m = 1;
+ if (context->listen_service_extraseen > 5)
+ m = 2;
+ while (m--) {
+ /* even with extpoll, we prepared this internal fds for listen */
+ n = poll(&context->fds[0], 1, 0);
+ if (n > 0) { /* there's a connection waiting for us */
+ libwebsocket_service_fd(context, &context->fds[0]);
+ context->listen_service_extraseen++;
+ } else {
+ if (context->listen_service_extraseen)
+ context->listen_service_extraseen--;
+ break;
+ }
+ }
+ }
+
+ }
+
+ /* okay, what we came here to do... */
+
wsi = wsi_from_fd(context, pollfd->fd);
if (wsi == NULL) {
@@ -3040,10 +3075,14 @@
wsi->mode = LWS_CONNMODE_SERVER_LISTENER;
insert_wsi(context, wsi);
+ context->listen_service_modulo = LWS_LISTEN_SERVICE_MODULO;
+ context->listen_service_count = 0;
+ context->listen_service_fd = sockfd;
+
listen(sockfd, LWS_SOMAXCONN);
lwsl_info(" Listening on port %d\n", port);
- /* list in the internal poll array */
+ /* list in the internal poll array - we're always first */
context->fds[context->fds_count].fd = sockfd;
context->fds[context->fds_count++].events = POLLIN;