webserver: Reduce amount of FD watcher resets during async I/O

While investigating an unrelated issue I noticed that the current
implementation in webservd performs too much work each time a socket
is triggered to be ready for reading and/or writing. Where there are
a lot of incoming requests, many sockets get triggered rather quickly
and we'd call ProtocolHandler::DoWork() for every one of the events
which completely destroys and recreates all the file descriptor watchers
for all outstanding sockets. This produces a lot of CPU load for no
apparent reason.

So the changes to metigate this somewhat are as follows:

1. Do not call DoWork() for each socket event directly. Schedule a
   message loop task and do not schedule another one if another
   task for DoWork() is pending to be processed. This effectively
   coalesces simultaneous events on multiple sockets and lets
   libmicrohttpd process them all at once.
2. Do not destroy and recreate socket file descriptor watchers from
   scratch a new I/O event occurs. Instead, store them in a map keyed
   by the file descriptor. And reuse the watchers already setup from
   previous iteration.
3. If a watcher is already monitoring a socket and no events have been
   triggered for it since last time the watcher was set up, do not tear
   down/recreate the watcher. Instead, keep watching the file descriptor
   unless the watch mode changes.
4. OnTimer() method wasn't used. However the timer_triggered_ was set
   the first time the timer was set up and never reset, effectively
   disabling any possible timer resets in the future. Remove all this.

BUG=brillo:990
TEST=`FEATURES=test emerge-link webserver`
     test_that -b gizmo 100.96.49.59 buffet_Registration --iterations 50
     Manually triggered 2000 simultaneous HTTP requests to the web server

Change-Id: Ia3a5007d46b15e6682b88ffc9e643d29249f707d
Reviewed-on: https://chromium-review.googlesource.com/272661
Reviewed-by: Alex Vakulenko <avakulenko@chromium.org>
Reviewed-by: Vitaly Buka <vitalybuka@chromium.org>
Commit-Queue: Vitaly Buka <vitalybuka@chromium.org>
Tested-by: Vitaly Buka <vitalybuka@chromium.org>
2 files changed
tree: 534e01e484ad6a6ea92227da5e586498aebb0b9d
  1. libwebserv/
  2. webservd/
  3. OWNERS
  4. README
  5. webserver.gyp