[autotest] Limit gs_offloader notifications to once per hour.

This change reworks gs_offloader e-mail notifications so that if
multiple directories fail in a single loop through current results,
all the failures will be summarized in a single e-mail message.
Moreover, once notification of a problem has been sent out, no new
notifications will be sent out for at least one hour.

Additionally, this restricts the definition of "offload failure" so
that a directory that fails to offload only 1 time will never
trigger notification at all.

Finally, this change includes new unit tests covering the new
functionality, as well as some pre-existing functionality.

BUG=chromium:232058
TEST=run the shiny, new unit tests
DEPLOY=gs_offloader

Change-Id: I27454de08ab75d822db10874a659627adc395aa8
Reviewed-on: https://chromium-review.googlesource.com/190440
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Commit-Queue: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
5 files changed