[autotest] Send an email if a drone is hitting process limit
Add a setting 'max_processes_warning_threshold' to global_config.ini.
If the raitio of active processes to max processes goes over
the threshold, DroneManager will send a email to
chromeos-lab-infrastructure@.
To prevent it from spamming our mailing list, only one email
will be sent for each drone within 24 hours.
BUG=chromium:277184
TEST=Test locally and confirm email is sent out when
the threshold is reached.
DEPLOY=scheduler
Change-Id: Id4a883ff6c26e9bba384974c255a0ce0f3cb4056
Reviewed-on: https://chromium-review.googlesource.com/168147
Reviewed-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
diff --git a/global_config.ini b/global_config.ini
index bdc634d..9dedf31 100644
--- a/global_config.ini
+++ b/global_config.ini
@@ -76,6 +76,7 @@
notify_email_from: chromeos-autotest@google.com
notify_email_statuses: Completed,Failed,Aborted
max_processes_per_drone: 1000
+max_processes_warning_threshold: 0.8
max_hostless_processes: 500
max_jobs_started_per_cycle: 100
max_parse_processes: 100