[autotest] Add a new thread to upload metadata reported by scheduler
Currently host state change was reported to metadb before the change is
committed to database. Each change makes a ES post call to send data. To avoid
performance overhead for scheduler, UDP is used. UDP has a data lost issue.
Especially that the ES server now lives in GCE, while scheduler runs in a
different network.
This CL attempts to fix the issue by reporting metadata in a separate thread
in bulk. The performance of ES bulk API is much better than individual calls.
For example, a single index request through HTTP might take 80ms. For bulk API,
1000 records can be indexed in less than 0.5 second.
BUG=chromium:471015
TEST=run local scheduler, make sure all metadata was uploaded. Also, confirm
scheduler can be properly shut down.
Change-Id: I38991b9e647bb7a6fcaade8e8ef9eea27d9aa035
Reviewed-on: https://chromium-review.googlesource.com/270074
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
diff --git a/scheduler/host_scheduler.py b/scheduler/host_scheduler.py
index c22b41c..e6c0faf 100755
--- a/scheduler/host_scheduler.py
+++ b/scheduler/host_scheduler.py
@@ -71,6 +71,7 @@
from autotest_lib.scheduler import scheduler_lib
from autotest_lib.scheduler import scheduler_models
from autotest_lib.site_utils import job_overhead
+from autotest_lib.site_utils import metadata_reporter
from autotest_lib.site_utils import server_manager_utils
_db_manager = None
@@ -475,6 +476,9 @@
initialize(options.testing)
+ # Start the thread to report metadata.
+ metadata_reporter.start()
+
host_scheduler = HostScheduler()
minimum_tick_sec = global_config.global_config.get_config_value(
'SCHEDULER', 'minimum_tick_sec', type=float)
@@ -494,6 +498,7 @@
email_manager.manager.send_queued_emails()
if _db_manager:
_db_manager.disconnect()
+ metadata_reporter.abort()
if __name__ == '__main__':