[autotest] HostScheduler maintain suite hosts assignment history.
This is part I of making host scheduler support a min_dut
requirement per suite.
With this CL, host scheduler is able to load and maintain
a record of current suite host assignment.
This CL:
1) Introduces host_scheduler.SuiteRecorder,
which holds suite host assignment
history, i.e which host has been assigned to which suite.
And provide functionality to update its record
when a host is assigned to a suite and when a host is released.
2) Adds some util methods in query_manager
- load suite host assignment on the start of host scheduler.
- load suite_min_duts job keyval for a suite from db.
CL-DEPEND=CL:231210
DEPLOY=host_scheduler
TEST=Add unittest for host_scheduler.
TEST=Schedule two suites with different "suite_min_duts".
Print out the current record hold by SuiteRecorder.
Confirm that the record is correct when a host
is assigned to a suite, when a host is released,
and when host_scheduler starts with the jobs already running.
TEST=Integration test with CL:231210 by running two bvt-cq suites
with different priority and suite_min_duts. Set testing_mode=True
and testing_exception=test_suites.
TEST=Integration test with CL:231210. Test the host scheduler
inline mode still works by setting inline_host_acquisition=True
and running two suites with different priority.
BUG=chromium:432648
Change-Id: I0045537fe69f5d0c06ea51c60ba254e9c505642c
Reviewed-on: https://chromium-review.googlesource.com/231139
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
diff --git a/scheduler/query_managers.py b/scheduler/query_managers.py
index aca504e..9cf0cdf 100644
--- a/scheduler/query_managers.py
+++ b/scheduler/query_managers.py
@@ -15,6 +15,7 @@
from autotest_lib.client.common_lib.cros.graphite import stats
from autotest_lib.frontend import setup_django_environment
from autotest_lib.frontend.afe import models
+from autotest_lib.server.cros.dynamic_suite import constants
from autotest_lib.scheduler import scheduler_models
from autotest_lib.scheduler import scheduler_lib
@@ -170,6 +171,45 @@
'id', 'job_id', 'host_id'))
+ @_job_timer.decorate
+ def get_suite_host_assignment(self):
+ """A helper method to get how many hosts each suite is holding.
+
+ @return: Two dictionaries (suite_host_num, hosts_to_suites)
+ suite_host_num maps suite job id to number of hosts
+ holding by its child jobs.
+ hosts_to_suites contains current hosts held by
+ any suites, and maps the host id to its parent_job_id.
+ """
+ query = models.HostQueueEntry.objects.filter(
+ host_id__isnull=False, complete=0, active=1,
+ job__parent_job_id__isnull=False)
+ suite_host_num = {}
+ hosts_to_suites = {}
+ for hqe in query:
+ host_id = hqe.host_id
+ parent_job_id = hqe.job.parent_job_id
+ count = suite_host_num.get(parent_job_id, 0)
+ suite_host_num[parent_job_id] = count + 1
+ hosts_to_suites[host_id] = parent_job_id
+ return suite_host_num, hosts_to_suites
+
+
+ @_job_timer.decorate
+ def get_min_duts_of_suites(self, suite_job_ids):
+ """Load suite_min_duts job keyval for a set of suites.
+
+ @param suite_job_ids: A set of suite job ids.
+
+ @return: A dictionary where the key is a suite job id,
+ the value is the value of 'suite_min_duts'.
+ """
+ query = models.JobKeyval.objects.filter(
+ job_id__in=suite_job_ids,
+ key=constants.SUITE_MIN_DUTS_KEY, value__isnull=False)
+ return dict((keyval.job_id, int(keyval.value)) for keyval in query)
+
+
_host_timer = stats.Timer('scheduler.host_query_manager')
class AFEHostQueryManager(object):
"""Query manager for AFE Hosts."""