[autotest] RDB Refactor II + Request/Response API.
Scheduler Refactor:
1. Batched processing of jobs.
2. Rdb hits the database instead of going through host_scheduler.
3. Migration to add a leased column.The scheduler released hosts
every tick, back to the rdb.
4. Client rdb host that queue_entries use to track a host, instead
of a database model.
Establishes a basic request/response api for the rdb:
rdb_utils:
1. Requests: Assert the format and fields of some basic request types.
2. Helper client/server modules to communicate with the rdb.
rdb_lib:
1. Request managers for rdb methods:
a. Match request-response
b. Abstract the batching of requests.
2. JobQueryManager: Regulates database access for job information.
rdb:
1. QueryManagers: Regulate database access
2. RequestHandlers: Use query managers to get things done.
3. Dispatchers: Send incoming requests to the appropriate handlers.
Ignores wire formats.
TEST=unittests, functional verification.
BUG=chromium:314081, chromium:314083, chromium:314084
DEPLOY=scheduler, migrate
Change-Id: Id174c663c6e78295d365142751053eae4023116d
Reviewed-on: https://chromium-review.googlesource.com/183385
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
diff --git a/frontend/afe/doctests/001_rpc_test.txt b/frontend/afe/doctests/001_rpc_test.txt
index e409036..605bdf7 100644
--- a/frontend/afe/doctests/001_rpc_test.txt
+++ b/frontend/afe/doctests/001_rpc_test.txt
@@ -110,7 +110,8 @@
... 'invalid': 0,
... 'protection': 'No protection',
... 'locked_by': 'debug_user',
-... 'dirty': True}]
+... 'dirty': True,
+... 'leased': 1}]
True
>>> rpc_interface.modify_host('ipaj1', status='Hello')
Traceback (most recent call last):
@@ -568,7 +569,8 @@
... 'protection': 'No protection',
... 'locked_by': None,
... 'lock_time': None,
-... 'dirty': True},
+... 'dirty': True,
+... 'leased': 1},
... 'id': 1,
... 'job': job, # full job info here
... 'meta_host': None,
diff --git a/frontend/afe/model_logic.py b/frontend/afe/model_logic.py
index 204799f..cefdae5 100644
--- a/frontend/afe/model_logic.py
+++ b/frontend/afe/model_logic.py
@@ -89,6 +89,14 @@
_make_queryset_readonly(self)
+class LeasedHostManager(dbmodels.Manager):
+ """Query manager for unleased, unlocked hosts.
+ """
+ def get_query_set(self):
+ return (super(LeasedHostManager, self).get_query_set().filter(
+ leased=0, locked=0))
+
+
class ExtendedManager(dbmodels.Manager):
"""\
Extended manager supporting subquery filtering.
diff --git a/frontend/afe/models.py b/frontend/afe/models.py
index d4ada02..bf0b3cb 100644
--- a/frontend/afe/models.py
+++ b/frontend/afe/models.py
@@ -367,6 +367,7 @@
labels = dbmodels.ManyToManyField(Label, blank=True,
db_table='afe_hosts_labels')
locked = dbmodels.BooleanField(default=False)
+ leased = dbmodels.BooleanField(default=True)
synch_id = dbmodels.IntegerField(blank=True, null=True,
editable=settings.FULL_ADMIN)
status = dbmodels.CharField(max_length=255, default=Status.READY,
@@ -384,6 +385,7 @@
name_field = 'hostname'
objects = model_logic.ModelWithInvalidManager()
valid_objects = model_logic.ValidObjectsManager()
+ leased_objects = model_logic.LeasedHostManager()
def __init__(self, *args, **kwargs):
diff --git a/frontend/migrations/085_lease_hosts.py b/frontend/migrations/085_lease_hosts.py
new file mode 100644
index 0000000..fd277bb
--- /dev/null
+++ b/frontend/migrations/085_lease_hosts.py
@@ -0,0 +1,9 @@
+UP_SQL = """
+ALTER TABLE afe_hosts ADD COLUMN leased TINYINT(1) NOT NULL DEFAULT '1';
+CREATE INDEX leased_hosts ON afe_hosts (leased, locked);
+"""
+
+DOWN_SQL = """
+DROP INDEX leased_hosts ON afe_hosts;
+ALTER TABLE afe_hosts DROP COLUMN leased;
+"""