[Autotest] Allow the syncing of labels added/removed.

This allows us to perform label add/remove actions
on the master and have it pass down to the shards those hosts
are on by introducing a generic decorator.

The cl also fixes some logging issues with the shard-client,
and a bug with propogating the invalid bit of a host.

TEST=Added/removed labels.
BUG=chromium:431786
DEPLOY=apache

Change-Id: Ic176054e04a5508bb3e967ecc902422628446d32
Reviewed-on: https://chromium-review.googlesource.com/231558
Reviewed-by: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
diff --git a/frontend/afe/rpc_utils.py b/frontend/afe/rpc_utils.py
index 5c0deb1..1f4a694 100644
--- a/frontend/afe/rpc_utils.py
+++ b/frontend/afe/rpc_utils.py
@@ -1056,6 +1056,47 @@
     return replacement
 
 
+def forward_multi_host_rpc_to_shards(func):
+    """This decorator forwards rpc calls that modify multiple hosts.
+
+    If a host is assigned to a shard, rpcs that change his attributes should be
+    forwarded to the shard. Some calls however, take a list of hosts and a
+    single id to modify, eg: label_add_hosts. This wrapper will sift through
+    the list of hosts, find each of their shards, and forward the rpc for
+    those hosts to that shard before calling the local version of the given rpc.
+
+    This assumes:
+        1. The rpc call uses `smart_get` to retrieve host objects, not the
+           stock django `get` call. This is true for most, if not all rpcs in
+           the rpc_interface.
+        2. The kwargs to the function contain either a list of host ids or
+           hostnames, keyed under 'hosts'. This is true for all the rpc
+           functions that use 'smart_get'.
+
+    @param func: The function to decorate
+
+    @returns: The function to replace func with.
+    """
+    def replacement(**kwargs):
+        if not is_shard():
+
+            # Figure out which hosts are on which shards.
+            shard_host_map = {}
+            for host in models.Host.smart_get_bulk(kwargs['hosts']):
+                if host.shard:
+                    shard_host_map.setdefault(
+                            host.shard.hostname, []).append(host.hostname)
+
+            # Execute the rpc against the appropriate shards.
+            for shard, hostnames in shard_host_map.iteritems():
+                kwargs['hosts'] = hostnames
+                run_rpc_on_multiple_hostnames(func.func_name, [shard],
+                                              **kwargs)
+        return func(**kwargs)
+
+    return replacement
+
+
 def run_rpc_on_multiple_hostnames(rpc_call, shard_hostnames, **kwargs):
     """Runs an rpc to multiple AFEs