Threaded generation of software paths

Re-land of: https://skia-review.googlesource.com/36560

All information needed by the thread is captured by the prepare
callback object, the lambda captures a pointer to that, and does the
mask render. Once it's done, it signals the semaphore (also owned by the
callback). The callback defers the semaphore wait even longer (into the
ASAP upload), so the odds of waiting for the thread are REALLY low.

Also did a bunch of cleanup along the way, and put in some trace markers
so we can monitor how well this is working.

Traces of a GM that includes GPU and SW path rendering (path-reverse):

Original:
    https://screenshot.googleplex.com/f5BG3901tQg.png
Threaded, with wait in the callback (notice pre flush callback blocking):
    https://screenshot.googleplex.com/htOSZFE2s04.png
Current version, with wait deferred to ASAP upload function:
    https://screenshot.googleplex.com/GHjD0U3C34q.png
Bug: skia:
Change-Id: Idb92f385590749f41328a9aec65b2a93f4775079
Reviewed-on: https://skia-review.googlesource.com/40775
Reviewed-by: Brian Salomon <bsalomon@google.com>
Commit-Queue: Brian Osman <brianosman@google.com>
diff --git a/infra/bots/recipes/test.py b/infra/bots/recipes/test.py
index 69ec34a..ad0334c 100644
--- a/infra/bots/recipes/test.py
+++ b/infra/bots/recipes/test.py
@@ -26,6 +26,15 @@
 
 def dm_flags(api, bot):
   args = []
+  configs = []
+  blacklisted = []
+
+  def blacklist(quad):
+    config, src, options, name = quad.split(' ') if type(quad) is str else quad
+    if (config == '_' or
+        config in configs or
+        (config[0] == '~' and config[1:] in configs)):
+      blacklisted.extend([config, src, options, name])
 
   # We've been spending lots of time writing out and especially uploading
   # .pdfs, but not doing anything further with them.  skia:6821
@@ -67,7 +76,6 @@
       'PixelC' in bot):
     args.append('--ignoreSigInt')
 
-  configs = []
   if api.vars.builder_cfg.get('cpu_or_gpu') == 'CPU':
     args.append('--nogpu')
 
@@ -148,8 +156,13 @@
 
     # We want to test both the OpenGL config and the GLES config on Linux Intel:
     # GL is used by Chrome, GLES is used by ChromeOS.
+    # Also do the Ganesh threading verification test (render with and without
+    # worker threads, using only the SW path renderer, and compare the results).
     if 'Intel' in bot and api.vars.is_linux:
-      configs.extend(['gles', 'glesdft', 'glessrgb'])
+      configs.extend(['gles', 'glesdft', 'glessrgb', 'gltestthreading'])
+      # skbug.com/6333, skbug.com/6419, skbug.com/6702
+      blacklist('gltestthreading gm _ lcdblendmodes')
+      blacklist('gltestthreading gm _ lcdoverlap')
 
     # The following devices do not support glessrgb.
     if 'glessrgb' in configs:
@@ -218,14 +231,6 @@
   if 'SK_FORCE_RASTER_PIPELINE_BLITTER' in bot:
     args.remove('tests')
 
-  blacklisted = []
-  def blacklist(quad):
-    config, src, options, name = quad.split(' ') if type(quad) is str else quad
-    if (config == '_' or
-        config in configs or
-        (config[0] == '~' and config[1:] in configs)):
-      blacklisted.extend([config, src, options, name])
-
   # Only run the 'svgparse_*' svgs on 8888.
   if api.vars.builder_cfg.get('cpu_or_gpu') == 'GPU':
     blacklist('_ svg _ svgparse_')
@@ -486,6 +491,7 @@
     match.extend(['~Once', '~Shared'])  # Not sure what's up with these tests.
 
   if 'TSAN' in bot:
+    args.extend(['--gpuThreads', '8'])
     match.extend(['~ReadWriteAlpha'])   # Flaky on TSAN-covered on nvidia bots.
     match.extend(['~RGBA4444TextureTest',  # Flakier than they are important.
                   '~RGB565TextureTest'])