Introduce KMP_DEVICE_THREAD_LIMIT

This change drops in KMP_DEVICE_THREAD_LIMIT to replace KMP_MAX_THREADS. It's
possible there will eventually be a OMP_DEVICE_THREAD_LIMIT, and we need
something to distinguish from OMP_THREAD_LIMIT, which is currently implemented
incorrectly (the fix for that will be added soon in a separate patch).
KMP_ALL_THREADS is deprecated here, but we can keep the "all" option on
KMP_DEVICE_THREAD_LIMIT to support that functionality. KMP_DEVICE_THREAD_LIMIT
now has priority over its deprecated rival KMP_ALL_THREADS. I also cleaned up
some comments that incorrectly referred to non-existent kmp_max_threads variable
instead of kmp_max_nth.

I've left the name of where this setting eventually ends up as
__kmp_max_nth, for now.

This change does not change much in the way of functionality. It does NOT change
OMP_THREAD_LIMIT. It's just cleaning up and setting up for that.

Patch by Terry Wilmarth

Differential Revision: https://reviews.llvm.org/D35860

llvm-svn: 309168
diff --git a/openmp/runtime/src/kmp_runtime.cpp b/openmp/runtime/src/kmp_runtime.cpp
index c9dba4e..286f4a7 100644
--- a/openmp/runtime/src/kmp_runtime.cpp
+++ b/openmp/runtime/src/kmp_runtime.cpp
@@ -881,7 +881,7 @@
     KMP_ASSERT(0);
   }
 
-  // Respect KMP_ALL_THREADS, KMP_MAX_THREADS, OMP_THREAD_LIMIT.
+  // Respect KMP_ALL_THREADS, KMP_DEVICE_THREAD_LIMIT, OMP_THREAD_LIMIT.
   if (__kmp_nth + new_nthreads -
           (root->r.r_active ? 1 : root->r.r_hot_team->t.t_nproc) >
       __kmp_max_nth) {
@@ -3513,7 +3513,7 @@
     //
     // 1) The initialization thread (gtid = 0) exits.  __kmp_threads[0]
     //    may not be resused by another thread, so we may need to increase
-    //    __kmp_threads_capacity to __kmp_max_threads + 1.
+    //    __kmp_threads_capacity to __kmp_max_nth + 1.
     //
     // 2) New foreign root(s) are encountered.  We always register new foreign
     //    roots. This may cause a smaller # of threads to be allocated at
@@ -3521,7 +3521,7 @@
     //    eventually go to sleep) and need slots in the __kmp_threads[] array.
     //
     // Anyway, that is the reason for moving the check to see if
-    // __kmp_max_threads was exceeded into __kmp_reseerve_threads()
+    // __kmp_max_nth was exceeded into __kmp_reserve_threads()
     // instead of having it performed here. -BB
     old_tp_cached = __kmp_tp_cached;
     __kmp_actual_max_nth =