Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / python / google-api-python-client / 85c2c6d80859c466d29946e66ccd6a1c3a2df748 / . / docs / dyn / dataproc_v1beta1.projects.jobs.html
blob: 6dfcc9c4d37c395aae85bcf08be5576605a9ef41 [file] [log] [blame]
Sai Cheemalapatidf613972016-10-21 13:59:49 -0700[diff] [blame]1<html><body>
2<style>
3
4body, h1, h2, h3, div, span, p, pre, a {
5 margin: 0;
6 padding: 0;
7 border: 0;
8 font-weight: inherit;
9 font-style: inherit;
10 font-size: 100%;
11 font-family: inherit;
12 vertical-align: baseline;
13}
14
15body {
16 font-size: 13px;
17 padding: 1em;
18}
19
20h1 {
21 font-size: 26px;
22 margin-bottom: 1em;
23}
24
25h2 {
26 font-size: 24px;
27 margin-bottom: 1em;
28}
29
30h3 {
31 font-size: 20px;
32 margin-bottom: 1em;
33 margin-top: 1em;
34}
35
36pre, code {
37 line-height: 1.5;
38 font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace;
39}
40
41pre {
42 margin-top: 0.5em;
43}
44
45h1, h2, h3, p {
46 font-family: Arial, sans serif;
47}
48
49h1, h2, h3 {
50 border-bottom: solid #CCC 1px;
51}
52
53.toc_element {
54 margin-top: 0.5em;
55}
56
57.firstline {
58 margin-left: 2 em;
59}
60
61.method {
62 margin-top: 1em;
63 border: solid 1px #CCC;
64 padding: 1em;
65 background: #EEE;
66}
67
68.details {
69 font-weight: bold;
70 font-size: 14px;
71}
72
73</style>
74
75<h1><a href="dataproc_v1beta1.html">Google Cloud Dataproc API</a> . <a href="dataproc_v1beta1.projects.html">projects</a> . <a href="dataproc_v1beta1.projects.jobs.html">jobs</a></h1>
76<h2>Instance Methods</h2>
77<p class="toc_element">
78 <code><a href="#cancel">cancel(projectId, jobId, body, x__xgafv=None)</a></code></p>
79<p class="firstline">Starts a job cancellation request. To access the job resource after cancellation, call [jobs.list](/dataproc/reference/rest/v1beta1/projects.jobs/list) or [jobs.get](/dataproc/reference/rest/v1beta1/projects.jobs/get).</p>
80<p class="toc_element">
81 <code><a href="#delete">delete(projectId, jobId, x__xgafv=None)</a></code></p>
82<p class="firstline">Deletes the job from the project. If the job is active, the delete fails, and the response returns `FAILED_PRECONDITION`.</p>
83<p class="toc_element">
84 <code><a href="#get">get(projectId, jobId, x__xgafv=None)</a></code></p>
85<p class="firstline">Gets the resource representation for a job in a project.</p>
86<p class="toc_element">
87 <code><a href="#list">list(projectId, jobStateMatcher=None, pageSize=None, clusterName=None, x__xgafv=None, pageToken=None)</a></code></p>
88<p class="firstline">Lists jobs in a project.</p>
89<p class="toc_element">
90 <code><a href="#list_next">list_next(previous_request, previous_response)</a></code></p>
91<p class="firstline">Retrieves the next page of results.</p>
92<p class="toc_element">
93 <code><a href="#submit">submit(projectId, body, x__xgafv=None)</a></code></p>
94<p class="firstline">Submits a job to a cluster.</p>
95<h3>Method Details</h3>
96<div class="method">
97 <code class="details" id="cancel">cancel(projectId, jobId, body, x__xgafv=None)</code>
98 <pre>Starts a job cancellation request. To access the job resource after cancellation, call [jobs.list](/dataproc/reference/rest/v1beta1/projects.jobs/list) or [jobs.get](/dataproc/reference/rest/v1beta1/projects.jobs/get).
99
100Args:
101 projectId: string, [Required] The ID of the Google Cloud Platform project that the job belongs to. (required)
102 jobId: string, [Required] The job ID. (required)
103 body: object, The request body. (required)
104 The object takes the form of:
105
106{ # A request to cancel a job.
107 }
108
109 x__xgafv: string, V1 error format.
110
111Returns:
112 An object of the form:
113
114 { # A Cloud Dataproc job resource.
115 "status": { # Cloud Dataproc job status. # [Output-only] The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
116 "state": "A String", # [Required] A state message specifying the overall job state.
117 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
118 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
119 },
120 "hadoopJob": { # A Cloud Dataproc job for running Hadoop MapReduce jobs on YARN. # Job is a Hadoop job.
121 "jarFileUris": [ # [Optional] Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
122 "A String",
123 ],
124 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `-libjars` or `-Dfoo=bar`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
125 "A String",
126 ],
127 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
128 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
129 "a_key": "A String",
130 },
131 },
132 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
133 "A String",
134 ],
135 "mainClass": "A String", # The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in `jar_file_uris`.
136 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
137 "A String",
138 ],
139 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
140 "properties": { # [Optional] A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
141 "a_key": "A String",
142 },
143 },
144 "statusHistory": [ # [Output-only] The previous job status.
145 { # Cloud Dataproc job status.
146 "state": "A String", # [Required] A state message specifying the overall job state.
147 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
148 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
149 },
150 ],
151 "placement": { # Cloud Dataproc job configuration. # [Required] Job information, including how, when, and where to run the job.
152 "clusterName": "A String", # [Required] The name of the cluster where the job will be submitted.
153 "clusterUuid": "A String", # [Output-only] A cluster UUID generated by the Dataproc service when the job is submitted.
154 },
155 "reference": { # Encapsulates the full scoping used to reference a job. # [Optional] The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
156 "projectId": "A String", # [Required] The ID of the Google Cloud Platform project that the job belongs to.
157 "jobId": "A String", # [Required] The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 512 characters.
158 },
159 "sparkSqlJob": { # A Cloud Dataproc job for running Spark SQL queries. # Job is a SparkSql job.
160 "queryFileUri": "A String", # The HCFS URI of the script that contains SQL queries.
161 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Spark SQL command: SET `name="value";`).
162 "a_key": "A String",
163 },
164 "jarFileUris": [ # [Optional] HCFS URIs of jar files to be added to the Spark CLASSPATH.
165 "A String",
166 ],
167 "queryList": { # A list of queries to run on a cluster. # A list of queries.
168 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
169 "A String",
170 ],
171 },
172 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
173 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
174 "a_key": "A String",
175 },
176 },
177 "properties": { # [Optional] A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
178 "a_key": "A String",
179 },
180 },
181 "pigJob": { # A Cloud Dataproc job for running Pig queries on YARN. # Job is a Pig job.
182 "queryFileUri": "A String", # The HCFS URI of the script that contains the Pig queries.
183 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Pig command: `name=[value]`).
184 "a_key": "A String",
185 },
186 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
187 "A String",
188 ],
189 "queryList": { # A list of queries to run on a cluster. # A list of queries.
190 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
191 "A String",
192 ],
193 },
194 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
195 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
196 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
197 "a_key": "A String",
198 },
199 },
200 "properties": { # [Optional] A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
201 "a_key": "A String",
202 },
203 },
204 "driverOutputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdout of the job's driver program.
205 "submittedBy": "A String", # [Output-only] The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
206 "driverInputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdin of the job's driver program, only set if the job is interactive.
207 "driverControlFilesUri": "A String", # [Output-only] If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as `driver_output_uri`.
208 "sparkJob": { # A Cloud Dataproc job for running Spark applications on YARN. # Job is a Spark job.
209 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
210 "A String",
211 ],
212 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
213 "A String",
214 ],
215 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
216 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
217 "a_key": "A String",
218 },
219 },
220 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
221 "A String",
222 ],
223 "mainClass": "A String", # The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in `jar_file_uris`.
224 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
225 "A String",
226 ],
227 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file that contains the main class.
228 "properties": { # [Optional] A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
229 "a_key": "A String",
230 },
231 },
232 "yarnApplications": [ # [Output-only] The collection of YARN applications spun up by this job.
233 { # A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
234 "progress": 3.14, # [Required] The numerical progress of the application, from 1 to 100.
235 "state": "A String", # [Required] The application state.
236 "name": "A String", # [Required] The application name.
237 "trackingUrl": "A String", # [Optional] The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
238 },
239 ],
240 "pysparkJob": { # A Cloud Dataproc job for running PySpark applications on YARN. # Job is a Pyspark job.
241 "mainPythonFileUri": "A String", # [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
242 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
243 "A String",
244 ],
245 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
246 "A String",
247 ],
248 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
249 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
250 "a_key": "A String",
251 },
252 },
253 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
254 "A String",
255 ],
256 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
257 "A String",
258 ],
259 "pythonFileUris": [ # [Optional] HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
260 "A String",
261 ],
262 "properties": { # [Optional] A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
263 "a_key": "A String",
264 },
265 },
266 "hiveJob": { # A Cloud Dataproc job for running Hive queries on YARN. # Job is a Hive job.
267 "queryFileUri": "A String", # The HCFS URI of the script that contains Hive queries.
268 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Hive command: `SET name="value";`).
269 "a_key": "A String",
270 },
271 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
272 "A String",
273 ],
274 "queryList": { # A list of queries to run on a cluster. # A list of queries.
275 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
276 "A String",
277 ],
278 },
279 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
280 "properties": { # [Optional] A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
281 "a_key": "A String",
282 },
283 },
284 "interactive": True or False, # [Optional] If set to `true`, the driver's stdin will be kept open and `driver_input_uri` will be set to provide a path at which additional input can be sent to the driver.
285 }</pre>
286</div>
287
288<div class="method">
289 <code class="details" id="delete">delete(projectId, jobId, x__xgafv=None)</code>
290 <pre>Deletes the job from the project. If the job is active, the delete fails, and the response returns `FAILED_PRECONDITION`.
291
292Args:
293 projectId: string, [Required] The ID of the Google Cloud Platform project that the job belongs to. (required)
294 jobId: string, [Required] The job ID. (required)
295 x__xgafv: string, V1 error format.
296
297Returns:
298 An object of the form:
299
300 { # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } The JSON representation for `Empty` is empty JSON object `{}`.
301 }</pre>
302</div>
303
304<div class="method">
305 <code class="details" id="get">get(projectId, jobId, x__xgafv=None)</code>
306 <pre>Gets the resource representation for a job in a project.
307
308Args:
309 projectId: string, [Required] The ID of the Google Cloud Platform project that the job belongs to. (required)
310 jobId: string, [Required] The job ID. (required)
311 x__xgafv: string, V1 error format.
312
313Returns:
314 An object of the form:
315
316 { # A Cloud Dataproc job resource.
317 "status": { # Cloud Dataproc job status. # [Output-only] The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
318 "state": "A String", # [Required] A state message specifying the overall job state.
319 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
320 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
321 },
322 "hadoopJob": { # A Cloud Dataproc job for running Hadoop MapReduce jobs on YARN. # Job is a Hadoop job.
323 "jarFileUris": [ # [Optional] Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
324 "A String",
325 ],
326 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `-libjars` or `-Dfoo=bar`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
327 "A String",
328 ],
329 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
330 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
331 "a_key": "A String",
332 },
333 },
334 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
335 "A String",
336 ],
337 "mainClass": "A String", # The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in `jar_file_uris`.
338 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
339 "A String",
340 ],
341 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
342 "properties": { # [Optional] A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
343 "a_key": "A String",
344 },
345 },
346 "statusHistory": [ # [Output-only] The previous job status.
347 { # Cloud Dataproc job status.
348 "state": "A String", # [Required] A state message specifying the overall job state.
349 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
350 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
351 },
352 ],
353 "placement": { # Cloud Dataproc job configuration. # [Required] Job information, including how, when, and where to run the job.
354 "clusterName": "A String", # [Required] The name of the cluster where the job will be submitted.
355 "clusterUuid": "A String", # [Output-only] A cluster UUID generated by the Dataproc service when the job is submitted.
356 },
357 "reference": { # Encapsulates the full scoping used to reference a job. # [Optional] The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
358 "projectId": "A String", # [Required] The ID of the Google Cloud Platform project that the job belongs to.
359 "jobId": "A String", # [Required] The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 512 characters.
360 },
361 "sparkSqlJob": { # A Cloud Dataproc job for running Spark SQL queries. # Job is a SparkSql job.
362 "queryFileUri": "A String", # The HCFS URI of the script that contains SQL queries.
363 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Spark SQL command: SET `name="value";`).
364 "a_key": "A String",
365 },
366 "jarFileUris": [ # [Optional] HCFS URIs of jar files to be added to the Spark CLASSPATH.
367 "A String",
368 ],
369 "queryList": { # A list of queries to run on a cluster. # A list of queries.
370 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
371 "A String",
372 ],
373 },
374 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
375 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
376 "a_key": "A String",
377 },
378 },
379 "properties": { # [Optional] A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
380 "a_key": "A String",
381 },
382 },
383 "pigJob": { # A Cloud Dataproc job for running Pig queries on YARN. # Job is a Pig job.
384 "queryFileUri": "A String", # The HCFS URI of the script that contains the Pig queries.
385 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Pig command: `name=[value]`).
386 "a_key": "A String",
387 },
388 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
389 "A String",
390 ],
391 "queryList": { # A list of queries to run on a cluster. # A list of queries.
392 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
393 "A String",
394 ],
395 },
396 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
397 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
398 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
399 "a_key": "A String",
400 },
401 },
402 "properties": { # [Optional] A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
403 "a_key": "A String",
404 },
405 },
406 "driverOutputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdout of the job's driver program.
407 "submittedBy": "A String", # [Output-only] The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
408 "driverInputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdin of the job's driver program, only set if the job is interactive.
409 "driverControlFilesUri": "A String", # [Output-only] If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as `driver_output_uri`.
410 "sparkJob": { # A Cloud Dataproc job for running Spark applications on YARN. # Job is a Spark job.
411 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
412 "A String",
413 ],
414 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
415 "A String",
416 ],
417 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
418 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
419 "a_key": "A String",
420 },
421 },
422 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
423 "A String",
424 ],
425 "mainClass": "A String", # The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in `jar_file_uris`.
426 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
427 "A String",
428 ],
429 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file that contains the main class.
430 "properties": { # [Optional] A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
431 "a_key": "A String",
432 },
433 },
434 "yarnApplications": [ # [Output-only] The collection of YARN applications spun up by this job.
435 { # A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
436 "progress": 3.14, # [Required] The numerical progress of the application, from 1 to 100.
437 "state": "A String", # [Required] The application state.
438 "name": "A String", # [Required] The application name.
439 "trackingUrl": "A String", # [Optional] The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
440 },
441 ],
442 "pysparkJob": { # A Cloud Dataproc job for running PySpark applications on YARN. # Job is a Pyspark job.
443 "mainPythonFileUri": "A String", # [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
444 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
445 "A String",
446 ],
447 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
448 "A String",
449 ],
450 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
451 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
452 "a_key": "A String",
453 },
454 },
455 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
456 "A String",
457 ],
458 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
459 "A String",
460 ],
461 "pythonFileUris": [ # [Optional] HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
462 "A String",
463 ],
464 "properties": { # [Optional] A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
465 "a_key": "A String",
466 },
467 },
468 "hiveJob": { # A Cloud Dataproc job for running Hive queries on YARN. # Job is a Hive job.
469 "queryFileUri": "A String", # The HCFS URI of the script that contains Hive queries.
470 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Hive command: `SET name="value";`).
471 "a_key": "A String",
472 },
473 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
474 "A String",
475 ],
476 "queryList": { # A list of queries to run on a cluster. # A list of queries.
477 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
478 "A String",
479 ],
480 },
481 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
482 "properties": { # [Optional] A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
483 "a_key": "A String",
484 },
485 },
486 "interactive": True or False, # [Optional] If set to `true`, the driver's stdin will be kept open and `driver_input_uri` will be set to provide a path at which additional input can be sent to the driver.
487 }</pre>
488</div>
489
490<div class="method">
491 <code class="details" id="list">list(projectId, jobStateMatcher=None, pageSize=None, clusterName=None, x__xgafv=None, pageToken=None)</code>
492 <pre>Lists jobs in a project.
493
494Args:
495 projectId: string, [Required] The ID of the Google Cloud Platform project that the job belongs to. (required)
496 jobStateMatcher: string, [Optional] Specifies enumerated categories of jobs to list.
497 pageSize: integer, [Optional] The number of results to return in each response.
498 clusterName: string, [Optional] If set, the returned jobs list includes only jobs that were submitted to the named cluster.
499 x__xgafv: string, V1 error format.
500 pageToken: string, [Optional] The page token, returned by a previous call, to request the next page of results.
501
502Returns:
503 An object of the form:
504
505 { # A list of jobs in a project.
506 "nextPageToken": "A String", # [Optional] This token is included in the response if there are more results to fetch. To fetch additional results, provide this value as the `page_token` in a subsequent ListJobsRequest.
507 "jobs": [ # [Output-only] Jobs list.
508 { # A Cloud Dataproc job resource.
509 "status": { # Cloud Dataproc job status. # [Output-only] The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
510 "state": "A String", # [Required] A state message specifying the overall job state.
511 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
512 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
513 },
514 "hadoopJob": { # A Cloud Dataproc job for running Hadoop MapReduce jobs on YARN. # Job is a Hadoop job.
515 "jarFileUris": [ # [Optional] Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
516 "A String",
517 ],
518 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `-libjars` or `-Dfoo=bar`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
519 "A String",
520 ],
521 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
522 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
523 "a_key": "A String",
524 },
525 },
526 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
527 "A String",
528 ],
529 "mainClass": "A String", # The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in `jar_file_uris`.
530 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
531 "A String",
532 ],
533 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
534 "properties": { # [Optional] A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
535 "a_key": "A String",
536 },
537 },
538 "statusHistory": [ # [Output-only] The previous job status.
539 { # Cloud Dataproc job status.
540 "state": "A String", # [Required] A state message specifying the overall job state.
541 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
542 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
543 },
544 ],
545 "placement": { # Cloud Dataproc job configuration. # [Required] Job information, including how, when, and where to run the job.
546 "clusterName": "A String", # [Required] The name of the cluster where the job will be submitted.
547 "clusterUuid": "A String", # [Output-only] A cluster UUID generated by the Dataproc service when the job is submitted.
548 },
549 "reference": { # Encapsulates the full scoping used to reference a job. # [Optional] The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
550 "projectId": "A String", # [Required] The ID of the Google Cloud Platform project that the job belongs to.
551 "jobId": "A String", # [Required] The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 512 characters.
552 },
553 "sparkSqlJob": { # A Cloud Dataproc job for running Spark SQL queries. # Job is a SparkSql job.
554 "queryFileUri": "A String", # The HCFS URI of the script that contains SQL queries.
555 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Spark SQL command: SET `name="value";`).
556 "a_key": "A String",
557 },
558 "jarFileUris": [ # [Optional] HCFS URIs of jar files to be added to the Spark CLASSPATH.
559 "A String",
560 ],
561 "queryList": { # A list of queries to run on a cluster. # A list of queries.
562 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
563 "A String",
564 ],
565 },
566 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
567 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
568 "a_key": "A String",
569 },
570 },
571 "properties": { # [Optional] A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
572 "a_key": "A String",
573 },
574 },
575 "pigJob": { # A Cloud Dataproc job for running Pig queries on YARN. # Job is a Pig job.
576 "queryFileUri": "A String", # The HCFS URI of the script that contains the Pig queries.
577 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Pig command: `name=[value]`).
578 "a_key": "A String",
579 },
580 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
581 "A String",
582 ],
583 "queryList": { # A list of queries to run on a cluster. # A list of queries.
584 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
585 "A String",
586 ],
587 },
588 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
589 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
590 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
591 "a_key": "A String",
592 },
593 },
594 "properties": { # [Optional] A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
595 "a_key": "A String",
596 },
597 },
598 "driverOutputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdout of the job's driver program.
599 "submittedBy": "A String", # [Output-only] The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
600 "driverInputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdin of the job's driver program, only set if the job is interactive.
601 "driverControlFilesUri": "A String", # [Output-only] If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as `driver_output_uri`.
602 "sparkJob": { # A Cloud Dataproc job for running Spark applications on YARN. # Job is a Spark job.
603 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
604 "A String",
605 ],
606 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
607 "A String",
608 ],
609 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
610 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
611 "a_key": "A String",
612 },
613 },
614 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
615 "A String",
616 ],
617 "mainClass": "A String", # The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in `jar_file_uris`.
618 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
619 "A String",
620 ],
621 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file that contains the main class.
622 "properties": { # [Optional] A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
623 "a_key": "A String",
624 },
625 },
626 "yarnApplications": [ # [Output-only] The collection of YARN applications spun up by this job.
627 { # A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
628 "progress": 3.14, # [Required] The numerical progress of the application, from 1 to 100.
629 "state": "A String", # [Required] The application state.
630 "name": "A String", # [Required] The application name.
631 "trackingUrl": "A String", # [Optional] The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
632 },
633 ],
634 "pysparkJob": { # A Cloud Dataproc job for running PySpark applications on YARN. # Job is a Pyspark job.
635 "mainPythonFileUri": "A String", # [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
636 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
637 "A String",
638 ],
639 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
640 "A String",
641 ],
642 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
643 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
644 "a_key": "A String",
645 },
646 },
647 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
648 "A String",
649 ],
650 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
651 "A String",
652 ],
653 "pythonFileUris": [ # [Optional] HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
654 "A String",
655 ],
656 "properties": { # [Optional] A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
657 "a_key": "A String",
658 },
659 },
660 "hiveJob": { # A Cloud Dataproc job for running Hive queries on YARN. # Job is a Hive job.
661 "queryFileUri": "A String", # The HCFS URI of the script that contains Hive queries.
662 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Hive command: `SET name="value";`).
663 "a_key": "A String",
664 },
665 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
666 "A String",
667 ],
668 "queryList": { # A list of queries to run on a cluster. # A list of queries.
669 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
670 "A String",
671 ],
672 },
673 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
674 "properties": { # [Optional] A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
675 "a_key": "A String",
676 },
677 },
678 "interactive": True or False, # [Optional] If set to `true`, the driver's stdin will be kept open and `driver_input_uri` will be set to provide a path at which additional input can be sent to the driver.
679 },
680 ],
681 }</pre>
682</div>
683
684<div class="method">
685 <code class="details" id="list_next">list_next(previous_request, previous_response)</code>
686 <pre>Retrieves the next page of results.
687
688Args:
689 previous_request: The request for the previous page. (required)
690 previous_response: The response from the request for the previous page. (required)
691
692Returns:
693 A request object that you can call 'execute()' on to request the next
694 page. Returns None if there are no more items in the collection.
695 </pre>
696</div>
697
698<div class="method">
699 <code class="details" id="submit">submit(projectId, body, x__xgafv=None)</code>
700 <pre>Submits a job to a cluster.
701
702Args:
703 projectId: string, [Required] The ID of the Google Cloud Platform project that the job belongs to. (required)
704 body: object, The request body. (required)
705 The object takes the form of:
706
707{ # A request to submit a job.
708 "job": { # A Cloud Dataproc job resource. # [Required] The job resource.
709 "status": { # Cloud Dataproc job status. # [Output-only] The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
710 "state": "A String", # [Required] A state message specifying the overall job state.
711 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
712 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
713 },
714 "hadoopJob": { # A Cloud Dataproc job for running Hadoop MapReduce jobs on YARN. # Job is a Hadoop job.
715 "jarFileUris": [ # [Optional] Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
716 "A String",
717 ],
718 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `-libjars` or `-Dfoo=bar`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
719 "A String",
720 ],
721 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
722 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
723 "a_key": "A String",
724 },
725 },
726 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
727 "A String",
728 ],
729 "mainClass": "A String", # The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in `jar_file_uris`.
730 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
731 "A String",
732 ],
733 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
734 "properties": { # [Optional] A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
735 "a_key": "A String",
736 },
737 },
738 "statusHistory": [ # [Output-only] The previous job status.
739 { # Cloud Dataproc job status.
740 "state": "A String", # [Required] A state message specifying the overall job state.
741 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
742 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
743 },
744 ],
745 "placement": { # Cloud Dataproc job configuration. # [Required] Job information, including how, when, and where to run the job.
746 "clusterName": "A String", # [Required] The name of the cluster where the job will be submitted.
747 "clusterUuid": "A String", # [Output-only] A cluster UUID generated by the Dataproc service when the job is submitted.
748 },
749 "reference": { # Encapsulates the full scoping used to reference a job. # [Optional] The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
750 "projectId": "A String", # [Required] The ID of the Google Cloud Platform project that the job belongs to.
751 "jobId": "A String", # [Required] The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 512 characters.
752 },
753 "sparkSqlJob": { # A Cloud Dataproc job for running Spark SQL queries. # Job is a SparkSql job.
754 "queryFileUri": "A String", # The HCFS URI of the script that contains SQL queries.
755 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Spark SQL command: SET `name="value";`).
756 "a_key": "A String",
757 },
758 "jarFileUris": [ # [Optional] HCFS URIs of jar files to be added to the Spark CLASSPATH.
759 "A String",
760 ],
761 "queryList": { # A list of queries to run on a cluster. # A list of queries.
762 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
763 "A String",
764 ],
765 },
766 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
767 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
768 "a_key": "A String",
769 },
770 },
771 "properties": { # [Optional] A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
772 "a_key": "A String",
773 },
774 },
775 "pigJob": { # A Cloud Dataproc job for running Pig queries on YARN. # Job is a Pig job.
776 "queryFileUri": "A String", # The HCFS URI of the script that contains the Pig queries.
777 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Pig command: `name=[value]`).
778 "a_key": "A String",
779 },
780 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
781 "A String",
782 ],
783 "queryList": { # A list of queries to run on a cluster. # A list of queries.
784 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
785 "A String",
786 ],
787 },
788 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
789 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
790 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
791 "a_key": "A String",
792 },
793 },
794 "properties": { # [Optional] A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
795 "a_key": "A String",
796 },
797 },
798 "driverOutputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdout of the job's driver program.
799 "submittedBy": "A String", # [Output-only] The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
800 "driverInputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdin of the job's driver program, only set if the job is interactive.
801 "driverControlFilesUri": "A String", # [Output-only] If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as `driver_output_uri`.
802 "sparkJob": { # A Cloud Dataproc job for running Spark applications on YARN. # Job is a Spark job.
803 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
804 "A String",
805 ],
806 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
807 "A String",
808 ],
809 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
810 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
811 "a_key": "A String",
812 },
813 },
814 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
815 "A String",
816 ],
817 "mainClass": "A String", # The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in `jar_file_uris`.
818 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
819 "A String",
820 ],
821 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file that contains the main class.
822 "properties": { # [Optional] A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
823 "a_key": "A String",
824 },
825 },
826 "yarnApplications": [ # [Output-only] The collection of YARN applications spun up by this job.
827 { # A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
828 "progress": 3.14, # [Required] The numerical progress of the application, from 1 to 100.
829 "state": "A String", # [Required] The application state.
830 "name": "A String", # [Required] The application name.
831 "trackingUrl": "A String", # [Optional] The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
832 },
833 ],
834 "pysparkJob": { # A Cloud Dataproc job for running PySpark applications on YARN. # Job is a Pyspark job.
835 "mainPythonFileUri": "A String", # [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
836 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
837 "A String",
838 ],
839 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
840 "A String",
841 ],
842 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
843 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
844 "a_key": "A String",
845 },
846 },
847 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
848 "A String",
849 ],
850 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
851 "A String",
852 ],
853 "pythonFileUris": [ # [Optional] HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
854 "A String",
855 ],
856 "properties": { # [Optional] A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
857 "a_key": "A String",
858 },
859 },
860 "hiveJob": { # A Cloud Dataproc job for running Hive queries on YARN. # Job is a Hive job.
861 "queryFileUri": "A String", # The HCFS URI of the script that contains Hive queries.
862 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Hive command: `SET name="value";`).
863 "a_key": "A String",
864 },
865 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
866 "A String",
867 ],
868 "queryList": { # A list of queries to run on a cluster. # A list of queries.
869 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
870 "A String",
871 ],
872 },
873 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
874 "properties": { # [Optional] A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
875 "a_key": "A String",
876 },
877 },
878 "interactive": True or False, # [Optional] If set to `true`, the driver's stdin will be kept open and `driver_input_uri` will be set to provide a path at which additional input can be sent to the driver.
879 },
880 }
881
882 x__xgafv: string, V1 error format.
883
884Returns:
885 An object of the form:
886
887 { # A Cloud Dataproc job resource.
888 "status": { # Cloud Dataproc job status. # [Output-only] The job status. Additional application-specific status information may be contained in the type_job and yarn_applications fields.
889 "state": "A String", # [Required] A state message specifying the overall job state.
890 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
891 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
892 },
893 "hadoopJob": { # A Cloud Dataproc job for running Hadoop MapReduce jobs on YARN. # Job is a Hadoop job.
894 "jarFileUris": [ # [Optional] Jar file URIs to add to the CLASSPATHs of the Hadoop driver and tasks.
895 "A String",
896 ],
897 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `-libjars` or `-Dfoo=bar`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
898 "A String",
899 ],
900 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
901 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
902 "a_key": "A String",
903 },
904 },
905 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Hadoop drivers and distributed tasks. Useful for naively parallel tasks.
906 "A String",
907 ],
908 "mainClass": "A String", # The name of the driver's main class. The jar file containing the class must be in the default CLASSPATH or specified in `jar_file_uris`.
909 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Hadoop drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, or .zip.
910 "A String",
911 ],
912 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file containing the main class. Examples: 'gs://foo-bucket/analytics-binaries/extract-useful-metrics-mr.jar' 'hdfs:/tmp/test-samples/custom-wordcount.jar' 'file:///home/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar'
913 "properties": { # [Optional] A mapping of property names to values, used to configure Hadoop. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site and classes in user code.
914 "a_key": "A String",
915 },
916 },
917 "statusHistory": [ # [Output-only] The previous job status.
918 { # Cloud Dataproc job status.
919 "state": "A String", # [Required] A state message specifying the overall job state.
920 "stateStartTime": "A String", # [Output-only] The time when this state was entered.
921 "details": "A String", # [Optional] Job state details, such as an error description if the state is ERROR.
922 },
923 ],
924 "placement": { # Cloud Dataproc job configuration. # [Required] Job information, including how, when, and where to run the job.
925 "clusterName": "A String", # [Required] The name of the cluster where the job will be submitted.
926 "clusterUuid": "A String", # [Output-only] A cluster UUID generated by the Dataproc service when the job is submitted.
927 },
928 "reference": { # Encapsulates the full scoping used to reference a job. # [Optional] The fully qualified reference to the job, which can be used to obtain the equivalent REST path of the job resource. If this property is not specified when a job is created, the server generates a job_id.
929 "projectId": "A String", # [Required] The ID of the Google Cloud Platform project that the job belongs to.
930 "jobId": "A String", # [Required] The job ID, which must be unique within the project. The job ID is generated by the server upon job submission or provided by the user as a means to perform retries without creating duplicate jobs. The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), or hyphens (-). The maximum length is 512 characters.
931 },
932 "sparkSqlJob": { # A Cloud Dataproc job for running Spark SQL queries. # Job is a SparkSql job.
933 "queryFileUri": "A String", # The HCFS URI of the script that contains SQL queries.
934 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Spark SQL command: SET `name="value";`).
935 "a_key": "A String",
936 },
937 "jarFileUris": [ # [Optional] HCFS URIs of jar files to be added to the Spark CLASSPATH.
938 "A String",
939 ],
940 "queryList": { # A list of queries to run on a cluster. # A list of queries.
941 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
942 "A String",
943 ],
944 },
945 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
946 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
947 "a_key": "A String",
948 },
949 },
950 "properties": { # [Optional] A mapping of property names to values, used to configure Spark SQL's SparkConf. Properties that conflict with values set by the Cloud Dataproc API may be overwritten.
951 "a_key": "A String",
952 },
953 },
954 "pigJob": { # A Cloud Dataproc job for running Pig queries on YARN. # Job is a Pig job.
955 "queryFileUri": "A String", # The HCFS URI of the script that contains the Pig queries.
956 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Pig command: `name=[value]`).
957 "a_key": "A String",
958 },
959 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Pig Client and Hadoop MapReduce (MR) tasks. Can contain Pig UDFs.
960 "A String",
961 ],
962 "queryList": { # A list of queries to run on a cluster. # A list of queries.
963 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
964 "A String",
965 ],
966 },
967 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
968 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
969 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
970 "a_key": "A String",
971 },
972 },
973 "properties": { # [Optional] A mapping of property names to values, used to configure Pig. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/pig/conf/pig.properties, and classes in user code.
974 "a_key": "A String",
975 },
976 },
977 "driverOutputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdout of the job's driver program.
978 "submittedBy": "A String", # [Output-only] The email address of the user submitting the job. For jobs submitted on the cluster, the address is username@hostname.
979 "driverInputResourceUri": "A String", # [Output-only] A URI pointing to the location of the stdin of the job's driver program, only set if the job is interactive.
980 "driverControlFilesUri": "A String", # [Output-only] If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as `driver_output_uri`.
981 "sparkJob": { # A Cloud Dataproc job for running Spark applications on YARN. # Job is a Spark job.
982 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Spark driver and tasks.
983 "A String",
984 ],
985 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
986 "A String",
987 ],
988 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
989 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
990 "a_key": "A String",
991 },
992 },
993 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Spark drivers and distributed tasks. Useful for naively parallel tasks.
994 "A String",
995 ],
996 "mainClass": "A String", # The name of the driver's main class. The jar file that contains the class must be in the default CLASSPATH or specified in `jar_file_uris`.
997 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of Spark drivers and tasks. Supported file types: .jar, .tar, .tar.gz, .tgz, and .zip.
998 "A String",
999 ],
1000 "mainJarFileUri": "A String", # The Hadoop Compatible Filesystem (HCFS) URI of the jar file that contains the main class.
1001 "properties": { # [Optional] A mapping of property names to values, used to configure Spark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
1002 "a_key": "A String",
1003 },
1004 },
1005 "yarnApplications": [ # [Output-only] The collection of YARN applications spun up by this job.
1006 { # A YARN application created by a job. Application information is a subset of org.apache.hadoop.yarn.proto.YarnProtos.ApplicationReportProto.
1007 "progress": 3.14, # [Required] The numerical progress of the application, from 1 to 100.
1008 "state": "A String", # [Required] The application state.
1009 "name": "A String", # [Required] The application name.
1010 "trackingUrl": "A String", # [Optional] The HTTP URL of the ApplicationMaster, HistoryServer, or TimelineServer that provides application-specific information. The URL uses the internal hostname, and requires a proxy server for resolution and, possibly, access.
1011 },
1012 ],
1013 "pysparkJob": { # A Cloud Dataproc job for running PySpark applications on YARN. # Job is a Pyspark job.
1014 "mainPythonFileUri": "A String", # [Required] The Hadoop Compatible Filesystem (HCFS) URI of the main Python file to use as the driver. Must be a .py file.
1015 "args": [ # [Optional] The arguments to pass to the driver. Do not include arguments, such as `--conf`, that can be set as job properties, since a collision may occur that causes an incorrect job submission.
1016 "A String",
1017 ],
1018 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
1019 "A String",
1020 ],
1021 "loggingConfiguration": { # The runtime logging configuration of the job. # [Optional] The runtime log configuration for job execution.
1022 "driverLogLevels": { # The per-package log levels for the driver. This may include "root" package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'
1023 "a_key": "A String",
1024 },
1025 },
1026 "fileUris": [ # [Optional] HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.
1027 "A String",
1028 ],
1029 "archiveUris": [ # [Optional] HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
1030 "A String",
1031 ],
1032 "pythonFileUris": [ # [Optional] HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.
1033 "A String",
1034 ],
1035 "properties": { # [Optional] A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
1036 "a_key": "A String",
1037 },
1038 },
1039 "hiveJob": { # A Cloud Dataproc job for running Hive queries on YARN. # Job is a Hive job.
1040 "queryFileUri": "A String", # The HCFS URI of the script that contains Hive queries.
1041 "scriptVariables": { # [Optional] Mapping of query variable names to values (equivalent to the Hive command: `SET name="value";`).
1042 "a_key": "A String",
1043 },
1044 "jarFileUris": [ # [Optional] HCFS URIs of jar files to add to the CLASSPATH of the Hive server and Hadoop MapReduce (MR) tasks. Can contain Hive SerDes and UDFs.
1045 "A String",
1046 ],
1047 "queryList": { # A list of queries to run on a cluster. # A list of queries.
1048 "queries": [ # [Required] The queries to execute. You do not need to terminate a query with a semicolon. Multiple queries can be specified in one string by separating each with a semicolon. Here is an example of an Cloud Dataproc API snippet that uses a QueryList to specify a HiveJob: "hiveJob": { "queryList": { "queries": [ "query1", "query2", "query3;query4", ] } }
1049 "A String",
1050 ],
1051 },
1052 "continueOnFailure": True or False, # [Optional] Whether to continue executing queries if a query fails. The default value is `false`. Setting to `true` can be useful when executing independent parallel queries.
1053 "properties": { # [Optional] A mapping of property names and values, used to configure Hive. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/hadoop/conf/*-site.xml, /etc/hive/conf/hive-site.xml, and classes in user code.
1054 "a_key": "A String",
1055 },
1056 },
1057 "interactive": True or False, # [Optional] If set to `true`, the driver's stdin will be kept open and `driver_input_uri` will be set to provide a path at which additional input can be sent to the driver.
1058 }</pre>
1059</div>
1060
1061</body></html>