Blame - docs/dyn/dataflow_v1b3.projects.locations.templates.html - platform/external/python/google-api-python-client

2017-06-06 18:46:08 -0400

[diff] [blame]

76

<h2>Instance Methods</h2>

77

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

78

<code><a href="#create">create(projectId, location, body=None, x__xgafv=None)</a></code></p>

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

79

<p class="firstline">Creates a Cloud Dataflow job from a template.</p>

80

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

81

<code><a href="#get">get(projectId, location, view=None, gcsPath=None, x__xgafv=None)</a></code></p>

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

82

<p class="firstline">Get the template associated with a template.</p>

83

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

84

<code><a href="#launch">launch(projectId, location, body=None, gcsPath=None, dynamicTemplate_gcsPath=None, dynamicTemplate_stagingLocation=None, validateOnly=None, x__xgafv=None)</a></code></p>

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

85

<p class="firstline">Launch a template.</p>

86

<h3>Method Details</h3>

87

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

88

<code class="details" id="create">create(projectId, location, body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

89

<pre>Creates a Cloud Dataflow job from a template.

90

91

Args:

92

projectId: string, Required. The ID of the Cloud Platform project that the job belongs to. (required)

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

93

location: string, The [regional endpoint]

94

(https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to

95

which to direct the request. (required)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

96

body: object, The request body.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

97

The object takes the form of:

98

99

{ # A request to create a Cloud Dataflow job from a template.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

100

"environment": { # The environment values to set at runtime. # The runtime environment for the job.

101

"workerRegion": "A String", # The Compute Engine region

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

102

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

103

# which worker processing should occur, e.g. "us-west1". Mutually exclusive

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

104

# with worker_zone. If neither worker_region nor worker_zone is specified,

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

105

# default to the control plane's region.

106

"numWorkers": 42, # The initial number of Google Compute Engine instnaces for the job.

107

"zone": "A String", # The Compute Engine [availability

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

108

# zone](https://cloud.google.com/compute/docs/regions-zones/regions-zones)

109

# for launching worker instances to run your pipeline.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

110

# In the future, worker_zone will take precedence.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

111

"workerZone": "A String", # The Compute Engine zone

112

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

113

# which worker processing should occur, e.g. "us-west1-a". Mutually exclusive

114

# with worker_region. If neither worker_region nor worker_zone is specified,

115

# a zone in the control plane's region is chosen based on available capacity.

116

# If both `worker_zone` and `zone` are set, `worker_zone` takes precedence.

117

"additionalUserLabels": { # Additional user labels to be specified for the job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

118

# Keys and values should follow the restrictions specified in the [labeling

119

# restrictions](https://cloud.google.com/compute/docs/labeling-resources#restrictions)

120

# page.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

121

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

122

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

123

"additionalExperiments": [ # Additional experiment flags for the job.

124

"A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

125

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

126

"maxWorkers": 42, # The maximum number of Google Compute Engine instances to be made

127

# available to your pipeline during execution, from 1 to 1000.

128

"serviceAccountEmail": "A String", # The email address of the service account to run the job as.

129

"machineType": "A String", # The machine type to use for the job. Defaults to the value from the

130

# template if not specified.

131

"subnetwork": "A String", # Subnetwork to which VMs will be assigned, if desired. Expected to be of

132

# the form "regions/REGION/subnetworks/SUBNETWORK".

133

"ipConfiguration": "A String", # Configuration for VM IPs.

134

"kmsKeyName": "A String", # Optional. Name for the Cloud KMS key for the job.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

135

# Key format is:

136

# projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

137

"bypassTempDirValidation": True or False, # Whether to bypass the safety checks for the job's temporary directory.

138

# Use with caution.

139

"tempLocation": "A String", # The Cloud Storage path to use for temporary files.

140

# Must be a valid Cloud Storage URL, beginning with `gs://`.

141

"network": "A String", # Network to which VMs will be assigned. If empty or unspecified,

142

# the service will use the network "default".

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

143

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

144

"location": "A String", # The [regional endpoint]

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

145

# (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to

146

# which to direct the request.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

147

"parameters": { # The runtime parameters to pass to the job.

148

"a_key": "A String",

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

149

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

150

"jobName": "A String", # Required. The job name to use for the created job.

151

"gcsPath": "A String", # Required. A Cloud Storage path to the template from which to

152

# create the job.

153

# Must be a valid Cloud Storage URL, beginning with `gs://`.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

154

}

155

156

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

163

164

{ # Defines a job to be run by the Cloud Dataflow service.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

165

"clientRequestId": "A String", # The client's unique identifier of the job, re-used across retried attempts.

166

# If this field is set, the service will ensure its uniqueness.

167

# The request to create a job will fail if the service has knowledge of a

168

# previously submitted job with the same client's ID and job name.

169

# The caller may use this field to ensure idempotence of job

170

# creation across retried attempts to create a job.

171

# By default, the field is empty and, in that case, the service ignores it.

172

"id": "A String", # The unique ID of this job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

173

#

174

# This field is set by the Cloud Dataflow service when the Job is

175

# created, and is immutable for the life of the job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

176

"currentStateTime": "A String", # The timestamp associated with the current state.

177

"transformNameMapping": { # The map of transform name prefixes of the job to be replaced to the

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

178

# corresponding name prefixes of the new job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

179

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

180

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

181

"environment": { # Describes the environment in which a Dataflow Job runs. # The environment for the job.

182

"sdkPipelineOptions": { # The Cloud Dataflow SDK pipeline options specified by the user. These

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

183

# options are passed through the service and are used to recreate the

184

# SDK pipeline options on the worker in a language agnostic and platform

185

# independent way.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

186

"a_key": "", # Properties of the object.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

187

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

188

"flexResourceSchedulingGoal": "A String", # Which Flexible Resource Scheduling mode to run in.

189

"workerPools": [ # The worker pools. At least one "harness" worker pool must be

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

190

# specified in order for the job to have workers.

191

{ # Describes one particular pool of Cloud Dataflow workers to be

192

# instantiated by the Cloud Dataflow service in order to perform the

193

# computations required by a job. Note that a workflow job may use

194

# multiple pools, in order to match the various computational

195

# requirements of the various stages of the job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

196

"defaultPackageSet": "A String", # The default package set to install. This allows the service to

197

# select a default set of packages which are useful to worker

198

# harnesses written in a particular language.

199

"network": "A String", # Network to which VMs will be assigned. If empty or unspecified,

200

# the service will use the network "default".

201

"zone": "A String", # Zone to run the worker pools in. If empty or unspecified, the service

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

202

# will attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

203

"numWorkers": 42, # Number of Google Compute Engine workers in this pool needed to

204

# execute the job. If zero or unspecified, the service will

205

# attempt to choose a reasonable default.

206

"numThreadsPerWorker": 42, # The number of threads per worker harness. If empty or unspecified, the

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

207

# service will choose a number of threads (according to the number of cores

208

# on the selected machine type for batch, or 1 by convention for streaming).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

209

"diskSourceImage": "A String", # Fully qualified source image for disks.

210

"packages": [ # Packages to be installed on workers.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

211

{ # The packages that must be installed in order for a worker to run the

212

# steps of the Cloud Dataflow job that will be assigned to its worker

213

# pool.

214

#

215

# This is the mechanism by which the Cloud Dataflow SDK causes code to

216

# be loaded onto the workers. For example, the Cloud Dataflow Java SDK

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

217

# might use this to install jars containing the user's code and all of the

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

218

# various dependencies (libraries, data files, etc.) required in order

219

# for that code to run.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

220

"location": "A String", # The resource to read the package from. The supported resource type is:

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

221

#

222

# Google Cloud Storage:

223

#

224

# storage.googleapis.com/{bucket}

225

# bucket.storage.googleapis.com/

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

226

"name": "A String", # The name of the package.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

227

},

228

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

229

"teardownPolicy": "A String", # Sets the policy for determining when to turndown worker pool.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

230

# Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and

231

# `TEARDOWN_NEVER`.

232

# `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether

233

# the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down

234

# if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn

235

# down.

236

#

237

# If the workers are not torn down by the service, they will

238

# continue to run and use Google Compute Engine VM resources in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

239

# user's project until they are explicitly terminated by the user.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

240

# Because of this, Google recommends using the `TEARDOWN_ALWAYS`

241

# policy except for small, manually supervised test jobs.

242

#

243

# If unknown or unspecified, the service will attempt to choose a reasonable

244

# default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

245

"onHostMaintenance": "A String", # The action to take on host maintenance, as defined by the Google

246

# Compute Engine API.

247

"poolArgs": { # Extra arguments for this worker pool.

248

"a_key": "", # Properties of the object. Contains field @type with type URL.

249

},

250

"diskSizeGb": 42, # Size of root disk for VMs, in GB. If zero or unspecified, the service will

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

251

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

252

"workerHarnessContainerImage": "A String", # Required. Docker container image that executes the Cloud Dataflow worker

253

# harness, residing in Google Container Registry.

254

#

255

# Deprecated for the Fn API path. Use sdk_harness_container_images instead.

256

"diskType": "A String", # Type of root disk for VMs. If empty or unspecified, the service will

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

257

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

258

"machineType": "A String", # Machine type (e.g. "n1-standard-1"). If empty or unspecified, the

259

# service will attempt to choose a reasonable default.

260

"kind": "A String", # The kind of the worker pool; currently only `harness` and `shuffle`

261

# are supported.

262

"dataDisks": [ # Data disks that are used by a VM in this workflow.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

263

{ # Describes the data disk used by a workflow job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

264

"sizeGb": 42, # Size of disk in GB. If zero or unspecified, the service will

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

265

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

266

"diskType": "A String", # Disk storage type, as defined by Google Compute Engine. This

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

267

# must be a disk type appropriate to the project and zone in which

268

# the workers will run. If unknown or unspecified, the service

269

# will attempt to choose a reasonable default.

270

#

271

# For example, the standard persistent disk type is a resource name

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

272

# typically ending in "pd-standard". If SSD persistent disks are

273

# available, the resource name typically ends with "pd-ssd". The

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

274

# actual valid values are defined the Google Compute Engine API,

275

# not by the Cloud Dataflow API; consult the Google Compute Engine

276

# documentation for more information about determining the set of

277

# available disk types for a particular project and zone.

278

#

279

# Google Compute Engine Disk types are local to a particular

280

# project in a particular zone, and so the resource name will

281

# typically look something like this:

282

#

283

# compute.googleapis.com/projects/project-id/zones/zone/diskTypes/pd-standard

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

284

"mountPoint": "A String", # Directory in a VM where disk is mounted.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

285

},

286

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

287

"sdkHarnessContainerImages": [ # Set of SDK harness containers needed to execute this pipeline. This will

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

288

# only be set in the Fn API path. For non-cross-language pipelines this

289

# should have only one entry. Cross-language pipelines will have two or more

290

# entries.

291

{ # Defines a SDK harness container for executing Dataflow pipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

292

"containerImage": "A String", # A docker container image that resides in Google Container Registry.

293

"useSingleCorePerContainer": True or False, # If true, recommends the Dataflow service to use only one core per SDK

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

294

# container instance with this image. If false (or unset) recommends using

295

# more than one core per SDK container instance with this image for

296

# efficiency. Note that Dataflow service may choose to override this property

297

# if needed.

298

},

299

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

300

"subnetwork": "A String", # Subnetwork to which VMs will be assigned, if desired. Expected to be of

301

# the form "regions/REGION/subnetworks/SUBNETWORK".

302

"ipConfiguration": "A String", # Configuration for VM IPs.

303

"taskrunnerSettings": { # Taskrunner configuration settings. # Settings passed through to Google Compute Engine workers when

304

# using the standard Dataflow task runner. Users should ignore

305

# this field.

306

"alsologtostderr": True or False, # Whether to also send taskrunner log info to stderr.

307

"taskGroup": "A String", # The UNIX group ID on the worker VM to use for tasks launched by

308

# taskrunner; e.g. "wheel".

309

"harnessCommand": "A String", # The command to launch the worker harness.

310

"logDir": "A String", # The directory on the VM to store logs.

311

"oauthScopes": [ # The OAuth2 scopes to be requested by the taskrunner in order to

312

# access the Cloud Dataflow API.

313

"A String",

314

],

315

"dataflowApiVersion": "A String", # The API version of endpoint, e.g. "v1b3"

316

"logUploadLocation": "A String", # Indicates where to put logs. If this is not specified, the logs

317

# will not be uploaded.

318

#

319

# The supported resource type is:

320

#

321

# Google Cloud Storage:

322

# storage.googleapis.com/{bucket}/{object}

323

# bucket.storage.googleapis.com/{object}

324

"streamingWorkerMainClass": "A String", # The streaming worker main class name.

325

"workflowFileName": "A String", # The file to store the workflow in.

326

"baseTaskDir": "A String", # The location on the worker for task-specific subdirectories.

327

"tempStoragePrefix": "A String", # The prefix of the resources the taskrunner should use for

328

# temporary storage.

329

#

330

# The supported resource type is:

331

#

332

# Google Cloud Storage:

333

# storage.googleapis.com/{bucket}/{object}

334

# bucket.storage.googleapis.com/{object}

335

"commandlinesFileName": "A String", # The file to store preprocessing commands in.

336

"languageHint": "A String", # The suggested backend language.

337

"baseUrl": "A String", # The base URL for the taskrunner to use when accessing Google Cloud APIs.

338

#

339

# When workers access Google Cloud APIs, they logically do so via

340

# relative URLs. If this field is specified, it supplies the base

341

# URL to use for resolving these relative URLs. The normative

342

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

343

# Locators".

344

#

345

# If not specified, the default value is "http://www.googleapis.com/"

346

"logToSerialconsole": True or False, # Whether to send taskrunner log info to Google Compute Engine VM serial

347

# console.

348

"continueOnException": True or False, # Whether to continue taskrunner if an exception is hit.

349

"parallelWorkerSettings": { # Provides data to pass through to the worker harness. # The settings to pass to the parallel worker harness.

350

"baseUrl": "A String", # The base URL for accessing Google Cloud APIs.

351

#

352

# When workers access Google Cloud APIs, they logically do so via

353

# relative URLs. If this field is specified, it supplies the base

354

# URL to use for resolving these relative URLs. The normative

355

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

356

# Locators".

357

#

358

# If not specified, the default value is "http://www.googleapis.com/"

359

"reportingEnabled": True or False, # Whether to send work progress updates to the service.

360

"servicePath": "A String", # The Cloud Dataflow service path relative to the root URL, for example,

361

# "dataflow/v1b3/projects".

362

"shuffleServicePath": "A String", # The Shuffle service path relative to the root URL, for example,

363

# "shuffle/v1beta1".

364

"workerId": "A String", # The ID of the worker running this pipeline.

365

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

366

# storage.

367

#

368

# The supported resource type is:

369

#

370

# Google Cloud Storage:

371

#

372

# storage.googleapis.com/{bucket}/{object}

373

# bucket.storage.googleapis.com/{object}

374

},

375

"vmId": "A String", # The ID string of the VM.

376

"taskUser": "A String", # The UNIX user ID on the worker VM to use for tasks launched by

377

# taskrunner; e.g. "root".

378

},

379

"autoscalingSettings": { # Settings for WorkerPool autoscaling. # Settings for autoscaling of this WorkerPool.

380

"maxNumWorkers": 42, # The maximum number of workers to cap scaling at.

381

"algorithm": "A String", # The algorithm to use for autoscaling.

382

},

383

"metadata": { # Metadata to set on the Google Compute Engine VMs.

384

"a_key": "A String",

385

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

386

},

387

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

388

"dataset": "A String", # The dataset for the current project where various workflow

389

# related tables are stored.

390

#

391

# The supported resource type is:

392

#

393

# Google BigQuery:

394

# bigquery.googleapis.com/{dataset}

395

"internalExperiments": { # Experimental settings.

396

"a_key": "", # Properties of the object. Contains field @type with type URL.

397

},

398

"workerRegion": "A String", # The Compute Engine region

399

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

400

# which worker processing should occur, e.g. "us-west1". Mutually exclusive

401

# with worker_zone. If neither worker_region nor worker_zone is specified,

402

# default to the control plane's region.

403

"serviceKmsKeyName": "A String", # If set, contains the Cloud KMS key identifier used to encrypt data

404

# at rest, AKA a Customer Managed Encryption Key (CMEK).

405

#

406

# Format:

407

# projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY

408

"userAgent": { # A description of the process that generated the request.

409

"a_key": "", # Properties of the object.

410

},

411

"workerZone": "A String", # The Compute Engine zone

412

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

413

# which worker processing should occur, e.g. "us-west1-a". Mutually exclusive

414

# with worker_region. If neither worker_region nor worker_zone is specified,

415

# a zone in the control plane's region is chosen based on available capacity.

416

"clusterManagerApiService": "A String", # The type of cluster manager API to use. If unknown or

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

417

# unspecified, the service will attempt to choose a reasonable

418

# default. This should be in the form of the API service name,

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

419

# e.g. "compute.googleapis.com".

420

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

421

# storage. The system will append the suffix "/temp-{JOBNAME} to

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

422

# this resource prefix, where {JOBNAME} is the value of the

423

# job_name field. The resulting bucket and object prefix is used

424

# as the prefix of the resources used to store temporary data

425

# needed during the job execution. NOTE: This will override the

426

# value in taskrunner_settings.

427

# The supported resource type is:

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

428

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

429

# Google Cloud Storage:

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

430

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

431

# storage.googleapis.com/{bucket}/{object}

432

# bucket.storage.googleapis.com/{object}

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

433

"experiments": [ # The list of experiments to enable.

434

"A String",

435

],

436

"version": { # A structure describing which components and their versions of the service

437

# are required in order to run the job.

438

"a_key": "", # Properties of the object.

439

},

440

"serviceAccountEmail": "A String", # Identity to run virtual machines as. Defaults to the default account.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

441

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

442

"stageStates": [ # This field may be mutated by the Cloud Dataflow service;

443

# callers cannot mutate it.

444

{ # A message describing the state of a particular execution stage.

445

"executionStageName": "A String", # The name of the execution stage.

446

"currentStateTime": "A String", # The time at which the stage transitioned to this state.

447

"executionStageState": "A String", # Executions stage states allow the same set of values as JobState.

448

},

449

],

450

"jobMetadata": { # Metadata available primarily for filtering jobs. Will be included in the # This field is populated by the Dataflow service to support filtering jobs

451

# by the metadata values provided here. Populated for ListJobs and all GetJob

452

# views SUMMARY and higher.

453

# ListJob response and Job SUMMARY view.

454

"bigTableDetails": [ # Identification of a BigTable source used in the Dataflow job.

455

{ # Metadata for a BigTable connector used by the job.

456

"tableId": "A String", # TableId accessed in the connection.

457

"projectId": "A String", # ProjectId accessed in the connection.

458

"instanceId": "A String", # InstanceId accessed in the connection.

459

},

460

],

461

"spannerDetails": [ # Identification of a Spanner source used in the Dataflow job.

462

{ # Metadata for a Spanner connector used by the job.

463

"databaseId": "A String", # DatabaseId accessed in the connection.

464

"instanceId": "A String", # InstanceId accessed in the connection.

465

"projectId": "A String", # ProjectId accessed in the connection.

466

},

467

],

468

"datastoreDetails": [ # Identification of a Datastore source used in the Dataflow job.

469

{ # Metadata for a Datastore connector used by the job.

470

"projectId": "A String", # ProjectId accessed in the connection.

471

"namespace": "A String", # Namespace used in the connection.

472

},

473

],

474

"sdkVersion": { # The version of the SDK used to run the job. # The SDK version used to run the job.

475

"versionDisplayName": "A String", # A readable string describing the version of the SDK.

476

"sdkSupportStatus": "A String", # The support status for this SDK version.

477

"version": "A String", # The version of the SDK used to run the job.

478

},

479

"bigqueryDetails": [ # Identification of a BigQuery source used in the Dataflow job.

480

{ # Metadata for a BigQuery connector used by the job.

481

"table": "A String", # Table accessed in the connection.

482

"dataset": "A String", # Dataset accessed in the connection.

483

"projectId": "A String", # Project accessed in the connection.

484

"query": "A String", # Query used to access data in the connection.

485

},

486

],

487

"fileDetails": [ # Identification of a File source used in the Dataflow job.

488

{ # Metadata for a File connector used by the job.

489

"filePattern": "A String", # File Pattern used to access files by the connector.

490

},

491

],

492

"pubsubDetails": [ # Identification of a PubSub source used in the Dataflow job.

493

{ # Metadata for a PubSub connector used by the job.

494

"subscription": "A String", # Subscription used in the connection.

495

"topic": "A String", # Topic accessed in the connection.

},

],

},

"createdFromSnapshotId": "A String", # If this is specified, the job's initial state is populated from the given

500

# snapshot.

501

"projectId": "A String", # The ID of the Cloud Platform project that the job belongs to.

502

"type": "A String", # The type of Cloud Dataflow job.

503

"pipelineDescription": { # A descriptive representation of submitted pipeline as well as the executed # Preliminary field: The format of this data may change at any time.

504

# A description of the user pipeline and stages through which it is executed.

505

# Created by Cloud Dataflow service. Only retrieved with

506

# JOB_VIEW_DESCRIPTION or JOB_VIEW_ALL.

507

# form. This data is provided by the Dataflow service for ease of visualizing

508

# the pipeline and interpreting Dataflow provided metrics.

509

"executionPipelineStage": [ # Description of each stage of execution of the pipeline.

510

{ # Description of the composing transforms, names/ids, and input/outputs of a

511

# stage of execution. Some composing transforms and sources may have been

512

# generated by the Dataflow service during execution planning.

513

"id": "A String", # Dataflow service generated id for this stage.

514

"componentTransform": [ # Transforms that comprise this execution stage.

515

{ # Description of a transform executed as part of an execution stage.

516

"originalTransform": "A String", # User name for the original user transform with which this transform is

517

# most closely associated.

518

"name": "A String", # Dataflow service generated name for this source.

519

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

520

},

521

],

522

"componentSource": [ # Collections produced and consumed by component transforms of this stage.

523

{ # Description of an interstitial value between transforms in an execution

524

# stage.

525

"name": "A String", # Dataflow service generated name for this source.

526

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

527

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

528

# source is most closely associated.

529

},

530

],

531

"kind": "A String", # Type of tranform this stage is executing.

532

"outputSource": [ # Output sources for this stage.

533

{ # Description of an input or output of an execution stage.

534

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

535

# source is most closely associated.

536

"name": "A String", # Dataflow service generated name for this source.

537

"sizeBytes": "A String", # Size of the source, if measurable.

538

"userName": "A String", # Human-readable name for this source; may be user or system generated.

539

},

540

],

541

"name": "A String", # Dataflow service generated name for this stage.

542

"inputSource": [ # Input sources for this stage.

543

{ # Description of an input or output of an execution stage.

544

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

545

# source is most closely associated.

546

"name": "A String", # Dataflow service generated name for this source.

547

"sizeBytes": "A String", # Size of the source, if measurable.

548

"userName": "A String", # Human-readable name for this source; may be user or system generated.

},

],

},

],

"originalPipelineTransform": [ # Description of each transform in the pipeline and collections between them.

554

{ # Description of the type, names/ids, and input/outputs for a transform.

555

"kind": "A String", # Type of transform.

556

"inputCollectionName": [ # User names for all collection inputs to this transform.

557

"A String",

558

],

559

"name": "A String", # User provided name for this transform instance.

560

"id": "A String", # SDK generated id of this transform instance.

561

"displayData": [ # Transform-specific display data.

562

{ # Data provided with a pipeline or transform to provide descriptive info.

563

"timestampValue": "A String", # Contains value if the data is of timestamp type.

564

"boolValue": True or False, # Contains value if the data is of a boolean type.

565

"javaClassValue": "A String", # Contains value if the data is of java class type.

566

"strValue": "A String", # Contains value if the data is of string type.

567

"int64Value": "A String", # Contains value if the data is of int64 type.

568

"durationValue": "A String", # Contains value if the data is of duration type.

569

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

570

# language namespace (i.e. python module) which defines the display data.

571

# This allows a dax monitoring system to specially handle the data

572

# and perform custom rendering.

573

"floatValue": 3.14, # Contains value if the data is of float type.

574

"key": "A String", # The key identifying the display data.

575

# This is intended to be used as a label for the display data

576

# when viewed in a dax monitoring system.

577

"shortStrValue": "A String", # A possible additional shorter value to display.

578

# For example a java_class_name_value of com.mypackage.MyDoFn

579

# will be stored with MyDoFn as the short_str_value and

580

# com.mypackage.MyDoFn as the java_class_name value.

581

# short_str_value can be displayed and java_class_name_value

582

# will be displayed as a tooltip.

583

"url": "A String", # An optional full URL.

584

"label": "A String", # An optional label to display in a dax UI for the element.

585

},

586

],

587

"outputCollectionName": [ # User names for all collection outputs to this transform.

"A String",

],

},

],

"displayData": [ # Pipeline level display data.

593

{ # Data provided with a pipeline or transform to provide descriptive info.

594

"timestampValue": "A String", # Contains value if the data is of timestamp type.

595

"boolValue": True or False, # Contains value if the data is of a boolean type.

596

"javaClassValue": "A String", # Contains value if the data is of java class type.

597

"strValue": "A String", # Contains value if the data is of string type.

598

"int64Value": "A String", # Contains value if the data is of int64 type.

599

"durationValue": "A String", # Contains value if the data is of duration type.

600

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

601

# language namespace (i.e. python module) which defines the display data.

602

# This allows a dax monitoring system to specially handle the data

603

# and perform custom rendering.

604

"floatValue": 3.14, # Contains value if the data is of float type.

605

"key": "A String", # The key identifying the display data.

606

# This is intended to be used as a label for the display data

607

# when viewed in a dax monitoring system.

608

"shortStrValue": "A String", # A possible additional shorter value to display.

609

# For example a java_class_name_value of com.mypackage.MyDoFn

610

# will be stored with MyDoFn as the short_str_value and

611

# com.mypackage.MyDoFn as the java_class_name value.

612

# short_str_value can be displayed and java_class_name_value

613

# will be displayed as a tooltip.

614

"url": "A String", # An optional full URL.

615

"label": "A String", # An optional label to display in a dax UI for the element.

},

],

},

"replaceJobId": "A String", # If this job is an update of an existing job, this field is the job ID

620

# of the job it replaced.

621

#

622

# When sending a `CreateJobRequest`, you can update a job by specifying it

623

# here. The job named here is stopped, and its intermediate state is

624

# transferred to this job.

625

"tempFiles": [ # A set of files the system should be aware of that are used

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

626

# for temporary storage. These temporary files will be

627

# removed on job completion.

628

# No duplicates are allowed.

629

# No file patterns are supported.

630

#

631

# The supported files are:

632

#

633

# Google Cloud Storage:

634

#

635

# storage.googleapis.com/{bucket}/{object}

636

# bucket.storage.googleapis.com/{object}

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

637

"A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

638

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

639

"name": "A String", # The user-specified Cloud Dataflow job name.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

640

#

641

# Only one Job with a given name may exist in a project at any

642

# given time. If a caller attempts to create a Job with the same

643

# name as an already-existing Job, the attempt returns the

644

# existing Job.

645

#

646

# The name must match the regular expression

647

# `[a-z]([-a-z0-9]{0,38}[a-z0-9])?`

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

648

"steps": [ # Exactly one of step or steps_location should be specified.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

649

#

650

# The top-level steps that constitute the entire job.

651

{ # Defines a particular step within a Cloud Dataflow job.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

652

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

653

# A job consists of multiple steps, each of which performs some

654

# specific operation as part of the overall job. Data is typically

655

# passed from one step to another as part of the job.

656

#

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

657

# Here's an example of a sequence of steps which together implement a

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

658

# Map-Reduce job:

659

#

660

# * Read a collection of data from some source, parsing the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

661

# collection's elements.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

662

#

663

# * Validate the elements.

664

#

665

# * Apply a user-defined function to map each element to some value

666

# and extract an element-specific key value.

667

#

668

# * Group elements with the same key into a single element with

669

# that key, transforming a multiply-keyed collection into a

670

# uniquely-keyed collection.

671

#

672

# * Write the elements out to some data sink.

673

#

674

# Note that the Cloud Dataflow service may be used to run many different

675

# types of jobs, not just Map-Reduce.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

676

"name": "A String", # The name that identifies the step. This must be unique for each

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

677

# step with respect to all other steps in the Cloud Dataflow job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

678

"kind": "A String", # The kind of step in the Cloud Dataflow job.

679

"properties": { # Named properties associated with the step. Each kind of

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

680

# predefined step has its own required set of properties.

681

# Must be provided on Create. Only retrieved with JOB_VIEW_ALL.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

682

"a_key": "", # Properties of the object.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

683

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

684

},

685

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

686

"replacedByJobId": "A String", # If another job is an update of this job (and thus, this job is in

687

# `JOB_STATE_UPDATED`), this field contains the ID of that job.

688

"executionInfo": { # Additional information about how a Cloud Dataflow job will be executed that # Deprecated.

689

# isn't contained in the submitted job.

690

"stages": { # A mapping from each stage to the information about that stage.

691

"a_key": { # Contains information about how a particular

692

# google.dataflow.v1beta3.Step will be executed.

693

"stepName": [ # The steps associated with the execution stage.

694

# Note that stages may have several steps, and that a given step

695

# might be run by more than one stage.

"A String",

],

},

},

},

"currentState": "A String", # The current state of the job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

702

#

703

# Jobs are created in the `JOB_STATE_STOPPED` state unless otherwise

704

# specified.

705

#

706

# A job in the `JOB_STATE_RUNNING` state may asynchronously enter a

707

# terminal state. After a job has reached a terminal state, no

708

# further state updates may be made.

709

#

710

# This field may be mutated by the Cloud Dataflow service;

711

# callers cannot mutate it.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

712

"location": "A String", # The [regional endpoint]

713

# (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that

714

# contains this job.

715

"startTime": "A String", # The timestamp when the job was started (transitioned to JOB_STATE_PENDING).

716

# Flexible resource scheduling jobs are started with some delay after job

717

# creation, so start_time is unset before start and is updated when the

718

# job is started by the Cloud Dataflow service. For other jobs, start_time

719

# always equals to create_time and is immutable and set by the Cloud Dataflow

720

# service.

721

"stepsLocation": "A String", # The GCS location where the steps are stored.

722

"labels": { # User-defined labels for this job.

723

#

724

# The labels map can contain no more than 64 entries. Entries of the labels

725

# map are UTF8 strings that comply with the following restrictions:

726

#

727

# * Keys must conform to regexp: \p{Ll}\p{Lo}{0,62}

728

# * Values must conform to regexp: [\p{Ll}\p{Lo}\p{N}_-]{0,63}

729

# * Both keys and values are additionally constrained to be <= 128 bytes in

730

# size.

731

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

732

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

733

"createTime": "A String", # The timestamp when the job was initially created. Immutable and set by the

734

# Cloud Dataflow service.

735

"requestedState": "A String", # The job's requested state.

736

#

737

# `UpdateJob` may be used to switch between the `JOB_STATE_STOPPED` and

738

# `JOB_STATE_RUNNING` states, by setting requested_state. `UpdateJob` may

739

# also be used to directly set a job's requested state to

740

# `JOB_STATE_CANCELLED` or `JOB_STATE_DONE`, irrevocably terminating the

741

# job if it has not already reached a terminal state.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

}</pre>

</div>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

746

<code class="details" id="get">get(projectId, location, view=None, gcsPath=None, x__xgafv=None)</code>

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

747

<pre>Get the template associated with a template.

748

749

Args:

750

projectId: string, Required. The ID of the Cloud Platform project that the job belongs to. (required)

751

location: string, The [regional endpoint]

752

(https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to

753

which to direct the request. (required)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

754

view: string, The view to retrieve. Defaults to METADATA_ONLY.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

755

gcsPath: string, Required. A Cloud Storage path to the template from which to

756

create the job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

757

Must be valid Cloud Storage URL, beginning with 'gs://'.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

758

x__xgafv: string, V1 error format.

759

Allowed values

760

1 - v1 error format

761

2 - v2 error format

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

762

763

Returns:

764

An object of the form:

765

766

{ # The response to a GetTemplate request.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

767

"status": { # The `Status` type defines a logical error model that is suitable for # The status of the get template request. Any problems with the

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

768

# request will be indicated in the error_details.

769

# different programming environments, including REST APIs and RPC APIs. It is

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

770

# used by [gRPC](https://github.com/grpc). Each `Status` message contains

771

# three pieces of data: error code, error message, and error details.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

772

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

773

# You can find out more about this error model and how to work with it in the

774

# [API Design Guide](https://cloud.google.com/apis/design/errors).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

775

"message": "A String", # A developer-facing error message, which should be in English. Any

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

776

# user-facing error message should be localized and sent in the

777

# google.rpc.Status.details field, or localized by the client.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

778

"details": [ # A list of messages that carry the error details. There is a common set of

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

779

# message types for APIs to use.

780

{

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

781

"a_key": "", # Properties of the object. Contains field @type with type URL.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

782

},

783

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

784

"code": 42, # The status code, which should be an enum value of google.rpc.Code.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

785

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

786

"templateType": "A String", # Template Type.

787

"metadata": { # Metadata describing a template. # The template metadata describing the template name, available

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

788

# parameters, etc.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

789

"name": "A String", # Required. The name of the template.

790

"parameters": [ # The parameters for the template.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

791

{ # Metadata for a specific parameter.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

792

"label": "A String", # Required. The label to display for the parameter.

793

"paramType": "A String", # Optional. The type of the parameter.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

794

# Used for selecting input picker.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

795

"helpText": "A String", # Required. The help text to display for the parameter.

796

"name": "A String", # Required. The name of the parameter.

797

"regexes": [ # Optional. Regexes that the parameter must match.

798

"A String",

799

],

800

"isOptional": True or False, # Optional. Whether the parameter is optional. Defaults to false.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

801

},

802

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

803

"description": "A String", # Optional. A description of the template.

804

},

805

"runtimeMetadata": { # RuntimeMetadata describing a runtime environment. # Describes the runtime metadata with SDKInfo and available parameters.

806

"sdkInfo": { # SDK Information. # SDK Info for the template.

807

"language": "A String", # Required. The SDK Language.

808

"version": "A String", # Optional. The SDK version.

809

},

810

"parameters": [ # The parameters for the template.

811

{ # Metadata for a specific parameter.

812

"label": "A String", # Required. The label to display for the parameter.

813

"paramType": "A String", # Optional. The type of the parameter.

814

# Used for selecting input picker.

815

"helpText": "A String", # Required. The help text to display for the parameter.

816

"name": "A String", # Required. The name of the parameter.

817

"regexes": [ # Optional. Regexes that the parameter must match.

818

"A String",

819

],

820

"isOptional": True or False, # Optional. Whether the parameter is optional. Defaults to false.

821

},

822

],

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

},

}</pre>

</div>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

828

<code class="details" id="launch">launch(projectId, location, body=None, gcsPath=None, dynamicTemplate_gcsPath=None, dynamicTemplate_stagingLocation=None, validateOnly=None, x__xgafv=None)</code>

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

829

<pre>Launch a template.

830

831

Args:

832

projectId: string, Required. The ID of the Cloud Platform project that the job belongs to. (required)

833

location: string, The [regional endpoint]

834

(https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to

835

which to direct the request. (required)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

836

body: object, The request body.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

837

The object takes the form of:

838

839

{ # Parameters to provide to the template being launched.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

840

"transformNameMapping": { # Only applicable when updating a pipeline. Map of transform name prefixes of

841

# the job to be replaced to the corresponding name prefixes of the new job.

842

"a_key": "A String",

843

},

844

"environment": { # The environment values to set at runtime. # The runtime environment for the job.

845

"workerRegion": "A String", # The Compute Engine region

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

846

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

847

# which worker processing should occur, e.g. "us-west1". Mutually exclusive

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

848

# with worker_zone. If neither worker_region nor worker_zone is specified,

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

849

# default to the control plane's region.

850

"numWorkers": 42, # The initial number of Google Compute Engine instnaces for the job.

851

"zone": "A String", # The Compute Engine [availability

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

852

# zone](https://cloud.google.com/compute/docs/regions-zones/regions-zones)

853

# for launching worker instances to run your pipeline.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

854

# In the future, worker_zone will take precedence.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

855

"workerZone": "A String", # The Compute Engine zone

856

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

857

# which worker processing should occur, e.g. "us-west1-a". Mutually exclusive

858

# with worker_region. If neither worker_region nor worker_zone is specified,

859

# a zone in the control plane's region is chosen based on available capacity.

860

# If both `worker_zone` and `zone` are set, `worker_zone` takes precedence.

861

"additionalUserLabels": { # Additional user labels to be specified for the job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

862

# Keys and values should follow the restrictions specified in the [labeling

863

# restrictions](https://cloud.google.com/compute/docs/labeling-resources#restrictions)

864

# page.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

865

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

866

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

867

"additionalExperiments": [ # Additional experiment flags for the job.

868

"A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

869

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

870

"maxWorkers": 42, # The maximum number of Google Compute Engine instances to be made

871

# available to your pipeline during execution, from 1 to 1000.

872

"serviceAccountEmail": "A String", # The email address of the service account to run the job as.

873

"machineType": "A String", # The machine type to use for the job. Defaults to the value from the

874

# template if not specified.

875

"subnetwork": "A String", # Subnetwork to which VMs will be assigned, if desired. Expected to be of

876

# the form "regions/REGION/subnetworks/SUBNETWORK".

877

"ipConfiguration": "A String", # Configuration for VM IPs.

878

"kmsKeyName": "A String", # Optional. Name for the Cloud KMS key for the job.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

879

# Key format is:

880

# projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

881

"bypassTempDirValidation": True or False, # Whether to bypass the safety checks for the job's temporary directory.

882

# Use with caution.

883

"tempLocation": "A String", # The Cloud Storage path to use for temporary files.

884

# Must be a valid Cloud Storage URL, beginning with `gs://`.

885

"network": "A String", # Network to which VMs will be assigned. If empty or unspecified,

886

# the service will use the network "default".

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

887

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

888

"update": True or False, # If set, replace the existing pipeline with the name specified by jobName

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

889

# with this pipeline, preserving state.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

890

"parameters": { # The runtime parameters to pass to the job.

891

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

892

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

893

"jobName": "A String", # Required. The job name to use for the created job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

894

}

895

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

896

gcsPath: string, A Cloud Storage path to the template from which to create

897

the job.

898

Must be valid Cloud Storage URL, beginning with 'gs://'.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

899

dynamicTemplate_gcsPath: string, Path to dynamic template spec file on GCS.

900

The file must be a Json serialized DynamicTemplateFieSpec object.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

901

dynamicTemplate_stagingLocation: string, Cloud Storage path for staging dependencies.

902

Must be a valid Cloud Storage URL, beginning with `gs://`.

903

validateOnly: boolean, If true, the request is validated but not actually executed.

904

Defaults to false.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

905

x__xgafv: string, V1 error format.

906

Allowed values

907

1 - v1 error format

908

2 - v2 error format

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

909

910

Returns:

911

An object of the form:

912

913

{ # Response to the request to launch a template.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

914

"job": { # Defines a job to be run by the Cloud Dataflow service. # The job that was launched, if the request was not a dry run and

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

915

# the job was successfully launched.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

916

"clientRequestId": "A String", # The client's unique identifier of the job, re-used across retried attempts.

917

# If this field is set, the service will ensure its uniqueness.

918

# The request to create a job will fail if the service has knowledge of a

919

# previously submitted job with the same client's ID and job name.

920

# The caller may use this field to ensure idempotence of job

921

# creation across retried attempts to create a job.

922

# By default, the field is empty and, in that case, the service ignores it.

923

"id": "A String", # The unique ID of this job.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

924

#

925

# This field is set by the Cloud Dataflow service when the Job is

926

# created, and is immutable for the life of the job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

927

"currentStateTime": "A String", # The timestamp associated with the current state.

928

"transformNameMapping": { # The map of transform name prefixes of the job to be replaced to the

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

929

# corresponding name prefixes of the new job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

930

"a_key": "A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

931

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

932

"environment": { # Describes the environment in which a Dataflow Job runs. # The environment for the job.

933

"sdkPipelineOptions": { # The Cloud Dataflow SDK pipeline options specified by the user. These

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

934

# options are passed through the service and are used to recreate the

935

# SDK pipeline options on the worker in a language agnostic and platform

936

# independent way.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

937

"a_key": "", # Properties of the object.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

938

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

939

"flexResourceSchedulingGoal": "A String", # Which Flexible Resource Scheduling mode to run in.

940

"workerPools": [ # The worker pools. At least one "harness" worker pool must be

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

941

# specified in order for the job to have workers.

942

{ # Describes one particular pool of Cloud Dataflow workers to be

943

# instantiated by the Cloud Dataflow service in order to perform the

944

# computations required by a job. Note that a workflow job may use

945

# multiple pools, in order to match the various computational

946

# requirements of the various stages of the job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

947

"defaultPackageSet": "A String", # The default package set to install. This allows the service to

948

# select a default set of packages which are useful to worker

949

# harnesses written in a particular language.

950

"network": "A String", # Network to which VMs will be assigned. If empty or unspecified,

951

# the service will use the network "default".

952

"zone": "A String", # Zone to run the worker pools in. If empty or unspecified, the service

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

953

# will attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

954

"numWorkers": 42, # Number of Google Compute Engine workers in this pool needed to

955

# execute the job. If zero or unspecified, the service will

956

# attempt to choose a reasonable default.

957

"numThreadsPerWorker": 42, # The number of threads per worker harness. If empty or unspecified, the

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

958

# service will choose a number of threads (according to the number of cores

959

# on the selected machine type for batch, or 1 by convention for streaming).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

960

"diskSourceImage": "A String", # Fully qualified source image for disks.

961

"packages": [ # Packages to be installed on workers.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

962

{ # The packages that must be installed in order for a worker to run the

963

# steps of the Cloud Dataflow job that will be assigned to its worker

964

# pool.

965

#

966

# This is the mechanism by which the Cloud Dataflow SDK causes code to

967

# be loaded onto the workers. For example, the Cloud Dataflow Java SDK

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

968

# might use this to install jars containing the user's code and all of the

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

969

# various dependencies (libraries, data files, etc.) required in order

970

# for that code to run.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

971

"location": "A String", # The resource to read the package from. The supported resource type is:

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

972

#

973

# Google Cloud Storage:

974

#

975

# storage.googleapis.com/{bucket}

976

# bucket.storage.googleapis.com/

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

977

"name": "A String", # The name of the package.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

978

},

979

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

980

"teardownPolicy": "A String", # Sets the policy for determining when to turndown worker pool.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

981

# Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and

982

# `TEARDOWN_NEVER`.

983

# `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether

984

# the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down

985

# if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn

986

# down.

987

#

988

# If the workers are not torn down by the service, they will

989

# continue to run and use Google Compute Engine VM resources in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

990

# user's project until they are explicitly terminated by the user.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

991

# Because of this, Google recommends using the `TEARDOWN_ALWAYS`

992

# policy except for small, manually supervised test jobs.

993

#

994

# If unknown or unspecified, the service will attempt to choose a reasonable

995

# default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

996

"onHostMaintenance": "A String", # The action to take on host maintenance, as defined by the Google

997

# Compute Engine API.

998

"poolArgs": { # Extra arguments for this worker pool.

999

"a_key": "", # Properties of the object. Contains field @type with type URL.

1000

},

1001

"diskSizeGb": 42, # Size of root disk for VMs, in GB. If zero or unspecified, the service will

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1002

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1003

"workerHarnessContainerImage": "A String", # Required. Docker container image that executes the Cloud Dataflow worker

1004

# harness, residing in Google Container Registry.

1005

#

1006

# Deprecated for the Fn API path. Use sdk_harness_container_images instead.

1007

"diskType": "A String", # Type of root disk for VMs. If empty or unspecified, the service will

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1008

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1009

"machineType": "A String", # Machine type (e.g. "n1-standard-1"). If empty or unspecified, the

1010

# service will attempt to choose a reasonable default.

1011

"kind": "A String", # The kind of the worker pool; currently only `harness` and `shuffle`

1012

# are supported.

1013

"dataDisks": [ # Data disks that are used by a VM in this workflow.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1014

{ # Describes the data disk used by a workflow job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1015

"sizeGb": 42, # Size of disk in GB. If zero or unspecified, the service will

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1016

# attempt to choose a reasonable default.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1017

"diskType": "A String", # Disk storage type, as defined by Google Compute Engine. This

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1018

# must be a disk type appropriate to the project and zone in which

1019

# the workers will run. If unknown or unspecified, the service

1020

# will attempt to choose a reasonable default.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1021

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1022

# For example, the standard persistent disk type is a resource name

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1023

# typically ending in "pd-standard". If SSD persistent disks are

1024

# available, the resource name typically ends with "pd-ssd". The

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1025

# actual valid values are defined the Google Compute Engine API,

1026

# not by the Cloud Dataflow API; consult the Google Compute Engine

1027

# documentation for more information about determining the set of

1028

# available disk types for a particular project and zone.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1029

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1030

# Google Compute Engine Disk types are local to a particular

1031

# project in a particular zone, and so the resource name will

1032

# typically look something like this:

1033

#

1034

# compute.googleapis.com/projects/project-id/zones/zone/diskTypes/pd-standard

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1035

"mountPoint": "A String", # Directory in a VM where disk is mounted.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1036

},

1037

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1038

"sdkHarnessContainerImages": [ # Set of SDK harness containers needed to execute this pipeline. This will

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1039

# only be set in the Fn API path. For non-cross-language pipelines this

1040

# should have only one entry. Cross-language pipelines will have two or more

1041

# entries.

1042

{ # Defines a SDK harness container for executing Dataflow pipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1043

"containerImage": "A String", # A docker container image that resides in Google Container Registry.

1044

"useSingleCorePerContainer": True or False, # If true, recommends the Dataflow service to use only one core per SDK

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1045

# container instance with this image. If false (or unset) recommends using

1046

# more than one core per SDK container instance with this image for

1047

# efficiency. Note that Dataflow service may choose to override this property

1048

# if needed.

1049

},

1050

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1051

"subnetwork": "A String", # Subnetwork to which VMs will be assigned, if desired. Expected to be of

1052

# the form "regions/REGION/subnetworks/SUBNETWORK".

1053

"ipConfiguration": "A String", # Configuration for VM IPs.

1054

"taskrunnerSettings": { # Taskrunner configuration settings. # Settings passed through to Google Compute Engine workers when

1055

# using the standard Dataflow task runner. Users should ignore

1056

# this field.

1057

"alsologtostderr": True or False, # Whether to also send taskrunner log info to stderr.

1058

"taskGroup": "A String", # The UNIX group ID on the worker VM to use for tasks launched by

1059

# taskrunner; e.g. "wheel".

1060

"harnessCommand": "A String", # The command to launch the worker harness.

1061

"logDir": "A String", # The directory on the VM to store logs.

1062

"oauthScopes": [ # The OAuth2 scopes to be requested by the taskrunner in order to

1063

# access the Cloud Dataflow API.

1064

"A String",

1065

],

1066

"dataflowApiVersion": "A String", # The API version of endpoint, e.g. "v1b3"

1067

"logUploadLocation": "A String", # Indicates where to put logs. If this is not specified, the logs

1068

# will not be uploaded.

1069

#

1070

# The supported resource type is:

1071

#

1072

# Google Cloud Storage:

1073

# storage.googleapis.com/{bucket}/{object}

1074

# bucket.storage.googleapis.com/{object}

1075

"streamingWorkerMainClass": "A String", # The streaming worker main class name.

1076

"workflowFileName": "A String", # The file to store the workflow in.

1077

"baseTaskDir": "A String", # The location on the worker for task-specific subdirectories.

1078

"tempStoragePrefix": "A String", # The prefix of the resources the taskrunner should use for

1079

# temporary storage.

1080

#

1081

# The supported resource type is:

1082

#

1083

# Google Cloud Storage:

1084

# storage.googleapis.com/{bucket}/{object}

1085

# bucket.storage.googleapis.com/{object}

1086

"commandlinesFileName": "A String", # The file to store preprocessing commands in.

1087

"languageHint": "A String", # The suggested backend language.

1088

"baseUrl": "A String", # The base URL for the taskrunner to use when accessing Google Cloud APIs.

1089

#

1090

# When workers access Google Cloud APIs, they logically do so via

1091

# relative URLs. If this field is specified, it supplies the base

1092

# URL to use for resolving these relative URLs. The normative

1093

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

1094

# Locators".

1095

#

1096

# If not specified, the default value is "http://www.googleapis.com/"

1097

"logToSerialconsole": True or False, # Whether to send taskrunner log info to Google Compute Engine VM serial

1098

# console.

1099

"continueOnException": True or False, # Whether to continue taskrunner if an exception is hit.

1100

"parallelWorkerSettings": { # Provides data to pass through to the worker harness. # The settings to pass to the parallel worker harness.

1101

"baseUrl": "A String", # The base URL for accessing Google Cloud APIs.

1102

#

1103

# When workers access Google Cloud APIs, they logically do so via

1104

# relative URLs. If this field is specified, it supplies the base

1105

# URL to use for resolving these relative URLs. The normative

1106

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

1107

# Locators".

1108

#

1109

# If not specified, the default value is "http://www.googleapis.com/"

1110

"reportingEnabled": True or False, # Whether to send work progress updates to the service.

1111

"servicePath": "A String", # The Cloud Dataflow service path relative to the root URL, for example,

1112

# "dataflow/v1b3/projects".

1113

"shuffleServicePath": "A String", # The Shuffle service path relative to the root URL, for example,

1114

# "shuffle/v1beta1".

1115

"workerId": "A String", # The ID of the worker running this pipeline.

1116

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

1117

# storage.

1118

#

1119

# The supported resource type is:

1120

#

1121

# Google Cloud Storage:

1122

#

1123

# storage.googleapis.com/{bucket}/{object}

1124

# bucket.storage.googleapis.com/{object}

1125

},

1126

"vmId": "A String", # The ID string of the VM.

1127

"taskUser": "A String", # The UNIX user ID on the worker VM to use for tasks launched by

1128

# taskrunner; e.g. "root".

1129

},

1130

"autoscalingSettings": { # Settings for WorkerPool autoscaling. # Settings for autoscaling of this WorkerPool.

1131

"maxNumWorkers": 42, # The maximum number of workers to cap scaling at.

1132

"algorithm": "A String", # The algorithm to use for autoscaling.

1133

},

1134

"metadata": { # Metadata to set on the Google Compute Engine VMs.

1135

"a_key": "A String",

1136

},

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1137

},

1138

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1139

"dataset": "A String", # The dataset for the current project where various workflow

1140

# related tables are stored.

1141

#

1142

# The supported resource type is:

1143

#

1144

# Google BigQuery:

1145

# bigquery.googleapis.com/{dataset}

1146

"internalExperiments": { # Experimental settings.

1147

"a_key": "", # Properties of the object. Contains field @type with type URL.

1148

},

1149

"workerRegion": "A String", # The Compute Engine region

1150

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

1151

# which worker processing should occur, e.g. "us-west1". Mutually exclusive

1152

# with worker_zone. If neither worker_region nor worker_zone is specified,

1153

# default to the control plane's region.

1154

"serviceKmsKeyName": "A String", # If set, contains the Cloud KMS key identifier used to encrypt data

1155

# at rest, AKA a Customer Managed Encryption Key (CMEK).

1156

#

1157

# Format:

1158

# projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY

1159

"userAgent": { # A description of the process that generated the request.

1160

"a_key": "", # Properties of the object.

1161

},

1162

"workerZone": "A String", # The Compute Engine zone

1163

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

1164

# which worker processing should occur, e.g. "us-west1-a". Mutually exclusive

1165

# with worker_region. If neither worker_region nor worker_zone is specified,

1166

# a zone in the control plane's region is chosen based on available capacity.

1167

"clusterManagerApiService": "A String", # The type of cluster manager API to use. If unknown or

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1168

# unspecified, the service will attempt to choose a reasonable

1169

# default. This should be in the form of the API service name,

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1170

# e.g. "compute.googleapis.com".

1171

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

1172

# storage. The system will append the suffix "/temp-{JOBNAME} to

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1173

# this resource prefix, where {JOBNAME} is the value of the

1174

# job_name field. The resulting bucket and object prefix is used

1175

# as the prefix of the resources used to store temporary data

1176

# needed during the job execution. NOTE: This will override the

1177

# value in taskrunner_settings.

1178

# The supported resource type is:

1179

#

1180

# Google Cloud Storage:

1181

#

1182

# storage.googleapis.com/{bucket}/{object}

1183

# bucket.storage.googleapis.com/{object}

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1184

"experiments": [ # The list of experiments to enable.

1185

"A String",

1186

],

1187

"version": { # A structure describing which components and their versions of the service

1188

# are required in order to run the job.

1189

"a_key": "", # Properties of the object.

1190

},

1191

"serviceAccountEmail": "A String", # Identity to run virtual machines as. Defaults to the default account.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1192

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1193

"stageStates": [ # This field may be mutated by the Cloud Dataflow service;

1194

# callers cannot mutate it.

1195

{ # A message describing the state of a particular execution stage.

1196

"executionStageName": "A String", # The name of the execution stage.

1197

"currentStateTime": "A String", # The time at which the stage transitioned to this state.

1198

"executionStageState": "A String", # Executions stage states allow the same set of values as JobState.

1199

},

1200

],

1201

"jobMetadata": { # Metadata available primarily for filtering jobs. Will be included in the # This field is populated by the Dataflow service to support filtering jobs

1202

# by the metadata values provided here. Populated for ListJobs and all GetJob

1203

# views SUMMARY and higher.

1204

# ListJob response and Job SUMMARY view.

1205

"bigTableDetails": [ # Identification of a BigTable source used in the Dataflow job.

1206

{ # Metadata for a BigTable connector used by the job.

1207

"tableId": "A String", # TableId accessed in the connection.

1208

"projectId": "A String", # ProjectId accessed in the connection.

1209

"instanceId": "A String", # InstanceId accessed in the connection.

1210

},

1211

],

1212

"spannerDetails": [ # Identification of a Spanner source used in the Dataflow job.

1213

{ # Metadata for a Spanner connector used by the job.

1214

"databaseId": "A String", # DatabaseId accessed in the connection.

1215

"instanceId": "A String", # InstanceId accessed in the connection.

1216

"projectId": "A String", # ProjectId accessed in the connection.

1217

},

1218

],

1219

"datastoreDetails": [ # Identification of a Datastore source used in the Dataflow job.

1220

{ # Metadata for a Datastore connector used by the job.

1221

"projectId": "A String", # ProjectId accessed in the connection.

1222

"namespace": "A String", # Namespace used in the connection.

1223

},

1224

],

1225

"sdkVersion": { # The version of the SDK used to run the job. # The SDK version used to run the job.

1226

"versionDisplayName": "A String", # A readable string describing the version of the SDK.

1227

"sdkSupportStatus": "A String", # The support status for this SDK version.

1228

"version": "A String", # The version of the SDK used to run the job.

1229

},

1230

"bigqueryDetails": [ # Identification of a BigQuery source used in the Dataflow job.

1231

{ # Metadata for a BigQuery connector used by the job.

1232

"table": "A String", # Table accessed in the connection.

1233

"dataset": "A String", # Dataset accessed in the connection.

1234

"projectId": "A String", # Project accessed in the connection.

1235

"query": "A String", # Query used to access data in the connection.

1236

},

1237

],

1238

"fileDetails": [ # Identification of a File source used in the Dataflow job.

1239

{ # Metadata for a File connector used by the job.

1240

"filePattern": "A String", # File Pattern used to access files by the connector.

1241

},

1242

],

1243

"pubsubDetails": [ # Identification of a PubSub source used in the Dataflow job.

1244

{ # Metadata for a PubSub connector used by the job.

1245

"subscription": "A String", # Subscription used in the connection.

1246

"topic": "A String", # Topic accessed in the connection.

},

],

},

"createdFromSnapshotId": "A String", # If this is specified, the job's initial state is populated from the given

1251

# snapshot.

1252

"projectId": "A String", # The ID of the Cloud Platform project that the job belongs to.

1253

"type": "A String", # The type of Cloud Dataflow job.

1254

"pipelineDescription": { # A descriptive representation of submitted pipeline as well as the executed # Preliminary field: The format of this data may change at any time.

1255

# A description of the user pipeline and stages through which it is executed.

1256

# Created by Cloud Dataflow service. Only retrieved with

1257

# JOB_VIEW_DESCRIPTION or JOB_VIEW_ALL.

1258

# form. This data is provided by the Dataflow service for ease of visualizing

1259

# the pipeline and interpreting Dataflow provided metrics.

1260

"executionPipelineStage": [ # Description of each stage of execution of the pipeline.

1261

{ # Description of the composing transforms, names/ids, and input/outputs of a

1262

# stage of execution. Some composing transforms and sources may have been

1263

# generated by the Dataflow service during execution planning.

1264

"id": "A String", # Dataflow service generated id for this stage.

1265

"componentTransform": [ # Transforms that comprise this execution stage.

1266

{ # Description of a transform executed as part of an execution stage.

1267

"originalTransform": "A String", # User name for the original user transform with which this transform is

1268

# most closely associated.

1269

"name": "A String", # Dataflow service generated name for this source.

1270

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

1271

},

1272

],

1273

"componentSource": [ # Collections produced and consumed by component transforms of this stage.

1274

{ # Description of an interstitial value between transforms in an execution

1275

# stage.

1276

"name": "A String", # Dataflow service generated name for this source.

1277

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

1278

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

1279

# source is most closely associated.

1280

},

1281

],

1282

"kind": "A String", # Type of tranform this stage is executing.

1283

"outputSource": [ # Output sources for this stage.

1284

{ # Description of an input or output of an execution stage.

1285

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

1286

# source is most closely associated.

1287

"name": "A String", # Dataflow service generated name for this source.

1288

"sizeBytes": "A String", # Size of the source, if measurable.

1289

"userName": "A String", # Human-readable name for this source; may be user or system generated.

1290

},

1291

],

1292

"name": "A String", # Dataflow service generated name for this stage.

1293

"inputSource": [ # Input sources for this stage.

1294

{ # Description of an input or output of an execution stage.

1295

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

1296

# source is most closely associated.

1297

"name": "A String", # Dataflow service generated name for this source.

1298

"sizeBytes": "A String", # Size of the source, if measurable.

1299

"userName": "A String", # Human-readable name for this source; may be user or system generated.

},

],

},

],

"originalPipelineTransform": [ # Description of each transform in the pipeline and collections between them.

1305

{ # Description of the type, names/ids, and input/outputs for a transform.

1306

"kind": "A String", # Type of transform.

1307

"inputCollectionName": [ # User names for all collection inputs to this transform.

1308

"A String",

1309

],

1310

"name": "A String", # User provided name for this transform instance.

1311

"id": "A String", # SDK generated id of this transform instance.

1312

"displayData": [ # Transform-specific display data.

1313

{ # Data provided with a pipeline or transform to provide descriptive info.

1314

"timestampValue": "A String", # Contains value if the data is of timestamp type.

1315

"boolValue": True or False, # Contains value if the data is of a boolean type.

1316

"javaClassValue": "A String", # Contains value if the data is of java class type.

1317

"strValue": "A String", # Contains value if the data is of string type.

1318

"int64Value": "A String", # Contains value if the data is of int64 type.

1319

"durationValue": "A String", # Contains value if the data is of duration type.

1320

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

1321

# language namespace (i.e. python module) which defines the display data.

1322

# This allows a dax monitoring system to specially handle the data

1323

# and perform custom rendering.

1324

"floatValue": 3.14, # Contains value if the data is of float type.

1325

"key": "A String", # The key identifying the display data.

1326

# This is intended to be used as a label for the display data

1327

# when viewed in a dax monitoring system.

1328

"shortStrValue": "A String", # A possible additional shorter value to display.

1329

# For example a java_class_name_value of com.mypackage.MyDoFn

1330

# will be stored with MyDoFn as the short_str_value and

1331

# com.mypackage.MyDoFn as the java_class_name value.

1332

# short_str_value can be displayed and java_class_name_value

1333

# will be displayed as a tooltip.

1334

"url": "A String", # An optional full URL.

1335

"label": "A String", # An optional label to display in a dax UI for the element.

1336

},

1337

],

1338

"outputCollectionName": [ # User names for all collection outputs to this transform.

"A String",

],

},

],

"displayData": [ # Pipeline level display data.

1344

{ # Data provided with a pipeline or transform to provide descriptive info.

1345

"timestampValue": "A String", # Contains value if the data is of timestamp type.

1346

"boolValue": True or False, # Contains value if the data is of a boolean type.

1347

"javaClassValue": "A String", # Contains value if the data is of java class type.

1348

"strValue": "A String", # Contains value if the data is of string type.

1349

"int64Value": "A String", # Contains value if the data is of int64 type.

1350

"durationValue": "A String", # Contains value if the data is of duration type.

1351

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

1352

# language namespace (i.e. python module) which defines the display data.

1353

# This allows a dax monitoring system to specially handle the data

1354

# and perform custom rendering.

1355

"floatValue": 3.14, # Contains value if the data is of float type.

1356

"key": "A String", # The key identifying the display data.

1357

# This is intended to be used as a label for the display data

1358

# when viewed in a dax monitoring system.

1359

"shortStrValue": "A String", # A possible additional shorter value to display.

1360

# For example a java_class_name_value of com.mypackage.MyDoFn

1361

# will be stored with MyDoFn as the short_str_value and

1362

# com.mypackage.MyDoFn as the java_class_name value.

1363

# short_str_value can be displayed and java_class_name_value

1364

# will be displayed as a tooltip.

1365

"url": "A String", # An optional full URL.

1366

"label": "A String", # An optional label to display in a dax UI for the element.

},

],

},

"replaceJobId": "A String", # If this job is an update of an existing job, this field is the job ID

1371

# of the job it replaced.

1372

#

1373

# When sending a `CreateJobRequest`, you can update a job by specifying it

1374

# here. The job named here is stopped, and its intermediate state is

1375

# transferred to this job.

1376

"tempFiles": [ # A set of files the system should be aware of that are used

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1377

# for temporary storage. These temporary files will be

1378

# removed on job completion.

1379

# No duplicates are allowed.

1380

# No file patterns are supported.

1381

#

1382

# The supported files are:

1383

#

1384

# Google Cloud Storage:

1385

#

1386

# storage.googleapis.com/{bucket}/{object}

1387

# bucket.storage.googleapis.com/{object}

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1388

"A String",

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1389

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1390

"name": "A String", # The user-specified Cloud Dataflow job name.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1391

#

1392

# Only one Job with a given name may exist in a project at any

1393

# given time. If a caller attempts to create a Job with the same

1394

# name as an already-existing Job, the attempt returns the

1395

# existing Job.

1396

#

1397

# The name must match the regular expression

1398

# `[a-z]([-a-z0-9]{0,38}[a-z0-9])?`

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1399

"steps": [ # Exactly one of step or steps_location should be specified.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1400

#

1401

# The top-level steps that constitute the entire job.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1402

{ # Defines a particular step within a Cloud Dataflow job.

1403

#

1404

# A job consists of multiple steps, each of which performs some

1405

# specific operation as part of the overall job. Data is typically

1406

# passed from one step to another as part of the job.

1407

#

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1408

# Here's an example of a sequence of steps which together implement a

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1409

# Map-Reduce job:

1410

#

1411

# * Read a collection of data from some source, parsing the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1412

# collection's elements.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1413

#

1414

# * Validate the elements.

1415

#

1416

# * Apply a user-defined function to map each element to some value

1417

# and extract an element-specific key value.

1418

#

1419

# * Group elements with the same key into a single element with

1420

# that key, transforming a multiply-keyed collection into a

1421

# uniquely-keyed collection.

1422

#

1423

# * Write the elements out to some data sink.

1424

#

1425

# Note that the Cloud Dataflow service may be used to run many different

1426

# types of jobs, not just Map-Reduce.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1427

"name": "A String", # The name that identifies the step. This must be unique for each

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1428

# step with respect to all other steps in the Cloud Dataflow job.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1429

"kind": "A String", # The kind of step in the Cloud Dataflow job.

1430

"properties": { # Named properties associated with the step. Each kind of

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1431

# predefined step has its own required set of properties.

1432

# Must be provided on Create. Only retrieved with JOB_VIEW_ALL.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1433

"a_key": "", # Properties of the object.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1434

},

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1435

},

1436

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1437

"replacedByJobId": "A String", # If another job is an update of this job (and thus, this job is in

1438

# `JOB_STATE_UPDATED`), this field contains the ID of that job.

1439

"executionInfo": { # Additional information about how a Cloud Dataflow job will be executed that # Deprecated.

1440

# isn't contained in the submitted job.

1441

"stages": { # A mapping from each stage to the information about that stage.

1442

"a_key": { # Contains information about how a particular

1443

# google.dataflow.v1beta3.Step will be executed.

1444

"stepName": [ # The steps associated with the execution stage.

1445

# Note that stages may have several steps, and that a given step

1446

# might be run by more than one stage.

"A String",

],

},

},

},

"currentState": "A String", # The current state of the job.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1453

#

1454

# Jobs are created in the `JOB_STATE_STOPPED` state unless otherwise

1455

# specified.

1456

#

1457

# A job in the `JOB_STATE_RUNNING` state may asynchronously enter a

1458

# terminal state. After a job has reached a terminal state, no

1459

# further state updates may be made.

1460

#

1461

# This field may be mutated by the Cloud Dataflow service;

1462

# callers cannot mutate it.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1463

"location": "A String", # The [regional endpoint]

1464

# (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that

1465

# contains this job.

1466

"startTime": "A String", # The timestamp when the job was started (transitioned to JOB_STATE_PENDING).

1467

# Flexible resource scheduling jobs are started with some delay after job

1468

# creation, so start_time is unset before start and is updated when the

1469

# job is started by the Cloud Dataflow service. For other jobs, start_time

1470

# always equals to create_time and is immutable and set by the Cloud Dataflow

1471

# service.

1472

"stepsLocation": "A String", # The GCS location where the steps are stored.

1473

"labels": { # User-defined labels for this job.

1474

#

1475

# The labels map can contain no more than 64 entries. Entries of the labels

1476

# map are UTF8 strings that comply with the following restrictions:

1477

#

1478

# * Keys must conform to regexp: \p{Ll}\p{Lo}{0,62}

1479

# * Values must conform to regexp: [\p{Ll}\p{Lo}\p{N}_-]{0,63}

1480

# * Both keys and values are additionally constrained to be <= 128 bytes in

1481

# size.

1482

"a_key": "A String",

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1483

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1484

"createTime": "A String", # The timestamp when the job was initially created. Immutable and set by the

1485

# Cloud Dataflow service.

1486

"requestedState": "A String", # The job's requested state.

1487

#

1488

# `UpdateJob` may be used to switch between the `JOB_STATE_STOPPED` and

1489

# `JOB_STATE_RUNNING` states, by setting requested_state. `UpdateJob` may

1490

# also be used to directly set a job's requested state to

1491

# `JOB_STATE_CANCELLED` or `JOB_STATE_DONE`, irrevocably terminating the

1492

# job if it has not already reached a terminal state.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1493

},

Sai Cheemalapati