Blame - docs/dyn/dataflow_v1b3.projects.locations.flexTemplates.html - platform/external/python/google-api-python-client

<h1><a href="dataflow_v1b3.html">Dataflow API</a> . <a href="dataflow_v1b3.projects.html">projects</a> . <a href="dataflow_v1b3.projects.locations.html">locations</a> . <a href="dataflow_v1b3.projects.locations.flexTemplates.html">flexTemplates</a></h1>

76

<h2>Instance Methods</h2>

77

78

<code><a href="#launch">launch(projectId, location, body=None, x__xgafv=None)</a></code></p>

79

<p class="firstline">Launch a job with a FlexTemplate.</p>

80

<h3>Method Details</h3>

81

82

<code class="details" id="launch">launch(projectId, location, body=None, x__xgafv=None)</code>

83

<pre>Launch a job with a FlexTemplate.

84

85

Args:

86

projectId: string, Required. The ID of the Cloud Platform project that the job belongs to. (required)

87

location: string, Required. The [regional endpoint]

88

(https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) to

89

which to direct the request. E.g., us-central1, us-west1. (required)

90

body: object, The request body.

91

The object takes the form of:

92

93

{ # A request to launch a Cloud Dataflow job from a FlexTemplate.

94

"validateOnly": True or False, # If true, the request is validated but not actually executed.

95

# Defaults to false.

96

"launchParameter": { # Launch FlexTemplate Parameter. # Required. Parameter to launch a job form Flex Template.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

97

"containerSpecGcsPath": "A String", # Gcs path to a file with json serialized ContainerSpec as content.

98

"parameters": { # The parameters for FlexTemplate.

99

# Ex. {"num_workers":"5"}

100

"a_key": "A String",

101

},

102

"jobName": "A String", # Required. The job name to use for the created job.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

103

"containerSpec": { # Container Spec. # Spec about the container image to launch.

104

"metadata": { # Metadata describing a template. # Metadata describing a template including description and validation rules.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

105

"parameters": [ # The parameters for the template.

106

{ # Metadata for a specific parameter.

107

"label": "A String", # Required. The label to display for the parameter.

108

"paramType": "A String", # Optional. The type of the parameter.

109

# Used for selecting input picker.

110

"helpText": "A String", # Required. The help text to display for the parameter.

111

"name": "A String", # Required. The name of the parameter.

112

"regexes": [ # Optional. Regexes that the parameter must match.

113

"A String",

114

],

115

"isOptional": True or False, # Optional. Whether the parameter is optional. Defaults to false.

116

},

117

],

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

118

"name": "A String", # Required. The name of the template.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

119

"description": "A String", # Optional. A description of the template.

120

},

121

"sdkInfo": { # SDK Information. # Required. SDK info of the Flex Template.

122

"language": "A String", # Required. The SDK Language.

123

"version": "A String", # Optional. The SDK version.

124

},

125

"image": "A String", # Name of the docker container image. E.g., gcr.io/project/some-image

126

},

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

},

}

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

137

138

{ # Response to the request to launch a job from Flex Template.

139

"job": { # Defines a job to be run by the Cloud Dataflow service. # The job that was launched, if the request was not a dry run and

140

# the job was successfully launched.

141

"clientRequestId": "A String", # The client's unique identifier of the job, re-used across retried attempts.

142

# If this field is set, the service will ensure its uniqueness.

143

# The request to create a job will fail if the service has knowledge of a

144

# previously submitted job with the same client's ID and job name.

145

# The caller may use this field to ensure idempotence of job

146

# creation across retried attempts to create a job.

147

# By default, the field is empty and, in that case, the service ignores it.

148

"id": "A String", # The unique ID of this job.

149

#

150

# This field is set by the Cloud Dataflow service when the Job is

151

# created, and is immutable for the life of the job.

152

"currentStateTime": "A String", # The timestamp associated with the current state.

153

"transformNameMapping": { # The map of transform name prefixes of the job to be replaced to the

154

# corresponding name prefixes of the new job.

155

"a_key": "A String",

156

},

157

"environment": { # Describes the environment in which a Dataflow Job runs. # The environment for the job.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

158

"internalExperiments": { # Experimental settings.

159

"a_key": "", # Properties of the object. Contains field @type with type URL.

160

},

161

"workerRegion": "A String", # The Compute Engine region

162

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

163

# which worker processing should occur, e.g. "us-west1". Mutually exclusive

164

# with worker_zone. If neither worker_region nor worker_zone is specified,

165

# default to the control plane's region.

166

"serviceKmsKeyName": "A String", # If set, contains the Cloud KMS key identifier used to encrypt data

167

# at rest, AKA a Customer Managed Encryption Key (CMEK).

168

#

169

# Format:

170

# projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY

171

"userAgent": { # A description of the process that generated the request.

172

"a_key": "", # Properties of the object.

173

},

174

"workerZone": "A String", # The Compute Engine zone

175

# (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in

176

# which worker processing should occur, e.g. "us-west1-a". Mutually exclusive

177

# with worker_region. If neither worker_region nor worker_zone is specified,

178

# a zone in the control plane's region is chosen based on available capacity.

179

"clusterManagerApiService": "A String", # The type of cluster manager API to use. If unknown or

180

# unspecified, the service will attempt to choose a reasonable

181

# default. This should be in the form of the API service name,

182

# e.g. "compute.googleapis.com".

183

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

184

# storage. The system will append the suffix "/temp-{JOBNAME} to

185

# this resource prefix, where {JOBNAME} is the value of the

186

# job_name field. The resulting bucket and object prefix is used

187

# as the prefix of the resources used to store temporary data

188

# needed during the job execution. NOTE: This will override the

189

# value in taskrunner_settings.

190

# The supported resource type is:

191

#

192

# Google Cloud Storage:

193

#

194

# storage.googleapis.com/{bucket}/{object}

195

# bucket.storage.googleapis.com/{object}

196

"experiments": [ # The list of experiments to enable.

197

"A String",

198

],

199

"version": { # A structure describing which components and their versions of the service

200

# are required in order to run the job.

201

"a_key": "", # Properties of the object.

202

},

203

"serviceAccountEmail": "A String", # Identity to run virtual machines as. Defaults to the default account.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

204

"sdkPipelineOptions": { # The Cloud Dataflow SDK pipeline options specified by the user. These

205

# options are passed through the service and are used to recreate the

206

# SDK pipeline options on the worker in a language agnostic and platform

207

# independent way.

208

"a_key": "", # Properties of the object.

209

},

210

"flexResourceSchedulingGoal": "A String", # Which Flexible Resource Scheduling mode to run in.

211

"workerPools": [ # The worker pools. At least one "harness" worker pool must be

212

# specified in order for the job to have workers.

213

{ # Describes one particular pool of Cloud Dataflow workers to be

214

# instantiated by the Cloud Dataflow service in order to perform the

215

# computations required by a job. Note that a workflow job may use

216

# multiple pools, in order to match the various computational

217

# requirements of the various stages of the job.

218

"numThreadsPerWorker": 42, # The number of threads per worker harness. If empty or unspecified, the

219

# service will choose a number of threads (according to the number of cores

220

# on the selected machine type for batch, or 1 by convention for streaming).

221

"numWorkers": 42, # Number of Google Compute Engine workers in this pool needed to

222

# execute the job. If zero or unspecified, the service will

223

# attempt to choose a reasonable default.

224

"zone": "A String", # Zone to run the worker pools in. If empty or unspecified, the service

225

# will attempt to choose a reasonable default.

226

"diskSourceImage": "A String", # Fully qualified source image for disks.

227

"packages": [ # Packages to be installed on workers.

228

{ # The packages that must be installed in order for a worker to run the

229

# steps of the Cloud Dataflow job that will be assigned to its worker

230

# pool.

231

#

232

# This is the mechanism by which the Cloud Dataflow SDK causes code to

233

# be loaded onto the workers. For example, the Cloud Dataflow Java SDK

234

# might use this to install jars containing the user's code and all of the

235

# various dependencies (libraries, data files, etc.) required in order

236

# for that code to run.

237

"name": "A String", # The name of the package.

238

"location": "A String", # The resource to read the package from. The supported resource type is:

239

#

240

# Google Cloud Storage:

241

#

242

# storage.googleapis.com/{bucket}

243

# bucket.storage.googleapis.com/

244

},

245

],

246

"teardownPolicy": "A String", # Sets the policy for determining when to turndown worker pool.

247

# Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and

248

# `TEARDOWN_NEVER`.

249

# `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether

250

# the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down

251

# if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn

252

# down.

253

#

254

# If the workers are not torn down by the service, they will

255

# continue to run and use Google Compute Engine VM resources in the

256

# user's project until they are explicitly terminated by the user.

257

# Because of this, Google recommends using the `TEARDOWN_ALWAYS`

258

# policy except for small, manually supervised test jobs.

259

#

260

# If unknown or unspecified, the service will attempt to choose a reasonable

261

# default.

262

"onHostMaintenance": "A String", # The action to take on host maintenance, as defined by the Google

263

# Compute Engine API.

264

"poolArgs": { # Extra arguments for this worker pool.

265

"a_key": "", # Properties of the object. Contains field @type with type URL.

266

},

267

"diskSizeGb": 42, # Size of root disk for VMs, in GB. If zero or unspecified, the service will

268

# attempt to choose a reasonable default.

269

"workerHarnessContainerImage": "A String", # Required. Docker container image that executes the Cloud Dataflow worker

270

# harness, residing in Google Container Registry.

271

#

272

# Deprecated for the Fn API path. Use sdk_harness_container_images instead.

273

"diskType": "A String", # Type of root disk for VMs. If empty or unspecified, the service will

274

# attempt to choose a reasonable default.

275

"machineType": "A String", # Machine type (e.g. "n1-standard-1"). If empty or unspecified, the

276

# service will attempt to choose a reasonable default.

277

"kind": "A String", # The kind of the worker pool; currently only `harness` and `shuffle`

278

# are supported.

279

"sdkHarnessContainerImages": [ # Set of SDK harness containers needed to execute this pipeline. This will

280

# only be set in the Fn API path. For non-cross-language pipelines this

281

# should have only one entry. Cross-language pipelines will have two or more

282

# entries.

283

{ # Defines a SDK harness container for executing Dataflow pipelines.

284

"containerImage": "A String", # A docker container image that resides in Google Container Registry.

285

"useSingleCorePerContainer": True or False, # If true, recommends the Dataflow service to use only one core per SDK

286

# container instance with this image. If false (or unset) recommends using

287

# more than one core per SDK container instance with this image for

288

# efficiency. Note that Dataflow service may choose to override this property

# if needed.

},

],

"dataDisks": [ # Data disks that are used by a VM in this workflow.

293

{ # Describes the data disk used by a workflow job.

294

"diskType": "A String", # Disk storage type, as defined by Google Compute Engine. This

295

# must be a disk type appropriate to the project and zone in which

296

# the workers will run. If unknown or unspecified, the service

297

# will attempt to choose a reasonable default.

298

#

299

# For example, the standard persistent disk type is a resource name

300

# typically ending in "pd-standard". If SSD persistent disks are

301

# available, the resource name typically ends with "pd-ssd". The

302

# actual valid values are defined the Google Compute Engine API,

303

# not by the Cloud Dataflow API; consult the Google Compute Engine

304

# documentation for more information about determining the set of

305

# available disk types for a particular project and zone.

306

#

307

# Google Compute Engine Disk types are local to a particular

308

# project in a particular zone, and so the resource name will

309

# typically look something like this:

310

#

311

# compute.googleapis.com/projects/project-id/zones/zone/diskTypes/pd-standard

312

"sizeGb": 42, # Size of disk in GB. If zero or unspecified, the service will

313

# attempt to choose a reasonable default.

314

"mountPoint": "A String", # Directory in a VM where disk is mounted.

315

},

316

],

317

"subnetwork": "A String", # Subnetwork to which VMs will be assigned, if desired. Expected to be of

318

# the form "regions/REGION/subnetworks/SUBNETWORK".

319

"ipConfiguration": "A String", # Configuration for VM IPs.

320

"taskrunnerSettings": { # Taskrunner configuration settings. # Settings passed through to Google Compute Engine workers when

321

# using the standard Dataflow task runner. Users should ignore

322

# this field.

323

"alsologtostderr": True or False, # Whether to also send taskrunner log info to stderr.

324

"taskGroup": "A String", # The UNIX group ID on the worker VM to use for tasks launched by

325

# taskrunner; e.g. "wheel".

326

"harnessCommand": "A String", # The command to launch the worker harness.

327

"logDir": "A String", # The directory on the VM to store logs.

328

"oauthScopes": [ # The OAuth2 scopes to be requested by the taskrunner in order to

329

# access the Cloud Dataflow API.

330

"A String",

331

],

332

"dataflowApiVersion": "A String", # The API version of endpoint, e.g. "v1b3"

333

"logUploadLocation": "A String", # Indicates where to put logs. If this is not specified, the logs

334

# will not be uploaded.

335

#

336

# The supported resource type is:

337

#

338

# Google Cloud Storage:

339

# storage.googleapis.com/{bucket}/{object}

340

# bucket.storage.googleapis.com/{object}

341

"streamingWorkerMainClass": "A String", # The streaming worker main class name.

342

"workflowFileName": "A String", # The file to store the workflow in.

343

"languageHint": "A String", # The suggested backend language.

344

"commandlinesFileName": "A String", # The file to store preprocessing commands in.

345

"baseTaskDir": "A String", # The location on the worker for task-specific subdirectories.

346

"tempStoragePrefix": "A String", # The prefix of the resources the taskrunner should use for

347

# temporary storage.

348

#

349

# The supported resource type is:

350

#

351

# Google Cloud Storage:

352

# storage.googleapis.com/{bucket}/{object}

353

# bucket.storage.googleapis.com/{object}

354

"baseUrl": "A String", # The base URL for the taskrunner to use when accessing Google Cloud APIs.

355

#

356

# When workers access Google Cloud APIs, they logically do so via

357

# relative URLs. If this field is specified, it supplies the base

358

# URL to use for resolving these relative URLs. The normative

359

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

360

# Locators".

361

#

362

# If not specified, the default value is "http://www.googleapis.com/"

363

"logToSerialconsole": True or False, # Whether to send taskrunner log info to Google Compute Engine VM serial

364

# console.

365

"continueOnException": True or False, # Whether to continue taskrunner if an exception is hit.

366

"parallelWorkerSettings": { # Provides data to pass through to the worker harness. # The settings to pass to the parallel worker harness.

367

"tempStoragePrefix": "A String", # The prefix of the resources the system should use for temporary

368

# storage.

369

#

370

# The supported resource type is:

371

#

372

# Google Cloud Storage:

373

#

374

# storage.googleapis.com/{bucket}/{object}

375

# bucket.storage.googleapis.com/{object}

376

"reportingEnabled": True or False, # Whether to send work progress updates to the service.

377

"baseUrl": "A String", # The base URL for accessing Google Cloud APIs.

378

#

379

# When workers access Google Cloud APIs, they logically do so via

380

# relative URLs. If this field is specified, it supplies the base

381

# URL to use for resolving these relative URLs. The normative

382

# algorithm used is defined by RFC 1808, "Relative Uniform Resource

383

# Locators".

384

#

385

# If not specified, the default value is "http://www.googleapis.com/"

386

"servicePath": "A String", # The Cloud Dataflow service path relative to the root URL, for example,

387

# "dataflow/v1b3/projects".

388

"shuffleServicePath": "A String", # The Shuffle service path relative to the root URL, for example,

389

# "shuffle/v1beta1".

390

"workerId": "A String", # The ID of the worker running this pipeline.

391

},

392

"taskUser": "A String", # The UNIX user ID on the worker VM to use for tasks launched by

393

# taskrunner; e.g. "root".

394

"vmId": "A String", # The ID string of the VM.

395

},

396

"autoscalingSettings": { # Settings for WorkerPool autoscaling. # Settings for autoscaling of this WorkerPool.

397

"algorithm": "A String", # The algorithm to use for autoscaling.

398

"maxNumWorkers": 42, # The maximum number of workers to cap scaling at.

399

},

400

"metadata": { # Metadata to set on the Google Compute Engine VMs.

401

"a_key": "A String",

402

},

403

"defaultPackageSet": "A String", # The default package set to install. This allows the service to

404

# select a default set of packages which are useful to worker

405

# harnesses written in a particular language.

406

"network": "A String", # Network to which VMs will be assigned. If empty or unspecified,

407

# the service will use the network "default".

408

},

409

],

410

"dataset": "A String", # The dataset for the current project where various workflow

411

# related tables are stored.

412

#

413

# The supported resource type is:

414

#

415

# Google BigQuery:

416

# bigquery.googleapis.com/{dataset}

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

417

},

418

"stageStates": [ # This field may be mutated by the Cloud Dataflow service;

419

# callers cannot mutate it.

420

{ # A message describing the state of a particular execution stage.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

421

"currentStateTime": "A String", # The time at which the stage transitioned to this state.

422

"executionStageState": "A String", # Executions stage states allow the same set of values as JobState.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

423

"executionStageName": "A String", # The name of the execution stage.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

424

},

425

],

426

"jobMetadata": { # Metadata available primarily for filtering jobs. Will be included in the # This field is populated by the Dataflow service to support filtering jobs

427

# by the metadata values provided here. Populated for ListJobs and all GetJob

428

# views SUMMARY and higher.

429

# ListJob response and Job SUMMARY view.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

430

"datastoreDetails": [ # Identification of a Datastore source used in the Dataflow job.

431

{ # Metadata for a Datastore connector used by the job.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

432

"namespace": "A String", # Namespace used in the connection.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

433

"projectId": "A String", # ProjectId accessed in the connection.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

434

},

435

],

436

"sdkVersion": { # The version of the SDK used to run the job. # The SDK version used to run the job.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

437

"version": "A String", # The version of the SDK used to run the job.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

438

"sdkSupportStatus": "A String", # The support status for this SDK version.

439

"versionDisplayName": "A String", # A readable string describing the version of the SDK.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

440

},

441

"bigqueryDetails": [ # Identification of a BigQuery source used in the Dataflow job.

442

{ # Metadata for a BigQuery connector used by the job.

443

"table": "A String", # Table accessed in the connection.

444

"dataset": "A String", # Dataset accessed in the connection.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

445

"query": "A String", # Query used to access data in the connection.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

446

"projectId": "A String", # Project accessed in the connection.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

447

},

448

],

449

"fileDetails": [ # Identification of a File source used in the Dataflow job.

450

{ # Metadata for a File connector used by the job.

451

"filePattern": "A String", # File Pattern used to access files by the connector.

452

},

453

],

454

"pubsubDetails": [ # Identification of a PubSub source used in the Dataflow job.

455

{ # Metadata for a PubSub connector used by the job.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

456

"topic": "A String", # Topic accessed in the connection.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

457

"subscription": "A String", # Subscription used in the connection.

458

},

459

],

460

"bigTableDetails": [ # Identification of a BigTable source used in the Dataflow job.

461

{ # Metadata for a BigTable connector used by the job.

462

"projectId": "A String", # ProjectId accessed in the connection.

463

"instanceId": "A String", # InstanceId accessed in the connection.

464

"tableId": "A String", # TableId accessed in the connection.

465

},

466

],

467

"spannerDetails": [ # Identification of a Spanner source used in the Dataflow job.

468

{ # Metadata for a Spanner connector used by the job.

469

"instanceId": "A String", # InstanceId accessed in the connection.

470

"projectId": "A String", # ProjectId accessed in the connection.

471

"databaseId": "A String", # DatabaseId accessed in the connection.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

472

},

473

],

474

},

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

475

"type": "A String", # The type of Cloud Dataflow job.

476

"projectId": "A String", # The ID of the Cloud Platform project that the job belongs to.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

477

"createdFromSnapshotId": "A String", # If this is specified, the job's initial state is populated from the given

478

# snapshot.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

479

"pipelineDescription": { # A descriptive representation of submitted pipeline as well as the executed # Preliminary field: The format of this data may change at any time.

480

# A description of the user pipeline and stages through which it is executed.

481

# Created by Cloud Dataflow service. Only retrieved with

482

# JOB_VIEW_DESCRIPTION or JOB_VIEW_ALL.

483

# form. This data is provided by the Dataflow service for ease of visualizing

484

# the pipeline and interpreting Dataflow provided metrics.

485

"executionPipelineStage": [ # Description of each stage of execution of the pipeline.

486

{ # Description of the composing transforms, names/ids, and input/outputs of a

487

# stage of execution. Some composing transforms and sources may have been

488

# generated by the Dataflow service during execution planning.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

489

"outputSource": [ # Output sources for this stage.

490

{ # Description of an input or output of an execution stage.

491

"sizeBytes": "A String", # Size of the source, if measurable.

492

"name": "A String", # Dataflow service generated name for this source.

493

"userName": "A String", # Human-readable name for this source; may be user or system generated.

494

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

495

# source is most closely associated.

496

},

497

],

498

"name": "A String", # Dataflow service generated name for this stage.

499

"inputSource": [ # Input sources for this stage.

500

{ # Description of an input or output of an execution stage.

501

"sizeBytes": "A String", # Size of the source, if measurable.

502

"name": "A String", # Dataflow service generated name for this source.

503

"userName": "A String", # Human-readable name for this source; may be user or system generated.

504

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

505

# source is most closely associated.

506

},

507

],

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

508

"id": "A String", # Dataflow service generated id for this stage.

509

"componentTransform": [ # Transforms that comprise this execution stage.

510

{ # Description of a transform executed as part of an execution stage.

511

"originalTransform": "A String", # User name for the original user transform with which this transform is

512

# most closely associated.

513

"name": "A String", # Dataflow service generated name for this source.

514

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

515

},

516

],

517

"componentSource": [ # Collections produced and consumed by component transforms of this stage.

518

{ # Description of an interstitial value between transforms in an execution

519

# stage.

520

"name": "A String", # Dataflow service generated name for this source.

521

"userName": "A String", # Human-readable name for this transform; may be user or system generated.

522

"originalTransformOrCollection": "A String", # User name for the original user transform or collection with which this

523

# source is most closely associated.

524

},

525

],

526

"kind": "A String", # Type of tranform this stage is executing.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

527

},

528

],

529

"originalPipelineTransform": [ # Description of each transform in the pipeline and collections between them.

530

{ # Description of the type, names/ids, and input/outputs for a transform.

531

"kind": "A String", # Type of transform.

532

"inputCollectionName": [ # User names for all collection inputs to this transform.

533

"A String",

534

],

535

"name": "A String", # User provided name for this transform instance.

536

"id": "A String", # SDK generated id of this transform instance.

537

"displayData": [ # Transform-specific display data.

538

{ # Data provided with a pipeline or transform to provide descriptive info.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

539

"durationValue": "A String", # Contains value if the data is of duration type.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

540

"int64Value": "A String", # Contains value if the data is of int64 type.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

541

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

542

# language namespace (i.e. python module) which defines the display data.

543

# This allows a dax monitoring system to specially handle the data

544

# and perform custom rendering.

545

"floatValue": 3.14, # Contains value if the data is of float type.

546

"key": "A String", # The key identifying the display data.

547

# This is intended to be used as a label for the display data

548

# when viewed in a dax monitoring system.

549

"shortStrValue": "A String", # A possible additional shorter value to display.

550

# For example a java_class_name_value of com.mypackage.MyDoFn

551

# will be stored with MyDoFn as the short_str_value and

552

# com.mypackage.MyDoFn as the java_class_name value.

553

# short_str_value can be displayed and java_class_name_value

554

# will be displayed as a tooltip.

555

"url": "A String", # An optional full URL.

556

"label": "A String", # An optional label to display in a dax UI for the element.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

557

"timestampValue": "A String", # Contains value if the data is of timestamp type.

558

"boolValue": True or False, # Contains value if the data is of a boolean type.

559

"javaClassValue": "A String", # Contains value if the data is of java class type.

560

"strValue": "A String", # Contains value if the data is of string type.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

561

},

562

],

563

"outputCollectionName": [ # User names for all collection outputs to this transform.

"A String",

],

},

],

"displayData": [ # Pipeline level display data.

569

{ # Data provided with a pipeline or transform to provide descriptive info.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

570

"durationValue": "A String", # Contains value if the data is of duration type.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

571

"int64Value": "A String", # Contains value if the data is of int64 type.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

572

"namespace": "A String", # The namespace for the key. This is usually a class name or programming

573

# language namespace (i.e. python module) which defines the display data.

574

# This allows a dax monitoring system to specially handle the data

575

# and perform custom rendering.

576

"floatValue": 3.14, # Contains value if the data is of float type.

577

"key": "A String", # The key identifying the display data.

578

# This is intended to be used as a label for the display data

579

# when viewed in a dax monitoring system.

580

"shortStrValue": "A String", # A possible additional shorter value to display.

581

# For example a java_class_name_value of com.mypackage.MyDoFn

582

# will be stored with MyDoFn as the short_str_value and

583

# com.mypackage.MyDoFn as the java_class_name value.

584

# short_str_value can be displayed and java_class_name_value

585

# will be displayed as a tooltip.

586

"url": "A String", # An optional full URL.

587

"label": "A String", # An optional label to display in a dax UI for the element.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

588

"timestampValue": "A String", # Contains value if the data is of timestamp type.

589

"boolValue": True or False, # Contains value if the data is of a boolean type.

590

"javaClassValue": "A String", # Contains value if the data is of java class type.

591

"strValue": "A String", # Contains value if the data is of string type.

Bu Sun Kim

6502091

2020-05-20 12:08:20 -0700

[diff] [blame]

},

],

},

"replaceJobId": "A String", # If this job is an update of an existing job, this field is the job ID

596

# of the job it replaced.

597

#

598

# When sending a `CreateJobRequest`, you can update a job by specifying it

599

# here. The job named here is stopped, and its intermediate state is

600

# transferred to this job.

601

"tempFiles": [ # A set of files the system should be aware of that are used

602

# for temporary storage. These temporary files will be

603

# removed on job completion.

604

# No duplicates are allowed.

605

# No file patterns are supported.

606

#

607

# The supported files are:

608

#

609

# Google Cloud Storage:

610

#

611

# storage.googleapis.com/{bucket}/{object}

612

# bucket.storage.googleapis.com/{object}

613

"A String",

614

],

615

"name": "A String", # The user-specified Cloud Dataflow job name.

616

#

617

# Only one Job with a given name may exist in a project at any

618

# given time. If a caller attempts to create a Job with the same

619

# name as an already-existing Job, the attempt returns the

620

# existing Job.

621

#

622

# The name must match the regular expression

623

# `[a-z]([-a-z0-9]{0,38}[a-z0-9])?`

624

"steps": [ # Exactly one of step or steps_location should be specified.

625

#

626

# The top-level steps that constitute the entire job.

627

{ # Defines a particular step within a Cloud Dataflow job.

628

#

629

# A job consists of multiple steps, each of which performs some

630

# specific operation as part of the overall job. Data is typically

631

# passed from one step to another as part of the job.

632

#

633

# Here's an example of a sequence of steps which together implement a

634

# Map-Reduce job:

635

#

636

# * Read a collection of data from some source, parsing the

637

# collection's elements.

638

#

639

# * Validate the elements.

640

#

641

# * Apply a user-defined function to map each element to some value

642

# and extract an element-specific key value.

643

#

644

# * Group elements with the same key into a single element with

645

# that key, transforming a multiply-keyed collection into a

646

# uniquely-keyed collection.

647

#

648

# * Write the elements out to some data sink.

649

#

650

# Note that the Cloud Dataflow service may be used to run many different

651

# types of jobs, not just Map-Reduce.

652

"name": "A String", # The name that identifies the step. This must be unique for each

653

# step with respect to all other steps in the Cloud Dataflow job.

654

"kind": "A String", # The kind of step in the Cloud Dataflow job.

655

"properties": { # Named properties associated with the step. Each kind of

656

# predefined step has its own required set of properties.

657

# Must be provided on Create. Only retrieved with JOB_VIEW_ALL.

658

"a_key": "", # Properties of the object.

},

},

],

"replacedByJobId": "A String", # If another job is an update of this job (and thus, this job is in

663

# `JOB_STATE_UPDATED`), this field contains the ID of that job.

664

"executionInfo": { # Additional information about how a Cloud Dataflow job will be executed that # Deprecated.

665

# isn't contained in the submitted job.

666

"stages": { # A mapping from each stage to the information about that stage.

667

"a_key": { # Contains information about how a particular

668

# google.dataflow.v1beta3.Step will be executed.

669

"stepName": [ # The steps associated with the execution stage.

670

# Note that stages may have several steps, and that a given step

671

# might be run by more than one stage.

"A String",

],

},

},

},

"currentState": "A String", # The current state of the job.

678

#

679

# Jobs are created in the `JOB_STATE_STOPPED` state unless otherwise

680

# specified.

681

#

682

# A job in the `JOB_STATE_RUNNING` state may asynchronously enter a

683

# terminal state. After a job has reached a terminal state, no

684

# further state updates may be made.

685

#

686

# This field may be mutated by the Cloud Dataflow service;

687

# callers cannot mutate it.

688

"location": "A String", # The [regional endpoint]

689

# (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that

690

# contains this job.

691

"startTime": "A String", # The timestamp when the job was started (transitioned to JOB_STATE_PENDING).

692

# Flexible resource scheduling jobs are started with some delay after job

693

# creation, so start_time is unset before start and is updated when the

694

# job is started by the Cloud Dataflow service. For other jobs, start_time

695

# always equals to create_time and is immutable and set by the Cloud Dataflow

696

# service.

697

"stepsLocation": "A String", # The GCS location where the steps are stored.

698

"labels": { # User-defined labels for this job.

699

#

700

# The labels map can contain no more than 64 entries. Entries of the labels

701

# map are UTF8 strings that comply with the following restrictions:

702

#

703

# * Keys must conform to regexp: \p{Ll}\p{Lo}{0,62}

704

# * Values must conform to regexp: [\p{Ll}\p{Lo}\p{N}_-]{0,63}

705

# * Both keys and values are additionally constrained to be <= 128 bytes in

# size.

"a_key": "A String",

},

"createTime": "A String", # The timestamp when the job was initially created. Immutable and set by the

710

# Cloud Dataflow service.

711

"requestedState": "A String", # The job's requested state.

712

#

713

# `UpdateJob` may be used to switch between the `JOB_STATE_STOPPED` and

714

# `JOB_STATE_RUNNING` states, by setting requested_state. `UpdateJob` may

715

# also be used to directly set a job's requested state to

716

# `JOB_STATE_CANCELLED` or `JOB_STATE_DONE`, irrevocably terminating the

717

# job if it has not already reached a terminal state.

},

}</pre>

</div>

</body></html>