Blame - docs/dyn/genomics_v1alpha2.pipelines.html - platform/external/python/google-api-python-client

2017-03-13 12:12:03 -0400

[diff] [blame]

220

# `LocalCopy` indicates where on the VM the file should be. The value

221

# given to this parameter (either at runtime or using `defaultValue`)

222

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

223

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

224

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

225

# Resources field, or "boot", which represents the Docker

226

# instance's boot disk and has a mount point of `/`.

227

"path": "A String", # Required. The path within the user's docker container where

228

# this input should be localized to and from, relative to the specified

229

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

230

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

231

},

232

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

233

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

234

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

235

# Users that run pipelines must have READ access to the image.

236

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

237

# string will be executed within a bash shell.

238

#

239

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

240

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

241

# `error` field will be populated.

242

#

243

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

244

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

245

"description": "A String", # User-specified description.

246

"inputParameters": [ # Input parameters of the pipeline.

247

{ # Parameters facilitate setting and delivering data into the

248

# pipeline's execution environment. They are defined at create time,

249

# with optional defaults, and can be overridden at run time.

250

#

251

# If `localCopy` is unset, then the parameter specifies a string that

252

# is passed as-is into the pipeline, as the value of the environment

253

# variable with the given name. A default value can be optionally

254

# specified at create time. The default can be overridden at run time

255

# using the inputs map. If no default is given, a value must be

256

# supplied at runtime.

257

#

258

# If `localCopy` is defined, then the parameter specifies a data

259

# source or sink, both in Google Cloud Storage and on the Docker container

260

# where the pipeline computation is run. The service account associated with

261

# the Pipeline (by

262

# default the project's Compute Engine service account) must have access to the

263

# Google Cloud Storage paths.

264

#

265

# At run time, the Google Cloud Storage paths can be overridden if a default

266

# was provided at create time, or must be set otherwise. The pipeline runner

267

# should add a key/value pair to either the inputs or outputs map. The

268

# indicated data copies will be carried out before/after pipeline execution,

269

# just as if the corresponding arguments were provided to `gsutil cp`.

270

#

271

# For example: Given the following `PipelineParameter`, specified

272

# in the `inputParameters` list:

273

#

274

# ```

275

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

276

# ```

277

#

278

# where `disk` is defined in the `PipelineResources` object as:

279

#

280

# ```

281

# {name: "pd1", mountPoint: "/mnt/disk/"}

282

# ```

283

#

284

# We create a disk named `pd1`, mount it on the host VM, and map

285

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

286

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

291

# ```

292

#

293

# This would generate the following gsutil call:

294

#

295

# ```

296

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

297

# ```

298

#

299

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

300

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

305

# </thead>

306

# <tbody>

307

# <tr><td>file</td><td>file</td></tr>

308

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

313

#

314

# ```

315

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

316

# ```

317

#

318

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

323

# </thead>

324

# <tbody>

325

# <tr><td>file</td><td>file</td></tr>

326

# <tr>

327

# <td>file</td>

328

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

337

# on the boot disk, the local path cannot be a glob and must be a file.

338

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

339

# If `localCopy` is present, then this must be a Google Cloud Storage path

340

# beginning with `gs://`.

341

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

342

# as the key to the input and output maps in RunPipeline.

343

"description": "A String", # Human-readable description.

344

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

345

# `LocalCopy` indicates where on the VM the file should be. The value

346

# given to this parameter (either at runtime or using `defaultValue`)

347

# must be the remote path where the file should be.

348

"disk": "A String", # Required. The name of the disk where this parameter is

349

# located. Can be the name of one of the disks specified in the

350

# Resources field, or "boot", which represents the Docker

351

# instance's boot disk and has a mount point of `/`.

352

"path": "A String", # Required. The path within the user's docker container where

353

# this input should be localized to and from, relative to the specified

354

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

366

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

367

# must be true for both create time and run time. Cannot be true at run time

368

# if false at create time.

369

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

370

"zones": [ # List of Google Compute Engine availability zones to which resource

371

# creation will restricted. If empty, any zone may be chosen.

372

"A String",

373

],

374

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

375

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

376

# By specifying this parameter, you will download and install the following

377

# third-party software onto your managed Compute Engine instances:

378

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

379

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

380

# By specifying this parameter, you will download and install the following

381

# third-party software onto your managed Compute Engine instances: NVIDIA®

382

# Tesla® drivers and NVIDIA® CUDA toolkit.

383

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

384

# available accelerator types.

385

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

386

# feature that may go away. Defaults to false.

387

# Corresponds to `--no_address` flag for [gcloud compute instances create]

388

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

389

# In order to use this, must be true for both create time and run time.

390

# Cannot be true at run time if false at create time. If you need to ssh into

391

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

392

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

393

# only load docker images from Google Container Registry and not Docker Hub.

394

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

395

# [configure access to Google services from internal

396

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

397

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

398

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

399

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

400

# Specifies the path in the docker container where files on

401

# this disk should be located. For example, if `mountPoint`

402

# is `/mnt/disk`, and the parameter has `localPath`

403

# `inputs/file.txt`, the docker container can access the data at

404

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

405

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

406

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

407

# for more details.

408

# Can only be set at create time.

409

"source": "A String", # The full or partial URL of the persistent disk to attach. See

410

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

411

# and

412

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

413

# for more details.

414

"name": "A String", # Required. The name of the disk that can be used in the pipeline

415

# parameters. Must be 1 - 63 characters.

416

# The name "boot" is reserved for system use.

417

"type": "A String", # Required. The type of the disk to create.

418

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

419

# of the pipeline run, regardless of what this field is set to.

420

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

421

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

422

},

423

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

424

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

425

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

426

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

427

# This name can be used for filtering Pipelines in ListPipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

428

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

429

# WRITE access.

430

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

431

# is called. Cannot be specified in the Pipeline used in the

432

# CreatePipelineRequest, and will be populated in the response to

433

# CreatePipeline and all subsequent Get and List calls. Indicates that the

434

# service has registered this pipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

435

}

436

437

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

438

Allowed values

439

1 - v1 error format

440

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

441

442

Returns:

443

An object of the form:

444

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

445

{ # The pipeline object. Represents a transformation from a set of input

446

# parameters to a set of output parameters. The transformation is defined

447

# as a docker image and command to run within that image. Each pipeline

448

# is run on a Google Compute Engine VM. A pipeline can be created with the

449

# `create` method and then later run with the `run` method, or a pipeline can

450

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

451

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

452

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

453

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

454

# with optional defaults, and can be overridden at run time.

455

#

456

# If `localCopy` is unset, then the parameter specifies a string that

457

# is passed as-is into the pipeline, as the value of the environment

458

# variable with the given name. A default value can be optionally

459

# specified at create time. The default can be overridden at run time

460

# using the inputs map. If no default is given, a value must be

461

# supplied at runtime.

462

#

463

# If `localCopy` is defined, then the parameter specifies a data

464

# source or sink, both in Google Cloud Storage and on the Docker container

465

# where the pipeline computation is run. The service account associated with

466

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

467

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

468

# Google Cloud Storage paths.

469

#

470

# At run time, the Google Cloud Storage paths can be overridden if a default

471

# was provided at create time, or must be set otherwise. The pipeline runner

472

# should add a key/value pair to either the inputs or outputs map. The

473

# indicated data copies will be carried out before/after pipeline execution,

474

# just as if the corresponding arguments were provided to `gsutil cp`.

475

#

476

# For example: Given the following `PipelineParameter`, specified

477

# in the `inputParameters` list:

478

#

479

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

480

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

481

# ```

482

#

483

# where `disk` is defined in the `PipelineResources` object as:

484

#

485

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

486

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

487

# ```

488

#

489

# We create a disk named `pd1`, mount it on the host VM, and map

490

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

491

# runtime, an entry for `input_file` would be required in the inputs

492

# map, such as:

493

#

494

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

495

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

496

# ```

497

#

498

# This would generate the following gsutil call:

499

#

500

# ```

501

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

502

# ```

503

#

504

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

505

# Docker container. Acceptable paths are:

506

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

507

# <table>

508

# <thead>

509

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

510

# </thead>

511

# <tbody>

512

# <tr><td>file</td><td>file</td></tr>

513

# <tr><td>glob</td><td>directory</td></tr>

514

# </tbody>

515

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

516

#

517

# For outputs, the direction of the copy is reversed:

518

#

519

# ```

520

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

521

# ```

522

#

523

# Acceptable paths are:

524

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

525

# <table>

526

# <thead>

527

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

528

# </thead>

529

# <tbody>

530

# <tr><td>file</td><td>file</td></tr>

531

# <tr>

532

# <td>file</td>

533

# <td>directory - directory must already exist</td>

534

# </tr>

535

# <tr>

536

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

537

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

538

# </tbody>

539

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

540

#

541

# One restriction due to docker limitations, is that for outputs that are found

542

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

543

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

544

# If `localCopy` is present, then this must be a Google Cloud Storage path

545

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

546

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

547

# as the key to the input and output maps in RunPipeline.

548

"description": "A String", # Human-readable description.

549

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

550

# `LocalCopy` indicates where on the VM the file should be. The value

551

# given to this parameter (either at runtime or using `defaultValue`)

552

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

553

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

554

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

555

# Resources field, or "boot", which represents the Docker

556

# instance's boot disk and has a mount point of `/`.

557

"path": "A String", # Required. The path within the user's docker container where

558

# this input should be localized to and from, relative to the specified

559

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

560

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

561

},

562

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

563

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

564

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

565

# Users that run pipelines must have READ access to the image.

566

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

567

# string will be executed within a bash shell.

568

#

569

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

570

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

571

# `error` field will be populated.

572

#

573

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

574

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

575

"description": "A String", # User-specified description.

576

"inputParameters": [ # Input parameters of the pipeline.

577

{ # Parameters facilitate setting and delivering data into the

578

# pipeline's execution environment. They are defined at create time,

579

# with optional defaults, and can be overridden at run time.

580

#

581

# If `localCopy` is unset, then the parameter specifies a string that

582

# is passed as-is into the pipeline, as the value of the environment

583

# variable with the given name. A default value can be optionally

584

# specified at create time. The default can be overridden at run time

585

# using the inputs map. If no default is given, a value must be

586

# supplied at runtime.

587

#

588

# If `localCopy` is defined, then the parameter specifies a data

589

# source or sink, both in Google Cloud Storage and on the Docker container

590

# where the pipeline computation is run. The service account associated with

591

# the Pipeline (by

592

# default the project's Compute Engine service account) must have access to the

593

# Google Cloud Storage paths.

594

#

595

# At run time, the Google Cloud Storage paths can be overridden if a default

596

# was provided at create time, or must be set otherwise. The pipeline runner

597

# should add a key/value pair to either the inputs or outputs map. The

598

# indicated data copies will be carried out before/after pipeline execution,

599

# just as if the corresponding arguments were provided to `gsutil cp`.

600

#

601

# For example: Given the following `PipelineParameter`, specified

602

# in the `inputParameters` list:

603

#

604

# ```

605

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

606

# ```

607

#

608

# where `disk` is defined in the `PipelineResources` object as:

609

#

610

# ```

611

# {name: "pd1", mountPoint: "/mnt/disk/"}

612

# ```

613

#

614

# We create a disk named `pd1`, mount it on the host VM, and map

615

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

616

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

621

# ```

622

#

623

# This would generate the following gsutil call:

624

#

625

# ```

626

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

627

# ```

628

#

629

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

630

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

635

# </thead>

636

# <tbody>

637

# <tr><td>file</td><td>file</td></tr>

638

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

643

#

644

# ```

645

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

646

# ```

647

#

648

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

653

# </thead>

654

# <tbody>

655

# <tr><td>file</td><td>file</td></tr>

656

# <tr>

657

# <td>file</td>

658

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

667

# on the boot disk, the local path cannot be a glob and must be a file.

668

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

669

# If `localCopy` is present, then this must be a Google Cloud Storage path

670

# beginning with `gs://`.

671

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

672

# as the key to the input and output maps in RunPipeline.

673

"description": "A String", # Human-readable description.

674

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

675

# `LocalCopy` indicates where on the VM the file should be. The value

676

# given to this parameter (either at runtime or using `defaultValue`)

677

# must be the remote path where the file should be.

678

"disk": "A String", # Required. The name of the disk where this parameter is

679

# located. Can be the name of one of the disks specified in the

680

# Resources field, or "boot", which represents the Docker

681

# instance's boot disk and has a mount point of `/`.

682

"path": "A String", # Required. The path within the user's docker container where

683

# this input should be localized to and from, relative to the specified

684

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

696

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

697

# must be true for both create time and run time. Cannot be true at run time

698

# if false at create time.

699

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

700

"zones": [ # List of Google Compute Engine availability zones to which resource

701

# creation will restricted. If empty, any zone may be chosen.

702

"A String",

703

],

704

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

705

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

706

# By specifying this parameter, you will download and install the following

707

# third-party software onto your managed Compute Engine instances:

708

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

709

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

710

# By specifying this parameter, you will download and install the following

711

# third-party software onto your managed Compute Engine instances: NVIDIA®

712

# Tesla® drivers and NVIDIA® CUDA toolkit.

713

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

714

# available accelerator types.

715

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

716

# feature that may go away. Defaults to false.

717

# Corresponds to `--no_address` flag for [gcloud compute instances create]

718

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

719

# In order to use this, must be true for both create time and run time.

720

# Cannot be true at run time if false at create time. If you need to ssh into

721

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

722

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

723

# only load docker images from Google Container Registry and not Docker Hub.

724

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

725

# [configure access to Google services from internal

726

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

727

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

728

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

729

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

730

# Specifies the path in the docker container where files on

731

# this disk should be located. For example, if `mountPoint`

732

# is `/mnt/disk`, and the parameter has `localPath`

733

# `inputs/file.txt`, the docker container can access the data at

734

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

735

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

736

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

737

# for more details.

738

# Can only be set at create time.

739

"source": "A String", # The full or partial URL of the persistent disk to attach. See

740

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

741

# and

742

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

743

# for more details.

744

"name": "A String", # Required. The name of the disk that can be used in the pipeline

745

# parameters. Must be 1 - 63 characters.

746

# The name "boot" is reserved for system use.

747

"type": "A String", # Required. The type of the disk to create.

748

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

749

# of the pipeline run, regardless of what this field is set to.

750

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

751

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

752

},

753

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

754

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

755

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

756

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

757

# This name can be used for filtering Pipelines in ListPipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

758

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

759

# WRITE access.

760

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

761

# is called. Cannot be specified in the Pipeline used in the

762

# CreatePipelineRequest, and will be populated in the response to

763

# CreatePipeline and all subsequent Get and List calls. Indicates that the

764

# service has registered this pipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

769

<code class="details" id="delete">delete(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

770

<pre>Deletes a pipeline based on ID.

771

772

Caller must have WRITE permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

773

774

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

775

pipelineId: string, Caller must have WRITE access to the project in which this pipeline

776

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

777

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

778

Allowed values

779

1 - v1 error format

780

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

781

782

Returns:

783

An object of the form:

784

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

785

{ # A generic empty message that you can re-use to avoid defining duplicated

786

# empty messages in your APIs. A typical example is to use it as the request

787

# or the response type of an API method. For instance:

788

#

789

# service Foo {

790

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

791

# }

792

#

793

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

798

<code class="details" id="get">get(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

799

<pre>Retrieves a pipeline based on ID.

800

801

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

802

803

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

804

pipelineId: string, Caller must have READ access to the project in which this pipeline

805

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

806

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

807

Allowed values

808

1 - v1 error format

809

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

810

811

Returns:

812

An object of the form:

813

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

814

{ # The pipeline object. Represents a transformation from a set of input

815

# parameters to a set of output parameters. The transformation is defined

816

# as a docker image and command to run within that image. Each pipeline

817

# is run on a Google Compute Engine VM. A pipeline can be created with the

818

# `create` method and then later run with the `run` method, or a pipeline can

819

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

820

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

821

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

822

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

823

# with optional defaults, and can be overridden at run time.

824

#

825

# If `localCopy` is unset, then the parameter specifies a string that

826

# is passed as-is into the pipeline, as the value of the environment

827

# variable with the given name. A default value can be optionally

828

# specified at create time. The default can be overridden at run time

829

# using the inputs map. If no default is given, a value must be

830

# supplied at runtime.

831

#

832

# If `localCopy` is defined, then the parameter specifies a data

833

# source or sink, both in Google Cloud Storage and on the Docker container

834

# where the pipeline computation is run. The service account associated with

835

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

836

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

837

# Google Cloud Storage paths.

838

#

839

# At run time, the Google Cloud Storage paths can be overridden if a default

840

# was provided at create time, or must be set otherwise. The pipeline runner

841

# should add a key/value pair to either the inputs or outputs map. The

842

# indicated data copies will be carried out before/after pipeline execution,

843

# just as if the corresponding arguments were provided to `gsutil cp`.

844

#

845

# For example: Given the following `PipelineParameter`, specified

846

# in the `inputParameters` list:

847

#

848

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

849

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

850

# ```

851

#

852

# where `disk` is defined in the `PipelineResources` object as:

853

#

854

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

855

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

856

# ```

857

#

858

# We create a disk named `pd1`, mount it on the host VM, and map

859

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

860

# runtime, an entry for `input_file` would be required in the inputs

861

# map, such as:

862

#

863

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

864

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

865

# ```

866

#

867

# This would generate the following gsutil call:

868

#

869

# ```

870

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

871

# ```

872

#

873

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

874

# Docker container. Acceptable paths are:

875

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

876

# <table>

877

# <thead>

878

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

879

# </thead>

880

# <tbody>

881

# <tr><td>file</td><td>file</td></tr>

882

# <tr><td>glob</td><td>directory</td></tr>

883

# </tbody>

884

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

885

#

886

# For outputs, the direction of the copy is reversed:

887

#

888

# ```

889

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

890

# ```

891

#

892

# Acceptable paths are:

893

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

894

# <table>

895

# <thead>

896

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

897

# </thead>

898

# <tbody>

899

# <tr><td>file</td><td>file</td></tr>

900

# <tr>

901

# <td>file</td>

902

# <td>directory - directory must already exist</td>

903

# </tr>

904

# <tr>

905

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

906

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

907

# </tbody>

908

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

909

#

910

# One restriction due to docker limitations, is that for outputs that are found

911

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

912

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

913

# If `localCopy` is present, then this must be a Google Cloud Storage path

914

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

915

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

916

# as the key to the input and output maps in RunPipeline.

917

"description": "A String", # Human-readable description.

918

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

919

# `LocalCopy` indicates where on the VM the file should be. The value

920

# given to this parameter (either at runtime or using `defaultValue`)

921

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

922

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

923

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

924

# Resources field, or "boot", which represents the Docker

925

# instance's boot disk and has a mount point of `/`.

926

"path": "A String", # Required. The path within the user's docker container where

927

# this input should be localized to and from, relative to the specified

928

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

929

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

930

},

931

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

932

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

933

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

934

# Users that run pipelines must have READ access to the image.

935

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

936

# string will be executed within a bash shell.

937

#

938

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

939

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

940

# `error` field will be populated.

941

#

942

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

943

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

944

"description": "A String", # User-specified description.

945

"inputParameters": [ # Input parameters of the pipeline.

946

{ # Parameters facilitate setting and delivering data into the

947

# pipeline's execution environment. They are defined at create time,

948

# with optional defaults, and can be overridden at run time.

949

#

950

# If `localCopy` is unset, then the parameter specifies a string that

951

# is passed as-is into the pipeline, as the value of the environment

952

# variable with the given name. A default value can be optionally

953

# specified at create time. The default can be overridden at run time

954

# using the inputs map. If no default is given, a value must be

955

# supplied at runtime.

956

#

957

# If `localCopy` is defined, then the parameter specifies a data

958

# source or sink, both in Google Cloud Storage and on the Docker container

959

# where the pipeline computation is run. The service account associated with

960

# the Pipeline (by

961

# default the project's Compute Engine service account) must have access to the

962

# Google Cloud Storage paths.

963

#

964

# At run time, the Google Cloud Storage paths can be overridden if a default

965

# was provided at create time, or must be set otherwise. The pipeline runner

966

# should add a key/value pair to either the inputs or outputs map. The

967

# indicated data copies will be carried out before/after pipeline execution,

968

# just as if the corresponding arguments were provided to `gsutil cp`.

969

#

970

# For example: Given the following `PipelineParameter`, specified

971

# in the `inputParameters` list:

972

#

973

# ```

974

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

975

# ```

976

#

977

# where `disk` is defined in the `PipelineResources` object as:

978

#

979

# ```

980

# {name: "pd1", mountPoint: "/mnt/disk/"}

981

# ```

982

#

983

# We create a disk named `pd1`, mount it on the host VM, and map

984

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

985

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

990

# ```

991

#

992

# This would generate the following gsutil call:

993

#

994

# ```

995

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

996

# ```

997

#

998

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

999

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1004

# </thead>

1005

# <tbody>

1006

# <tr><td>file</td><td>file</td></tr>

1007

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1012

#

1013

# ```

1014

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1015

# ```

1016

#

1017

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1022

# </thead>

1023

# <tbody>

1024

# <tr><td>file</td><td>file</td></tr>

1025

# <tr>

1026

# <td>file</td>

1027

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1036

# on the boot disk, the local path cannot be a glob and must be a file.

1037

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1038

# If `localCopy` is present, then this must be a Google Cloud Storage path

1039

# beginning with `gs://`.

1040

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1041

# as the key to the input and output maps in RunPipeline.

1042

"description": "A String", # Human-readable description.

1043

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1044

# `LocalCopy` indicates where on the VM the file should be. The value

1045

# given to this parameter (either at runtime or using `defaultValue`)

1046

# must be the remote path where the file should be.

1047

"disk": "A String", # Required. The name of the disk where this parameter is

1048

# located. Can be the name of one of the disks specified in the

1049

# Resources field, or "boot", which represents the Docker

1050

# instance's boot disk and has a mount point of `/`.

1051

"path": "A String", # Required. The path within the user's docker container where

1052

# this input should be localized to and from, relative to the specified

1053

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1065

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1066

# must be true for both create time and run time. Cannot be true at run time

1067

# if false at create time.

1068

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1069

"zones": [ # List of Google Compute Engine availability zones to which resource

1070

# creation will restricted. If empty, any zone may be chosen.

1071

"A String",

1072

],

1073

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1074

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1075

# By specifying this parameter, you will download and install the following

1076

# third-party software onto your managed Compute Engine instances:

1077

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1078

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1079

# By specifying this parameter, you will download and install the following

1080

# third-party software onto your managed Compute Engine instances: NVIDIA®

1081

# Tesla® drivers and NVIDIA® CUDA toolkit.

1082

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1083

# available accelerator types.

1084

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1085

# feature that may go away. Defaults to false.

1086

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1087

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1088

# In order to use this, must be true for both create time and run time.

1089

# Cannot be true at run time if false at create time. If you need to ssh into

1090

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1091

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1092

# only load docker images from Google Container Registry and not Docker Hub.

1093

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1094

# [configure access to Google services from internal

1095

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1096

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1097

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1098

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1099

# Specifies the path in the docker container where files on

1100

# this disk should be located. For example, if `mountPoint`

1101

# is `/mnt/disk`, and the parameter has `localPath`

1102

# `inputs/file.txt`, the docker container can access the data at

1103

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1104

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1105

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1106

# for more details.

1107

# Can only be set at create time.

1108

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1109

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1110

# and

1111

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1112

# for more details.

1113

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1114

# parameters. Must be 1 - 63 characters.

1115

# The name "boot" is reserved for system use.

1116

"type": "A String", # Required. The type of the disk to create.

1117

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1118

# of the pipeline run, regardless of what this field is set to.

1119

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1120

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1121

},

1122

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1123

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1124

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1125

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1126

# This name can be used for filtering Pipelines in ListPipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1127

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1128

# WRITE access.

1129

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1130

# is called. Cannot be specified in the Pipeline used in the

1131

# CreatePipelineRequest, and will be populated in the response to

1132

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1133

# service has registered this pipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1138

<code class="details" id="getControllerConfig">getControllerConfig(operationId=None, validationToken=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1139

<pre>Gets controller configuration information. Should only be called

1140

by VMs created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1141

1142

Args:

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1143

operationId: string, The operation to retrieve controller configuration for.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1144

validationToken: string, A parameter

1145

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1146

Allowed values

1147

1 - v1 error format

1148

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1149

1150

Returns:

1151

An object of the form:

1152

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1153

{ # Stores the information that the controller will fetch from the

1154

# server in order to run. Should only be used by VMs created by the

1155

# Pipelines Service and not by end users.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1156

"vars": {

1157

"a_key": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1158

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1159

"image": "A String",

1160

"gcsLogPath": "A String",

"gcsSources": {

"a_key": {

"values": [

"A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1165

],

1166

},

1167

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

"gcsSinks": {

"a_key": {

"values": [

"A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1172

],

1173

},

1174

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

"disks": {

"a_key": "A String",

},

"machineType": "A String",

1179

"cmd": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1184

<code class="details" id="list">list(namePrefix=None, pageToken=None, pageSize=None, projectId=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1185

<pre>Lists pipelines.

1186

1187

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1188

1189

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1190

namePrefix: string, Pipelines with names that match this prefix should be

1191

returned. If unspecified, all pipelines in the project, up to

1192

`pageSize`, will be returned.

1193

pageToken: string, Token to use to indicate where to start getting results.

1194

If unspecified, returns the first page of results.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1195

pageSize: integer, Number of pipelines to return at once. Defaults to 256, and max

1196

is 2048.

1197

projectId: string, Required. The name of the project to search for pipelines. Caller

1198

must have READ access to this project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1199

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1200

Allowed values

1201

1 - v1 error format

1202

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1203

1204

Returns:

1205

An object of the form:

1206

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1207

{ # The response of ListPipelines. Contains at most `pageSize`

1208

# pipelines. If it contains `pageSize` pipelines, and more pipelines

1209

# exist, then `nextPageToken` will be populated and should be

1210

# used as the `pageToken` argument to a subsequent ListPipelines

1211

# request.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1212

"pipelines": [ # The matched pipelines.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1213

{ # The pipeline object. Represents a transformation from a set of input

1214

# parameters to a set of output parameters. The transformation is defined

1215

# as a docker image and command to run within that image. Each pipeline

1216

# is run on a Google Compute Engine VM. A pipeline can be created with the

1217

# `create` method and then later run with the `run` method, or a pipeline can

1218

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1219

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1220

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1221

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1222

# with optional defaults, and can be overridden at run time.

1223

#

1224

# If `localCopy` is unset, then the parameter specifies a string that

1225

# is passed as-is into the pipeline, as the value of the environment

1226

# variable with the given name. A default value can be optionally

1227

# specified at create time. The default can be overridden at run time

1228

# using the inputs map. If no default is given, a value must be

1229

# supplied at runtime.

1230

#

1231

# If `localCopy` is defined, then the parameter specifies a data

1232

# source or sink, both in Google Cloud Storage and on the Docker container

1233

# where the pipeline computation is run. The service account associated with

1234

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1235

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1236

# Google Cloud Storage paths.

1237

#

1238

# At run time, the Google Cloud Storage paths can be overridden if a default

1239

# was provided at create time, or must be set otherwise. The pipeline runner

1240

# should add a key/value pair to either the inputs or outputs map. The

1241

# indicated data copies will be carried out before/after pipeline execution,

1242

# just as if the corresponding arguments were provided to `gsutil cp`.

1243

#

1244

# For example: Given the following `PipelineParameter`, specified

1245

# in the `inputParameters` list:

1246

#

1247

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1248

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1249

# ```

1250

#

1251

# where `disk` is defined in the `PipelineResources` object as:

1252

#

1253

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1254

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1255

# ```

1256

#

1257

# We create a disk named `pd1`, mount it on the host VM, and map

1258

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1259

# runtime, an entry for `input_file` would be required in the inputs

1260

# map, such as:

1261

#

1262

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1263

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1264

# ```

1265

#

1266

# This would generate the following gsutil call:

1267

#

1268

# ```

1269

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1270

# ```

1271

#

1272

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1273

# Docker container. Acceptable paths are:

1274

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1275

# <table>

1276

# <thead>

1277

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1278

# </thead>

1279

# <tbody>

1280

# <tr><td>file</td><td>file</td></tr>

1281

# <tr><td>glob</td><td>directory</td></tr>

1282

# </tbody>

1283

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1284

#

1285

# For outputs, the direction of the copy is reversed:

1286

#

1287

# ```

1288

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1289

# ```

1290

#

1291

# Acceptable paths are:

1292

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1293

# <table>

1294

# <thead>

1295

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1296

# </thead>

1297

# <tbody>

1298

# <tr><td>file</td><td>file</td></tr>

1299

# <tr>

1300

# <td>file</td>

1301

# <td>directory - directory must already exist</td>

1302

# </tr>

1303

# <tr>

1304

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1305

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1306

# </tbody>

1307

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1308

#

1309

# One restriction due to docker limitations, is that for outputs that are found

1310

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1311

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1312

# If `localCopy` is present, then this must be a Google Cloud Storage path

1313

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1314

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1315

# as the key to the input and output maps in RunPipeline.

1316

"description": "A String", # Human-readable description.

1317

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1318

# `LocalCopy` indicates where on the VM the file should be. The value

1319

# given to this parameter (either at runtime or using `defaultValue`)

1320

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1321

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1322

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1323

# Resources field, or "boot", which represents the Docker

1324

# instance's boot disk and has a mount point of `/`.

1325

"path": "A String", # Required. The path within the user's docker container where

1326

# this input should be localized to and from, relative to the specified

1327

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1328

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1329

},

1330

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1331

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

1332

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1333

# Users that run pipelines must have READ access to the image.

1334

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1335

# string will be executed within a bash shell.

1336

#

1337

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1338

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1339

# `error` field will be populated.

1340

#

1341

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1342

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1343

"description": "A String", # User-specified description.

1344

"inputParameters": [ # Input parameters of the pipeline.

1345

{ # Parameters facilitate setting and delivering data into the

1346

# pipeline's execution environment. They are defined at create time,

1347

# with optional defaults, and can be overridden at run time.

1348

#

1349

# If `localCopy` is unset, then the parameter specifies a string that

1350

# is passed as-is into the pipeline, as the value of the environment

1351

# variable with the given name. A default value can be optionally

1352

# specified at create time. The default can be overridden at run time

1353

# using the inputs map. If no default is given, a value must be

1354

# supplied at runtime.

1355

#

1356

# If `localCopy` is defined, then the parameter specifies a data

1357

# source or sink, both in Google Cloud Storage and on the Docker container

1358

# where the pipeline computation is run. The service account associated with

1359

# the Pipeline (by

1360

# default the project's Compute Engine service account) must have access to the

1361

# Google Cloud Storage paths.

1362

#

1363

# At run time, the Google Cloud Storage paths can be overridden if a default

1364

# was provided at create time, or must be set otherwise. The pipeline runner

1365

# should add a key/value pair to either the inputs or outputs map. The

1366

# indicated data copies will be carried out before/after pipeline execution,

1367

# just as if the corresponding arguments were provided to `gsutil cp`.

1368

#

1369

# For example: Given the following `PipelineParameter`, specified

1370

# in the `inputParameters` list:

1371

#

1372

# ```

1373

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1374

# ```

1375

#

1376

# where `disk` is defined in the `PipelineResources` object as:

1377

#

1378

# ```

1379

# {name: "pd1", mountPoint: "/mnt/disk/"}

1380

# ```

1381

#

1382

# We create a disk named `pd1`, mount it on the host VM, and map

1383

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1384

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1389

# ```

1390

#

1391

# This would generate the following gsutil call:

1392

#

1393

# ```

1394

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1395

# ```

1396

#

1397

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1398

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1403

# </thead>

1404

# <tbody>

1405

# <tr><td>file</td><td>file</td></tr>

1406

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1411

#

1412

# ```

1413

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1414

# ```

1415

#

1416

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1421

# </thead>

1422

# <tbody>

1423

# <tr><td>file</td><td>file</td></tr>

1424

# <tr>

1425

# <td>file</td>

1426

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1435

# on the boot disk, the local path cannot be a glob and must be a file.

1436

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1437

# If `localCopy` is present, then this must be a Google Cloud Storage path

1438

# beginning with `gs://`.

1439

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1440

# as the key to the input and output maps in RunPipeline.

1441

"description": "A String", # Human-readable description.

1442

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1443

# `LocalCopy` indicates where on the VM the file should be. The value

1444

# given to this parameter (either at runtime or using `defaultValue`)

1445

# must be the remote path where the file should be.

1446

"disk": "A String", # Required. The name of the disk where this parameter is

1447

# located. Can be the name of one of the disks specified in the

1448

# Resources field, or "boot", which represents the Docker

1449

# instance's boot disk and has a mount point of `/`.

1450

"path": "A String", # Required. The path within the user's docker container where

1451

# this input should be localized to and from, relative to the specified

1452

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1464

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1465

# must be true for both create time and run time. Cannot be true at run time

1466

# if false at create time.

1467

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1468

"zones": [ # List of Google Compute Engine availability zones to which resource

1469

# creation will restricted. If empty, any zone may be chosen.

1470

"A String",

1471

],

1472

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1473

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1474

# By specifying this parameter, you will download and install the following

1475

# third-party software onto your managed Compute Engine instances:

1476

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1477

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1478

# By specifying this parameter, you will download and install the following

1479

# third-party software onto your managed Compute Engine instances: NVIDIA®

1480

# Tesla® drivers and NVIDIA® CUDA toolkit.

1481

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1482

# available accelerator types.

1483

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1484

# feature that may go away. Defaults to false.

1485

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1486

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1487

# In order to use this, must be true for both create time and run time.

1488

# Cannot be true at run time if false at create time. If you need to ssh into

1489

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1490

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1491

# only load docker images from Google Container Registry and not Docker Hub.

1492

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1493

# [configure access to Google services from internal

1494

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1495

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1496

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1497

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1498

# Specifies the path in the docker container where files on

1499

# this disk should be located. For example, if `mountPoint`

1500

# is `/mnt/disk`, and the parameter has `localPath`

1501

# `inputs/file.txt`, the docker container can access the data at

1502

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1503

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1504

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1505

# for more details.

1506

# Can only be set at create time.

1507

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1508

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1509

# and

1510

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1511

# for more details.

1512

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1513

# parameters. Must be 1 - 63 characters.

1514

# The name "boot" is reserved for system use.

1515

"type": "A String", # Required. The type of the disk to create.

1516

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1517

# of the pipeline run, regardless of what this field is set to.

1518

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1519

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1520

},

1521

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1522

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1523

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1524

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1525

# This name can be used for filtering Pipelines in ListPipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1526

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1527

# WRITE access.

1528

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1529

# is called. Cannot be specified in the Pipeline used in the

1530

# CreatePipelineRequest, and will be populated in the response to

1531

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1532

# service has registered this pipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1533

},

1534

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1535

"nextPageToken": "A String", # The token to use to get the next page of results.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

<code class="details" id="list_next">list_next(previous_request, previous_response)</code>

1541

<pre>Retrieves the next page of results.

1542

1543

Args:

1544

previous_request: The request for the previous page. (required)

1545

previous_response: The response from the request for the previous page. (required)

1546

1547

Returns:

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1548

A request object that you can call 'execute()' on to request the next

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1549

page. Returns None if there are no more items in the collection.

</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1554

<code class="details" id="run">run(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1555

<pre>Runs a pipeline. If `pipelineId` is specified in the request, then

1556

run a saved pipeline. If `ephemeralPipeline` is specified, then run

1557

that pipeline once without saving a copy.

1558

1559

The caller must have READ permission to the project where the pipeline

1560

is stored and WRITE permission to the project where the pipeline will be

1561

run, as VMs will be created and storage will be used.

1562

1563

If a pipeline operation is still running after 6 days, it will be canceled.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1564

1565

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1566

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1567

The object takes the form of:

1568

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1569

{ # The request to run a pipeline. If `pipelineId` is specified, it

1570

# refers to a saved pipeline created with CreatePipeline and set as

1571

# the `pipelineId` of the returned Pipeline object. If

1572

# `ephemeralPipeline` is specified, that pipeline is run once

1573

# with the given args and not saved. It is an error to specify both

1574

# `pipelineId` and `ephemeralPipeline`. `pipelineArgs`

1575

# must be specified.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1576

"pipelineArgs": { # The pipeline run arguments. # The arguments to use when running this pipeline.

1577

"projectId": "A String", # Required. The project in which to run the pipeline. The caller must have

1578

# WRITER access to all Google Cloud services and resources (e.g. Google

1579

# Compute Engine) will be used.

1580

"clientId": "A String", # This field is deprecated. Use `labels` instead. Client-specified pipeline

1581

# operation identifier.

1582

"inputs": { # Pipeline input arguments; keys are defined in the pipeline documentation.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1583

# All input parameters that do not have default values must be specified.

1584

# If parameters with defaults are specified here, the defaults will be

1585

# overridden.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1586

"a_key": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1587

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1588

"serviceAccount": { # A Google Cloud Service Account. # The Google Cloud Service Account that will be used to access data and

1589

# services. By default, the compute service account associated with

1590

# `projectId` is used.

1591

"email": "A String", # Email address of the service account. Defaults to `default`,

1592

# which uses the compute service account associated with the project.

1593

"scopes": [ # List of scopes to be enabled for this service account on the VM.

1594

# The following scopes are automatically included:

1595

#

1596

# * https://www.googleapis.com/auth/compute

1597

# * https://www.googleapis.com/auth/devstorage.full_control

1598

# * https://www.googleapis.com/auth/genomics

1599

# * https://www.googleapis.com/auth/logging.write

1600

# * https://www.googleapis.com/auth/monitoring.write

"A String",

],

},

"logging": { # The logging options for the pipeline run. # Required. Logging options. Used by the service to communicate results

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1605

# to the user.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1606

"gcsPath": "A String", # The location in Google Cloud Storage to which the pipeline logs

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1607

# will be copied. Can be specified as a fully qualified directory

1608

# path, in which case logs will be output with a unique identifier

1609

# as the filename in that directory, or as a fully specified path,

1610

# which must end in `.log`, in which case that path will be

1611

# used, and the user must ensure that logs are not

1612

# overwritten. Stdout and stderr logs from the run are also

1613

# generated and output as `-stdout.log` and `-stderr.log`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1614

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1615

"labels": { # Labels to apply to this pipeline run. Labels will also be applied to

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1616

# compute resources (VM, disks) created by this pipeline run. When listing

1617

# operations, operations can filtered by labels.

1618

# Label keys may not be empty; label values may be empty. Non-empty labels

1619

# must be 1-63 characters long, and comply with [RFC1035]

1620

# (https://www.ietf.org/rfc/rfc1035.txt).

1621

# Specifically, the name must be 1-63 characters long and match the regular

1622

# expression `[a-z]([-a-z0-9]*[a-z0-9])?` which means the first

1623

# character must be a lowercase letter, and all following characters must be

1624

# a dash, lowercase letter, or digit, except the last character, which cannot

1625

# be a dash.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1626

"a_key": "A String",

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1627

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1628

"outputs": { # Pipeline output arguments; keys are defined in the pipeline

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1629

# documentation. All output parameters of without default values

1630

# must be specified. If parameters with defaults are specified

1631

# here, the defaults will be overridden.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1632

"a_key": "A String",

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1633

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1634

"resources": { # The system resources for the pipeline run. # Specifies resource requirements/overrides for the pipeline run.

1635

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1636

# must be true for both create time and run time. Cannot be true at run time

1637

# if false at create time.

1638

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1639

"zones": [ # List of Google Compute Engine availability zones to which resource

1640

# creation will restricted. If empty, any zone may be chosen.

1641

"A String",

1642

],

1643

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1644

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1645

# By specifying this parameter, you will download and install the following

1646

# third-party software onto your managed Compute Engine instances:

1647

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1648

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1649

# By specifying this parameter, you will download and install the following

1650

# third-party software onto your managed Compute Engine instances: NVIDIA®

1651

# Tesla® drivers and NVIDIA® CUDA toolkit.

1652

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1653

# available accelerator types.

1654

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1655

# feature that may go away. Defaults to false.

1656

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1657

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1658

# In order to use this, must be true for both create time and run time.

1659

# Cannot be true at run time if false at create time. If you need to ssh into

1660

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1661

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1662

# only load docker images from Google Container Registry and not Docker Hub.

1663

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1664

# [configure access to Google services from internal

1665

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1666

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1667

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1668

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1669

# Specifies the path in the docker container where files on

1670

# this disk should be located. For example, if `mountPoint`

1671

# is `/mnt/disk`, and the parameter has `localPath`

1672

# `inputs/file.txt`, the docker container can access the data at

1673

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1674

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1675

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1676

# for more details.

1677

# Can only be set at create time.

1678

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1679

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1680

# and

1681

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1682

# for more details.

1683

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1684

# parameters. Must be 1 - 63 characters.

1685

# The name "boot" is reserved for system use.

1686

"type": "A String", # Required. The type of the disk to create.

1687

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1688

# of the pipeline run, regardless of what this field is set to.

1689

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1690

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1691

},

1692

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1693

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1694

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1695

"keepVmAliveOnFailureDuration": "A String", # How long to keep the VM up after a failure (for example docker command

1696

# failed, copying input or output files failed, etc). While the VM is up, one

1697

# can ssh into the VM to debug. Default is 0; maximum allowed value is 1 day.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1698

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1699

"pipelineId": "A String", # The already created pipeline to run.

1700

"ephemeralPipeline": { # The pipeline object. Represents a transformation from a set of input # A new pipeline object to run once and then delete.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1701

# parameters to a set of output parameters. The transformation is defined

1702

# as a docker image and command to run within that image. Each pipeline

1703

# is run on a Google Compute Engine VM. A pipeline can be created with the

1704

# `create` method and then later run with the `run` method, or a pipeline can

1705

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1706

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1707

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1708

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1709

# with optional defaults, and can be overridden at run time.

1710

#

1711

# If `localCopy` is unset, then the parameter specifies a string that

1712

# is passed as-is into the pipeline, as the value of the environment

1713

# variable with the given name. A default value can be optionally

1714

# specified at create time. The default can be overridden at run time

1715

# using the inputs map. If no default is given, a value must be

1716

# supplied at runtime.

1717

#

1718

# If `localCopy` is defined, then the parameter specifies a data

1719

# source or sink, both in Google Cloud Storage and on the Docker container

1720

# where the pipeline computation is run. The service account associated with

1721

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1722

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1723

# Google Cloud Storage paths.

1724

#

1725

# At run time, the Google Cloud Storage paths can be overridden if a default

1726

# was provided at create time, or must be set otherwise. The pipeline runner

1727

# should add a key/value pair to either the inputs or outputs map. The

1728

# indicated data copies will be carried out before/after pipeline execution,

1729

# just as if the corresponding arguments were provided to `gsutil cp`.

1730

#

1731

# For example: Given the following `PipelineParameter`, specified

1732

# in the `inputParameters` list:

1733

#

1734

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1735

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1736

# ```

1737

#

1738

# where `disk` is defined in the `PipelineResources` object as:

1739

#

1740

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1741

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1742

# ```

1743

#

1744

# We create a disk named `pd1`, mount it on the host VM, and map

1745

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1746

# runtime, an entry for `input_file` would be required in the inputs

1747

# map, such as:

1748

#

1749

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1750

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1751

# ```

1752

#

1753

# This would generate the following gsutil call:

1754

#

1755

# ```

1756

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1757

# ```

1758

#

1759

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1760

# Docker container. Acceptable paths are:

1761

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1762

# <table>

1763

# <thead>

1764

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1765

# </thead>

1766

# <tbody>

1767

# <tr><td>file</td><td>file</td></tr>

1768

# <tr><td>glob</td><td>directory</td></tr>

1769

# </tbody>

1770

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1771

#

1772

# For outputs, the direction of the copy is reversed:

1773

#

1774

# ```

1775

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1776

# ```

1777

#

1778

# Acceptable paths are:

1779

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1780

# <table>

1781

# <thead>

1782

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1783

# </thead>

1784

# <tbody>

1785

# <tr><td>file</td><td>file</td></tr>

1786

# <tr>

1787

# <td>file</td>

1788

# <td>directory - directory must already exist</td>

1789

# </tr>

1790

# <tr>

1791

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1792

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1793

# </tbody>

1794

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1795

#

1796

# One restriction due to docker limitations, is that for outputs that are found

1797

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1798

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1799

# If `localCopy` is present, then this must be a Google Cloud Storage path

1800

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1801

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1802

# as the key to the input and output maps in RunPipeline.

1803

"description": "A String", # Human-readable description.

1804

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1805

# `LocalCopy` indicates where on the VM the file should be. The value

1806

# given to this parameter (either at runtime or using `defaultValue`)

1807

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1808

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1809

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1810

# Resources field, or "boot", which represents the Docker

1811

# instance's boot disk and has a mount point of `/`.

1812

"path": "A String", # Required. The path within the user's docker container where

1813

# this input should be localized to and from, relative to the specified

1814

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1815

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1816

},

1817

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1818

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

1819

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1820

# Users that run pipelines must have READ access to the image.

1821

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1822

# string will be executed within a bash shell.

1823

#

1824

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1825

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1826

# `error` field will be populated.

1827

#

1828

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1829

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1830

"description": "A String", # User-specified description.

1831

"inputParameters": [ # Input parameters of the pipeline.

1832

{ # Parameters facilitate setting and delivering data into the

1833

# pipeline's execution environment. They are defined at create time,

1834

# with optional defaults, and can be overridden at run time.

1835

#

1836

# If `localCopy` is unset, then the parameter specifies a string that

1837

# is passed as-is into the pipeline, as the value of the environment

1838

# variable with the given name. A default value can be optionally

1839

# specified at create time. The default can be overridden at run time

1840

# using the inputs map. If no default is given, a value must be

1841

# supplied at runtime.

1842

#

1843

# If `localCopy` is defined, then the parameter specifies a data

1844

# source or sink, both in Google Cloud Storage and on the Docker container

1845

# where the pipeline computation is run. The service account associated with

1846

# the Pipeline (by

1847

# default the project's Compute Engine service account) must have access to the

1848

# Google Cloud Storage paths.

1849

#

1850

# At run time, the Google Cloud Storage paths can be overridden if a default

1851

# was provided at create time, or must be set otherwise. The pipeline runner

1852

# should add a key/value pair to either the inputs or outputs map. The

1853

# indicated data copies will be carried out before/after pipeline execution,

1854

# just as if the corresponding arguments were provided to `gsutil cp`.

1855

#

1856

# For example: Given the following `PipelineParameter`, specified

1857

# in the `inputParameters` list:

1858

#

1859

# ```

1860

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1861

# ```

1862

#

1863

# where `disk` is defined in the `PipelineResources` object as:

1864

#

1865

# ```

1866

# {name: "pd1", mountPoint: "/mnt/disk/"}

1867

# ```

1868

#

1869

# We create a disk named `pd1`, mount it on the host VM, and map

1870

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1871

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1876

# ```

1877

#

1878

# This would generate the following gsutil call:

1879

#

1880

# ```

1881

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1882

# ```

1883

#

1884

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1885

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1890

# </thead>

1891

# <tbody>

1892

# <tr><td>file</td><td>file</td></tr>

1893

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1898

#

1899

# ```

1900

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1901

# ```

1902

#

1903

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1908

# </thead>

1909

# <tbody>

1910

# <tr><td>file</td><td>file</td></tr>

1911

# <tr>

1912

# <td>file</td>

1913

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1922

# on the boot disk, the local path cannot be a glob and must be a file.

1923

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1924

# If `localCopy` is present, then this must be a Google Cloud Storage path

1925

# beginning with `gs://`.

1926

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1927

# as the key to the input and output maps in RunPipeline.

1928

"description": "A String", # Human-readable description.

1929

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1930

# `LocalCopy` indicates where on the VM the file should be. The value

1931

# given to this parameter (either at runtime or using `defaultValue`)

1932

# must be the remote path where the file should be.

1933

"disk": "A String", # Required. The name of the disk where this parameter is

1934

# located. Can be the name of one of the disks specified in the

1935

# Resources field, or "boot", which represents the Docker

1936

# instance's boot disk and has a mount point of `/`.

1937

"path": "A String", # Required. The path within the user's docker container where

1938

# this input should be localized to and from, relative to the specified

1939

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1951

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1952

# must be true for both create time and run time. Cannot be true at run time

1953

# if false at create time.

1954

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1955

"zones": [ # List of Google Compute Engine availability zones to which resource

1956

# creation will restricted. If empty, any zone may be chosen.

1957

"A String",

1958

],

1959

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1960

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1961

# By specifying this parameter, you will download and install the following

1962

# third-party software onto your managed Compute Engine instances:

1963

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1964

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1965

# By specifying this parameter, you will download and install the following

1966

# third-party software onto your managed Compute Engine instances: NVIDIA®

1967

# Tesla® drivers and NVIDIA® CUDA toolkit.

1968

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1969

# available accelerator types.

1970

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1971

# feature that may go away. Defaults to false.

1972

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1973

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1974

# In order to use this, must be true for both create time and run time.

1975

# Cannot be true at run time if false at create time. If you need to ssh into

1976

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1977

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1978

# only load docker images from Google Container Registry and not Docker Hub.

1979

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1980

# [configure access to Google services from internal

1981

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1982

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1983

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1984

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1985

# Specifies the path in the docker container where files on

1986

# this disk should be located. For example, if `mountPoint`

1987

# is `/mnt/disk`, and the parameter has `localPath`

1988

# `inputs/file.txt`, the docker container can access the data at

1989

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1990

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1991

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1992

# for more details.

1993

# Can only be set at create time.

1994

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1995

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1996

# and

1997

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1998

# for more details.

1999

"name": "A String", # Required. The name of the disk that can be used in the pipeline

2000

# parameters. Must be 1 - 63 characters.

2001

# The name "boot" is reserved for system use.

2002

"type": "A String", # Required. The type of the disk to create.

2003

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

2004

# of the pipeline run, regardless of what this field is set to.

2005

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

2006

# This field is not applicable for local SSD.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2007

},

2008

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2009

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2010

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2011

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2012

# This name can be used for filtering Pipelines in ListPipelines.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2013

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

2014

# WRITE access.

2015

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

2016

# is called. Cannot be specified in the Pipeline used in the

2017

# CreatePipelineRequest, and will be populated in the response to

2018

# CreatePipeline and all subsequent Get and List calls. Indicates that the

2019

# service has registered this pipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2020

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2021

}

2022

2023

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2024

Allowed values

2025

1 - v1 error format

2026

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2027

2028

Returns:

2029

An object of the form:

2030

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2031

{ # This resource represents a long-running operation that is the result of a

2032

# network API call.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2033

"done": True or False, # If the value is `false`, it means the operation is still in progress.

2034

# If `true`, the operation is completed, and either `error` or `response` is

2035

# available.

2036

"response": { # An Empty object.

2037

"a_key": "", # Properties of the object. Contains field @type with type URL.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2038

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2039

"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. For example&#58; `operations/CJHU7Oi_ChDrveSpBRjfuL-qzoWAgEw`

2040

"error": { # The `Status` type defines a logical error model that is suitable for # The error result of the operation in case of failure or cancellation.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2041

# different programming environments, including REST APIs and RPC APIs. It is

2042

# used by [gRPC](https://github.com/grpc). Each `Status` message contains

2043

# three pieces of data: error code, error message, and error details.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2044

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2045

# You can find out more about this error model and how to work with it in the

2046

# [API Design Guide](https://cloud.google.com/apis/design/errors).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2047

"message": "A String", # A developer-facing error message, which should be in English. Any

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2048

# user-facing error message should be localized and sent in the

2049

# google.rpc.Status.details field, or localized by the client.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2050

"details": [ # A list of messages that carry the error details. There is a common set of

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2051

# message types for APIs to use.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2052

{

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2053

"a_key": "", # Properties of the object. Contains field @type with type URL.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2054

},

2055

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2056

"code": 42, # The status code, which should be an enum value of google.rpc.Code.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2057

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2058

"metadata": { # An OperationMetadata or Metadata object. This will always be returned with the Operation.

2059

"a_key": "", # Properties of the object. Contains field @type with type URL.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2060

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2065

<code class="details" id="setOperationStatus">setOperationStatus(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2066

<pre>Sets status of a given operation. Any new timestamps (as determined by

2067

description) are appended to TimestampEvents. Should only be called by VMs

2068

created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2069

2070

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2071

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2072

The object takes the form of:

2073

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2074

{ # Request to set operation status. Should only be used by VMs

2075

# created by the Pipelines Service and not by end users.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2076

"operationId": "A String",

2077

"errorMessage": "A String",

2078

"validationToken": "A String",

2079

"errorCode": "A String",

2080

"timestampEvents": [

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2081

{ # Stores the list of events and times they occured for major events in job

2082

# execution.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2083

"timestamp": "A String", # The time this event occured.

2084

"description": "A String", # String indicating the type of event

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2085

},

2086

],

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2087

}

2088

2089

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2090

Allowed values

2091

1 - v1 error format

2092

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2093

2094

Returns:

2095

An object of the form:

2096

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2097

{ # A generic empty message that you can re-use to avoid defining duplicated

2098

# empty messages in your APIs. A typical example is to use it as the request

2099

# or the response type of an API method. For instance:

2100

#

2101

# service Foo {

2102

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

2103

# }

2104

#

2105

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati