Blame - docs/dyn/genomics_v1alpha2.pipelines.html - platform/external/python/google-api-python-client

2020-07-22 17:02:09 -0700

[diff] [blame^]

300

"description": "A String", # Human-readable description.

301

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

302

# If `localCopy` is present, then this must be a Google Cloud Storage path

303

# beginning with `gs://`.

304

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

305

# as the key to the input and output maps in RunPipeline.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

306

},

307

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

308

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

309

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

310

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

311

# with optional defaults, and can be overridden at run time.

312

#

313

# If `localCopy` is unset, then the parameter specifies a string that

314

# is passed as-is into the pipeline, as the value of the environment

315

# variable with the given name. A default value can be optionally

316

# specified at create time. The default can be overridden at run time

317

# using the inputs map. If no default is given, a value must be

318

# supplied at runtime.

319

#

320

# If `localCopy` is defined, then the parameter specifies a data

321

# source or sink, both in Google Cloud Storage and on the Docker container

322

# where the pipeline computation is run. The service account associated with

323

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

324

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

325

# Google Cloud Storage paths.

326

#

327

# At run time, the Google Cloud Storage paths can be overridden if a default

328

# was provided at create time, or must be set otherwise. The pipeline runner

329

# should add a key/value pair to either the inputs or outputs map. The

330

# indicated data copies will be carried out before/after pipeline execution,

331

# just as if the corresponding arguments were provided to `gsutil cp`.

332

#

333

# For example: Given the following `PipelineParameter`, specified

334

# in the `inputParameters` list:

335

#

336

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

337

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

338

# ```

339

#

340

# where `disk` is defined in the `PipelineResources` object as:

341

#

342

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

343

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

344

# ```

345

#

346

# We create a disk named `pd1`, mount it on the host VM, and map

347

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

348

# runtime, an entry for `input_file` would be required in the inputs

349

# map, such as:

350

#

351

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

352

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

353

# ```

354

#

355

# This would generate the following gsutil call:

356

#

357

# ```

358

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

359

# ```

360

#

361

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

362

# Docker container. Acceptable paths are:

363

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

364

# <table>

365

# <thead>

366

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

367

# </thead>

368

# <tbody>

369

# <tr><td>file</td><td>file</td></tr>

370

# <tr><td>glob</td><td>directory</td></tr>

371

# </tbody>

372

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

373

#

374

# For outputs, the direction of the copy is reversed:

375

#

376

# ```

377

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

378

# ```

379

#

380

# Acceptable paths are:

381

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

382

# <table>

383

# <thead>

384

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

385

# </thead>

386

# <tbody>

387

# <tr><td>file</td><td>file</td></tr>

388

# <tr>

389

# <td>file</td>

390

# <td>directory - directory must already exist</td>

391

# </tr>

392

# <tr>

393

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

394

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

395

# </tbody>

396

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

397

#

398

# One restriction due to docker limitations, is that for outputs that are found

399

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

400

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

401

# `LocalCopy` indicates where on the VM the file should be. The value

402

# given to this parameter (either at runtime or using `defaultValue`)

403

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

404

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

405

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

406

# Resources field, or "boot", which represents the Docker

407

# instance's boot disk and has a mount point of `/`.

408

"path": "A String", # Required. The path within the user's docker container where

409

# this input should be localized to and from, relative to the specified

410

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

411

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

412

"description": "A String", # Human-readable description.

413

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

414

# If `localCopy` is present, then this must be a Google Cloud Storage path

415

# beginning with `gs://`.

416

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

417

# as the key to the input and output maps in RunPipeline.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

418

},

419

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

420

"description": "A String", # User-specified description.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

421

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

422

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

423

# Users that run pipelines must have READ access to the image.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

424

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

425

# string will be executed within a bash shell.

426

#

427

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

428

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

429

# `error` field will be populated.

430

#

431

# Maximum command string length is 16384.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

432

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

433

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

434

# This name can be used for filtering Pipelines in ListPipelines.

435

}

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

436

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

437

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

438

Allowed values

439

1 - v1 error format

440

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

441

442

Returns:

443

An object of the form:

444

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

445

{ # The pipeline object. Represents a transformation from a set of input

446

# parameters to a set of output parameters. The transformation is defined

447

# as a docker image and command to run within that image. Each pipeline

448

# is run on a Google Compute Engine VM. A pipeline can be created with the

449

# `create` method and then later run with the `run` method, or a pipeline can

450

# be defined and run all at once with the `run` method.

451

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

459

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

460

# must be true for both create time and run time. Cannot be true at run time

461

# if false at create time.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

462

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

463

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

464

# By specifying this parameter, you will download and install the following

465

# third-party software onto your managed Compute Engine instances:

466

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

467

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

468

# feature that may go away. Defaults to false.

469

# Corresponds to `--no_address` flag for [gcloud compute instances create]

470

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

471

# In order to use this, must be true for both create time and run time.

472

# Cannot be true at run time if false at create time. If you need to ssh into

473

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

474

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

475

# only load docker images from Google Container Registry and not Docker Hub.

476

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

477

# [configure access to Google services from internal

478

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

479

"zones": [ # List of Google Compute Engine availability zones to which resource

480

# creation will restricted. If empty, any zone may be chosen.

481

"A String",

482

],

483

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

484

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

485

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

486

"source": "A String", # The full or partial URL of the persistent disk to attach. See

487

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

488

# and

489

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

490

# for more details.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

491

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

492

# Specifies the path in the docker container where files on

493

# this disk should be located. For example, if `mountPoint`

494

# is `/mnt/disk`, and the parameter has `localPath`

495

# `inputs/file.txt`, the docker container can access the data at

496

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

497

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

498

# of the pipeline run, regardless of what this field is set to.

499

"name": "A String", # Required. The name of the disk that can be used in the pipeline

500

# parameters. Must be 1 - 63 characters.

501

# The name "boot" is reserved for system use.

502

"type": "A String", # Required. The type of the disk to create.

503

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

504

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

505

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

506

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

507

# for more details.

508

# Can only be set at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

509

},

510

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

511

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

512

# By specifying this parameter, you will download and install the following

513

# third-party software onto your managed Compute Engine instances: NVIDIA®

514

# Tesla® drivers and NVIDIA® CUDA toolkit.

515

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

516

# available accelerator types.

517

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

518

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

519

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

520

# WRITE access.

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

521

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

522

# is called. Cannot be specified in the Pipeline used in the

523

# CreatePipelineRequest, and will be populated in the response to

524

# CreatePipeline and all subsequent Get and List calls. Indicates that the

525

# service has registered this pipeline.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

526

"inputParameters": [ # Input parameters of the pipeline.

527

{ # Parameters facilitate setting and delivering data into the

528

# pipeline's execution environment. They are defined at create time,

529

# with optional defaults, and can be overridden at run time.

530

#

531

# If `localCopy` is unset, then the parameter specifies a string that

532

# is passed as-is into the pipeline, as the value of the environment

533

# variable with the given name. A default value can be optionally

534

# specified at create time. The default can be overridden at run time

535

# using the inputs map. If no default is given, a value must be

536

# supplied at runtime.

537

#

538

# If `localCopy` is defined, then the parameter specifies a data

539

# source or sink, both in Google Cloud Storage and on the Docker container

540

# where the pipeline computation is run. The service account associated with

541

# the Pipeline (by

542

# default the project's Compute Engine service account) must have access to the

543

# Google Cloud Storage paths.

544

#

545

# At run time, the Google Cloud Storage paths can be overridden if a default

546

# was provided at create time, or must be set otherwise. The pipeline runner

547

# should add a key/value pair to either the inputs or outputs map. The

548

# indicated data copies will be carried out before/after pipeline execution,

549

# just as if the corresponding arguments were provided to `gsutil cp`.

550

#

551

# For example: Given the following `PipelineParameter`, specified

552

# in the `inputParameters` list:

553

#

554

# ```

555

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

556

# ```

557

#

558

# where `disk` is defined in the `PipelineResources` object as:

559

#

560

# ```

561

# {name: "pd1", mountPoint: "/mnt/disk/"}

562

# ```

563

#

564

# We create a disk named `pd1`, mount it on the host VM, and map

565

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

566

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

571

# ```

572

#

573

# This would generate the following gsutil call:

574

#

575

# ```

576

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

577

# ```

578

#

579

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

580

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

585

# </thead>

586

# <tbody>

587

# <tr><td>file</td><td>file</td></tr>

588

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

593

#

594

# ```

595

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

596

# ```

597

#

598

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

603

# </thead>

604

# <tbody>

605

# <tr><td>file</td><td>file</td></tr>

606

# <tr>

607

# <td>file</td>

608

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

617

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

618

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

619

# `LocalCopy` indicates where on the VM the file should be. The value

620

# given to this parameter (either at runtime or using `defaultValue`)

621

# must be the remote path where the file should be.

622

"disk": "A String", # Required. The name of the disk where this parameter is

623

# located. Can be the name of one of the disks specified in the

624

# Resources field, or "boot", which represents the Docker

625

# instance's boot disk and has a mount point of `/`.

626

"path": "A String", # Required. The path within the user's docker container where

627

# this input should be localized to and from, relative to the specified

628

# disk's mount point. For example: file.txt,

629

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

630

"description": "A String", # Human-readable description.

631

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

632

# If `localCopy` is present, then this must be a Google Cloud Storage path

633

# beginning with `gs://`.

634

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

635

# as the key to the input and output maps in RunPipeline.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

636

},

637

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

638

"outputParameters": [ # Output parameters of the pipeline.

639

{ # Parameters facilitate setting and delivering data into the

640

# pipeline's execution environment. They are defined at create time,

641

# with optional defaults, and can be overridden at run time.

642

#

643

# If `localCopy` is unset, then the parameter specifies a string that

644

# is passed as-is into the pipeline, as the value of the environment

645

# variable with the given name. A default value can be optionally

646

# specified at create time. The default can be overridden at run time

647

# using the inputs map. If no default is given, a value must be

648

# supplied at runtime.

649

#

650

# If `localCopy` is defined, then the parameter specifies a data

651

# source or sink, both in Google Cloud Storage and on the Docker container

652

# where the pipeline computation is run. The service account associated with

653

# the Pipeline (by

654

# default the project's Compute Engine service account) must have access to the

655

# Google Cloud Storage paths.

656

#

657

# At run time, the Google Cloud Storage paths can be overridden if a default

658

# was provided at create time, or must be set otherwise. The pipeline runner

659

# should add a key/value pair to either the inputs or outputs map. The

660

# indicated data copies will be carried out before/after pipeline execution,

661

# just as if the corresponding arguments were provided to `gsutil cp`.

662

#

663

# For example: Given the following `PipelineParameter`, specified

664

# in the `inputParameters` list:

665

#

666

# ```

667

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

668

# ```

669

#

670

# where `disk` is defined in the `PipelineResources` object as:

671

#

672

# ```

673

# {name: "pd1", mountPoint: "/mnt/disk/"}

674

# ```

675

#

676

# We create a disk named `pd1`, mount it on the host VM, and map

677

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

678

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

683

# ```

684

#

685

# This would generate the following gsutil call:

686

#

687

# ```

688

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

689

# ```

690

#

691

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

692

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

697

# </thead>

698

# <tbody>

699

# <tr><td>file</td><td>file</td></tr>

700

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

705

#

706

# ```

707

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

708

# ```

709

#

710

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

715

# </thead>

716

# <tbody>

717

# <tr><td>file</td><td>file</td></tr>

718

# <tr>

719

# <td>file</td>

720

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

729

# on the boot disk, the local path cannot be a glob and must be a file.

730

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

731

# `LocalCopy` indicates where on the VM the file should be. The value

732

# given to this parameter (either at runtime or using `defaultValue`)

733

# must be the remote path where the file should be.

734

"disk": "A String", # Required. The name of the disk where this parameter is

735

# located. Can be the name of one of the disks specified in the

736

# Resources field, or "boot", which represents the Docker

737

# instance's boot disk and has a mount point of `/`.

738

"path": "A String", # Required. The path within the user's docker container where

739

# this input should be localized to and from, relative to the specified

740

# disk's mount point. For example: file.txt,

741

},

742

"description": "A String", # Human-readable description.

743

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

744

# If `localCopy` is present, then this must be a Google Cloud Storage path

745

# beginning with `gs://`.

746

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

747

# as the key to the input and output maps in RunPipeline.

748

},

749

],

750

"description": "A String", # User-specified description.

751

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

752

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

753

# Users that run pipelines must have READ access to the image.

754

"cmd": "A String", # Required. The command or newline delimited script to run. The command

755

# string will be executed within a bash shell.

756

#

757

# If the command exits with a non-zero exit code, output parameter

758

# de-localization will be skipped and the pipeline operation's

759

# `error` field will be populated.

760

#

761

# Maximum command string length is 16384.

762

},

763

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

764

# This name can be used for filtering Pipelines in ListPipelines.

}</pre>

</div>

<code class="details" id="delete">delete(pipelineId, x__xgafv=None)</code>

770

<pre>Deletes a pipeline based on ID.

771

772

Caller must have WRITE permission to the project.

773

774

Args:

775

pipelineId: string, Caller must have WRITE access to the project in which this pipeline

776

is defined. (required)

777

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

784

785

{ # A generic empty message that you can re-use to avoid defining duplicated

786

# empty messages in your APIs. A typical example is to use it as the request

787

# or the response type of an API method. For instance:

788

#

789

# service Foo {

790

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

791

# }

792

#

793

# The JSON representation for `Empty` is empty JSON object `{}`.

}</pre>

</div>

<code class="details" id="get">get(pipelineId, x__xgafv=None)</code>

799

<pre>Retrieves a pipeline based on ID.

800

801

Caller must have READ permission to the project.

802

803

Args:

804

pipelineId: string, Caller must have READ access to the project in which this pipeline

805

is defined. (required)

806

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

813

814

{ # The pipeline object. Represents a transformation from a set of input

815

# parameters to a set of output parameters. The transformation is defined

816

# as a docker image and command to run within that image. Each pipeline

817

# is run on a Google Compute Engine VM. A pipeline can be created with the

818

# `create` method and then later run with the `run` method, or a pipeline can

819

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

820

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

828

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

829

# must be true for both create time and run time. Cannot be true at run time

830

# if false at create time.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

831

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

832

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

833

# By specifying this parameter, you will download and install the following

834

# third-party software onto your managed Compute Engine instances:

835

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

836

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

837

# feature that may go away. Defaults to false.

838

# Corresponds to `--no_address` flag for [gcloud compute instances create]

839

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

840

# In order to use this, must be true for both create time and run time.

841

# Cannot be true at run time if false at create time. If you need to ssh into

842

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

843

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

844

# only load docker images from Google Container Registry and not Docker Hub.

845

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

846

# [configure access to Google services from internal

847

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

848

"zones": [ # List of Google Compute Engine availability zones to which resource

849

# creation will restricted. If empty, any zone may be chosen.

850

"A String",

851

],

852

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

853

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

854

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

855

"source": "A String", # The full or partial URL of the persistent disk to attach. See

856

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

857

# and

858

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

859

# for more details.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

860

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

861

# Specifies the path in the docker container where files on

862

# this disk should be located. For example, if `mountPoint`

863

# is `/mnt/disk`, and the parameter has `localPath`

864

# `inputs/file.txt`, the docker container can access the data at

865

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

866

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

867

# of the pipeline run, regardless of what this field is set to.

868

"name": "A String", # Required. The name of the disk that can be used in the pipeline

869

# parameters. Must be 1 - 63 characters.

870

# The name "boot" is reserved for system use.

871

"type": "A String", # Required. The type of the disk to create.

872

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

873

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

874

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

875

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

876

# for more details.

877

# Can only be set at create time.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

878

},

879

],

880

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

881

# By specifying this parameter, you will download and install the following

882

# third-party software onto your managed Compute Engine instances: NVIDIA®

883

# Tesla® drivers and NVIDIA® CUDA toolkit.

884

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

885

# available accelerator types.

886

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

887

},

888

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

889

# WRITE access.

890

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

891

# is called. Cannot be specified in the Pipeline used in the

892

# CreatePipelineRequest, and will be populated in the response to

893

# CreatePipeline and all subsequent Get and List calls. Indicates that the

894

# service has registered this pipeline.

895

"inputParameters": [ # Input parameters of the pipeline.

896

{ # Parameters facilitate setting and delivering data into the

897

# pipeline's execution environment. They are defined at create time,

898

# with optional defaults, and can be overridden at run time.

899

#

900

# If `localCopy` is unset, then the parameter specifies a string that

901

# is passed as-is into the pipeline, as the value of the environment

902

# variable with the given name. A default value can be optionally

903

# specified at create time. The default can be overridden at run time

904

# using the inputs map. If no default is given, a value must be

905

# supplied at runtime.

906

#

907

# If `localCopy` is defined, then the parameter specifies a data

908

# source or sink, both in Google Cloud Storage and on the Docker container

909

# where the pipeline computation is run. The service account associated with

910

# the Pipeline (by

911

# default the project's Compute Engine service account) must have access to the

912

# Google Cloud Storage paths.

913

#

914

# At run time, the Google Cloud Storage paths can be overridden if a default

915

# was provided at create time, or must be set otherwise. The pipeline runner

916

# should add a key/value pair to either the inputs or outputs map. The

917

# indicated data copies will be carried out before/after pipeline execution,

918

# just as if the corresponding arguments were provided to `gsutil cp`.

919

#

920

# For example: Given the following `PipelineParameter`, specified

921

# in the `inputParameters` list:

922

#

923

# ```

924

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

925

# ```

926

#

927

# where `disk` is defined in the `PipelineResources` object as:

928

#

929

# ```

930

# {name: "pd1", mountPoint: "/mnt/disk/"}

931

# ```

932

#

933

# We create a disk named `pd1`, mount it on the host VM, and map

934

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

935

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

940

# ```

941

#

942

# This would generate the following gsutil call:

943

#

944

# ```

945

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

946

# ```

947

#

948

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

949

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

954

# </thead>

955

# <tbody>

956

# <tr><td>file</td><td>file</td></tr>

957

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

962

#

963

# ```

964

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

965

# ```

966

#

967

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

972

# </thead>

973

# <tbody>

974

# <tr><td>file</td><td>file</td></tr>

975

# <tr>

976

# <td>file</td>

977

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

986

# on the boot disk, the local path cannot be a glob and must be a file.

987

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

988

# `LocalCopy` indicates where on the VM the file should be. The value

989

# given to this parameter (either at runtime or using `defaultValue`)

990

# must be the remote path where the file should be.

991

"disk": "A String", # Required. The name of the disk where this parameter is

992

# located. Can be the name of one of the disks specified in the

993

# Resources field, or "boot", which represents the Docker

994

# instance's boot disk and has a mount point of `/`.

995

"path": "A String", # Required. The path within the user's docker container where

996

# this input should be localized to and from, relative to the specified

997

# disk's mount point. For example: file.txt,

998

},

999

"description": "A String", # Human-readable description.

1000

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1001

# If `localCopy` is present, then this must be a Google Cloud Storage path

1002

# beginning with `gs://`.

1003

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1004

# as the key to the input and output maps in RunPipeline.

1005

},

1006

],

1007

"outputParameters": [ # Output parameters of the pipeline.

1008

{ # Parameters facilitate setting and delivering data into the

1009

# pipeline's execution environment. They are defined at create time,

1010

# with optional defaults, and can be overridden at run time.

1011

#

1012

# If `localCopy` is unset, then the parameter specifies a string that

1013

# is passed as-is into the pipeline, as the value of the environment

1014

# variable with the given name. A default value can be optionally

1015

# specified at create time. The default can be overridden at run time

1016

# using the inputs map. If no default is given, a value must be

1017

# supplied at runtime.

1018

#

1019

# If `localCopy` is defined, then the parameter specifies a data

1020

# source or sink, both in Google Cloud Storage and on the Docker container

1021

# where the pipeline computation is run. The service account associated with

1022

# the Pipeline (by

1023

# default the project's Compute Engine service account) must have access to the

1024

# Google Cloud Storage paths.

1025

#

1026

# At run time, the Google Cloud Storage paths can be overridden if a default

1027

# was provided at create time, or must be set otherwise. The pipeline runner

1028

# should add a key/value pair to either the inputs or outputs map. The

1029

# indicated data copies will be carried out before/after pipeline execution,

1030

# just as if the corresponding arguments were provided to `gsutil cp`.

1031

#

1032

# For example: Given the following `PipelineParameter`, specified

1033

# in the `inputParameters` list:

1034

#

1035

# ```

1036

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1037

# ```

1038

#

1039

# where `disk` is defined in the `PipelineResources` object as:

1040

#

1041

# ```

1042

# {name: "pd1", mountPoint: "/mnt/disk/"}

1043

# ```

1044

#

1045

# We create a disk named `pd1`, mount it on the host VM, and map

1046

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1047

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1052

# ```

1053

#

1054

# This would generate the following gsutil call:

1055

#

1056

# ```

1057

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1058

# ```

1059

#

1060

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1061

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1066

# </thead>

1067

# <tbody>

1068

# <tr><td>file</td><td>file</td></tr>

1069

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1074

#

1075

# ```

1076

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1077

# ```

1078

#

1079

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1084

# </thead>

1085

# <tbody>

1086

# <tr><td>file</td><td>file</td></tr>

1087

# <tr>

1088

# <td>file</td>

1089

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1098

# on the boot disk, the local path cannot be a glob and must be a file.

1099

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1100

# `LocalCopy` indicates where on the VM the file should be. The value

1101

# given to this parameter (either at runtime or using `defaultValue`)

1102

# must be the remote path where the file should be.

1103

"disk": "A String", # Required. The name of the disk where this parameter is

1104

# located. Can be the name of one of the disks specified in the

1105

# Resources field, or "boot", which represents the Docker

1106

# instance's boot disk and has a mount point of `/`.

1107

"path": "A String", # Required. The path within the user's docker container where

1108

# this input should be localized to and from, relative to the specified

1109

# disk's mount point. For example: file.txt,

1110

},

1111

"description": "A String", # Human-readable description.

1112

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1113

# If `localCopy` is present, then this must be a Google Cloud Storage path

1114

# beginning with `gs://`.

1115

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1116

# as the key to the input and output maps in RunPipeline.

1117

},

1118

],

1119

"description": "A String", # User-specified description.

1120

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

1121

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1122

# Users that run pipelines must have READ access to the image.

1123

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1124

# string will be executed within a bash shell.

1125

#

1126

# If the command exits with a non-zero exit code, output parameter

1127

# de-localization will be skipped and the pipeline operation's

1128

# `error` field will be populated.

1129

#

1130

# Maximum command string length is 16384.

1131

},

1132

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1133

# This name can be used for filtering Pipelines in ListPipelines.

}</pre>

</div>

<code class="details" id="getControllerConfig">getControllerConfig(validationToken=None, operationId=None, x__xgafv=None)</code>

1139

<pre>Gets controller configuration information. Should only be called

1140

by VMs created by the Pipelines Service and not by end users.

1141

1142

Args:

1143

validationToken: string, A parameter

1144

operationId: string, The operation to retrieve controller configuration for.

1145

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

1152

1153

{ # Stores the information that the controller will fetch from the

1154

# server in order to run. Should only be used by VMs created by the

1155

# Pipelines Service and not by end users.

"cmd": "A String",

"gcsSinks": {

"a_key": {

"values": [

"A String",

],

},

},

"gcsLogPath": "A String",

1165

"machineType": "A String",

"disks": {

"a_key": "A String",

},

"vars": {

"a_key": "A String",

},

"image": "A String",

"gcsSources": {

"a_key": {

"values": [

"A String",

],

},

},

}</pre>

</div>

<code class="details" id="list">list(namePrefix=None, pageSize=None, projectId=None, pageToken=None, x__xgafv=None)</code>

1185

<pre>Lists pipelines.

1186

1187

Caller must have READ permission to the project.

1188

1189

Args:

1190

namePrefix: string, Pipelines with names that match this prefix should be

1191

returned. If unspecified, all pipelines in the project, up to

1192

`pageSize`, will be returned.

1193

pageSize: integer, Number of pipelines to return at once. Defaults to 256, and max

1194

is 2048.

1195

projectId: string, Required. The name of the project to search for pipelines. Caller

1196

must have READ access to this project.

1197

pageToken: string, Token to use to indicate where to start getting results.

1198

If unspecified, returns the first page of results.

1199

x__xgafv: string, V1 error format.

Allowed values

1 - v1 error format

2 - v2 error format

Returns:

An object of the form:

1206

1207

{ # The response of ListPipelines. Contains at most `pageSize`

1208

# pipelines. If it contains `pageSize` pipelines, and more pipelines

1209

# exist, then `nextPageToken` will be populated and should be

1210

# used as the `pageToken` argument to a subsequent ListPipelines

1211

# request.

1212

"nextPageToken": "A String", # The token to use to get the next page of results.

1213

"pipelines": [ # The matched pipelines.

1214

{ # The pipeline object. Represents a transformation from a set of input

1215

# parameters to a set of output parameters. The transformation is defined

1216

# as a docker image and command to run within that image. Each pipeline

1217

# is run on a Google Compute Engine VM. A pipeline can be created with the

1218

# `create` method and then later run with the `run` method, or a pipeline can

1219

# be defined and run all at once with the `run` method.

1220

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1229

# must be true for both create time and run time. Cannot be true at run time

1230

# if false at create time.

1231

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

1232

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1233

# By specifying this parameter, you will download and install the following

1234

# third-party software onto your managed Compute Engine instances:

1235

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1236

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1237

# feature that may go away. Defaults to false.

1238

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1239

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1240

# In order to use this, must be true for both create time and run time.

1241

# Cannot be true at run time if false at create time. If you need to ssh into

1242

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1243

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1244

# only load docker images from Google Container Registry and not Docker Hub.

1245

# Before using this, you must

1246

# [configure access to Google services from internal

1247

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

1248

"zones": [ # List of Google Compute Engine availability zones to which resource

1249

# creation will restricted. If empty, any zone may be chosen.

1250

"A String",

1251

],

1252

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1253

"disks": [ # Disks to attach.

1254

{ # A Google Compute Engine disk resource specification.

1255

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1256

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1257

# and

1258

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1259

# for more details.

1260

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1261

# Specifies the path in the docker container where files on

1262

# this disk should be located. For example, if `mountPoint`

1263

# is `/mnt/disk`, and the parameter has `localPath`

1264

# `inputs/file.txt`, the docker container can access the data at

1265

# `/mnt/disk/inputs/file.txt`.

1266

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1267

# of the pipeline run, regardless of what this field is set to.

1268

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1269

# parameters. Must be 1 - 63 characters.

1270

# The name "boot" is reserved for system use.

1271

"type": "A String", # Required. The type of the disk to create.

1272

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1273

# This field is not applicable for local SSD.

1274

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1275

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1276

# for more details.

1277

# Can only be set at create time.

1278

},

1279

],

1280

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1281

# By specifying this parameter, you will download and install the following

1282

# third-party software onto your managed Compute Engine instances: NVIDIA®

1283

# Tesla® drivers and NVIDIA® CUDA toolkit.

1284

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1285

# available accelerator types.

1286

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1287

},

1288

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1289

# WRITE access.

1290

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1291

# is called. Cannot be specified in the Pipeline used in the

1292

# CreatePipelineRequest, and will be populated in the response to

1293

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1294

# service has registered this pipeline.

1295

"inputParameters": [ # Input parameters of the pipeline.

1296

{ # Parameters facilitate setting and delivering data into the

1297

# pipeline's execution environment. They are defined at create time,

1298

# with optional defaults, and can be overridden at run time.

1299

#

1300

# If `localCopy` is unset, then the parameter specifies a string that

1301

# is passed as-is into the pipeline, as the value of the environment

1302

# variable with the given name. A default value can be optionally

1303

# specified at create time. The default can be overridden at run time

1304

# using the inputs map. If no default is given, a value must be

1305

# supplied at runtime.

1306

#

1307

# If `localCopy` is defined, then the parameter specifies a data

1308

# source or sink, both in Google Cloud Storage and on the Docker container

1309

# where the pipeline computation is run. The service account associated with

1310

# the Pipeline (by

1311

# default the project's Compute Engine service account) must have access to the

1312

# Google Cloud Storage paths.

1313

#

1314

# At run time, the Google Cloud Storage paths can be overridden if a default

1315

# was provided at create time, or must be set otherwise. The pipeline runner

1316

# should add a key/value pair to either the inputs or outputs map. The

1317

# indicated data copies will be carried out before/after pipeline execution,

1318

# just as if the corresponding arguments were provided to `gsutil cp`.

1319

#

1320

# For example: Given the following `PipelineParameter`, specified

1321

# in the `inputParameters` list:

1322

#

1323

# ```

1324

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1325

# ```

1326

#

1327

# where `disk` is defined in the `PipelineResources` object as:

1328

#

1329

# ```

1330

# {name: "pd1", mountPoint: "/mnt/disk/"}

1331

# ```

1332

#

1333

# We create a disk named `pd1`, mount it on the host VM, and map

1334

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1335

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1340

# ```

1341

#

1342

# This would generate the following gsutil call:

1343

#

1344

# ```

1345

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1346

# ```

1347

#

1348

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1349

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1354

# </thead>

1355

# <tbody>

1356

# <tr><td>file</td><td>file</td></tr>

1357

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1362

#

1363

# ```

1364

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1365

# ```

1366

#

1367

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1372

# </thead>

1373

# <tbody>

1374

# <tr><td>file</td><td>file</td></tr>

1375

# <tr>

1376

# <td>file</td>

1377

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1386

# on the boot disk, the local path cannot be a glob and must be a file.

1387

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1388

# `LocalCopy` indicates where on the VM the file should be. The value

1389

# given to this parameter (either at runtime or using `defaultValue`)

1390

# must be the remote path where the file should be.

1391

"disk": "A String", # Required. The name of the disk where this parameter is

1392

# located. Can be the name of one of the disks specified in the

1393

# Resources field, or "boot", which represents the Docker

1394

# instance's boot disk and has a mount point of `/`.

1395

"path": "A String", # Required. The path within the user's docker container where

1396

# this input should be localized to and from, relative to the specified

1397

# disk's mount point. For example: file.txt,

1398

},

1399

"description": "A String", # Human-readable description.

1400

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1401

# If `localCopy` is present, then this must be a Google Cloud Storage path

1402

# beginning with `gs://`.

1403

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1404

# as the key to the input and output maps in RunPipeline.

1405

},

1406

],

1407

"outputParameters": [ # Output parameters of the pipeline.

1408

{ # Parameters facilitate setting and delivering data into the

1409

# pipeline's execution environment. They are defined at create time,

1410

# with optional defaults, and can be overridden at run time.

1411

#

1412

# If `localCopy` is unset, then the parameter specifies a string that

1413

# is passed as-is into the pipeline, as the value of the environment

1414

# variable with the given name. A default value can be optionally

1415

# specified at create time. The default can be overridden at run time

1416

# using the inputs map. If no default is given, a value must be

1417

# supplied at runtime.

1418

#

1419

# If `localCopy` is defined, then the parameter specifies a data

1420

# source or sink, both in Google Cloud Storage and on the Docker container

1421

# where the pipeline computation is run. The service account associated with

1422

# the Pipeline (by

1423

# default the project's Compute Engine service account) must have access to the

1424

# Google Cloud Storage paths.

1425

#

1426

# At run time, the Google Cloud Storage paths can be overridden if a default

1427

# was provided at create time, or must be set otherwise. The pipeline runner

1428

# should add a key/value pair to either the inputs or outputs map. The

1429

# indicated data copies will be carried out before/after pipeline execution,

1430

# just as if the corresponding arguments were provided to `gsutil cp`.

1431

#

1432

# For example: Given the following `PipelineParameter`, specified

1433

# in the `inputParameters` list:

1434

#

1435

# ```

1436

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1437

# ```

1438

#

1439

# where `disk` is defined in the `PipelineResources` object as:

1440

#

1441

# ```

1442

# {name: "pd1", mountPoint: "/mnt/disk/"}

1443

# ```

1444

#

1445

# We create a disk named `pd1`, mount it on the host VM, and map

1446

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1447

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1452

# ```

1453

#

1454

# This would generate the following gsutil call:

1455

#

1456

# ```

1457

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1458

# ```

1459

#

1460

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1461

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1466

# </thead>

1467

# <tbody>

1468

# <tr><td>file</td><td>file</td></tr>

1469

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1474

#

1475

# ```

1476

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1477

# ```

1478

#

1479

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1484

# </thead>

1485

# <tbody>

1486

# <tr><td>file</td><td>file</td></tr>

1487

# <tr>

1488

# <td>file</td>

1489

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1498

# on the boot disk, the local path cannot be a glob and must be a file.

1499

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1500

# `LocalCopy` indicates where on the VM the file should be. The value

1501

# given to this parameter (either at runtime or using `defaultValue`)

1502

# must be the remote path where the file should be.

1503

"disk": "A String", # Required. The name of the disk where this parameter is

1504

# located. Can be the name of one of the disks specified in the

1505

# Resources field, or "boot", which represents the Docker

1506

# instance's boot disk and has a mount point of `/`.

1507

"path": "A String", # Required. The path within the user's docker container where

1508

# this input should be localized to and from, relative to the specified

1509

# disk's mount point. For example: file.txt,

1510

},

1511

"description": "A String", # Human-readable description.

1512

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1513

# If `localCopy` is present, then this must be a Google Cloud Storage path

1514

# beginning with `gs://`.

1515

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1516

# as the key to the input and output maps in RunPipeline.

1517

},

1518

],

1519

"description": "A String", # User-specified description.

1520

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

1521

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1522

# Users that run pipelines must have READ access to the image.

1523

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1524

# string will be executed within a bash shell.

1525

#

1526

# If the command exits with a non-zero exit code, output parameter

1527

# de-localization will be skipped and the pipeline operation's

1528

# `error` field will be populated.

1529

#

1530

# Maximum command string length is 16384.

1531

},

1532

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1533

# This name can be used for filtering Pipelines in ListPipelines.

},

],

}</pre>

</div>

<code class="details" id="list_next">list_next(previous_request, previous_response)</code>

1541

<pre>Retrieves the next page of results.

1542

1543

Args:

1544

previous_request: The request for the previous page. (required)

1545

previous_response: The response from the request for the previous page. (required)

1546

1547

Returns:

1548

A request object that you can call 'execute()' on to request the next

1549

page. Returns None if there are no more items in the collection.

</pre>

</div>

<code class="details" id="run">run(body=None, x__xgafv=None)</code>

1555

<pre>Runs a pipeline. If `pipelineId` is specified in the request, then

1556

run a saved pipeline. If `ephemeralPipeline` is specified, then run

1557

that pipeline once without saving a copy.

1558

1559

The caller must have READ permission to the project where the pipeline

1560

is stored and WRITE permission to the project where the pipeline will be

1561

run, as VMs will be created and storage will be used.

1562

1563

If a pipeline operation is still running after 6 days, it will be canceled.

1564

1565

Args:

1566

body: object, The request body.

1567

The object takes the form of:

1568

1569

{ # The request to run a pipeline. If `pipelineId` is specified, it

1570

# refers to a saved pipeline created with CreatePipeline and set as

1571

# the `pipelineId` of the returned Pipeline object. If

1572

# `ephemeralPipeline` is specified, that pipeline is run once

1573

# with the given args and not saved. It is an error to specify both

1574

# `pipelineId` and `ephemeralPipeline`. `pipelineArgs`

1575

# must be specified.

1576

"pipelineId": "A String", # The already created pipeline to run.

1577

"ephemeralPipeline": { # The pipeline object. Represents a transformation from a set of input # A new pipeline object to run once and then delete.

1578

# parameters to a set of output parameters. The transformation is defined

1579

# as a docker image and command to run within that image. Each pipeline

1580

# is run on a Google Compute Engine VM. A pipeline can be created with the

1581

# `create` method and then later run with the `run` method, or a pipeline can

1582

# be defined and run all at once with the `run` method.

1583

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1592

# must be true for both create time and run time. Cannot be true at run time

1593

# if false at create time.

1594

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

1595

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1596

# By specifying this parameter, you will download and install the following

1597

# third-party software onto your managed Compute Engine instances:

1598

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1599

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1600

# feature that may go away. Defaults to false.

1601

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1602

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1603

# In order to use this, must be true for both create time and run time.

1604

# Cannot be true at run time if false at create time. If you need to ssh into

1605

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1606

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1607

# only load docker images from Google Container Registry and not Docker Hub.

1608

# Before using this, you must

1609

# [configure access to Google services from internal

1610

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

1611

"zones": [ # List of Google Compute Engine availability zones to which resource

1612

# creation will restricted. If empty, any zone may be chosen.

1613

"A String",

1614

],

1615

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1616

"disks": [ # Disks to attach.

1617

{ # A Google Compute Engine disk resource specification.

1618

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1619

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1620

# and

1621

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1622

# for more details.

1623

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1624

# Specifies the path in the docker container where files on

1625

# this disk should be located. For example, if `mountPoint`

1626

# is `/mnt/disk`, and the parameter has `localPath`

1627

# `inputs/file.txt`, the docker container can access the data at

1628

# `/mnt/disk/inputs/file.txt`.

1629

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1630

# of the pipeline run, regardless of what this field is set to.

1631

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1632

# parameters. Must be 1 - 63 characters.

1633

# The name "boot" is reserved for system use.

1634

"type": "A String", # Required. The type of the disk to create.

1635

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1636

# This field is not applicable for local SSD.

1637

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1638

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1639

# for more details.

1640

# Can only be set at create time.

1641

},

1642

],

1643

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1644

# By specifying this parameter, you will download and install the following

1645

# third-party software onto your managed Compute Engine instances: NVIDIA®

1646

# Tesla® drivers and NVIDIA® CUDA toolkit.

1647

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1648

# available accelerator types.

1649

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

1650

},

1651

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1652

# WRITE access.

1653

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1654

# is called. Cannot be specified in the Pipeline used in the

1655

# CreatePipelineRequest, and will be populated in the response to

1656

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1657

# service has registered this pipeline.

1658

"inputParameters": [ # Input parameters of the pipeline.

1659

{ # Parameters facilitate setting and delivering data into the

1660

# pipeline's execution environment. They are defined at create time,

1661

# with optional defaults, and can be overridden at run time.

1662

#

1663

# If `localCopy` is unset, then the parameter specifies a string that

1664

# is passed as-is into the pipeline, as the value of the environment

1665

# variable with the given name. A default value can be optionally

1666

# specified at create time. The default can be overridden at run time

1667

# using the inputs map. If no default is given, a value must be

1668

# supplied at runtime.

1669

#

1670

# If `localCopy` is defined, then the parameter specifies a data

1671

# source or sink, both in Google Cloud Storage and on the Docker container

1672

# where the pipeline computation is run. The service account associated with

1673

# the Pipeline (by

1674

# default the project's Compute Engine service account) must have access to the

1675

# Google Cloud Storage paths.

1676

#

1677

# At run time, the Google Cloud Storage paths can be overridden if a default

1678

# was provided at create time, or must be set otherwise. The pipeline runner

1679

# should add a key/value pair to either the inputs or outputs map. The

1680

# indicated data copies will be carried out before/after pipeline execution,

1681

# just as if the corresponding arguments were provided to `gsutil cp`.

1682

#

1683

# For example: Given the following `PipelineParameter`, specified

1684

# in the `inputParameters` list:

1685

#

1686

# ```

1687

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1688

# ```

1689

#

1690

# where `disk` is defined in the `PipelineResources` object as:

1691

#

1692

# ```

1693

# {name: "pd1", mountPoint: "/mnt/disk/"}

1694

# ```

1695

#

1696

# We create a disk named `pd1`, mount it on the host VM, and map

1697

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1698

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1703

# ```

1704

#

1705

# This would generate the following gsutil call:

1706

#

1707

# ```

1708

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1709

# ```

1710

#

1711

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1712

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1717

# </thead>

1718

# <tbody>

1719

# <tr><td>file</td><td>file</td></tr>

1720

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1725

#

1726

# ```

1727

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1728

# ```

1729

#

1730

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1735

# </thead>

1736

# <tbody>

1737

# <tr><td>file</td><td>file</td></tr>

1738

# <tr>

1739

# <td>file</td>

1740

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1749

# on the boot disk, the local path cannot be a glob and must be a file.

1750

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1751

# `LocalCopy` indicates where on the VM the file should be. The value

1752

# given to this parameter (either at runtime or using `defaultValue`)

1753

# must be the remote path where the file should be.

1754

"disk": "A String", # Required. The name of the disk where this parameter is

1755

# located. Can be the name of one of the disks specified in the

1756

# Resources field, or "boot", which represents the Docker

1757

# instance's boot disk and has a mount point of `/`.

1758

"path": "A String", # Required. The path within the user's docker container where

1759

# this input should be localized to and from, relative to the specified

1760

# disk's mount point. For example: file.txt,

1761

},

1762

"description": "A String", # Human-readable description.

1763

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1764

# If `localCopy` is present, then this must be a Google Cloud Storage path

1765

# beginning with `gs://`.

1766

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1767

# as the key to the input and output maps in RunPipeline.

1768

},

1769

],

1770

"outputParameters": [ # Output parameters of the pipeline.

1771

{ # Parameters facilitate setting and delivering data into the

1772

# pipeline's execution environment. They are defined at create time,

1773

# with optional defaults, and can be overridden at run time.

1774

#

1775

# If `localCopy` is unset, then the parameter specifies a string that

1776

# is passed as-is into the pipeline, as the value of the environment

1777

# variable with the given name. A default value can be optionally

1778

# specified at create time. The default can be overridden at run time

1779

# using the inputs map. If no default is given, a value must be

1780

# supplied at runtime.

1781

#

1782

# If `localCopy` is defined, then the parameter specifies a data

1783

# source or sink, both in Google Cloud Storage and on the Docker container

1784

# where the pipeline computation is run. The service account associated with

1785

# the Pipeline (by

1786

# default the project's Compute Engine service account) must have access to the

1787

# Google Cloud Storage paths.

1788

#

1789

# At run time, the Google Cloud Storage paths can be overridden if a default

1790

# was provided at create time, or must be set otherwise. The pipeline runner

1791

# should add a key/value pair to either the inputs or outputs map. The

1792

# indicated data copies will be carried out before/after pipeline execution,

1793

# just as if the corresponding arguments were provided to `gsutil cp`.

1794

#

1795

# For example: Given the following `PipelineParameter`, specified

1796

# in the `inputParameters` list:

1797

#

1798

# ```

1799

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1800

# ```

1801

#

1802

# where `disk` is defined in the `PipelineResources` object as:

1803

#

1804

# ```

1805

# {name: "pd1", mountPoint: "/mnt/disk/"}

1806

# ```

1807

#

1808

# We create a disk named `pd1`, mount it on the host VM, and map

1809

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1810

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1815

# ```

1816

#

1817

# This would generate the following gsutil call:

1818

#

1819

# ```

1820

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1821

# ```

1822

#

1823

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1824

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1829

# </thead>

1830

# <tbody>

1831

# <tr><td>file</td><td>file</td></tr>

1832

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1837

#

1838

# ```

1839

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1840

# ```

1841

#

1842

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1847

# </thead>

1848

# <tbody>

1849

# <tr><td>file</td><td>file</td></tr>

1850

# <tr>

1851

# <td>file</td>

1852

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1861

# on the boot disk, the local path cannot be a glob and must be a file.

1862

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1863

# `LocalCopy` indicates where on the VM the file should be. The value

1864

# given to this parameter (either at runtime or using `defaultValue`)

1865

# must be the remote path where the file should be.

1866

"disk": "A String", # Required. The name of the disk where this parameter is

1867

# located. Can be the name of one of the disks specified in the

1868

# Resources field, or "boot", which represents the Docker

1869

# instance's boot disk and has a mount point of `/`.

1870

"path": "A String", # Required. The path within the user's docker container where

1871

# this input should be localized to and from, relative to the specified

1872

# disk's mount point. For example: file.txt,

1873

},

1874

"description": "A String", # Human-readable description.

1875

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1876

# If `localCopy` is present, then this must be a Google Cloud Storage path

1877

# beginning with `gs://`.

1878

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1879

# as the key to the input and output maps in RunPipeline.

1880

},

1881

],

1882

"description": "A String", # User-specified description.

1883

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

1884

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1885

# Users that run pipelines must have READ access to the image.

1886

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1887

# string will be executed within a bash shell.

1888

#

1889

# If the command exits with a non-zero exit code, output parameter

1890

# de-localization will be skipped and the pipeline operation's

1891

# `error` field will be populated.

1892

#

1893

# Maximum command string length is 16384.

1894

},

1895

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1896

# This name can be used for filtering Pipelines in ListPipelines.

1897

},

1898

"pipelineArgs": { # The pipeline run arguments. # The arguments to use when running this pipeline.

1899

"logging": { # The logging options for the pipeline run. # Required. Logging options. Used by the service to communicate results

1900

# to the user.

1901

"gcsPath": "A String", # The location in Google Cloud Storage to which the pipeline logs

1902

# will be copied. Can be specified as a fully qualified directory

1903

# path, in which case logs will be output with a unique identifier

1904

# as the filename in that directory, or as a fully specified path,

1905

# which must end in `.log`, in which case that path will be

1906

# used, and the user must ensure that logs are not

1907

# overwritten. Stdout and stderr logs from the run are also

1908

# generated and output as `-stdout.log` and `-stderr.log`.

1909

},

1910

"inputs": { # Pipeline input arguments; keys are defined in the pipeline documentation.

1911

# All input parameters that do not have default values must be specified.

1912

# If parameters with defaults are specified here, the defaults will be

# overridden.

"a_key": "A String",

},

"resources": { # The system resources for the pipeline run. # Specifies resource requirements/overrides for the pipeline run.

1917

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1918

# must be true for both create time and run time. Cannot be true at run time

1919

# if false at create time.

1920

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

1921

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1922

# By specifying this parameter, you will download and install the following

1923

# third-party software onto your managed Compute Engine instances:

1924

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1925

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1926

# feature that may go away. Defaults to false.

1927

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1928

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1929

# In order to use this, must be true for both create time and run time.

1930

# Cannot be true at run time if false at create time. If you need to ssh into

1931

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1932

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1933

# only load docker images from Google Container Registry and not Docker Hub.

1934

# Before using this, you must

1935

# [configure access to Google services from internal

1936

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

1937

"zones": [ # List of Google Compute Engine availability zones to which resource

1938

# creation will restricted. If empty, any zone may be chosen.

1939

"A String",

1940

],

1941

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

1942

"disks": [ # Disks to attach.

1943

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1944

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1945

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1946

# and

1947

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1948

# for more details.

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

1949

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1950

# Specifies the path in the docker container where files on

1951

# this disk should be located. For example, if `mountPoint`

1952

# is `/mnt/disk`, and the parameter has `localPath`

1953

# `inputs/file.txt`, the docker container can access the data at

1954

# `/mnt/disk/inputs/file.txt`.

1955

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1956

# of the pipeline run, regardless of what this field is set to.

1957

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1958

# parameters. Must be 1 - 63 characters.

1959

# The name "boot" is reserved for system use.

1960

"type": "A String", # Required. The type of the disk to create.

1961

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1962

# This field is not applicable for local SSD.

1963

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1964

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1965

# for more details.

1966

# Can only be set at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1967

},

1968

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

1969

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1970

# By specifying this parameter, you will download and install the following

1971

# third-party software onto your managed Compute Engine instances: NVIDIA®

1972

# Tesla® drivers and NVIDIA® CUDA toolkit.

1973

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1974

# available accelerator types.

1975

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1976

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

1977

"labels": { # Labels to apply to this pipeline run. Labels will also be applied to

1978

# compute resources (VM, disks) created by this pipeline run. When listing

1979

# operations, operations can filtered by labels.

1980

# Label keys may not be empty; label values may be empty. Non-empty labels

1981

# must be 1-63 characters long, and comply with [RFC1035]

1982

# (https://www.ietf.org/rfc/rfc1035.txt).

1983

# Specifically, the name must be 1-63 characters long and match the regular

1984

# expression `[a-z]([-a-z0-9]*[a-z0-9])?` which means the first

1985

# character must be a lowercase letter, and all following characters must be

1986

# a dash, lowercase letter, or digit, except the last character, which cannot

# be a dash.

"a_key": "A String",

},

"serviceAccount": { # A Google Cloud Service Account. # The Google Cloud Service Account that will be used to access data and

1991

# services. By default, the compute service account associated with

1992

# `projectId` is used.

1993

"scopes": [ # List of scopes to be enabled for this service account on the VM.

1994

# The following scopes are automatically included:

1995

#

1996

# * https://www.googleapis.com/auth/compute

1997

# * https://www.googleapis.com/auth/devstorage.full_control

1998

# * https://www.googleapis.com/auth/genomics

1999

# * https://www.googleapis.com/auth/logging.write

2000

# * https://www.googleapis.com/auth/monitoring.write

2001

"A String",

2002

],

2003

"email": "A String", # Email address of the service account. Defaults to `default`,

2004

# which uses the compute service account associated with the project.

2005

},

2006

"clientId": "A String", # This field is deprecated. Use `labels` instead. Client-specified pipeline

2007

# operation identifier.

2008

"projectId": "A String", # Required. The project in which to run the pipeline. The caller must have

2009

# WRITER access to all Google Cloud services and resources (e.g. Google

2010

# Compute Engine) will be used.

2011

"outputs": { # Pipeline output arguments; keys are defined in the pipeline

2012

# documentation. All output parameters of without default values

2013

# must be specified. If parameters with defaults are specified

2014

# here, the defaults will be overridden.

2015

"a_key": "A String",

2016

},

2017

"keepVmAliveOnFailureDuration": "A String", # How long to keep the VM up after a failure (for example docker command

2018

# failed, copying input or output files failed, etc). While the VM is up, one

2019

# can ssh into the VM to debug. Default is 0; maximum allowed value is 1 day.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2020

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2021

}

2022

2023

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2024

Allowed values

2025

1 - v1 error format

2026

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2027

2028

Returns:

2029

An object of the form:

2030

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2031

{ # This resource represents a long-running operation that is the result of a

2032

# network API call.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2033

"error": { # The `Status` type defines a logical error model that is suitable for # The error result of the operation in case of failure or cancellation.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2034

# different programming environments, including REST APIs and RPC APIs. It is

2035

# used by [gRPC](https://github.com/grpc). Each `Status` message contains

2036

# three pieces of data: error code, error message, and error details.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2037

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2038

# You can find out more about this error model and how to work with it in the

2039

# [API Design Guide](https://cloud.google.com/apis/design/errors).

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

2040

"code": 42, # The status code, which should be an enum value of google.rpc.Code.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2041

"details": [ # A list of messages that carry the error details. There is a common set of

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2042

# message types for APIs to use.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2043

{

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2044

"a_key": "", # Properties of the object. Contains field @type with type URL.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2045

},

2046

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

2047

"message": "A String", # A developer-facing error message, which should be in English. Any

2048

# user-facing error message should be localized and sent in the

2049

# google.rpc.Status.details field, or localized by the client.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2050

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2051

"metadata": { # An OperationMetadata or Metadata object. This will always be returned with the Operation.

2052

"a_key": "", # Properties of the object. Contains field @type with type URL.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2053

},

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

2054

"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. For example&#58; `operations/CJHU7Oi_ChDrveSpBRjfuL-qzoWAgEw`

Bu Sun Kim

4ed7d3f

2020-05-27 12:20:54 -0700

[diff] [blame]

2055

"done": True or False, # If the value is `false`, it means the operation is still in progress.

2056

# If `true`, the operation is completed, and either `error` or `response` is

2057

# available.

2058

"response": { # An Empty object.

2059

"a_key": "", # Properties of the object. Contains field @type with type URL.

2060

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2065

<code class="details" id="setOperationStatus">setOperationStatus(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2066

<pre>Sets status of a given operation. Any new timestamps (as determined by

2067

description) are appended to TimestampEvents. Should only be called by VMs

2068

created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2069

2070

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2071

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2072

The object takes the form of:

2073

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2074

{ # Request to set operation status. Should only be used by VMs

2075

# created by the Pipelines Service and not by end users.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2076

"errorCode": "A String",

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

2077

"operationId": "A String",

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2078

"timestampEvents": [

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2079

{ # Stores the list of events and times they occured for major events in job

2080

# execution.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2081

"timestamp": "A String", # The time this event occured.

2082

"description": "A String", # String indicating the type of event

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2083

},

2084

],

Bu Sun Kim

2020-07-22 17:02:09 -0700

[diff] [blame^]

2085

"errorMessage": "A String",

2086

"validationToken": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2087

}

2088

2089

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2090

Allowed values

2091

1 - v1 error format

2092

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2093

2094

Returns:

2095

An object of the form:

2096

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2097

{ # A generic empty message that you can re-use to avoid defining duplicated

2098

# empty messages in your APIs. A typical example is to use it as the request

2099

# or the response type of an API method. For instance:

2100

#

2101

# service Foo {

2102

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

2103

# }

2104

#

2105

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati