Blame - docs/dyn/genomics_v1alpha2.pipelines.html - platform/external/python/google-api-python-client

2016-10-21 13:59:49 -0700

[diff] [blame]

235

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

236

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

237

# as the key to the input and output maps in RunPipeline.

238

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

239

},

240

],

241

"inputParameters": [ # Input parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

242

{ # Parameters facilitate setting and delivering data into the

243

# pipeline's execution environment. They are defined at create time,

244

# with optional defaults, and can be overridden at run time.

245

#

246

# If `localCopy` is unset, then the parameter specifies a string that

247

# is passed as-is into the pipeline, as the value of the environment

248

# variable with the given name. A default value can be optionally

249

# specified at create time. The default can be overridden at run time

250

# using the inputs map. If no default is given, a value must be

251

# supplied at runtime.

252

#

253

# If `localCopy` is defined, then the parameter specifies a data

254

# source or sink, both in Google Cloud Storage and on the Docker container

255

# where the pipeline computation is run. The service account associated with

256

# the Pipeline (by

257

# default the project's Compute Engine service account) must have access to the

258

# Google Cloud Storage paths.

259

#

260

# At run time, the Google Cloud Storage paths can be overridden if a default

261

# was provided at create time, or must be set otherwise. The pipeline runner

262

# should add a key/value pair to either the inputs or outputs map. The

263

# indicated data copies will be carried out before/after pipeline execution,

264

# just as if the corresponding arguments were provided to `gsutil cp`.

265

#

266

# For example: Given the following `PipelineParameter`, specified

267

# in the `inputParameters` list:

268

#

269

# ```

270

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

271

# ```

272

#

273

# where `disk` is defined in the `PipelineResources` object as:

274

#

275

# ```

276

# {name: "pd1", mountPoint: "/mnt/disk/"}

277

# ```

278

#

279

# We create a disk named `pd1`, mount it on the host VM, and map

280

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

281

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

286

# ```

287

#

288

# This would generate the following gsutil call:

289

#

290

# ```

291

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

292

# ```

293

#

294

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

295

# Docker container. Acceptable paths are:

296

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

297

# <table>

298

# <thead>

299

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

300

# </thead>

301

# <tbody>

302

# <tr><td>file</td><td>file</td></tr>

303

# <tr><td>glob</td><td>directory</td></tr>

304

# </tbody>

305

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

306

#

307

# For outputs, the direction of the copy is reversed:

308

#

309

# ```

310

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

311

# ```

312

#

313

# Acceptable paths are:

314

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

315

# <table>

316

# <thead>

317

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

318

# </thead>

319

# <tbody>

320

# <tr><td>file</td><td>file</td></tr>

321

# <tr>

322

# <td>file</td>

323

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

328

# </tbody>

329

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

330

#

331

# One restriction due to docker limitations, is that for outputs that are found

332

# on the boot disk, the local path cannot be a glob and must be a file.

333

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

334

# If `localCopy` is present, then this must be a Google Cloud Storage path

335

# beginning with `gs://`.

336

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

337

# `LocalCopy` indicates where on the VM the file should be. The value

338

# given to this parameter (either at runtime or using `defaultValue`)

339

# must be the remote path where the file should be.

340

"path": "A String", # Required. The path within the user's docker container where

341

# this input should be localized to and from, relative to the specified

342

# disk's mount point. For example: file.txt,

343

"disk": "A String", # Required. The name of the disk where this parameter is

344

# located. Can be the name of one of the disks specified in the

345

# Resources field, or "boot", which represents the Docker

346

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

347

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

348

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

349

# as the key to the input and output maps in RunPipeline.

350

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

351

},

352

],

353

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

354

"cmd": "A String", # Required. The command or newline delimited script to run. The command

355

# string will be executed within a bash shell.

356

#

357

# If the command exits with a non-zero exit code, output parameter

358

# de-localization will be skipped and the pipeline operation's

359

# `error` field will be populated.

360

#

361

# Maximum command string length is 16384.

362

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

363

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

364

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

365

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

374

# feature that may go away. Defaults to false.

375

# Corresponds to `--no_address` flag for [gcloud compute instances create]

376

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

377

# In order to use this, must be true for both create time and run time.

378

# Cannot be true at run time if false at create time. If you need to ssh into

379

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

380

# the private VM's Internal IP. If noAddress is set, this pipeline run may

381

# only load docker images from Google Container Registry and not Docker Hub.

382

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

383

# [configure access to Google services from internal

384

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

385

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

386

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

387

# By specifying this parameter, you will download and install the following

388

# third-party software onto your managed Compute Engine instances:

389

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

390

"disks": [ # Disks to attach.

391

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

392

"name": "A String", # Required. The name of the disk that can be used in the pipeline

393

# parameters. Must be 1 - 63 characters.

394

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

395

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

396

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

397

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

398

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

399

"source": "A String", # The full or partial URL of the persistent disk to attach. See

400

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

401

# and

402

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

403

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

404

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

405

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

406

# for more details.

407

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

408

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

409

# Specifies the path in the docker container where files on

410

# this disk should be located. For example, if `mountPoint`

411

# is `/mnt/disk`, and the parameter has `localPath`

412

# `inputs/file.txt`, the docker container can access the data at

413

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

414

"type": "A String", # Required. The type of the disk to create.

415

},

416

],

417

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

418

"zones": [ # List of Google Compute Engine availability zones to which resource

419

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

420

"A String",

421

],

422

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

423

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

424

# By specifying this parameter, you will download and install the following

425

# third-party software onto your managed Compute Engine instances: NVIDIA®

426

# Tesla® drivers and NVIDIA® CUDA toolkit.

427

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

428

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

429

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

430

# must be true for both create time and run time. Cannot be true at run time

431

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

432

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

433

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

434

# This name can be used for filtering Pipelines in ListPipelines.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

435

}

436

437

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

438

Allowed values

439

1 - v1 error format

440

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

441

442

Returns:

443

An object of the form:

444

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

445

{ # The pipeline object. Represents a transformation from a set of input

446

# parameters to a set of output parameters. The transformation is defined

447

# as a docker image and command to run within that image. Each pipeline

448

# is run on a Google Compute Engine VM. A pipeline can be created with the

449

# `create` method and then later run with the `run` method, or a pipeline can

450

# be defined and run all at once with the `run` method.

451

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

452

# is called. Cannot be specified in the Pipeline used in the

453

# CreatePipelineRequest, and will be populated in the response to

454

# CreatePipeline and all subsequent Get and List calls. Indicates that the

455

# service has registered this pipeline.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

456

"description": "A String", # User-specified description.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

457

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

458

# WRITE access.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

459

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

460

{ # Parameters facilitate setting and delivering data into the

461

# pipeline's execution environment. They are defined at create time,

462

# with optional defaults, and can be overridden at run time.

463

#

464

# If `localCopy` is unset, then the parameter specifies a string that

465

# is passed as-is into the pipeline, as the value of the environment

466

# variable with the given name. A default value can be optionally

467

# specified at create time. The default can be overridden at run time

468

# using the inputs map. If no default is given, a value must be

469

# supplied at runtime.

470

#

471

# If `localCopy` is defined, then the parameter specifies a data

472

# source or sink, both in Google Cloud Storage and on the Docker container

473

# where the pipeline computation is run. The service account associated with

474

# the Pipeline (by

475

# default the project's Compute Engine service account) must have access to the

476

# Google Cloud Storage paths.

477

#

478

# At run time, the Google Cloud Storage paths can be overridden if a default

479

# was provided at create time, or must be set otherwise. The pipeline runner

480

# should add a key/value pair to either the inputs or outputs map. The

481

# indicated data copies will be carried out before/after pipeline execution,

482

# just as if the corresponding arguments were provided to `gsutil cp`.

483

#

484

# For example: Given the following `PipelineParameter`, specified

485

# in the `inputParameters` list:

486

#

487

# ```

488

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

489

# ```

490

#

491

# where `disk` is defined in the `PipelineResources` object as:

492

#

493

# ```

494

# {name: "pd1", mountPoint: "/mnt/disk/"}

495

# ```

496

#

497

# We create a disk named `pd1`, mount it on the host VM, and map

498

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

499

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

504

# ```

505

#

506

# This would generate the following gsutil call:

507

#

508

# ```

509

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

510

# ```

511

#

512

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

513

# Docker container. Acceptable paths are:

514

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

515

# <table>

516

# <thead>

517

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

518

# </thead>

519

# <tbody>

520

# <tr><td>file</td><td>file</td></tr>

521

# <tr><td>glob</td><td>directory</td></tr>

522

# </tbody>

523

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

524

#

525

# For outputs, the direction of the copy is reversed:

526

#

527

# ```

528

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

529

# ```

530

#

531

# Acceptable paths are:

532

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

533

# <table>

534

# <thead>

535

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

536

# </thead>

537

# <tbody>

538

# <tr><td>file</td><td>file</td></tr>

539

# <tr>

540

# <td>file</td>

541

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

546

# </tbody>

547

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

548

#

549

# One restriction due to docker limitations, is that for outputs that are found

550

# on the boot disk, the local path cannot be a glob and must be a file.

551

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

552

# If `localCopy` is present, then this must be a Google Cloud Storage path

553

# beginning with `gs://`.

554

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

555

# `LocalCopy` indicates where on the VM the file should be. The value

556

# given to this parameter (either at runtime or using `defaultValue`)

557

# must be the remote path where the file should be.

558

"path": "A String", # Required. The path within the user's docker container where

559

# this input should be localized to and from, relative to the specified

560

# disk's mount point. For example: file.txt,

561

"disk": "A String", # Required. The name of the disk where this parameter is

562

# located. Can be the name of one of the disks specified in the

563

# Resources field, or "boot", which represents the Docker

564

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

565

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

566

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

567

# as the key to the input and output maps in RunPipeline.

568

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

569

},

570

],

571

"inputParameters": [ # Input parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

572

{ # Parameters facilitate setting and delivering data into the

573

# pipeline's execution environment. They are defined at create time,

574

# with optional defaults, and can be overridden at run time.

575

#

576

# If `localCopy` is unset, then the parameter specifies a string that

577

# is passed as-is into the pipeline, as the value of the environment

578

# variable with the given name. A default value can be optionally

579

# specified at create time. The default can be overridden at run time

580

# using the inputs map. If no default is given, a value must be

581

# supplied at runtime.

582

#

583

# If `localCopy` is defined, then the parameter specifies a data

584

# source or sink, both in Google Cloud Storage and on the Docker container

585

# where the pipeline computation is run. The service account associated with

586

# the Pipeline (by

587

# default the project's Compute Engine service account) must have access to the

588

# Google Cloud Storage paths.

589

#

590

# At run time, the Google Cloud Storage paths can be overridden if a default

591

# was provided at create time, or must be set otherwise. The pipeline runner

592

# should add a key/value pair to either the inputs or outputs map. The

593

# indicated data copies will be carried out before/after pipeline execution,

594

# just as if the corresponding arguments were provided to `gsutil cp`.

595

#

596

# For example: Given the following `PipelineParameter`, specified

597

# in the `inputParameters` list:

598

#

599

# ```

600

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

601

# ```

602

#

603

# where `disk` is defined in the `PipelineResources` object as:

604

#

605

# ```

606

# {name: "pd1", mountPoint: "/mnt/disk/"}

607

# ```

608

#

609

# We create a disk named `pd1`, mount it on the host VM, and map

610

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

611

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

616

# ```

617

#

618

# This would generate the following gsutil call:

619

#

620

# ```

621

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

622

# ```

623

#

624

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

625

# Docker container. Acceptable paths are:

626

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

627

# <table>

628

# <thead>

629

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

630

# </thead>

631

# <tbody>

632

# <tr><td>file</td><td>file</td></tr>

633

# <tr><td>glob</td><td>directory</td></tr>

634

# </tbody>

635

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

636

#

637

# For outputs, the direction of the copy is reversed:

638

#

639

# ```

640

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

641

# ```

642

#

643

# Acceptable paths are:

644

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

645

# <table>

646

# <thead>

647

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

648

# </thead>

649

# <tbody>

650

# <tr><td>file</td><td>file</td></tr>

651

# <tr>

652

# <td>file</td>

653

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

658

# </tbody>

659

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

660

#

661

# One restriction due to docker limitations, is that for outputs that are found

662

# on the boot disk, the local path cannot be a glob and must be a file.

663

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

664

# If `localCopy` is present, then this must be a Google Cloud Storage path

665

# beginning with `gs://`.

666

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

667

# `LocalCopy` indicates where on the VM the file should be. The value

668

# given to this parameter (either at runtime or using `defaultValue`)

669

# must be the remote path where the file should be.

670

"path": "A String", # Required. The path within the user's docker container where

671

# this input should be localized to and from, relative to the specified

672

# disk's mount point. For example: file.txt,

673

"disk": "A String", # Required. The name of the disk where this parameter is

674

# located. Can be the name of one of the disks specified in the

675

# Resources field, or "boot", which represents the Docker

676

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

677

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

678

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

679

# as the key to the input and output maps in RunPipeline.

680

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

681

},

682

],

683

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

684

"cmd": "A String", # Required. The command or newline delimited script to run. The command

685

# string will be executed within a bash shell.

686

#

687

# If the command exits with a non-zero exit code, output parameter

688

# de-localization will be skipped and the pipeline operation's

689

# `error` field will be populated.

690

#

691

# Maximum command string length is 16384.

692

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

693

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

694

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

695

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

704

# feature that may go away. Defaults to false.

705

# Corresponds to `--no_address` flag for [gcloud compute instances create]

706

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

707

# In order to use this, must be true for both create time and run time.

708

# Cannot be true at run time if false at create time. If you need to ssh into

709

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

710

# the private VM's Internal IP. If noAddress is set, this pipeline run may

711

# only load docker images from Google Container Registry and not Docker Hub.

712

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

713

# [configure access to Google services from internal

714

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

715

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

716

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

717

# By specifying this parameter, you will download and install the following

718

# third-party software onto your managed Compute Engine instances:

719

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

720

"disks": [ # Disks to attach.

721

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

722

"name": "A String", # Required. The name of the disk that can be used in the pipeline

723

# parameters. Must be 1 - 63 characters.

724

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

725

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

726

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

727

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

728

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

729

"source": "A String", # The full or partial URL of the persistent disk to attach. See

730

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

731

# and

732

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

733

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

734

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

735

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

736

# for more details.

737

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

738

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

739

# Specifies the path in the docker container where files on

740

# this disk should be located. For example, if `mountPoint`

741

# is `/mnt/disk`, and the parameter has `localPath`

742

# `inputs/file.txt`, the docker container can access the data at

743

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

744

"type": "A String", # Required. The type of the disk to create.

745

},

746

],

747

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

748

"zones": [ # List of Google Compute Engine availability zones to which resource

749

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

750

"A String",

751

],

752

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

753

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

754

# By specifying this parameter, you will download and install the following

755

# third-party software onto your managed Compute Engine instances: NVIDIA®

756

# Tesla® drivers and NVIDIA® CUDA toolkit.

757

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

758

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

759

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

760

# must be true for both create time and run time. Cannot be true at run time

761

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

762

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

763

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

764

# This name can be used for filtering Pipelines in ListPipelines.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

769

<code class="details" id="delete">delete(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

770

<pre>Deletes a pipeline based on ID.

771

772

Caller must have WRITE permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

773

774

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

775

pipelineId: string, Caller must have WRITE access to the project in which this pipeline

776

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

777

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

778

Allowed values

779

1 - v1 error format

780

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

781

782

Returns:

783

An object of the form:

784

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

785

{ # A generic empty message that you can re-use to avoid defining duplicated

786

# empty messages in your APIs. A typical example is to use it as the request

787

# or the response type of an API method. For instance:

788

#

789

# service Foo {

790

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

791

# }

792

#

793

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

798

<code class="details" id="get">get(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

799

<pre>Retrieves a pipeline based on ID.

800

801

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

802

803

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

804

pipelineId: string, Caller must have READ access to the project in which this pipeline

805

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

806

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

807

Allowed values

808

1 - v1 error format

809

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

810

811

Returns:

812

An object of the form:

813

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

814

{ # The pipeline object. Represents a transformation from a set of input

815

# parameters to a set of output parameters. The transformation is defined

816

# as a docker image and command to run within that image. Each pipeline

817

# is run on a Google Compute Engine VM. A pipeline can be created with the

818

# `create` method and then later run with the `run` method, or a pipeline can

819

# be defined and run all at once with the `run` method.

820

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

821

# is called. Cannot be specified in the Pipeline used in the

822

# CreatePipelineRequest, and will be populated in the response to

823

# CreatePipeline and all subsequent Get and List calls. Indicates that the

824

# service has registered this pipeline.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

825

"description": "A String", # User-specified description.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

826

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

827

# WRITE access.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

828

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

829

{ # Parameters facilitate setting and delivering data into the

830

# pipeline's execution environment. They are defined at create time,

831

# with optional defaults, and can be overridden at run time.

832

#

833

# If `localCopy` is unset, then the parameter specifies a string that

834

# is passed as-is into the pipeline, as the value of the environment

835

# variable with the given name. A default value can be optionally

836

# specified at create time. The default can be overridden at run time

837

# using the inputs map. If no default is given, a value must be

838

# supplied at runtime.

839

#

840

# If `localCopy` is defined, then the parameter specifies a data

841

# source or sink, both in Google Cloud Storage and on the Docker container

842

# where the pipeline computation is run. The service account associated with

843

# the Pipeline (by

844

# default the project's Compute Engine service account) must have access to the

845

# Google Cloud Storage paths.

846

#

847

# At run time, the Google Cloud Storage paths can be overridden if a default

848

# was provided at create time, or must be set otherwise. The pipeline runner

849

# should add a key/value pair to either the inputs or outputs map. The

850

# indicated data copies will be carried out before/after pipeline execution,

851

# just as if the corresponding arguments were provided to `gsutil cp`.

852

#

853

# For example: Given the following `PipelineParameter`, specified

854

# in the `inputParameters` list:

855

#

856

# ```

857

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

858

# ```

859

#

860

# where `disk` is defined in the `PipelineResources` object as:

861

#

862

# ```

863

# {name: "pd1", mountPoint: "/mnt/disk/"}

864

# ```

865

#

866

# We create a disk named `pd1`, mount it on the host VM, and map

867

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

868

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

873

# ```

874

#

875

# This would generate the following gsutil call:

876

#

877

# ```

878

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

879

# ```

880

#

881

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

882

# Docker container. Acceptable paths are:

883

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

884

# <table>

885

# <thead>

886

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

887

# </thead>

888

# <tbody>

889

# <tr><td>file</td><td>file</td></tr>

890

# <tr><td>glob</td><td>directory</td></tr>

891

# </tbody>

892

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

893

#

894

# For outputs, the direction of the copy is reversed:

895

#

896

# ```

897

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

898

# ```

899

#

900

# Acceptable paths are:

901

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

902

# <table>

903

# <thead>

904

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

905

# </thead>

906

# <tbody>

907

# <tr><td>file</td><td>file</td></tr>

908

# <tr>

909

# <td>file</td>

910

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

915

# </tbody>

916

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

917

#

918

# One restriction due to docker limitations, is that for outputs that are found

919

# on the boot disk, the local path cannot be a glob and must be a file.

920

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

921

# If `localCopy` is present, then this must be a Google Cloud Storage path

922

# beginning with `gs://`.

923

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

924

# `LocalCopy` indicates where on the VM the file should be. The value

925

# given to this parameter (either at runtime or using `defaultValue`)

926

# must be the remote path where the file should be.

927

"path": "A String", # Required. The path within the user's docker container where

928

# this input should be localized to and from, relative to the specified

929

# disk's mount point. For example: file.txt,

930

"disk": "A String", # Required. The name of the disk where this parameter is

931

# located. Can be the name of one of the disks specified in the

932

# Resources field, or "boot", which represents the Docker

933

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

934

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

935

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

936

# as the key to the input and output maps in RunPipeline.

937

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

938

},

939

],

940

"inputParameters": [ # Input parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

941

{ # Parameters facilitate setting and delivering data into the

942

# pipeline's execution environment. They are defined at create time,

943

# with optional defaults, and can be overridden at run time.

944

#

945

# If `localCopy` is unset, then the parameter specifies a string that

946

# is passed as-is into the pipeline, as the value of the environment

947

# variable with the given name. A default value can be optionally

948

# specified at create time. The default can be overridden at run time

949

# using the inputs map. If no default is given, a value must be

950

# supplied at runtime.

951

#

952

# If `localCopy` is defined, then the parameter specifies a data

953

# source or sink, both in Google Cloud Storage and on the Docker container

954

# where the pipeline computation is run. The service account associated with

955

# the Pipeline (by

956

# default the project's Compute Engine service account) must have access to the

957

# Google Cloud Storage paths.

958

#

959

# At run time, the Google Cloud Storage paths can be overridden if a default

960

# was provided at create time, or must be set otherwise. The pipeline runner

961

# should add a key/value pair to either the inputs or outputs map. The

962

# indicated data copies will be carried out before/after pipeline execution,

963

# just as if the corresponding arguments were provided to `gsutil cp`.

964

#

965

# For example: Given the following `PipelineParameter`, specified

966

# in the `inputParameters` list:

967

#

968

# ```

969

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

970

# ```

971

#

972

# where `disk` is defined in the `PipelineResources` object as:

973

#

974

# ```

975

# {name: "pd1", mountPoint: "/mnt/disk/"}

976

# ```

977

#

978

# We create a disk named `pd1`, mount it on the host VM, and map

979

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

980

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

985

# ```

986

#

987

# This would generate the following gsutil call:

988

#

989

# ```

990

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

991

# ```

992

#

993

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

994

# Docker container. Acceptable paths are:

995

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

996

# <table>

997

# <thead>

998

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

999

# </thead>

1000

# <tbody>

1001

# <tr><td>file</td><td>file</td></tr>

1002

# <tr><td>glob</td><td>directory</td></tr>

1003

# </tbody>

1004

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1005

#

1006

# For outputs, the direction of the copy is reversed:

1007

#

1008

# ```

1009

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1010

# ```

1011

#

1012

# Acceptable paths are:

1013

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1014

# <table>

1015

# <thead>

1016

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1017

# </thead>

1018

# <tbody>

1019

# <tr><td>file</td><td>file</td></tr>

1020

# <tr>

1021

# <td>file</td>

1022

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

1027

# </tbody>

1028

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1029

#

1030

# One restriction due to docker limitations, is that for outputs that are found

1031

# on the boot disk, the local path cannot be a glob and must be a file.

1032

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1033

# If `localCopy` is present, then this must be a Google Cloud Storage path

1034

# beginning with `gs://`.

1035

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1036

# `LocalCopy` indicates where on the VM the file should be. The value

1037

# given to this parameter (either at runtime or using `defaultValue`)

1038

# must be the remote path where the file should be.

1039

"path": "A String", # Required. The path within the user's docker container where

1040

# this input should be localized to and from, relative to the specified

1041

# disk's mount point. For example: file.txt,

1042

"disk": "A String", # Required. The name of the disk where this parameter is

1043

# located. Can be the name of one of the disks specified in the

1044

# Resources field, or "boot", which represents the Docker

1045

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1046

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1047

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1048

# as the key to the input and output maps in RunPipeline.

1049

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1050

},

1051

],

1052

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1053

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1054

# string will be executed within a bash shell.

1055

#

1056

# If the command exits with a non-zero exit code, output parameter

1057

# de-localization will be skipped and the pipeline operation's

1058

# `error` field will be populated.

1059

#

1060

# Maximum command string length is 16384.

1061

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1062

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1063

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1064

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1073

# feature that may go away. Defaults to false.

1074

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1075

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1076

# In order to use this, must be true for both create time and run time.

1077

# Cannot be true at run time if false at create time. If you need to ssh into

1078

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1079

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1080

# only load docker images from Google Container Registry and not Docker Hub.

1081

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1082

# [configure access to Google services from internal

1083

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1084

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1085

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1086

# By specifying this parameter, you will download and install the following

1087

# third-party software onto your managed Compute Engine instances:

1088

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1089

"disks": [ # Disks to attach.

1090

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1091

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1092

# parameters. Must be 1 - 63 characters.

1093

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1094

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1095

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1096

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1097

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1098

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1099

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1100

# and

1101

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1102

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1103

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1104

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1105

# for more details.

1106

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1107

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1108

# Specifies the path in the docker container where files on

1109

# this disk should be located. For example, if `mountPoint`

1110

# is `/mnt/disk`, and the parameter has `localPath`

1111

# `inputs/file.txt`, the docker container can access the data at

1112

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1113

"type": "A String", # Required. The type of the disk to create.

1114

},

1115

],

1116

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1117

"zones": [ # List of Google Compute Engine availability zones to which resource

1118

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1119

"A String",

1120

],

1121

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1122

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1123

# By specifying this parameter, you will download and install the following

1124

# third-party software onto your managed Compute Engine instances: NVIDIA®

1125

# Tesla® drivers and NVIDIA® CUDA toolkit.

1126

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1127

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1128

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1129

# must be true for both create time and run time. Cannot be true at run time

1130

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1131

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1132

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1133

# This name can be used for filtering Pipelines in ListPipelines.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

<code class="details" id="getControllerConfig">getControllerConfig(validationToken=None, x__xgafv=None, operationId=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1139

<pre>Gets controller configuration information. Should only be called

1140

by VMs created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1141

1142

Args:

1143

validationToken: string, A parameter

1144

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1145

Allowed values

1146

1 - v1 error format

1147

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1148

operationId: string, The operation to retrieve controller configuration for.

1149

1150

Returns:

1151

An object of the form:

1152

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1153

{ # Stores the information that the controller will fetch from the

1154

# server in order to run. Should only be used by VMs created by the

1155

# Pipelines Service and not by end users.

Thomas Coffee

2f24537

2017-03-27 10:39:26 -0700

[diff] [blame]

1156

"cmd": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1157

"machineType": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1158

"vars": {

1159

"a_key": "A String",

1160

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1161

"gcsLogPath": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

"gcsSinks": {

"a_key": {

"values": [

"A String",

],

},

},

"image": "A String",

Thomas Coffee

2f24537

2017-03-27 10:39:26 -0700

[diff] [blame]

1170

"disks": {

1171

"a_key": "A String",

1172

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

"gcsSources": {

"a_key": {

"values": [

"A String",

],

},

},

}</pre>

</div>

<code class="details" id="list">list(pageSize=None, projectId=None, namePrefix=None, pageToken=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1185

<pre>Lists pipelines.

1186

1187

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1188

1189

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1190

pageSize: integer, Number of pipelines to return at once. Defaults to 256, and max

1191

is 2048.

1192

projectId: string, Required. The name of the project to search for pipelines. Caller

1193

must have READ access to this project.

1194

namePrefix: string, Pipelines with names that match this prefix should be

1195

returned. If unspecified, all pipelines in the project, up to

1196

`pageSize`, will be returned.

1197

pageToken: string, Token to use to indicate where to start getting results.

1198

If unspecified, returns the first page of results.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1199

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1200

Allowed values

1201

1 - v1 error format

1202

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1203

1204

Returns:

1205

An object of the form:

1206

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1207

{ # The response of ListPipelines. Contains at most `pageSize`

1208

# pipelines. If it contains `pageSize` pipelines, and more pipelines

1209

# exist, then `nextPageToken` will be populated and should be

1210

# used as the `pageToken` argument to a subsequent ListPipelines

1211

# request.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1212

"nextPageToken": "A String", # The token to use to get the next page of results.

1213

"pipelines": [ # The matched pipelines.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1214

{ # The pipeline object. Represents a transformation from a set of input

1215

# parameters to a set of output parameters. The transformation is defined

1216

# as a docker image and command to run within that image. Each pipeline

1217

# is run on a Google Compute Engine VM. A pipeline can be created with the

1218

# `create` method and then later run with the `run` method, or a pipeline can

1219

# be defined and run all at once with the `run` method.

1220

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1221

# is called. Cannot be specified in the Pipeline used in the

1222

# CreatePipelineRequest, and will be populated in the response to

1223

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1224

# service has registered this pipeline.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1225

"description": "A String", # User-specified description.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1226

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1227

# WRITE access.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1228

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1229

{ # Parameters facilitate setting and delivering data into the

1230

# pipeline's execution environment. They are defined at create time,

1231

# with optional defaults, and can be overridden at run time.

1232

#

1233

# If `localCopy` is unset, then the parameter specifies a string that

1234

# is passed as-is into the pipeline, as the value of the environment

1235

# variable with the given name. A default value can be optionally

1236

# specified at create time. The default can be overridden at run time

1237

# using the inputs map. If no default is given, a value must be

1238

# supplied at runtime.

1239

#

1240

# If `localCopy` is defined, then the parameter specifies a data

1241

# source or sink, both in Google Cloud Storage and on the Docker container

1242

# where the pipeline computation is run. The service account associated with

1243

# the Pipeline (by

1244

# default the project's Compute Engine service account) must have access to the

1245

# Google Cloud Storage paths.

1246

#

1247

# At run time, the Google Cloud Storage paths can be overridden if a default

1248

# was provided at create time, or must be set otherwise. The pipeline runner

1249

# should add a key/value pair to either the inputs or outputs map. The

1250

# indicated data copies will be carried out before/after pipeline execution,

1251

# just as if the corresponding arguments were provided to `gsutil cp`.

1252

#

1253

# For example: Given the following `PipelineParameter`, specified

1254

# in the `inputParameters` list:

1255

#

1256

# ```

1257

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1258

# ```

1259

#

1260

# where `disk` is defined in the `PipelineResources` object as:

1261

#

1262

# ```

1263

# {name: "pd1", mountPoint: "/mnt/disk/"}

1264

# ```

1265

#

1266

# We create a disk named `pd1`, mount it on the host VM, and map

1267

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1268

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1273

# ```

1274

#

1275

# This would generate the following gsutil call:

1276

#

1277

# ```

1278

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1279

# ```

1280

#

1281

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1282

# Docker container. Acceptable paths are:

1283

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1284

# <table>

1285

# <thead>

1286

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1287

# </thead>

1288

# <tbody>

1289

# <tr><td>file</td><td>file</td></tr>

1290

# <tr><td>glob</td><td>directory</td></tr>

1291

# </tbody>

1292

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1293

#

1294

# For outputs, the direction of the copy is reversed:

1295

#

1296

# ```

1297

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1298

# ```

1299

#

1300

# Acceptable paths are:

1301

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1302

# <table>

1303

# <thead>

1304

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1305

# </thead>

1306

# <tbody>

1307

# <tr><td>file</td><td>file</td></tr>

1308

# <tr>

1309

# <td>file</td>

1310

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

1315

# </tbody>

1316

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1317

#

1318

# One restriction due to docker limitations, is that for outputs that are found

1319

# on the boot disk, the local path cannot be a glob and must be a file.

1320

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1321

# If `localCopy` is present, then this must be a Google Cloud Storage path

1322

# beginning with `gs://`.

1323

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1324

# `LocalCopy` indicates where on the VM the file should be. The value

1325

# given to this parameter (either at runtime or using `defaultValue`)

1326

# must be the remote path where the file should be.

1327

"path": "A String", # Required. The path within the user's docker container where

1328

# this input should be localized to and from, relative to the specified

1329

# disk's mount point. For example: file.txt,

1330

"disk": "A String", # Required. The name of the disk where this parameter is

1331

# located. Can be the name of one of the disks specified in the

1332

# Resources field, or "boot", which represents the Docker

1333

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1334

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1335

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1336

# as the key to the input and output maps in RunPipeline.

1337

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1338

},

1339

],

1340

"inputParameters": [ # Input parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1341

{ # Parameters facilitate setting and delivering data into the

1342

# pipeline's execution environment. They are defined at create time,

1343

# with optional defaults, and can be overridden at run time.

1344

#

1345

# If `localCopy` is unset, then the parameter specifies a string that

1346

# is passed as-is into the pipeline, as the value of the environment

1347

# variable with the given name. A default value can be optionally

1348

# specified at create time. The default can be overridden at run time

1349

# using the inputs map. If no default is given, a value must be

1350

# supplied at runtime.

1351

#

1352

# If `localCopy` is defined, then the parameter specifies a data

1353

# source or sink, both in Google Cloud Storage and on the Docker container

1354

# where the pipeline computation is run. The service account associated with

1355

# the Pipeline (by

1356

# default the project's Compute Engine service account) must have access to the

1357

# Google Cloud Storage paths.

1358

#

1359

# At run time, the Google Cloud Storage paths can be overridden if a default

1360

# was provided at create time, or must be set otherwise. The pipeline runner

1361

# should add a key/value pair to either the inputs or outputs map. The

1362

# indicated data copies will be carried out before/after pipeline execution,

1363

# just as if the corresponding arguments were provided to `gsutil cp`.

1364

#

1365

# For example: Given the following `PipelineParameter`, specified

1366

# in the `inputParameters` list:

1367

#

1368

# ```

1369

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1370

# ```

1371

#

1372

# where `disk` is defined in the `PipelineResources` object as:

1373

#

1374

# ```

1375

# {name: "pd1", mountPoint: "/mnt/disk/"}

1376

# ```

1377

#

1378

# We create a disk named `pd1`, mount it on the host VM, and map

1379

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1380

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1385

# ```

1386

#

1387

# This would generate the following gsutil call:

1388

#

1389

# ```

1390

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1391

# ```

1392

#

1393

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1394

# Docker container. Acceptable paths are:

1395

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1396

# <table>

1397

# <thead>

1398

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1399

# </thead>

1400

# <tbody>

1401

# <tr><td>file</td><td>file</td></tr>

1402

# <tr><td>glob</td><td>directory</td></tr>

1403

# </tbody>

1404

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1405

#

1406

# For outputs, the direction of the copy is reversed:

1407

#

1408

# ```

1409

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1410

# ```

1411

#

1412

# Acceptable paths are:

1413

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1414

# <table>

1415

# <thead>

1416

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1417

# </thead>

1418

# <tbody>

1419

# <tr><td>file</td><td>file</td></tr>

1420

# <tr>

1421

# <td>file</td>

1422

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

1427

# </tbody>

1428

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1429

#

1430

# One restriction due to docker limitations, is that for outputs that are found

1431

# on the boot disk, the local path cannot be a glob and must be a file.

1432

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1433

# If `localCopy` is present, then this must be a Google Cloud Storage path

1434

# beginning with `gs://`.

1435

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1436

# `LocalCopy` indicates where on the VM the file should be. The value

1437

# given to this parameter (either at runtime or using `defaultValue`)

1438

# must be the remote path where the file should be.

1439

"path": "A String", # Required. The path within the user's docker container where

1440

# this input should be localized to and from, relative to the specified

1441

# disk's mount point. For example: file.txt,

1442

"disk": "A String", # Required. The name of the disk where this parameter is

1443

# located. Can be the name of one of the disks specified in the

1444

# Resources field, or "boot", which represents the Docker

1445

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1446

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1447

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1448

# as the key to the input and output maps in RunPipeline.

1449

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1450

},

1451

],

1452

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1453

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1454

# string will be executed within a bash shell.

1455

#

1456

# If the command exits with a non-zero exit code, output parameter

1457

# de-localization will be skipped and the pipeline operation's

1458

# `error` field will be populated.

1459

#

1460

# Maximum command string length is 16384.

1461

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1462

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1463

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1464

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1473

# feature that may go away. Defaults to false.

1474

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1475

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1476

# In order to use this, must be true for both create time and run time.

1477

# Cannot be true at run time if false at create time. If you need to ssh into

1478

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1479

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1480

# only load docker images from Google Container Registry and not Docker Hub.

1481

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1482

# [configure access to Google services from internal

1483

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1484

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1485

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1486

# By specifying this parameter, you will download and install the following

1487

# third-party software onto your managed Compute Engine instances:

1488

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1489

"disks": [ # Disks to attach.

1490

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1491

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1492

# parameters. Must be 1 - 63 characters.

1493

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1494

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1495

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1496

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1497

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1498

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1499

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1500

# and

1501

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1502

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1503

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1504

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1505

# for more details.

1506

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1507

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1508

# Specifies the path in the docker container where files on

1509

# this disk should be located. For example, if `mountPoint`

1510

# is `/mnt/disk`, and the parameter has `localPath`

1511

# `inputs/file.txt`, the docker container can access the data at

1512

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1513

"type": "A String", # Required. The type of the disk to create.

1514

},

1515

],

1516

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1517

"zones": [ # List of Google Compute Engine availability zones to which resource

1518

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1519

"A String",

1520

],

1521

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1522

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1523

# By specifying this parameter, you will download and install the following

1524

# third-party software onto your managed Compute Engine instances: NVIDIA®

1525

# Tesla® drivers and NVIDIA® CUDA toolkit.

1526

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1527

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1528

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1529

# must be true for both create time and run time. Cannot be true at run time

1530

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1531

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1532

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1533

# This name can be used for filtering Pipelines in ListPipelines.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

},

],

}</pre>

</div>

<code class="details" id="list_next">list_next(previous_request, previous_response)</code>

1541

<pre>Retrieves the next page of results.

1542

1543

Args:

1544

previous_request: The request for the previous page. (required)

1545

previous_response: The response from the request for the previous page. (required)

1546

1547

Returns:

1548

A request object that you can call 'execute()' on to request the next

1549

page. Returns None if there are no more items in the collection.

</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1554

<code class="details" id="run">run(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1555

<pre>Runs a pipeline. If `pipelineId` is specified in the request, then

1556

run a saved pipeline. If `ephemeralPipeline` is specified, then run

1557

that pipeline once without saving a copy.

1558

1559

The caller must have READ permission to the project where the pipeline

1560

is stored and WRITE permission to the project where the pipeline will be

1561

run, as VMs will be created and storage will be used.

1562

1563

If a pipeline operation is still running after 6 days, it will be canceled.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1564

1565

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1566

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1567

The object takes the form of:

1568

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1569

{ # The request to run a pipeline. If `pipelineId` is specified, it

1570

# refers to a saved pipeline created with CreatePipeline and set as

1571

# the `pipelineId` of the returned Pipeline object. If

1572

# `ephemeralPipeline` is specified, that pipeline is run once

1573

# with the given args and not saved. It is an error to specify both

1574

# `pipelineId` and `ephemeralPipeline`. `pipelineArgs`

1575

# must be specified.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1576

"pipelineArgs": { # The pipeline run arguments. # The arguments to use when running this pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1577

"inputs": { # Pipeline input arguments; keys are defined in the pipeline documentation.

1578

# All input parameters that do not have default values must be specified.

1579

# If parameters with defaults are specified here, the defaults will be

1580

# overridden.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1581

"a_key": "A String",

1582

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1583

"logging": { # The logging options for the pipeline run. # Required. Logging options. Used by the service to communicate results

1584

# to the user.

1585

"gcsPath": "A String", # The location in Google Cloud Storage to which the pipeline logs

1586

# will be copied. Can be specified as a fully qualified directory

1587

# path, in which case logs will be output with a unique identifier

1588

# as the filename in that directory, or as a fully specified path,

1589

# which must end in `.log`, in which case that path will be

1590

# used, and the user must ensure that logs are not

1591

# overwritten. Stdout and stderr logs from the run are also

1592

# generated and output as `-stdout.log` and `-stderr.log`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1593

},

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1594

"projectId": "A String", # Required. The project in which to run the pipeline. The caller must have

1595

# WRITER access to all Google Cloud services and resources (e.g. Google

1596

# Compute Engine) will be used.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1597

"labels": { # Labels to apply to this pipeline run. Labels will also be applied to

1598

# compute resources (VM, disks) created by this pipeline run. When listing

1599

# operations, operations can filtered by labels.

1600

# Label keys may not be empty; label values may be empty. Non-empty labels

1601

# must be 1-63 characters long, and comply with [RFC1035]

1602

# (https://www.ietf.org/rfc/rfc1035.txt).

1603

# Specifically, the name must be 1-63 characters long and match the regular

1604

# expression `[a-z]([-a-z0-9]*[a-z0-9])?` which means the first

1605

# character must be a lowercase letter, and all following characters must be

1606

# a dash, lowercase letter, or digit, except the last character, which cannot

# be a dash.

"a_key": "A String",

},

"serviceAccount": { # A Google Cloud Service Account. # The Google Cloud Service Account that will be used to access data and

1611

# services. By default, the compute service account associated with

1612

# `projectId` is used.

1613

"scopes": [ # List of scopes to be enabled for this service account on the VM.

1614

# The following scopes are automatically included:

1615

#

1616

# * https://www.googleapis.com/auth/compute

1617

# * https://www.googleapis.com/auth/devstorage.full_control

1618

# * https://www.googleapis.com/auth/genomics

1619

# * https://www.googleapis.com/auth/logging.write

1620

# * https://www.googleapis.com/auth/monitoring.write

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1621

"A String",

1622

],

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1623

"email": "A String", # Email address of the service account. Defaults to `default`,

1624

# which uses the compute service account associated with the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1625

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1626

"clientId": "A String", # This field is deprecated. Use `labels` instead. Client-specified pipeline

1627

# operation identifier.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1628

"outputs": { # Pipeline output arguments; keys are defined in the pipeline

1629

# documentation. All output parameters of without default values

1630

# must be specified. If parameters with defaults are specified

1631

# here, the defaults will be overridden.

1632

"a_key": "A String",

1633

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1634

"keepVmAliveOnFailureDuration": "A String", # How long to keep the VM up after a failure (for example docker command

1635

# failed, copying input or output files failed, etc). While the VM is up, one

1636

# can ssh into the VM to debug. Default is 0; maximum allowed value is 1 day.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1637

"resources": { # The system resources for the pipeline run. # Specifies resource requirements/overrides for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1638

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1639

# feature that may go away. Defaults to false.

1640

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1641

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1642

# In order to use this, must be true for both create time and run time.

1643

# Cannot be true at run time if false at create time. If you need to ssh into

1644

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1645

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1646

# only load docker images from Google Container Registry and not Docker Hub.

1647

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1648

# [configure access to Google services from internal

1649

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1650

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1651

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1652

# By specifying this parameter, you will download and install the following

1653

# third-party software onto your managed Compute Engine instances:

1654

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1655

"disks": [ # Disks to attach.

1656

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1657

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1658

# parameters. Must be 1 - 63 characters.

1659

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1660

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1661

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1662

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1663

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1664

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1665

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1666

# and

1667

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1668

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1669

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1670

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1671

# for more details.

1672

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1673

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1674

# Specifies the path in the docker container where files on

1675

# this disk should be located. For example, if `mountPoint`

1676

# is `/mnt/disk`, and the parameter has `localPath`

1677

# `inputs/file.txt`, the docker container can access the data at

1678

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1679

"type": "A String", # Required. The type of the disk to create.

1680

},

1681

],

1682

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1683

"zones": [ # List of Google Compute Engine availability zones to which resource

1684

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1685

"A String",

1686

],

1687

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1688

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1689

# By specifying this parameter, you will download and install the following

1690

# third-party software onto your managed Compute Engine instances: NVIDIA®

1691

# Tesla® drivers and NVIDIA® CUDA toolkit.

1692

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1693

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1694

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1695

# must be true for both create time and run time. Cannot be true at run time

1696

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1697

},

1698

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1699

"ephemeralPipeline": { # The pipeline object. Represents a transformation from a set of input # A new pipeline object to run once and then delete.

1700

# parameters to a set of output parameters. The transformation is defined

1701

# as a docker image and command to run within that image. Each pipeline

1702

# is run on a Google Compute Engine VM. A pipeline can be created with the

1703

# `create` method and then later run with the `run` method, or a pipeline can

1704

# be defined and run all at once with the `run` method.

1705

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1706

# is called. Cannot be specified in the Pipeline used in the

1707

# CreatePipelineRequest, and will be populated in the response to

1708

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1709

# service has registered this pipeline.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1710

"description": "A String", # User-specified description.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1711

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1712

# WRITE access.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1713

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1714

{ # Parameters facilitate setting and delivering data into the

1715

# pipeline's execution environment. They are defined at create time,

1716

# with optional defaults, and can be overridden at run time.

1717

#

1718

# If `localCopy` is unset, then the parameter specifies a string that

1719

# is passed as-is into the pipeline, as the value of the environment

1720

# variable with the given name. A default value can be optionally

1721

# specified at create time. The default can be overridden at run time

1722

# using the inputs map. If no default is given, a value must be

1723

# supplied at runtime.

1724

#

1725

# If `localCopy` is defined, then the parameter specifies a data

1726

# source or sink, both in Google Cloud Storage and on the Docker container

1727

# where the pipeline computation is run. The service account associated with

1728

# the Pipeline (by

1729

# default the project's Compute Engine service account) must have access to the

1730

# Google Cloud Storage paths.

1731

#

1732

# At run time, the Google Cloud Storage paths can be overridden if a default

1733

# was provided at create time, or must be set otherwise. The pipeline runner

1734

# should add a key/value pair to either the inputs or outputs map. The

1735

# indicated data copies will be carried out before/after pipeline execution,

1736

# just as if the corresponding arguments were provided to `gsutil cp`.

1737

#

1738

# For example: Given the following `PipelineParameter`, specified

1739

# in the `inputParameters` list:

1740

#

1741

# ```

1742

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1743

# ```

1744

#

1745

# where `disk` is defined in the `PipelineResources` object as:

1746

#

1747

# ```

1748

# {name: "pd1", mountPoint: "/mnt/disk/"}

1749

# ```

1750

#

1751

# We create a disk named `pd1`, mount it on the host VM, and map

1752

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1753

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1758

# ```

1759

#

1760

# This would generate the following gsutil call:

1761

#

1762

# ```

1763

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1764

# ```

1765

#

1766

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1767

# Docker container. Acceptable paths are:

1768

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1769

# <table>

1770

# <thead>

1771

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1772

# </thead>

1773

# <tbody>

1774

# <tr><td>file</td><td>file</td></tr>

1775

# <tr><td>glob</td><td>directory</td></tr>

1776

# </tbody>

1777

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1778

#

1779

# For outputs, the direction of the copy is reversed:

1780

#

1781

# ```

1782

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1783

# ```

1784

#

1785

# Acceptable paths are:

1786

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1787

# <table>

1788

# <thead>

1789

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1790

# </thead>

1791

# <tbody>

1792

# <tr><td>file</td><td>file</td></tr>

1793

# <tr>

1794

# <td>file</td>

1795

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

1800

# </tbody>

1801

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1802

#

1803

# One restriction due to docker limitations, is that for outputs that are found

1804

# on the boot disk, the local path cannot be a glob and must be a file.

1805

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1806

# If `localCopy` is present, then this must be a Google Cloud Storage path

1807

# beginning with `gs://`.

1808

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1809

# `LocalCopy` indicates where on the VM the file should be. The value

1810

# given to this parameter (either at runtime or using `defaultValue`)

1811

# must be the remote path where the file should be.

1812

"path": "A String", # Required. The path within the user's docker container where

1813

# this input should be localized to and from, relative to the specified

1814

# disk's mount point. For example: file.txt,

1815

"disk": "A String", # Required. The name of the disk where this parameter is

1816

# located. Can be the name of one of the disks specified in the

1817

# Resources field, or "boot", which represents the Docker

1818

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1819

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1820

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1821

# as the key to the input and output maps in RunPipeline.

1822

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1823

},

1824

],

1825

"inputParameters": [ # Input parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1826

{ # Parameters facilitate setting and delivering data into the

1827

# pipeline's execution environment. They are defined at create time,

1828

# with optional defaults, and can be overridden at run time.

1829

#

1830

# If `localCopy` is unset, then the parameter specifies a string that

1831

# is passed as-is into the pipeline, as the value of the environment

1832

# variable with the given name. A default value can be optionally

1833

# specified at create time. The default can be overridden at run time

1834

# using the inputs map. If no default is given, a value must be

1835

# supplied at runtime.

1836

#

1837

# If `localCopy` is defined, then the parameter specifies a data

1838

# source or sink, both in Google Cloud Storage and on the Docker container

1839

# where the pipeline computation is run. The service account associated with

1840

# the Pipeline (by

1841

# default the project's Compute Engine service account) must have access to the

1842

# Google Cloud Storage paths.

1843

#

1844

# At run time, the Google Cloud Storage paths can be overridden if a default

1845

# was provided at create time, or must be set otherwise. The pipeline runner

1846

# should add a key/value pair to either the inputs or outputs map. The

1847

# indicated data copies will be carried out before/after pipeline execution,

1848

# just as if the corresponding arguments were provided to `gsutil cp`.

1849

#

1850

# For example: Given the following `PipelineParameter`, specified

1851

# in the `inputParameters` list:

1852

#

1853

# ```

1854

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1855

# ```

1856

#

1857

# where `disk` is defined in the `PipelineResources` object as:

1858

#

1859

# ```

1860

# {name: "pd1", mountPoint: "/mnt/disk/"}

1861

# ```

1862

#

1863

# We create a disk named `pd1`, mount it on the host VM, and map

1864

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1865

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1870

# ```

1871

#

1872

# This would generate the following gsutil call:

1873

#

1874

# ```

1875

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1876

# ```

1877

#

1878

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1879

# Docker container. Acceptable paths are:

1880

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1881

# <table>

1882

# <thead>

1883

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1884

# </thead>

1885

# <tbody>

1886

# <tr><td>file</td><td>file</td></tr>

1887

# <tr><td>glob</td><td>directory</td></tr>

1888

# </tbody>

1889

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1890

#

1891

# For outputs, the direction of the copy is reversed:

1892

#

1893

# ```

1894

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1895

# ```

1896

#

1897

# Acceptable paths are:

1898

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1899

# <table>

1900

# <thead>

1901

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1902

# </thead>

1903

# <tbody>

1904

# <tr><td>file</td><td>file</td></tr>

1905

# <tr>

1906

# <td>file</td>

1907

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

1912

# </tbody>

1913

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1914

#

1915

# One restriction due to docker limitations, is that for outputs that are found

1916

# on the boot disk, the local path cannot be a glob and must be a file.

1917

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1918

# If `localCopy` is present, then this must be a Google Cloud Storage path

1919

# beginning with `gs://`.

1920

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1921

# `LocalCopy` indicates where on the VM the file should be. The value

1922

# given to this parameter (either at runtime or using `defaultValue`)

1923

# must be the remote path where the file should be.

1924

"path": "A String", # Required. The path within the user's docker container where

1925

# this input should be localized to and from, relative to the specified

1926

# disk's mount point. For example: file.txt,

1927

"disk": "A String", # Required. The name of the disk where this parameter is

1928

# located. Can be the name of one of the disks specified in the

1929

# Resources field, or "boot", which represents the Docker

1930

# instance's boot disk and has a mount point of `/`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1931

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1932

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1933

# as the key to the input and output maps in RunPipeline.

1934

"description": "A String", # Human-readable description.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1935

},

1936

],

1937

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1938

"cmd": "A String", # Required. The command or newline delimited script to run. The command

1939

# string will be executed within a bash shell.

1940

#

1941

# If the command exits with a non-zero exit code, output parameter

1942

# de-localization will be skipped and the pipeline operation's

1943

# `error` field will be populated.

1944

#

1945

# Maximum command string length is 16384.

1946

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1947

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1948

},

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1949

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

1958

# feature that may go away. Defaults to false.

1959

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1960

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1961

# In order to use this, must be true for both create time and run time.

1962

# Cannot be true at run time if false at create time. If you need to ssh into

1963

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

1964

# the private VM's Internal IP. If noAddress is set, this pipeline run may

1965

# only load docker images from Google Container Registry and not Docker Hub.

1966

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1967

# [configure access to Google services from internal

1968

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1969

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

1970

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1971

# By specifying this parameter, you will download and install the following

1972

# third-party software onto your managed Compute Engine instances:

1973

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1974

"disks": [ # Disks to attach.

1975

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1976

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1977

# parameters. Must be 1 - 63 characters.

1978

# The name "boot" is reserved for system use.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1979

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1980

# This field is not applicable for local SSD.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1981

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1982

# of the pipeline run, regardless of what this field is set to.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1983

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1984

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1985

# and

1986

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1987

# for more details.

Sai Cheemalapati

2017-06-06 18:46:08 -0400

[diff] [blame]

1988

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1989

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1990

# for more details.

1991

# Can only be set at create time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1992

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

1993

# Specifies the path in the docker container where files on

1994

# this disk should be located. For example, if `mountPoint`

1995

# is `/mnt/disk`, and the parameter has `localPath`

1996

# `inputs/file.txt`, the docker container can access the data at

1997

# `/mnt/disk/inputs/file.txt`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1998

"type": "A String", # Required. The type of the disk to create.

1999

},

2000

],

2001

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2002

"zones": [ # List of Google Compute Engine availability zones to which resource

2003

# creation will restricted. If empty, any zone may be chosen.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2004

"A String",

2005

],

2006

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2007

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

2008

# By specifying this parameter, you will download and install the following

2009

# third-party software onto your managed Compute Engine instances: NVIDIA®

2010

# Tesla® drivers and NVIDIA® CUDA toolkit.

2011

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

2012

# available accelerator types.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2013

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

2014

# must be true for both create time and run time. Cannot be true at run time

2015

# if false at create time.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2016

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2017

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

2018

# This name can be used for filtering Pipelines in ListPipelines.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2019

},

2020

"pipelineId": "A String", # The already created pipeline to run.

2021

}

2022

2023

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2024

Allowed values

2025

1 - v1 error format

2026

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2027

2028

Returns:

2029

An object of the form:

2030

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2031

{ # This resource represents a long-running operation that is the result of a

2032

# network API call.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2033

"metadata": { # An OperationMetadata or Metadata object. This will always be returned with the Operation.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2034

"a_key": "", # Properties of the object. Contains field @type with type URL.

2035

},

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2036

"error": { # The `Status` type defines a logical error model that is suitable for # The error result of the operation in case of failure or cancellation.

2037

# different programming environments, including REST APIs and RPC APIs. It is

2038

# used by [gRPC](https://github.com/grpc). Each `Status` message contains

2039

# three pieces of data: error code, error message, and error details.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2040

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2041

# You can find out more about this error model and how to work with it in the

2042

# [API Design Guide](https://cloud.google.com/apis/design/errors).

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2043

"message": "A String", # A developer-facing error message, which should be in English. Any

2044

# user-facing error message should be localized and sent in the

2045

# google.rpc.Status.details field, or localized by the client.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2046

"code": 42, # The status code, which should be an enum value of google.rpc.Code.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2047

"details": [ # A list of messages that carry the error details. There is a common set of

2048

# message types for APIs to use.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2049

{

2050

"a_key": "", # Properties of the object. Contains field @type with type URL.

2051

},

2052

],

2053

},

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2054

"done": True or False, # If the value is `false`, it means the operation is still in progress.

2055

# If `true`, the operation is completed, and either `error` or `response` is

2056

# available.

2057

"response": { # An Empty object.

2058

"a_key": "", # Properties of the object. Contains field @type with type URL.

2059

},

2060

"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. For example&#58; `operations/CJHU7Oi_ChDrveSpBRjfuL-qzoWAgEw`

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2065

<code class="details" id="setOperationStatus">setOperationStatus(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2066

<pre>Sets status of a given operation. Any new timestamps (as determined by

2067

description) are appended to TimestampEvents. Should only be called by VMs

2068

created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2069

2070

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2071

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2072

The object takes the form of:

2073

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2074

{ # Request to set operation status. Should only be used by VMs

2075

# created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2076

"errorCode": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2077

"timestampEvents": [

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2078

{ # Stores the list of events and times they occured for major events in job

2079

# execution.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2080

"timestamp": "A String", # The time this event occured.

2081

"description": "A String", # String indicating the type of event

2082

},

2083

],

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2084

"errorMessage": "A String",

Thomas Coffee

2f24537

2017-03-27 10:39:26 -0700

[diff] [blame]

2085

"validationToken": "A String",

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame^]

2086

"operationId": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2087

}

2088

2089

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2090

Allowed values

2091

1 - v1 error format

2092

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2093

2094

Returns:

2095

An object of the form:

2096

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2097

{ # A generic empty message that you can re-use to avoid defining duplicated

2098

# empty messages in your APIs. A typical example is to use it as the request

2099

# or the response type of an API method. For instance:

2100

#

2101

# service Foo {

2102

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

2103

# }

2104

#

2105

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati