Blame - docs/dyn/genomics_v1alpha2.pipelines.html - platform/external/python/google-api-python-client

2017-03-13 12:12:03 -0400

[diff] [blame]

229

# `LocalCopy` indicates where on the VM the file should be. The value

230

# given to this parameter (either at runtime or using `defaultValue`)

231

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

232

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

233

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

234

# Resources field, or "boot", which represents the Docker

235

# instance's boot disk and has a mount point of `/`.

236

"path": "A String", # Required. The path within the user's docker container where

237

# this input should be localized to and from, relative to the specified

238

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

239

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

240

},

241

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

242

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

243

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

244

# string will be executed within a bash shell.

245

#

246

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

247

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

248

# `error` field will be populated.

249

#

250

# Maximum command string length is 16384.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

251

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

252

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

253

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

254

"description": "A String", # User-specified description.

255

"inputParameters": [ # Input parameters of the pipeline.

256

{ # Parameters facilitate setting and delivering data into the

257

# pipeline's execution environment. They are defined at create time,

258

# with optional defaults, and can be overridden at run time.

259

#

260

# If `localCopy` is unset, then the parameter specifies a string that

261

# is passed as-is into the pipeline, as the value of the environment

262

# variable with the given name. A default value can be optionally

263

# specified at create time. The default can be overridden at run time

264

# using the inputs map. If no default is given, a value must be

265

# supplied at runtime.

266

#

267

# If `localCopy` is defined, then the parameter specifies a data

268

# source or sink, both in Google Cloud Storage and on the Docker container

269

# where the pipeline computation is run. The service account associated with

270

# the Pipeline (by

271

# default the project's Compute Engine service account) must have access to the

272

# Google Cloud Storage paths.

273

#

274

# At run time, the Google Cloud Storage paths can be overridden if a default

275

# was provided at create time, or must be set otherwise. The pipeline runner

276

# should add a key/value pair to either the inputs or outputs map. The

277

# indicated data copies will be carried out before/after pipeline execution,

278

# just as if the corresponding arguments were provided to `gsutil cp`.

279

#

280

# For example: Given the following `PipelineParameter`, specified

281

# in the `inputParameters` list:

282

#

283

# ```

284

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

285

# ```

286

#

287

# where `disk` is defined in the `PipelineResources` object as:

288

#

289

# ```

290

# {name: "pd1", mountPoint: "/mnt/disk/"}

291

# ```

292

#

293

# We create a disk named `pd1`, mount it on the host VM, and map

294

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

295

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

300

# ```

301

#

302

# This would generate the following gsutil call:

303

#

304

# ```

305

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

306

# ```

307

#

308

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

309

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

314

# </thead>

315

# <tbody>

316

# <tr><td>file</td><td>file</td></tr>

317

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

322

#

323

# ```

324

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

325

# ```

326

#

327

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

332

# </thead>

333

# <tbody>

334

# <tr><td>file</td><td>file</td></tr>

335

# <tr>

336

# <td>file</td>

337

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

346

# on the boot disk, the local path cannot be a glob and must be a file.

347

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

348

# If `localCopy` is present, then this must be a Google Cloud Storage path

349

# beginning with `gs://`.

350

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

351

# as the key to the input and output maps in RunPipeline.

352

"description": "A String", # Human-readable description.

353

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

354

# `LocalCopy` indicates where on the VM the file should be. The value

355

# given to this parameter (either at runtime or using `defaultValue`)

356

# must be the remote path where the file should be.

357

"disk": "A String", # Required. The name of the disk where this parameter is

358

# located. Can be the name of one of the disks specified in the

359

# Resources field, or "boot", which represents the Docker

360

# instance's boot disk and has a mount point of `/`.

361

"path": "A String", # Required. The path within the user's docker container where

362

# this input should be localized to and from, relative to the specified

363

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

375

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

376

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

377

# must be true for both create time and run time. Cannot be true at run time

378

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

379

"zones": [ # List of Google Compute Engine availability zones to which resource

380

# creation will restricted. If empty, any zone may be chosen.

381

"A String",

382

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

383

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

384

# By specifying this parameter, you will download and install the following

385

# third-party software onto your managed Compute Engine instances:

386

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

387

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

388

# By specifying this parameter, you will download and install the following

389

# third-party software onto your managed Compute Engine instances: NVIDIA®

390

# Tesla® drivers and NVIDIA® CUDA toolkit.

391

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

392

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

393

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

394

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

395

# feature that may go away. Defaults to false.

396

# Corresponds to `--no_address` flag for [gcloud compute instances create]

397

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

398

# In order to use this, must be true for both create time and run time.

399

# Cannot be true at run time if false at create time. If you need to ssh into

400

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

401

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

402

# only load docker images from Google Container Registry and not Docker Hub.

403

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

404

# [configure access to Google services from internal

405

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

406

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

407

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

408

"name": "A String", # Required. The name of the disk that can be used in the pipeline

409

# parameters. Must be 1 - 63 characters.

410

# The name "boot" is reserved for system use.

411

"type": "A String", # Required. The type of the disk to create.

412

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

413

# of the pipeline run, regardless of what this field is set to.

414

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

415

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

416

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

417

# Specifies the path in the docker container where files on

418

# this disk should be located. For example, if `mountPoint`

419

# is `/mnt/disk`, and the parameter has `localPath`

420

# `inputs/file.txt`, the docker container can access the data at

421

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

422

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

423

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

424

# for more details.

425

# Can only be set at create time.

426

"source": "A String", # The full or partial URL of the persistent disk to attach. See

427

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

428

# and

429

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

430

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

431

},

432

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

433

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

434

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

435

}

436

437

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

438

Allowed values

439

1 - v1 error format

440

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

441

442

Returns:

443

An object of the form:

444

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

445

{ # The pipeline object. Represents a transformation from a set of input

446

# parameters to a set of output parameters. The transformation is defined

447

# as a docker image and command to run within that image. Each pipeline

448

# is run on a Google Compute Engine VM. A pipeline can be created with the

449

# `create` method and then later run with the `run` method, or a pipeline can

450

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

451

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

452

# This name can be used for filtering Pipelines in ListPipelines.

453

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

454

# is called. Cannot be specified in the Pipeline used in the

455

# CreatePipelineRequest, and will be populated in the response to

456

# CreatePipeline and all subsequent Get and List calls. Indicates that the

457

# service has registered this pipeline.

458

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

459

# WRITE access.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

460

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

461

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

462

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

463

# with optional defaults, and can be overridden at run time.

464

#

465

# If `localCopy` is unset, then the parameter specifies a string that

466

# is passed as-is into the pipeline, as the value of the environment

467

# variable with the given name. A default value can be optionally

468

# specified at create time. The default can be overridden at run time

469

# using the inputs map. If no default is given, a value must be

470

# supplied at runtime.

471

#

472

# If `localCopy` is defined, then the parameter specifies a data

473

# source or sink, both in Google Cloud Storage and on the Docker container

474

# where the pipeline computation is run. The service account associated with

475

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

476

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

477

# Google Cloud Storage paths.

478

#

479

# At run time, the Google Cloud Storage paths can be overridden if a default

480

# was provided at create time, or must be set otherwise. The pipeline runner

481

# should add a key/value pair to either the inputs or outputs map. The

482

# indicated data copies will be carried out before/after pipeline execution,

483

# just as if the corresponding arguments were provided to `gsutil cp`.

484

#

485

# For example: Given the following `PipelineParameter`, specified

486

# in the `inputParameters` list:

487

#

488

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

489

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

490

# ```

491

#

492

# where `disk` is defined in the `PipelineResources` object as:

493

#

494

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

495

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

496

# ```

497

#

498

# We create a disk named `pd1`, mount it on the host VM, and map

499

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

500

# runtime, an entry for `input_file` would be required in the inputs

501

# map, such as:

502

#

503

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

504

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

505

# ```

506

#

507

# This would generate the following gsutil call:

508

#

509

# ```

510

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

511

# ```

512

#

513

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

514

# Docker container. Acceptable paths are:

515

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

516

# <table>

517

# <thead>

518

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

519

# </thead>

520

# <tbody>

521

# <tr><td>file</td><td>file</td></tr>

522

# <tr><td>glob</td><td>directory</td></tr>

523

# </tbody>

524

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

525

#

526

# For outputs, the direction of the copy is reversed:

527

#

528

# ```

529

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

530

# ```

531

#

532

# Acceptable paths are:

533

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

534

# <table>

535

# <thead>

536

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

537

# </thead>

538

# <tbody>

539

# <tr><td>file</td><td>file</td></tr>

540

# <tr>

541

# <td>file</td>

542

# <td>directory - directory must already exist</td>

543

# </tr>

544

# <tr>

545

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

546

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

547

# </tbody>

548

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

549

#

550

# One restriction due to docker limitations, is that for outputs that are found

551

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

552

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

553

# If `localCopy` is present, then this must be a Google Cloud Storage path

554

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

555

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

556

# as the key to the input and output maps in RunPipeline.

557

"description": "A String", # Human-readable description.

558

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

559

# `LocalCopy` indicates where on the VM the file should be. The value

560

# given to this parameter (either at runtime or using `defaultValue`)

561

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

562

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

563

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

564

# Resources field, or "boot", which represents the Docker

565

# instance's boot disk and has a mount point of `/`.

566

"path": "A String", # Required. The path within the user's docker container where

567

# this input should be localized to and from, relative to the specified

568

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

569

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

570

},

571

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

572

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

573

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

574

# string will be executed within a bash shell.

575

#

576

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

577

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

578

# `error` field will be populated.

579

#

580

# Maximum command string length is 16384.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

581

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

582

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

583

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

584

"description": "A String", # User-specified description.

585

"inputParameters": [ # Input parameters of the pipeline.

586

{ # Parameters facilitate setting and delivering data into the

587

# pipeline's execution environment. They are defined at create time,

588

# with optional defaults, and can be overridden at run time.

589

#

590

# If `localCopy` is unset, then the parameter specifies a string that

591

# is passed as-is into the pipeline, as the value of the environment

592

# variable with the given name. A default value can be optionally

593

# specified at create time. The default can be overridden at run time

594

# using the inputs map. If no default is given, a value must be

595

# supplied at runtime.

596

#

597

# If `localCopy` is defined, then the parameter specifies a data

598

# source or sink, both in Google Cloud Storage and on the Docker container

599

# where the pipeline computation is run. The service account associated with

600

# the Pipeline (by

601

# default the project's Compute Engine service account) must have access to the

602

# Google Cloud Storage paths.

603

#

604

# At run time, the Google Cloud Storage paths can be overridden if a default

605

# was provided at create time, or must be set otherwise. The pipeline runner

606

# should add a key/value pair to either the inputs or outputs map. The

607

# indicated data copies will be carried out before/after pipeline execution,

608

# just as if the corresponding arguments were provided to `gsutil cp`.

609

#

610

# For example: Given the following `PipelineParameter`, specified

611

# in the `inputParameters` list:

612

#

613

# ```

614

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

615

# ```

616

#

617

# where `disk` is defined in the `PipelineResources` object as:

618

#

619

# ```

620

# {name: "pd1", mountPoint: "/mnt/disk/"}

621

# ```

622

#

623

# We create a disk named `pd1`, mount it on the host VM, and map

624

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

625

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

630

# ```

631

#

632

# This would generate the following gsutil call:

633

#

634

# ```

635

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

636

# ```

637

#

638

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

639

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

644

# </thead>

645

# <tbody>

646

# <tr><td>file</td><td>file</td></tr>

647

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

652

#

653

# ```

654

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

655

# ```

656

#

657

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

662

# </thead>

663

# <tbody>

664

# <tr><td>file</td><td>file</td></tr>

665

# <tr>

666

# <td>file</td>

667

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

676

# on the boot disk, the local path cannot be a glob and must be a file.

677

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

678

# If `localCopy` is present, then this must be a Google Cloud Storage path

679

# beginning with `gs://`.

680

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

681

# as the key to the input and output maps in RunPipeline.

682

"description": "A String", # Human-readable description.

683

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

684

# `LocalCopy` indicates where on the VM the file should be. The value

685

# given to this parameter (either at runtime or using `defaultValue`)

686

# must be the remote path where the file should be.

687

"disk": "A String", # Required. The name of the disk where this parameter is

688

# located. Can be the name of one of the disks specified in the

689

# Resources field, or "boot", which represents the Docker

690

# instance's boot disk and has a mount point of `/`.

691

"path": "A String", # Required. The path within the user's docker container where

692

# this input should be localized to and from, relative to the specified

693

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

705

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

706

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

707

# must be true for both create time and run time. Cannot be true at run time

708

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

709

"zones": [ # List of Google Compute Engine availability zones to which resource

710

# creation will restricted. If empty, any zone may be chosen.

711

"A String",

712

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

713

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

714

# By specifying this parameter, you will download and install the following

715

# third-party software onto your managed Compute Engine instances:

716

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

717

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

718

# By specifying this parameter, you will download and install the following

719

# third-party software onto your managed Compute Engine instances: NVIDIA®

720

# Tesla® drivers and NVIDIA® CUDA toolkit.

721

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

722

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

723

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

724

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

725

# feature that may go away. Defaults to false.

726

# Corresponds to `--no_address` flag for [gcloud compute instances create]

727

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

728

# In order to use this, must be true for both create time and run time.

729

# Cannot be true at run time if false at create time. If you need to ssh into

730

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

731

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

732

# only load docker images from Google Container Registry and not Docker Hub.

733

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

734

# [configure access to Google services from internal

735

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

736

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

737

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

738

"name": "A String", # Required. The name of the disk that can be used in the pipeline

739

# parameters. Must be 1 - 63 characters.

740

# The name "boot" is reserved for system use.

741

"type": "A String", # Required. The type of the disk to create.

742

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

743

# of the pipeline run, regardless of what this field is set to.

744

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

745

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

746

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

747

# Specifies the path in the docker container where files on

748

# this disk should be located. For example, if `mountPoint`

749

# is `/mnt/disk`, and the parameter has `localPath`

750

# `inputs/file.txt`, the docker container can access the data at

751

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

752

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

753

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

754

# for more details.

755

# Can only be set at create time.

756

"source": "A String", # The full or partial URL of the persistent disk to attach. See

757

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

758

# and

759

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

760

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

761

},

762

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

763

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

764

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

769

<code class="details" id="delete">delete(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

770

<pre>Deletes a pipeline based on ID.

771

772

Caller must have WRITE permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

773

774

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

775

pipelineId: string, Caller must have WRITE access to the project in which this pipeline

776

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

777

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

778

Allowed values

779

1 - v1 error format

780

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

781

782

Returns:

783

An object of the form:

784

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

785

{ # A generic empty message that you can re-use to avoid defining duplicated

786

# empty messages in your APIs. A typical example is to use it as the request

787

# or the response type of an API method. For instance:

788

#

789

# service Foo {

790

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

791

# }

792

#

793

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Thomas Coffee

2017-03-27 10:39:26 -0700

[diff] [blame]

798

<code class="details" id="get">get(pipelineId, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

799

<pre>Retrieves a pipeline based on ID.

800

801

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

802

803

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

804

pipelineId: string, Caller must have READ access to the project in which this pipeline

805

is defined. (required)

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

806

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

807

Allowed values

808

1 - v1 error format

809

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

810

811

Returns:

812

An object of the form:

813

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

814

{ # The pipeline object. Represents a transformation from a set of input

815

# parameters to a set of output parameters. The transformation is defined

816

# as a docker image and command to run within that image. Each pipeline

817

# is run on a Google Compute Engine VM. A pipeline can be created with the

818

# `create` method and then later run with the `run` method, or a pipeline can

819

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

820

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

821

# This name can be used for filtering Pipelines in ListPipelines.

822

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

823

# is called. Cannot be specified in the Pipeline used in the

824

# CreatePipelineRequest, and will be populated in the response to

825

# CreatePipeline and all subsequent Get and List calls. Indicates that the

826

# service has registered this pipeline.

827

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

828

# WRITE access.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

829

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

830

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

831

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

832

# with optional defaults, and can be overridden at run time.

833

#

834

# If `localCopy` is unset, then the parameter specifies a string that

835

# is passed as-is into the pipeline, as the value of the environment

836

# variable with the given name. A default value can be optionally

837

# specified at create time. The default can be overridden at run time

838

# using the inputs map. If no default is given, a value must be

839

# supplied at runtime.

840

#

841

# If `localCopy` is defined, then the parameter specifies a data

842

# source or sink, both in Google Cloud Storage and on the Docker container

843

# where the pipeline computation is run. The service account associated with

844

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

845

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

846

# Google Cloud Storage paths.

847

#

848

# At run time, the Google Cloud Storage paths can be overridden if a default

849

# was provided at create time, or must be set otherwise. The pipeline runner

850

# should add a key/value pair to either the inputs or outputs map. The

851

# indicated data copies will be carried out before/after pipeline execution,

852

# just as if the corresponding arguments were provided to `gsutil cp`.

853

#

854

# For example: Given the following `PipelineParameter`, specified

855

# in the `inputParameters` list:

856

#

857

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

858

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

859

# ```

860

#

861

# where `disk` is defined in the `PipelineResources` object as:

862

#

863

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

864

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

865

# ```

866

#

867

# We create a disk named `pd1`, mount it on the host VM, and map

868

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

869

# runtime, an entry for `input_file` would be required in the inputs

870

# map, such as:

871

#

872

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

873

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

874

# ```

875

#

876

# This would generate the following gsutil call:

877

#

878

# ```

879

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

880

# ```

881

#

882

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

883

# Docker container. Acceptable paths are:

884

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

885

# <table>

886

# <thead>

887

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

888

# </thead>

889

# <tbody>

890

# <tr><td>file</td><td>file</td></tr>

891

# <tr><td>glob</td><td>directory</td></tr>

892

# </tbody>

893

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

894

#

895

# For outputs, the direction of the copy is reversed:

896

#

897

# ```

898

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

899

# ```

900

#

901

# Acceptable paths are:

902

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

903

# <table>

904

# <thead>

905

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

906

# </thead>

907

# <tbody>

908

# <tr><td>file</td><td>file</td></tr>

909

# <tr>

910

# <td>file</td>

911

# <td>directory - directory must already exist</td>

912

# </tr>

913

# <tr>

914

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

915

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

916

# </tbody>

917

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

918

#

919

# One restriction due to docker limitations, is that for outputs that are found

920

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

921

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

922

# If `localCopy` is present, then this must be a Google Cloud Storage path

923

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

924

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

925

# as the key to the input and output maps in RunPipeline.

926

"description": "A String", # Human-readable description.

927

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

928

# `LocalCopy` indicates where on the VM the file should be. The value

929

# given to this parameter (either at runtime or using `defaultValue`)

930

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

931

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

932

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

933

# Resources field, or "boot", which represents the Docker

934

# instance's boot disk and has a mount point of `/`.

935

"path": "A String", # Required. The path within the user's docker container where

936

# this input should be localized to and from, relative to the specified

937

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

938

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

939

},

940

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

941

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

942

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

943

# string will be executed within a bash shell.

944

#

945

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

946

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

947

# `error` field will be populated.

948

#

949

# Maximum command string length is 16384.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

950

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

951

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

952

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

953

"description": "A String", # User-specified description.

954

"inputParameters": [ # Input parameters of the pipeline.

955

{ # Parameters facilitate setting and delivering data into the

956

# pipeline's execution environment. They are defined at create time,

957

# with optional defaults, and can be overridden at run time.

958

#

959

# If `localCopy` is unset, then the parameter specifies a string that

960

# is passed as-is into the pipeline, as the value of the environment

961

# variable with the given name. A default value can be optionally

962

# specified at create time. The default can be overridden at run time

963

# using the inputs map. If no default is given, a value must be

964

# supplied at runtime.

965

#

966

# If `localCopy` is defined, then the parameter specifies a data

967

# source or sink, both in Google Cloud Storage and on the Docker container

968

# where the pipeline computation is run. The service account associated with

969

# the Pipeline (by

970

# default the project's Compute Engine service account) must have access to the

971

# Google Cloud Storage paths.

972

#

973

# At run time, the Google Cloud Storage paths can be overridden if a default

974

# was provided at create time, or must be set otherwise. The pipeline runner

975

# should add a key/value pair to either the inputs or outputs map. The

976

# indicated data copies will be carried out before/after pipeline execution,

977

# just as if the corresponding arguments were provided to `gsutil cp`.

978

#

979

# For example: Given the following `PipelineParameter`, specified

980

# in the `inputParameters` list:

981

#

982

# ```

983

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

984

# ```

985

#

986

# where `disk` is defined in the `PipelineResources` object as:

987

#

988

# ```

989

# {name: "pd1", mountPoint: "/mnt/disk/"}

990

# ```

991

#

992

# We create a disk named `pd1`, mount it on the host VM, and map

993

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

994

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

999

# ```

1000

#

1001

# This would generate the following gsutil call:

1002

#

1003

# ```

1004

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1005

# ```

1006

#

1007

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1008

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1013

# </thead>

1014

# <tbody>

1015

# <tr><td>file</td><td>file</td></tr>

1016

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1021

#

1022

# ```

1023

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1024

# ```

1025

#

1026

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1031

# </thead>

1032

# <tbody>

1033

# <tr><td>file</td><td>file</td></tr>

1034

# <tr>

1035

# <td>file</td>

1036

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1045

# on the boot disk, the local path cannot be a glob and must be a file.

1046

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1047

# If `localCopy` is present, then this must be a Google Cloud Storage path

1048

# beginning with `gs://`.

1049

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1050

# as the key to the input and output maps in RunPipeline.

1051

"description": "A String", # Human-readable description.

1052

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1053

# `LocalCopy` indicates where on the VM the file should be. The value

1054

# given to this parameter (either at runtime or using `defaultValue`)

1055

# must be the remote path where the file should be.

1056

"disk": "A String", # Required. The name of the disk where this parameter is

1057

# located. Can be the name of one of the disks specified in the

1058

# Resources field, or "boot", which represents the Docker

1059

# instance's boot disk and has a mount point of `/`.

1060

"path": "A String", # Required. The path within the user's docker container where

1061

# this input should be localized to and from, relative to the specified

1062

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1074

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1075

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1076

# must be true for both create time and run time. Cannot be true at run time

1077

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1078

"zones": [ # List of Google Compute Engine availability zones to which resource

1079

# creation will restricted. If empty, any zone may be chosen.

1080

"A String",

1081

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1082

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1083

# By specifying this parameter, you will download and install the following

1084

# third-party software onto your managed Compute Engine instances:

1085

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1086

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1087

# By specifying this parameter, you will download and install the following

1088

# third-party software onto your managed Compute Engine instances: NVIDIA®

1089

# Tesla® drivers and NVIDIA® CUDA toolkit.

1090

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1091

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1092

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1093

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1094

# feature that may go away. Defaults to false.

1095

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1096

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1097

# In order to use this, must be true for both create time and run time.

1098

# Cannot be true at run time if false at create time. If you need to ssh into

1099

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1100

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1101

# only load docker images from Google Container Registry and not Docker Hub.

1102

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1103

# [configure access to Google services from internal

1104

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1105

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1106

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1107

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1108

# parameters. Must be 1 - 63 characters.

1109

# The name "boot" is reserved for system use.

1110

"type": "A String", # Required. The type of the disk to create.

1111

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1112

# of the pipeline run, regardless of what this field is set to.

1113

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1114

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1115

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1116

# Specifies the path in the docker container where files on

1117

# this disk should be located. For example, if `mountPoint`

1118

# is `/mnt/disk`, and the parameter has `localPath`

1119

# `inputs/file.txt`, the docker container can access the data at

1120

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1121

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1122

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1123

# for more details.

1124

# Can only be set at create time.

1125

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1126

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1127

# and

1128

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1129

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1130

},

1131

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1132

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1133

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1138

<code class="details" id="getControllerConfig">getControllerConfig(operationId=None, validationToken=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1139

<pre>Gets controller configuration information. Should only be called

1140

by VMs created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1141

1142

Args:

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1143

operationId: string, The operation to retrieve controller configuration for.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1144

validationToken: string, A parameter

1145

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1146

Allowed values

1147

1 - v1 error format

1148

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1149

1150

Returns:

1151

An object of the form:

1152

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1153

{ # Stores the information that the controller will fetch from the

1154

# server in order to run. Should only be used by VMs created by the

1155

# Pipelines Service and not by end users.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1156

"machineType": "A String",

1157

"cmd": "A String",

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1158

"vars": {

1159

"a_key": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1160

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1161

"image": "A String",

1162

"gcsLogPath": "A String",

"gcsSources": {

"a_key": {

"values": [

"A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1167

],

1168

},

1169

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

"gcsSinks": {

"a_key": {

"values": [

"A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1174

],

1175

},

1176

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1177

"disks": {

1178

"a_key": "A String",

1179

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1184

<code class="details" id="list">list(pageToken=None, pageSize=None, projectId=None, namePrefix=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1185

<pre>Lists pipelines.

1186

1187

Caller must have READ permission to the project.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1188

1189

Args:

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1190

pageToken: string, Token to use to indicate where to start getting results.

1191

If unspecified, returns the first page of results.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1192

pageSize: integer, Number of pipelines to return at once. Defaults to 256, and max

1193

is 2048.

1194

projectId: string, Required. The name of the project to search for pipelines. Caller

1195

must have READ access to this project.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1196

namePrefix: string, Pipelines with names that match this prefix should be

1197

returned. If unspecified, all pipelines in the project, up to

1198

`pageSize`, will be returned.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1199

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1200

Allowed values

1201

1 - v1 error format

1202

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1203

1204

Returns:

1205

An object of the form:

1206

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1207

{ # The response of ListPipelines. Contains at most `pageSize`

1208

# pipelines. If it contains `pageSize` pipelines, and more pipelines

1209

# exist, then `nextPageToken` will be populated and should be

1210

# used as the `pageToken` argument to a subsequent ListPipelines

1211

# request.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1212

"nextPageToken": "A String", # The token to use to get the next page of results.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1213

"pipelines": [ # The matched pipelines.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1214

{ # The pipeline object. Represents a transformation from a set of input

1215

# parameters to a set of output parameters. The transformation is defined

1216

# as a docker image and command to run within that image. Each pipeline

1217

# is run on a Google Compute Engine VM. A pipeline can be created with the

1218

# `create` method and then later run with the `run` method, or a pipeline can

1219

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1220

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1221

# This name can be used for filtering Pipelines in ListPipelines.

1222

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1223

# is called. Cannot be specified in the Pipeline used in the

1224

# CreatePipelineRequest, and will be populated in the response to

1225

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1226

# service has registered this pipeline.

1227

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1228

# WRITE access.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1229

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1230

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1231

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1232

# with optional defaults, and can be overridden at run time.

1233

#

1234

# If `localCopy` is unset, then the parameter specifies a string that

1235

# is passed as-is into the pipeline, as the value of the environment

1236

# variable with the given name. A default value can be optionally

1237

# specified at create time. The default can be overridden at run time

1238

# using the inputs map. If no default is given, a value must be

1239

# supplied at runtime.

1240

#

1241

# If `localCopy` is defined, then the parameter specifies a data

1242

# source or sink, both in Google Cloud Storage and on the Docker container

1243

# where the pipeline computation is run. The service account associated with

1244

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1245

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1246

# Google Cloud Storage paths.

1247

#

1248

# At run time, the Google Cloud Storage paths can be overridden if a default

1249

# was provided at create time, or must be set otherwise. The pipeline runner

1250

# should add a key/value pair to either the inputs or outputs map. The

1251

# indicated data copies will be carried out before/after pipeline execution,

1252

# just as if the corresponding arguments were provided to `gsutil cp`.

1253

#

1254

# For example: Given the following `PipelineParameter`, specified

1255

# in the `inputParameters` list:

1256

#

1257

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1258

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1259

# ```

1260

#

1261

# where `disk` is defined in the `PipelineResources` object as:

1262

#

1263

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1264

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1265

# ```

1266

#

1267

# We create a disk named `pd1`, mount it on the host VM, and map

1268

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1269

# runtime, an entry for `input_file` would be required in the inputs

1270

# map, such as:

1271

#

1272

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1273

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1274

# ```

1275

#

1276

# This would generate the following gsutil call:

1277

#

1278

# ```

1279

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1280

# ```

1281

#

1282

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1283

# Docker container. Acceptable paths are:

1284

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1285

# <table>

1286

# <thead>

1287

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1288

# </thead>

1289

# <tbody>

1290

# <tr><td>file</td><td>file</td></tr>

1291

# <tr><td>glob</td><td>directory</td></tr>

1292

# </tbody>

1293

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1294

#

1295

# For outputs, the direction of the copy is reversed:

1296

#

1297

# ```

1298

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1299

# ```

1300

#

1301

# Acceptable paths are:

1302

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1303

# <table>

1304

# <thead>

1305

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1306

# </thead>

1307

# <tbody>

1308

# <tr><td>file</td><td>file</td></tr>

1309

# <tr>

1310

# <td>file</td>

1311

# <td>directory - directory must already exist</td>

1312

# </tr>

1313

# <tr>

1314

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1315

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1316

# </tbody>

1317

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1318

#

1319

# One restriction due to docker limitations, is that for outputs that are found

1320

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1321

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1322

# If `localCopy` is present, then this must be a Google Cloud Storage path

1323

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1324

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1325

# as the key to the input and output maps in RunPipeline.

1326

"description": "A String", # Human-readable description.

1327

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1328

# `LocalCopy` indicates where on the VM the file should be. The value

1329

# given to this parameter (either at runtime or using `defaultValue`)

1330

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1331

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1332

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1333

# Resources field, or "boot", which represents the Docker

1334

# instance's boot disk and has a mount point of `/`.

1335

"path": "A String", # Required. The path within the user's docker container where

1336

# this input should be localized to and from, relative to the specified

1337

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1338

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1339

},

1340

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1341

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1342

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1343

# string will be executed within a bash shell.

1344

#

1345

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1346

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1347

# `error` field will be populated.

1348

#

1349

# Maximum command string length is 16384.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1350

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1351

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1352

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1353

"description": "A String", # User-specified description.

1354

"inputParameters": [ # Input parameters of the pipeline.

1355

{ # Parameters facilitate setting and delivering data into the

1356

# pipeline's execution environment. They are defined at create time,

1357

# with optional defaults, and can be overridden at run time.

1358

#

1359

# If `localCopy` is unset, then the parameter specifies a string that

1360

# is passed as-is into the pipeline, as the value of the environment

1361

# variable with the given name. A default value can be optionally

1362

# specified at create time. The default can be overridden at run time

1363

# using the inputs map. If no default is given, a value must be

1364

# supplied at runtime.

1365

#

1366

# If `localCopy` is defined, then the parameter specifies a data

1367

# source or sink, both in Google Cloud Storage and on the Docker container

1368

# where the pipeline computation is run. The service account associated with

1369

# the Pipeline (by

1370

# default the project's Compute Engine service account) must have access to the

1371

# Google Cloud Storage paths.

1372

#

1373

# At run time, the Google Cloud Storage paths can be overridden if a default

1374

# was provided at create time, or must be set otherwise. The pipeline runner

1375

# should add a key/value pair to either the inputs or outputs map. The

1376

# indicated data copies will be carried out before/after pipeline execution,

1377

# just as if the corresponding arguments were provided to `gsutil cp`.

1378

#

1379

# For example: Given the following `PipelineParameter`, specified

1380

# in the `inputParameters` list:

1381

#

1382

# ```

1383

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1384

# ```

1385

#

1386

# where `disk` is defined in the `PipelineResources` object as:

1387

#

1388

# ```

1389

# {name: "pd1", mountPoint: "/mnt/disk/"}

1390

# ```

1391

#

1392

# We create a disk named `pd1`, mount it on the host VM, and map

1393

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1394

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1399

# ```

1400

#

1401

# This would generate the following gsutil call:

1402

#

1403

# ```

1404

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1405

# ```

1406

#

1407

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1408

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1413

# </thead>

1414

# <tbody>

1415

# <tr><td>file</td><td>file</td></tr>

1416

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1421

#

1422

# ```

1423

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1424

# ```

1425

#

1426

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1431

# </thead>

1432

# <tbody>

1433

# <tr><td>file</td><td>file</td></tr>

1434

# <tr>

1435

# <td>file</td>

1436

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1445

# on the boot disk, the local path cannot be a glob and must be a file.

1446

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1447

# If `localCopy` is present, then this must be a Google Cloud Storage path

1448

# beginning with `gs://`.

1449

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1450

# as the key to the input and output maps in RunPipeline.

1451

"description": "A String", # Human-readable description.

1452

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1453

# `LocalCopy` indicates where on the VM the file should be. The value

1454

# given to this parameter (either at runtime or using `defaultValue`)

1455

# must be the remote path where the file should be.

1456

"disk": "A String", # Required. The name of the disk where this parameter is

1457

# located. Can be the name of one of the disks specified in the

1458

# Resources field, or "boot", which represents the Docker

1459

# instance's boot disk and has a mount point of `/`.

1460

"path": "A String", # Required. The path within the user's docker container where

1461

# this input should be localized to and from, relative to the specified

1462

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1474

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1475

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1476

# must be true for both create time and run time. Cannot be true at run time

1477

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1478

"zones": [ # List of Google Compute Engine availability zones to which resource

1479

# creation will restricted. If empty, any zone may be chosen.

1480

"A String",

1481

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1482

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1483

# By specifying this parameter, you will download and install the following

1484

# third-party software onto your managed Compute Engine instances:

1485

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1486

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1487

# By specifying this parameter, you will download and install the following

1488

# third-party software onto your managed Compute Engine instances: NVIDIA®

1489

# Tesla® drivers and NVIDIA® CUDA toolkit.

1490

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1491

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1492

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1493

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1494

# feature that may go away. Defaults to false.

1495

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1496

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1497

# In order to use this, must be true for both create time and run time.

1498

# Cannot be true at run time if false at create time. If you need to ssh into

1499

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1500

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1501

# only load docker images from Google Container Registry and not Docker Hub.

1502

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1503

# [configure access to Google services from internal

1504

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1505

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1506

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1507

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1508

# parameters. Must be 1 - 63 characters.

1509

# The name "boot" is reserved for system use.

1510

"type": "A String", # Required. The type of the disk to create.

1511

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1512

# of the pipeline run, regardless of what this field is set to.

1513

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1514

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1515

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1516

# Specifies the path in the docker container where files on

1517

# this disk should be located. For example, if `mountPoint`

1518

# is `/mnt/disk`, and the parameter has `localPath`

1519

# `inputs/file.txt`, the docker container can access the data at

1520

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1521

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1522

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1523

# for more details.

1524

# Can only be set at create time.

1525

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1526

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1527

# and

1528

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1529

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1530

},

1531

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1532

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1533

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

},

],

}</pre>

</div>

<code class="details" id="list_next">list_next(previous_request, previous_response)</code>

1541

<pre>Retrieves the next page of results.

1542

1543

Args:

1544

previous_request: The request for the previous page. (required)

1545

previous_response: The response from the request for the previous page. (required)

1546

1547

Returns:

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1548

A request object that you can call 'execute()' on to request the next

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1549

page. Returns None if there are no more items in the collection.

</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1554

<code class="details" id="run">run(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1555

<pre>Runs a pipeline. If `pipelineId` is specified in the request, then

1556

run a saved pipeline. If `ephemeralPipeline` is specified, then run

1557

that pipeline once without saving a copy.

1558

1559

The caller must have READ permission to the project where the pipeline

1560

is stored and WRITE permission to the project where the pipeline will be

1561

run, as VMs will be created and storage will be used.

1562

1563

If a pipeline operation is still running after 6 days, it will be canceled.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1564

1565

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1566

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1567

The object takes the form of:

1568

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1569

{ # The request to run a pipeline. If `pipelineId` is specified, it

1570

# refers to a saved pipeline created with CreatePipeline and set as

1571

# the `pipelineId` of the returned Pipeline object. If

1572

# `ephemeralPipeline` is specified, that pipeline is run once

1573

# with the given args and not saved. It is an error to specify both

1574

# `pipelineId` and `ephemeralPipeline`. `pipelineArgs`

1575

# must be specified.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1576

"pipelineArgs": { # The pipeline run arguments. # The arguments to use when running this pipeline.

1577

"projectId": "A String", # Required. The project in which to run the pipeline. The caller must have

1578

# WRITER access to all Google Cloud services and resources (e.g. Google

1579

# Compute Engine) will be used.

1580

"clientId": "A String", # This field is deprecated. Use `labels` instead. Client-specified pipeline

1581

# operation identifier.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1582

"serviceAccount": { # A Google Cloud Service Account. # The Google Cloud Service Account that will be used to access data and

1583

# services. By default, the compute service account associated with

1584

# `projectId` is used.

1585

"email": "A String", # Email address of the service account. Defaults to `default`,

1586

# which uses the compute service account associated with the project.

1587

"scopes": [ # List of scopes to be enabled for this service account on the VM.

1588

# The following scopes are automatically included:

1589

#

1590

# * https://www.googleapis.com/auth/compute

1591

# * https://www.googleapis.com/auth/devstorage.full_control

1592

# * https://www.googleapis.com/auth/genomics

1593

# * https://www.googleapis.com/auth/logging.write

1594

# * https://www.googleapis.com/auth/monitoring.write

1595

"A String",

1596

],

1597

},

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1598

"inputs": { # Pipeline input arguments; keys are defined in the pipeline documentation.

1599

# All input parameters that do not have default values must be specified.

1600

# If parameters with defaults are specified here, the defaults will be

1601

# overridden.

1602

"a_key": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1603

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1604

"labels": { # Labels to apply to this pipeline run. Labels will also be applied to

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1605

# compute resources (VM, disks) created by this pipeline run. When listing

1606

# operations, operations can filtered by labels.

1607

# Label keys may not be empty; label values may be empty. Non-empty labels

1608

# must be 1-63 characters long, and comply with [RFC1035]

1609

# (https://www.ietf.org/rfc/rfc1035.txt).

1610

# Specifically, the name must be 1-63 characters long and match the regular

1611

# expression `[a-z]([-a-z0-9]*[a-z0-9])?` which means the first

1612

# character must be a lowercase letter, and all following characters must be

1613

# a dash, lowercase letter, or digit, except the last character, which cannot

1614

# be a dash.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1615

"a_key": "A String",

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1616

},

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1617

"logging": { # The logging options for the pipeline run. # Required. Logging options. Used by the service to communicate results

1618

# to the user.

1619

"gcsPath": "A String", # The location in Google Cloud Storage to which the pipeline logs

1620

# will be copied. Can be specified as a fully qualified directory

1621

# path, in which case logs will be output with a unique identifier

1622

# as the filename in that directory, or as a fully specified path,

1623

# which must end in `.log`, in which case that path will be

1624

# used, and the user must ensure that logs are not

1625

# overwritten. Stdout and stderr logs from the run are also

1626

# generated and output as `-stdout.log` and `-stderr.log`.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1627

},

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1628

"keepVmAliveOnFailureDuration": "A String", # How long to keep the VM up after a failure (for example docker command

1629

# failed, copying input or output files failed, etc). While the VM is up, one

1630

# can ssh into the VM to debug. Default is 0; maximum allowed value is 1 day.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1631

"resources": { # The system resources for the pipeline run. # Specifies resource requirements/overrides for the pipeline run.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1632

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1633

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1634

# must be true for both create time and run time. Cannot be true at run time

1635

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1636

"zones": [ # List of Google Compute Engine availability zones to which resource

1637

# creation will restricted. If empty, any zone may be chosen.

1638

"A String",

1639

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1640

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1641

# By specifying this parameter, you will download and install the following

1642

# third-party software onto your managed Compute Engine instances:

1643

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1644

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1645

# By specifying this parameter, you will download and install the following

1646

# third-party software onto your managed Compute Engine instances: NVIDIA®

1647

# Tesla® drivers and NVIDIA® CUDA toolkit.

1648

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1649

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1650

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1651

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1652

# feature that may go away. Defaults to false.

1653

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1654

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1655

# In order to use this, must be true for both create time and run time.

1656

# Cannot be true at run time if false at create time. If you need to ssh into

1657

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1658

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1659

# only load docker images from Google Container Registry and not Docker Hub.

1660

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1661

# [configure access to Google services from internal

1662

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1663

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1664

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1665

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1666

# parameters. Must be 1 - 63 characters.

1667

# The name "boot" is reserved for system use.

1668

"type": "A String", # Required. The type of the disk to create.

1669

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1670

# of the pipeline run, regardless of what this field is set to.

1671

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

1672

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1673

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1674

# Specifies the path in the docker container where files on

1675

# this disk should be located. For example, if `mountPoint`

1676

# is `/mnt/disk`, and the parameter has `localPath`

1677

# `inputs/file.txt`, the docker container can access the data at

1678

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1679

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

1680

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

1681

# for more details.

1682

# Can only be set at create time.

1683

"source": "A String", # The full or partial URL of the persistent disk to attach. See

1684

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

1685

# and

1686

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

1687

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1688

},

1689

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1690

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1691

},

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1692

"outputs": { # Pipeline output arguments; keys are defined in the pipeline

1693

# documentation. All output parameters of without default values

1694

# must be specified. If parameters with defaults are specified

1695

# here, the defaults will be overridden.

1696

"a_key": "A String",

1697

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1698

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1699

"pipelineId": "A String", # The already created pipeline to run.

1700

"ephemeralPipeline": { # The pipeline object. Represents a transformation from a set of input # A new pipeline object to run once and then delete.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1701

# parameters to a set of output parameters. The transformation is defined

1702

# as a docker image and command to run within that image. Each pipeline

1703

# is run on a Google Compute Engine VM. A pipeline can be created with the

1704

# `create` method and then later run with the `run` method, or a pipeline can

1705

# be defined and run all at once with the `run` method.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1706

"name": "A String", # Required. A user specified pipeline name that does not have to be unique.

1707

# This name can be used for filtering Pipelines in ListPipelines.

1708

"pipelineId": "A String", # Unique pipeline id that is generated by the service when CreatePipeline

1709

# is called. Cannot be specified in the Pipeline used in the

1710

# CreatePipelineRequest, and will be populated in the response to

1711

# CreatePipeline and all subsequent Get and List calls. Indicates that the

1712

# service has registered this pipeline.

1713

"projectId": "A String", # Required. The project in which to create the pipeline. The caller must have

1714

# WRITE access.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1715

"outputParameters": [ # Output parameters of the pipeline.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1716

{ # Parameters facilitate setting and delivering data into the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1717

# pipeline's execution environment. They are defined at create time,

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1718

# with optional defaults, and can be overridden at run time.

1719

#

1720

# If `localCopy` is unset, then the parameter specifies a string that

1721

# is passed as-is into the pipeline, as the value of the environment

1722

# variable with the given name. A default value can be optionally

1723

# specified at create time. The default can be overridden at run time

1724

# using the inputs map. If no default is given, a value must be

1725

# supplied at runtime.

1726

#

1727

# If `localCopy` is defined, then the parameter specifies a data

1728

# source or sink, both in Google Cloud Storage and on the Docker container

1729

# where the pipeline computation is run. The service account associated with

1730

# the Pipeline (by

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1731

# default the project's Compute Engine service account) must have access to the

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1732

# Google Cloud Storage paths.

1733

#

1734

# At run time, the Google Cloud Storage paths can be overridden if a default

1735

# was provided at create time, or must be set otherwise. The pipeline runner

1736

# should add a key/value pair to either the inputs or outputs map. The

1737

# indicated data copies will be carried out before/after pipeline execution,

1738

# just as if the corresponding arguments were provided to `gsutil cp`.

1739

#

1740

# For example: Given the following `PipelineParameter`, specified

1741

# in the `inputParameters` list:

1742

#

1743

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1744

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1745

# ```

1746

#

1747

# where `disk` is defined in the `PipelineResources` object as:

1748

#

1749

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1750

# {name: "pd1", mountPoint: "/mnt/disk/"}

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1751

# ```

1752

#

1753

# We create a disk named `pd1`, mount it on the host VM, and map

1754

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1755

# runtime, an entry for `input_file` would be required in the inputs

1756

# map, such as:

1757

#

1758

# ```

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1759

# inputs["input_file"] = "gs://my-bucket/bar.txt"

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1760

# ```

1761

#

1762

# This would generate the following gsutil call:

1763

#

1764

# ```

1765

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1766

# ```

1767

#

1768

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1769

# Docker container. Acceptable paths are:

1770

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1771

# <table>

1772

# <thead>

1773

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1774

# </thead>

1775

# <tbody>

1776

# <tr><td>file</td><td>file</td></tr>

1777

# <tr><td>glob</td><td>directory</td></tr>

1778

# </tbody>

1779

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1780

#

1781

# For outputs, the direction of the copy is reversed:

1782

#

1783

# ```

1784

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1785

# ```

1786

#

1787

# Acceptable paths are:

1788

#

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1789

# <table>

1790

# <thead>

1791

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1792

# </thead>

1793

# <tbody>

1794

# <tr><td>file</td><td>file</td></tr>

1795

# <tr>

1796

# <td>file</td>

1797

# <td>directory - directory must already exist</td>

1798

# </tr>

1799

# <tr>

1800

# <td>glob</td>

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1801

# <td>directory - directory will be created if it doesn't exist</td></tr>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

1802

# </tbody>

1803

# </table>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1804

#

1805

# One restriction due to docker limitations, is that for outputs that are found

1806

# on the boot disk, the local path cannot be a glob and must be a file.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1807

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1808

# If `localCopy` is present, then this must be a Google Cloud Storage path

1809

# beginning with `gs://`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1810

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1811

# as the key to the input and output maps in RunPipeline.

1812

"description": "A String", # Human-readable description.

1813

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1814

# `LocalCopy` indicates where on the VM the file should be. The value

1815

# given to this parameter (either at runtime or using `defaultValue`)

1816

# must be the remote path where the file should be.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1817

"disk": "A String", # Required. The name of the disk where this parameter is

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1818

# located. Can be the name of one of the disks specified in the

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1819

# Resources field, or "boot", which represents the Docker

1820

# instance's boot disk and has a mount point of `/`.

1821

"path": "A String", # Required. The path within the user's docker container where

1822

# this input should be localized to and from, relative to the specified

1823

# disk's mount point. For example: file.txt,

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1824

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1825

},

1826

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1827

"docker": { # The Docker execuctor specification. # Specifies the docker run information.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1828

"cmd": "A String", # Required. The command or newline delimited script to run. The command

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1829

# string will be executed within a bash shell.

1830

#

1831

# If the command exits with a non-zero exit code, output parameter

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1832

# de-localization will be skipped and the pipeline operation's

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1833

# `error` field will be populated.

1834

#

1835

# Maximum command string length is 16384.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1836

"imageName": "A String", # Required. Image name from either Docker Hub or Google Container Registry.

1837

# Users that run pipelines must have READ access to the image.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1838

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1839

"description": "A String", # User-specified description.

1840

"inputParameters": [ # Input parameters of the pipeline.

1841

{ # Parameters facilitate setting and delivering data into the

1842

# pipeline's execution environment. They are defined at create time,

1843

# with optional defaults, and can be overridden at run time.

1844

#

1845

# If `localCopy` is unset, then the parameter specifies a string that

1846

# is passed as-is into the pipeline, as the value of the environment

1847

# variable with the given name. A default value can be optionally

1848

# specified at create time. The default can be overridden at run time

1849

# using the inputs map. If no default is given, a value must be

1850

# supplied at runtime.

1851

#

1852

# If `localCopy` is defined, then the parameter specifies a data

1853

# source or sink, both in Google Cloud Storage and on the Docker container

1854

# where the pipeline computation is run. The service account associated with

1855

# the Pipeline (by

1856

# default the project's Compute Engine service account) must have access to the

1857

# Google Cloud Storage paths.

1858

#

1859

# At run time, the Google Cloud Storage paths can be overridden if a default

1860

# was provided at create time, or must be set otherwise. The pipeline runner

1861

# should add a key/value pair to either the inputs or outputs map. The

1862

# indicated data copies will be carried out before/after pipeline execution,

1863

# just as if the corresponding arguments were provided to `gsutil cp`.

1864

#

1865

# For example: Given the following `PipelineParameter`, specified

1866

# in the `inputParameters` list:

1867

#

1868

# ```

1869

# {name: "input_file", localCopy: {path: "file.txt", disk: "pd1"}}

1870

# ```

1871

#

1872

# where `disk` is defined in the `PipelineResources` object as:

1873

#

1874

# ```

1875

# {name: "pd1", mountPoint: "/mnt/disk/"}

1876

# ```

1877

#

1878

# We create a disk named `pd1`, mount it on the host VM, and map

1879

# `/mnt/pd1` to `/mnt/disk` in the docker container. At

1880

# runtime, an entry for `input_file` would be required in the inputs

# map, such as:

#

# ```

# inputs["input_file"] = "gs://my-bucket/bar.txt"

1885

# ```

1886

#

1887

# This would generate the following gsutil call:

1888

#

1889

# ```

1890

# gsutil cp gs://my-bucket/bar.txt /mnt/pd1/file.txt

1891

# ```

1892

#

1893

# The file `/mnt/pd1/file.txt` maps to `/mnt/disk/file.txt` in the

1894

# Docker container. Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Google Cloud storage path</th><th>Local path</th></tr>

1899

# </thead>

1900

# <tbody>

1901

# <tr><td>file</td><td>file</td></tr>

1902

# <tr><td>glob</td><td>directory</td></tr>

# </tbody>

# </table>

#

# For outputs, the direction of the copy is reversed:

1907

#

1908

# ```

1909

# gsutil cp /mnt/disk/file.txt gs://my-bucket/bar.txt

1910

# ```

1911

#

1912

# Acceptable paths are:

#

# <table>

# <thead>

# <tr><th>Local path</th><th>Google Cloud Storage path</th></tr>

1917

# </thead>

1918

# <tbody>

1919

# <tr><td>file</td><td>file</td></tr>

1920

# <tr>

1921

# <td>file</td>

1922

# <td>directory - directory must already exist</td>

# </tr>

# <tr>

# <td>glob</td>

# <td>directory - directory will be created if it doesn't exist</td></tr>

# </tbody>

# </table>

#

# One restriction due to docker limitations, is that for outputs that are found

1931

# on the boot disk, the local path cannot be a glob and must be a file.

1932

"defaultValue": "A String", # The default value for this parameter. Can be overridden at runtime.

1933

# If `localCopy` is present, then this must be a Google Cloud Storage path

1934

# beginning with `gs://`.

1935

"name": "A String", # Required. Name of the parameter - the pipeline runner uses this string

1936

# as the key to the input and output maps in RunPipeline.

1937

"description": "A String", # Human-readable description.

1938

"localCopy": { # LocalCopy defines how a remote file should be copied to and from the VM. # If present, this parameter is marked for copying to and from the VM.

1939

# `LocalCopy` indicates where on the VM the file should be. The value

1940

# given to this parameter (either at runtime or using `defaultValue`)

1941

# must be the remote path where the file should be.

1942

"disk": "A String", # Required. The name of the disk where this parameter is

1943

# located. Can be the name of one of the disks specified in the

1944

# Resources field, or "boot", which represents the Docker

1945

# instance's boot disk and has a mount point of `/`.

1946

"path": "A String", # Required. The path within the user's docker container where

1947

# this input should be localized to and from, relative to the specified

1948

# disk's mount point. For example: file.txt,

},

},

],

"resources": { # The system resources for the pipeline run. # Required. Specifies resource requirements for the pipeline run.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

# Required fields:

#

# *

# minimumCpuCores

#

# *

# minimumRamGb

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1960

"minimumRamGb": 3.14, # The minimum amount of RAM to use. Defaults to 3.75 (GB)

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1961

"preemptible": True or False, # Whether to use preemptible VMs. Defaults to `false`. In order to use this,

1962

# must be true for both create time and run time. Cannot be true at run time

1963

# if false at create time.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1964

"zones": [ # List of Google Compute Engine availability zones to which resource

1965

# creation will restricted. If empty, any zone may be chosen.

1966

"A String",

1967

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1968

"acceleratorCount": "A String", # Optional. The number of accelerators of the specified type to attach.

1969

# By specifying this parameter, you will download and install the following

1970

# third-party software onto your managed Compute Engine instances:

1971

# NVIDIA® Tesla® drivers and NVIDIA® CUDA toolkit.

1972

"acceleratorType": "A String", # Optional. The Compute Engine defined accelerator type.

1973

# By specifying this parameter, you will download and install the following

1974

# third-party software onto your managed Compute Engine instances: NVIDIA®

1975

# Tesla® drivers and NVIDIA® CUDA toolkit.

1976

# Please see https://cloud.google.com/compute/docs/gpus/ for a list of

1977

# available accelerator types.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1978

"minimumCpuCores": 42, # The minimum number of cores to use. Defaults to 1.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1979

"noAddress": True or False, # Whether to assign an external IP to the instance. This is an experimental

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1980

# feature that may go away. Defaults to false.

1981

# Corresponds to `--no_address` flag for [gcloud compute instances create]

1982

# (https://cloud.google.com/sdk/gcloud/reference/compute/instances/create).

1983

# In order to use this, must be true for both create time and run time.

1984

# Cannot be true at run time if false at create time. If you need to ssh into

1985

# a private IP VM for debugging, you can ssh to a public VM and then ssh into

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1986

# the private VM's Internal IP. If noAddress is set, this pipeline run may

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

1987

# only load docker images from Google Container Registry and not Docker Hub.

1988

# Before using this, you must

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

1989

# [configure access to Google services from internal

1990

# IPs](https://cloud.google.com/compute/docs/configure-private-google-access#configuring_access_to_google_services_from_internal_ips).

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

1991

"disks": [ # Disks to attach.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

1992

{ # A Google Compute Engine disk resource specification.

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

1993

"name": "A String", # Required. The name of the disk that can be used in the pipeline

1994

# parameters. Must be 1 - 63 characters.

1995

# The name "boot" is reserved for system use.

1996

"type": "A String", # Required. The type of the disk to create.

1997

"autoDelete": True or False, # Deprecated. Disks created by the Pipelines API will be deleted at the end

1998

# of the pipeline run, regardless of what this field is set to.

1999

"sizeGb": 42, # The size of the disk. Defaults to 500 (GB).

2000

# This field is not applicable for local SSD.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2001

"mountPoint": "A String", # Required at create time and cannot be overridden at run time.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2002

# Specifies the path in the docker container where files on

2003

# this disk should be located. For example, if `mountPoint`

2004

# is `/mnt/disk`, and the parameter has `localPath`

2005

# `inputs/file.txt`, the docker container can access the data at

2006

# `/mnt/disk/inputs/file.txt`.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2007

"readOnly": True or False, # Specifies how a sourced-base persistent disk will be mounted. See

2008

# https://cloud.google.com/compute/docs/disks/persistent-disks#use_multi_instances

2009

# for more details.

2010

# Can only be set at create time.

2011

"source": "A String", # The full or partial URL of the persistent disk to attach. See

2012

# https://cloud.google.com/compute/docs/reference/latest/instances#resource

2013

# and

2014

# https://cloud.google.com/compute/docs/disks/persistent-disks#snapshots

2015

# for more details.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2016

},

2017

],

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2018

"bootDiskSizeGb": 42, # The size of the boot disk. Defaults to 10 (GB).

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2019

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2020

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2021

}

2022

2023

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2024

Allowed values

2025

1 - v1 error format

2026

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2027

2028

Returns:

2029

An object of the form:

2030

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2031

{ # This resource represents a long-running operation that is the result of a

2032

# network API call.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2033

"name": "A String", # The server-assigned name, which is only unique within the same service that originally returns it. For example&#58; `operations/CJHU7Oi_ChDrveSpBRjfuL-qzoWAgEw`

2034

"error": { # The `Status` type defines a logical error model that is suitable for # The error result of the operation in case of failure or cancellation.

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2035

# different programming environments, including REST APIs and RPC APIs. It is

2036

# used by [gRPC](https://github.com/grpc). Each `Status` message contains

2037

# three pieces of data: error code, error message, and error details.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2038

#

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2039

# You can find out more about this error model and how to work with it in the

2040

# [API Design Guide](https://cloud.google.com/apis/design/errors).

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

2041

"code": 42, # The status code, which should be an enum value of google.rpc.Code.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2042

"message": "A String", # A developer-facing error message, which should be in English. Any

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2043

# user-facing error message should be localized and sent in the

2044

# google.rpc.Status.details field, or localized by the client.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2045

"details": [ # A list of messages that carry the error details. There is a common set of

Bu Sun Kim

2019-06-14 16:50:42 -0700

[diff] [blame]

2046

# message types for APIs to use.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2047

{

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2048

"a_key": "", # Properties of the object. Contains field @type with type URL.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2049

},

2050

],

2051

},

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2052

"metadata": { # An OperationMetadata or Metadata object. This will always be returned with the Operation.

2053

"a_key": "", # Properties of the object. Contains field @type with type URL.

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2054

},

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

2055

"done": True or False, # If the value is `false`, it means the operation is still in progress.

2056

# If `true`, the operation is completed, and either `error` or `response` is

2057

# available.

2058

"response": { # An Empty object.

2059

"a_key": "", # Properties of the object. Contains field @type with type URL.

2060

},

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

}</pre>

</div>

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2065

<code class="details" id="setOperationStatus">setOperationStatus(body=None, x__xgafv=None)</code>

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2066

<pre>Sets status of a given operation. Any new timestamps (as determined by

2067

description) are appended to TimestampEvents. Should only be called by VMs

2068

created by the Pipelines Service and not by end users.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2069

2070

Args:

Dan O'Meara

2020-05-01 07:42:23 -0700

[diff] [blame]

2071

body: object, The request body.

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2072

The object takes the form of:

2073

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2074

{ # Request to set operation status. Should only be used by VMs

2075

# created by the Pipelines Service and not by end users.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2076

"errorMessage": "A String",

2077

"validationToken": "A String",

2078

"errorCode": "A String",

2079

"timestampEvents": [

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2080

{ # Stores the list of events and times they occured for major events in job

2081

# execution.

Bu Sun Kim

2020-05-20 12:08:20 -0700

[diff] [blame]

2082

"timestamp": "A String", # The time this event occured.

2083

"description": "A String", # String indicating the type of event

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2084

},

2085

],

Bu Sun Kim

2020-05-27 12:20:54 -0700

[diff] [blame^]

2086

"operationId": "A String",

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2087

}

2088

2089

x__xgafv: string, V1 error format.

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2090

Allowed values

2091

1 - v1 error format

2092

2 - v2 error format

Sai Cheemalapati

2016-10-21 13:59:49 -0700

[diff] [blame]

2093

2094

Returns:

2095

An object of the form:

2096

Sai Cheemalapati

2017-03-13 12:12:03 -0400

[diff] [blame]

2097

{ # A generic empty message that you can re-use to avoid defining duplicated

2098

# empty messages in your APIs. A typical example is to use it as the request

2099

# or the response type of an API method. For instance:

2100

#

2101

# service Foo {

2102

# rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty);

2103

# }

2104

#

2105

# The JSON representation for `Empty` is empty JSON object `{}`.

Sai Cheemalapati