test: Create BYOID Integration tests (#719)
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index ac65343..175e766 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -42,9 +42,9 @@
To run a single session, specify it with ``nox -s``::
$ nox -f system_tests/noxfile.py -s service_account
-
-First, set the environemnt variable ``GOOGLE_APPLICATION_CREDENTIALS`` to a valid service account.
-See `Creating and Managing Service Account Keys`_ for how to obtain a service account.
+
+First, set the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` to a valid service account.
+See `Creating and Managing Service Account Keys`_ for how to obtain a service account.
Project and Credentials Setup
-------------------------------
@@ -86,26 +86,40 @@
``service_account.json``
~~~~~~~~~~~~~~~~~~~~~~~~
-Follow `Creating and Managing Service Account Keys`_ to create a service account.
+Follow `Creating and Managing Service Account Keys`_ to create a service account.
Copy the credentials file to ``service_account.json``.
Grant the account associated with ``service_account.json`` the following roles.
- App Engine Admin (for App Engine tests)
-- Service Account Token Creator (for impersonated credentials tests)
+- Service Account Token Creator (for impersonated credentials and workload identity federation tests)
- Pub/Sub Viewer (for gRPC tests)
- Storage Object Viewer (for impersonated credentials tests)
+- DNS Viewer (for workload identity federation tests)
``impersonated_service_account.json``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Follow `Creating and Managing Service Account Keys`_ to create a service account.
+Follow `Creating and Managing Service Account Keys`_ to create a service account.
Copy the credentials file to ``impersonated_service_account.json``.
.. _Creating and Managing Service Account Keys: https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+``setup_external_accounts``
+~~~~~~~~~~~~~~~~
+
+In order to run the workload identity federation tests, you will need to set up
+a Workload Identity Pool, as well as attach relevant policy bindings for this
+new resource to our service account. To do this, make sure you have IAM Workload
+Identity Pool Admin and Security Admin permissions, and then run:
+
+ $ ./scripts/setup_external_accounts.sh
+
+and then use the output to replace the variables near
+the top of system_tests/system_tests_sync/test_external_accounts.py
+
App Engine System Tests
~~~~~~~~~~~~~~~~~~~~~~~~
@@ -118,16 +132,16 @@
$ pip install --target lib -r requirements.txt
$ gcloud app deploy -q app.yaml
-After the app is deployed, change ``service`` in ``app.yaml`` back to ``google-auth-system-tests``.
+After the app is deployed, change ``service`` in ``app.yaml`` back to ``google-auth-system-tests``.
You can now run the App Engine tests: ::
$ nox -f system_tests/noxfile.py -s app_engine
-
+
Compute Engine Tests
^^^^^^^^^^^^^^^^^^^^
These tests cannot be run locally and will be skipped if they are run outside of Google Compute Engine.
-
+
grpc Tests
^^^^^^^^^^^^
diff --git a/scripts/setup_external_accounts.sh b/scripts/setup_external_accounts.sh
new file mode 100644
index 0000000..2fd04e2
--- /dev/null
+++ b/scripts/setup_external_accounts.sh
@@ -0,0 +1,112 @@
+#!/bin/bash
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is a mostly common setup file to ensure all workload identity
+# federation integration tests are set up in a consistent fashion across the
+# languages in our various client libraries. It assumes that the current user
+# has the relevant permissions to run each of the commands listed.
+
+# This script needs to be run once. It will do the following:
+# 1. Create a random workload identity pool.
+# 2. Create a random OIDC provider in that pool which uses the
+# accounts.google.com as the issuer and the default STS audience as the
+# allowed audience. This audience will be validated on STS token exchange.
+# 3. Enable OIDC tokens generated by the current service account to impersonate
+# the service account. (Identified by the OIDC token sub field which is the
+# service account client ID).
+# 4. Create a random AWS provider in that pool which uses the provided AWS
+# account ID.
+# 5. Enable AWS provider to impersonate the service account. (Principal is
+# identified by the AWS role name).
+# 6. Print out the STS audience fields associated with the created providers
+# after the setup completes successfully so that they can be used in the
+# tests. These will be copied and used as the global _AUDIENCE_OIDC and
+# _AUDIENCE_AWS constants in system_tests/system_tests_sync/test_external_accounts.py.
+#
+# It is safe to run the setup script again. A new pool is created and new
+# audiences are printed. If run multiple times, it is advisable to delete
+# unused pools. Note that deleted pools are soft deleted and may remain for
+# a while before they are completely deleted. The old pool ID cannot be used
+# in the meantime.
+#
+# For AWS tests, an AWS developer account is needed.
+# The following AWS prerequisite setup is needed.
+# 1. An OIDC Google identity provider needs to be created with the following:
+# issuer: accounts.google.com
+# audience: Use the client_id of the service account.
+# 2. A role for OIDC web identity federation is needed with the created Google
+# provider as a trusted entity:
+# "accounts.google.com:aud": "$CLIENT_ID"
+# The steps are documented at:
+# https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html
+
+suffix=""
+
+function generate_random_string () {
+ local valid_chars=abcdefghijklmnopqrstuvwxyz0123456789
+ for i in {1..8} ; do
+ suffix+="${valid_chars:RANDOM%${#valid_chars}:1}"
+ done
+}
+
+generate_random_string
+
+pool_id="pool-"$suffix
+oidc_provider_id="oidc-"$suffix
+aws_provider_id="aws-"$suffix
+
+# TODO: Fill in.
+project_id="stellar-day-254222"
+project_number="79992041559"
+aws_account_id="077071391996"
+aws_role_name="ci-python-test"
+service_account_email="kokoro@stellar-day-254222.iam.gserviceaccount.com"
+sub="104692443208068386138"
+
+oidc_aud="//iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/providers/$oidc_provider_id"
+aws_aud="//iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/providers/$aws_provider_id"
+
+gcloud config set project $project_id
+
+# Create the Workload Identity Pool.
+gcloud beta iam workload-identity-pools create $pool_id \
+ --location="global" \
+ --description="Test pool" \
+ --display-name="Test pool for Python"
+
+# Create the OIDC Provider.
+gcloud beta iam workload-identity-pools providers create-oidc $oidc_provider_id \
+ --workload-identity-pool=$pool_id \
+ --issuer-uri="https://accounts.google.com" \
+ --location="global" \
+ --attribute-mapping="google.subject=assertion.sub"
+
+# Create the AWS Provider.
+gcloud beta iam workload-identity-pools providers create-aws $aws_provider_id \
+ --workload-identity-pool=$pool_id \
+ --account-id=$aws_account_id \
+ --location="global"
+
+# Give permission to impersonate the service account.
+gcloud iam service-accounts add-iam-policy-binding $service_account_email \
+--role roles/iam.workloadIdentityUser \
+--member "principal://iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/subject/$sub"
+
+gcloud iam service-accounts add-iam-policy-binding $service_account_email \
+ --role roles/iam.workloadIdentityUser \
+ --member "principalSet://iam.googleapis.com/projects/$project_number/locations/global/workloadIdentityPools/$pool_id/attribute.aws_role/arn:aws:sts::$aws_account_id:assumed-role/$aws_role_name"
+
+echo "OIDC audience: "$oidc_aud
+echo "AWS audience: "$aws_aud
diff --git a/system_tests/noxfile.py b/system_tests/noxfile.py
index 6e2e1b8..f177dcd 100644
--- a/system_tests/noxfile.py
+++ b/system_tests/noxfile.py
@@ -354,9 +354,14 @@
)
-# ASYNC SYSTEM TESTS
+@nox.session(python=PYTHON_VERSIONS_SYNC)
+def external_accounts(session):
+ session.install(*TEST_DEPENDENCIES_SYNC, "google-auth", "google-api-python-client", "enum34")
+ default(session, "system_tests_sync/test_external_accounts.py")
+# ASYNC SYSTEM TESTS
+
@nox.session(python=PYTHON_VERSIONS_ASYNC)
def service_account_async(session):
session.install(*(TEST_DEPENDENCIES_SYNC + TEST_DEPENDENCIES_ASYNC))
@@ -374,7 +379,7 @@
session.install(LIBRARY_DIR)
default(
session,
- "system_tests_async/test_default.py",
+ "system_tests_async/test_default.py",
"system_tests_async/test_id_token.py",
)
diff --git a/system_tests/system_tests_sync/test_external_accounts.py b/system_tests/system_tests_sync/test_external_accounts.py
new file mode 100644
index 0000000..0e8ad16
--- /dev/null
+++ b/system_tests/system_tests_sync/test_external_accounts.py
@@ -0,0 +1,138 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Prerequisites:
+# Make sure to run the setup in scripts/setup_external_accounts.sh
+# and copy the logged constant strings (_AUDIENCE_OIDC, _AUDIENCE_AWS)
+# into this file before running this test suite.
+# Once that is done, this test can be run indefinitely.
+#
+# The only requirement for this test suite to run is to set the environment
+# variable GOOGLE_APPLICATION_CREDENTIALS to point to the expected service
+# account keys whose email is referred to in the setup script.
+#
+# This script follows the following logic.
+# OIDC provider (file-sourced and url-sourced credentials):
+# Use the service account keys to generate a Google ID token using the
+# iamcredentials generateIdToken API, using the default STS audience.
+# This will use the service account client ID as the sub field of the token.
+# This OIDC token will be used as the external subject token to be exchanged
+# for a Google access token via GCP STS endpoint and then to impersonate the
+# original service account key.
+
+
+import json
+import os
+from tempfile import NamedTemporaryFile
+
+import sys
+import google.auth
+from googleapiclient import discovery
+from google.oauth2 import service_account
+import pytest
+from mock import patch
+
+# Populate values from the output of scripts/setup_external_accounts.sh.
+_AUDIENCE_OIDC = "//iam.googleapis.com/projects/79992041559/locations/global/workloadIdentityPools/pool-73wslmxn/providers/oidc-73wslmxn"
+
+
+def dns_access_direct(request, project_id):
+ # First, get the default credentials.
+ credentials, _ = google.auth.default(
+ scopes=["https://www.googleapis.com/auth/cloud-platform.read-only"],
+ request=request,
+ )
+
+ # Apply the default credentials to the headers to make the request.
+ headers = {}
+ credentials.apply(headers)
+ response = request(
+ url="https://dns.googleapis.com/dns/v1/projects/{}".format(project_id),
+ headers=headers,
+ )
+
+ if response.status == 200:
+ return response.data
+
+
+def dns_access_client_library(_, project_id):
+ service = discovery.build("dns", "v1")
+ request = service.projects().get(project=project_id)
+ return request.execute()
+
+
+@pytest.fixture(params=[dns_access_direct, dns_access_client_library])
+def dns_access(request, http_request, service_account_info):
+ # Fill in the fixtures on the functions,
+ # so that we don't have to fill in the parameters manually.
+ def wrapper():
+ return request.param(http_request, service_account_info["project_id"])
+
+ yield wrapper
+
+
+@pytest.fixture
+def oidc_credentials(service_account_file, http_request):
+ result = service_account.IDTokenCredentials.from_service_account_file(
+ service_account_file, target_audience=_AUDIENCE_OIDC
+ )
+ result.refresh(http_request)
+ yield result
+
+
+@pytest.fixture
+def service_account_info(service_account_file):
+ with open(service_account_file) as f:
+ yield json.load(f)
+
+
+# Our external accounts tests involve setting up some preconditions, setting a
+# credential file, and then making sure that our client libraries can work with
+# the set credentials.
+def get_project_dns(dns_access, credential_data):
+ with NamedTemporaryFile() as credfile:
+ credfile.write(json.dumps(credential_data).encode("utf-8"))
+ credfile.flush()
+ old_credentials = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
+
+ with patch.dict(os.environ, {"GOOGLE_APPLICATION_CREDENTIALS": credfile.name}):
+ # If our setup and credential file are correct,
+ # discovery.build should be able to establish these as the default credentials.
+ return dns_access()
+
+
+# This test makes sure that setting an accesible credential file
+# works to allow access to Google resources.
+def test_file_based_external_account(
+ oidc_credentials, service_account_info, dns_access
+):
+ with NamedTemporaryFile() as tmpfile:
+ tmpfile.write(oidc_credentials.token.encode("utf-8"))
+ tmpfile.flush()
+
+ assert get_project_dns(
+ dns_access,
+ {
+ "type": "external_account",
+ "audience": _AUDIENCE_OIDC,
+ "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
+ "token_url": "https://sts.googleapis.com/v1/token",
+ "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{}:generateAccessToken".format(
+ oidc_credentials.service_account_email
+ ),
+ "credential_source": {
+ "file": tmpfile.name,
+ },
+ },
+ )