The Good, The Bad, and The Vulnerable: Breaking Down GCP Tenant Projects

The Good, The Bad, and The Vulnerable: Breaking Down GCP Tenant Projects

Speakers: Ofir Balassiano; Ofir Shaty

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=WUO_-Agpcxs

Overview

Ofir Balassiano and Ofir Shaty, security researchers at Palo Alto Networks, present a deep investigation into GCP tenant projects -- hidden, Google-managed projects that are provisioned behind the scenes when customers use managed services like Vertex AI, Cloud Composer, Cloud SQL, and BigQuery. The researchers reverse-engineered the internal architecture of Vertex AI's tenant projects by exploiting a vulnerability that gave them code execution inside the tenant environment. From there, they mapped the service accounts, permissions, data flows, and cross-project access patterns, discovering multiple privilege escalation paths, data exposure risks, and a previously undocumented attack surface that most GCP customers do not know exists. The research demonstrates that these "shadow projects" hold customer data, connect to customer VPCs, and operate with broad permissions -- all outside the customer's visibility and control.

Background

▶ Watch: What are tenant projects: the producer-consumer model explained (1:30)

GCP's managed services frequently use a producer-consumer architecture in which Google (the producer) provisions dedicated tenant projects to deliver service functionality for each customer (the consumer). These tenant projects live in Google's organization, not the customer's, but they connect to the customer's infrastructure through VPC peering, Private Service Connect (PSC), and IAM service agents with cross-project permissions.

This architecture exists for legitimate reasons. From the customer's perspective, it simplifies operations by offloading infrastructure management to Google. From Google's perspective, it provides lifecycle management control, granular billing, and isolation between customers. Services known to use this model include Vertex AI (separate tenant projects for training and deployment), Cloud Composer, Cloud SQL, Filestore, and parts of BigQuery.

The security implications are significant: these tenant projects contain production customer data, have network connectivity to customer VPCs, and operate with elevated IAM permissions -- yet they exist entirely outside the customer's visibility and governance. There is no "list all tenant projects" button in GCP.

Key Findings

▶ Watch: Service agents and P4SA: cross-project permissions exposed (5:30)

The research produced several significant discoveries:

1. Code execution in Vertex AI tenant projects. The researchers exploited a standard Vertex AI feature -- custom training jobs that accept user-provided container images -- to achieve a reverse shell inside the tenant project. No vulnerability was needed for initial access; Vertex AI runs arbitrary container images by design. The shell revealed they were running as the gcp-sa-aiplatform-cc service agent inside a Google-managed worker.

1. Three categories of service accounts in tenant projects. Beyond the expected per-project service agents, the researchers discovered system service accounts (like training-jms and pipeline-robot) that appear to have cross-tenant privileges, and a Google internal group called two-sync. The system service accounts handle critical operations: creating Kubernetes clusters, granting permissions to other service agents, and managing pipeline synchronization.

1. Privilege escalation to consumer resources. Using the service agent credentials from within the tenant project, the researchers accessed consumer resources (Cloud Storage, BigQuery) that the consumer user they were operating as did not have direct permission to access. This represents a classic privilege escalation from tenant project to consumer project.

1. Full tenant project takeover via machine type bypass. The initial compute had limited IAM scope (read-only). By switching to a different Vertex AI machine type, the researchers discovered that the scope limitation was removed, allowing them to create a new machine with the default editor role and take over the tenant project completely.

1. Data movement between tenant projects. Training data is copied from the consumer project to tenant project #1 (training). Model weights (adapter layers) are then copied to tenant project #2 (deployment). The researchers confirmed that consumer data -- both training datasets and model artifacts -- resides in tenant projects, and that data flows between tenant projects as well.

1. Access to internal Google container images. From within the tenant project, the researchers discovered artifact URIs pointing to restricted Google repositories. Using the service agent credentials, they accessed some of these internal images located in what they term the "porta project."

Technical Deep Dive

▶ Watch: Three categories of service accounts: agents, system accounts, internal groups (10:00)

The tenant project architecture follows a consistent pattern across GCP services. When a customer enables a managed service, Google provisions one or more tenant projects in its own organization. These projects are linked to the consumer through a tenancy unit resource. Network connectivity is established through either VPC peering (used by Cloud SQL and Filestore) or Private Service Connect forwarding rules (used by Cloud Composer).

Service agents (also called P4SA -- Per-Product Per-Project Service Accounts) are automatically created when a GCP service is enabled. They follow a naming convention like service-{project-number}@gcp-sa-{service}.iam.gserviceaccount.com and are granted roles, often broad ones, in the customer's project to allow the service to function. The researchers mapped the specific service agents operating within Vertex AI's tenant projects using Cloud Audit Logs.

The Vertex AI vulnerability chain proceeded as follows:

1. Create a custom training job with a container image containing a reverse shell.
2. Obtain code execution inside the tenant project as the gcp-sa-aiplatform-cc service agent.
3. Enumerate all permissions, resources, and service accounts in the tenant project.
4. Discover that the initial compute has read-only scope limiting the service agent's permissions.
5. Configure a custom job with a different machine type that removes the scope restriction.
6. Use the unrestricted service agent to create a new machine with the default editor role.
7. Achieve full tenant project takeover and access to consumer resources.

The researchers classified tenant project entities into three categories: service agents (project-dedicated, one per service), system service accounts (cross-tenant, handling infrastructure operations), and Google internal groups (like two-sync, which was not observed actively operating). Key system service accounts included the training-jms (Job Management Service, responsible for cluster creation), the service-agent-manager (granting permissions to other service agents), and the pipeline-robot (permission setting and sync management).

Demo / Proof of Concept

▶ Watch: Privilege escalation: accessing consumer Cloud Storage and BigQuery from tena... (12:00)

The researchers demonstrated the full attack chain through screenshots and walkthroughs rather than a live demo (they presented remotely). The demonstration included obtaining a reverse shell in the tenant project, enumerating service accounts and permissions, accessing consumer resources through privilege escalation, and tracing data movement between tenant projects. Screenshots showed the adapter layer model artifacts stored in tenant project paths and training data copied from the consumer project.

Defensive Implications

▶ Watch: Discovery and monitoring: how to find tenant projects in your environment (16:30)

GCP customers should take several immediate actions to assess and mitigate tenant project risks:

• Discover your tenant projects. While no single "list all tenant projects" API exists, customers can identify them by examining VPC peering configurations, PSC forwarding rules, IAM policy bindings for service agents and system service accounts, and Cloud Audit Logs for cross-project access patterns.
• Audit service agent permissions. Review the roles granted to P4SA service agents in your consumer projects. These often have overly broad permissions that could be scoped down.
• Monitor Cloud Audit Logs for cross-project access. Build detections for access to consumer resources originating from service agents in tenant projects that may indicate compromise.
• Use CMEK (Customer-Managed Encryption Keys) to encrypt data that flows to tenant projects. This prevents resources outside the tenant project from accessing the data without your key.
• Restrict network connectivity. Use organization policies to restrict VPC peering and PSC forwarding rules. Scope down network access between consumer and tenant projects to the minimum required.
• Restrict data flow. Minimize the data transferred from consumer projects to tenant projects through IAM policies and data access controls.
• Recognize that the shared responsibility model has a gap. Tenant projects exist in Google's organization but hold your data and connect to your network. Actively manage this boundary rather than assuming Google's isolation is sufficient.

Key Takeaways

• GCP managed services like Vertex AI, Cloud Composer, and Cloud SQL create hidden "tenant projects" in Google's organization that hold customer data, connect to customer VPCs, and operate with elevated permissions.
• A vulnerability in Vertex AI's custom training job feature allowed the researchers to obtain code execution inside the tenant project and escalate to full tenant project takeover.
• Consumer data (training datasets, model weights, adapter layers) is copied to and between tenant projects, creating a data exposure surface that customers typically do not monitor.
• Service agents in tenant projects can access consumer resources (Cloud Storage, BigQuery) through cross-project permissions, enabling privilege escalation.
• There is no native "list all tenant projects" function, but customers can discover them through VPC peering configs, PSC forwarding rules, IAM policy bindings, and Cloud Audit Logs.
• Encrypt data with CMEK before it flows to tenant projects to maintain control over data access.

About the Speaker(s)

Ofir Balassiano and Ofir Shaty are security researchers at Palo Alto Networks. They presented remotely at the conference. Their research focuses on cloud security architecture and vulnerability research in GCP managed services. The Vertex AI tenant project research began during their broader investigation into Vertex AI's security posture and expanded into a systematic examination of the tenant project model across GCP services.