PortablePortable Logo

Using Google Cloud Storage as a Destination with Parquet Files

Denis
Staff Software Engineer

Portable allows you to send data to Google Cloud Storage (GCS) as a destination. By configuring your GCS bucket and service account credentials, you can seamlessly store Parquet files in your preferred folder structure.

This guide walks you through the setup process, including permissions, configuration, and troubleshooting.

Step 1: Create a Google Cloud Storage Bucket

If you don’t already have a GCS bucket, follow these steps:

  1. Go to the Google Cloud Console.
  2. Navigate to Cloud Storage.
  3. Click Create Bucket.
  4. Enter a unique bucket name (e.g., portable-data-bucket).
  5. Choose a storage class (e.g., Standard).
  6. Set access control to Uniform or Fine-grained.
  7. Click Create.

Step 2: Generate Service Account JSON Credentials

Portable requires a service account JSON key to authenticate with your GCS bucket.

Create a Service Account

  1. Go to the Google Cloud Console.
  2. Navigate to IAM & Admin → Service Accounts.
  3. Click Create Service Account.
  4. Give it a name (e.g., portable-gcs-access).
  5. Click Create & Continue.
  6. Assign the role Storage Admin (roles/storage.admin) for full access, or Storage Object Admin (roles/storage.objectAdmin) if you only need file management.
  7. Click Continue and then Done.

Generate and Download JSON Credentials

  1. In the Service Accounts page, click on the newly created account.
  2. Go to the Keys tab.
  3. Click Add Key → Create New Key.
  4. Select JSON.
  5. Click Create → Your JSON file will download automatically.
  6. Store this file securely.

Step 3: Configure Google Cloud Storage in Portable

Now, let's configure Google Cloud Storage as a destination in Portable.

1. Navigate to the GCS Destination Setup

  • Go to Portable’s Destinations Page and click in new (destination) button.
  • Select Google Cloud Storage.

2. Enter Your Connection Details

FieldDescription
BucketThe name of your GCS bucket (e.g., test-dor-1).
GCP Service Account JSONPaste the entire contents of your JSON key here.
Upload Path(Optional) Folder path inside your bucket (e.g., folder1).

🔹 Note: Portable uses the static IP 34.122.15.109. If you have a firewall, add this to your allowlist.

3. Save Your Configuration

  • Click Save.
  • Portable will validate your credentials and confirm that GCS Storage access is correctly set up.

Step 4: Test Your Configuration

To confirm that your setup is working:

  1. Click Save in Portable.
  2. You should see "GCP Storage access is correctly set up" under Status.

Step 5: Troubleshooting

If you run into issues, check these common fixes:

Invalid Credentials

Error: Failed to configure GCP access. Ensure that your credentials are correctly set.
✅ Fix: Verify that your service account JSON is correctly copied.

Permission Denied

Error: Failed to access GCP Storage: bucket does not exist or permission denied.
✅ Fix: Ensure the service account has the correct role (roles/storage.admin).

Bucket Not Found

Error: Bucket does not exist.
✅ Fix: Double-check that the bucket name matches exactly.

Additional Resources