The Cloud Verse
How to transfer data from GCS to Google Drive
Updated On Mar 12, 2023
Simple, cost-effective and scalable way to automatically transfer the data from GCS to Google Drive using cloud functions.
Most of the time the output of the data pipelines running in GCP in file format needs to be available to Google Drive for the general users to consume it. This is important because the users does not need to have access to the data pipeline as well as to the GCS Bucket.
So in this post, We will be using a cloud function to upload files google drive when it is created in the GCS. But this guide can be used to upload from Compute Engine, Kubernetes etc.
Google Drive Sync Architechture
The architecture is simple. Cloud function triggered by an event from Cloud storage. To upload files in the google drive, the function’s service account needs to have access to the drive. The service account should have an editor role on the drive folder.
Enable the required APIs
Enable the following APIs in the GCP project
- cloudfunctions.googleapis.com
- drive.googleapis.com
This command can be used to enable the APIs:
gcloud services enable <service_name>
Create GCS Bucket
GCS Bucket is created to host the files. These files will be moved to drive. The bucket can be created using the following command
gsutil mb -l us-central1 gs://source-bucket-41ljczocug
Create Service account
Cloud function service uses App Engine default service account while deploying the function which is not recommended. So it’s better to have a dedicated service account for the function.
The service account can be created by the command: gcloud iam service-accounts create gcs-drive-function-sa
Grant service account access to download files from GCS Bucket. Run the following command
gcloud projects add-iam-policy-binding quiet-antler-350415 --member=serviceAccount:gcs-drive-function-sa@quiet-antler-350415.iam.gserviceaccount.com --role=roles/storage.objectViewer
Grant service account permission
Now, the service account needs to have access to upload the files in the google drive. Create a folder in the drive and grant the editor role to the service account.
Grant service account access to google drive
Note down the drive folder ID of the drive. That will be used while calling the Drive API. You can get the folder ID by opening the folder.
Get the google drive folder ID
In the URL: 1hvU85TjN9_Qpefba-YKZoCxoDaL_FgCN is the folder ID
Deploy cloud function
Open cloud shell and create a directory. Create a file named main.py and paste the following code into it
import random
import json
import os
import string
import traceback
from googleapiclient import discovery
import google.auth
from googleapiclient.http import MediaFileUpload
from google.cloud import storage
def upload_to_drive(local_path, remote_path):
GDRIVE_FOLDER_IDS = str(os.environ["GDRIVE_FOLDER_IDS"]).split(",")
credentials, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/drive"])
service = discovery.build('drive', 'v3', credentials=credentials)
media = MediaFileUpload(local_path)
print("Starting to upload in the drive")
response = service.files().create(media_body=media, body={"name": remote_path.split("/")[-1], "parents": GDRIVE_FOLDER_IDS}).execute()
print(response)
def download_file_from_gcs(bucket, key):
print("Downloading key: {} in bucket: {}".format(key, bucket))
client = storage.Client()
source_bucket = client.bucket(bucket)
blob_object = source_bucket.blob(key)
tmpdir = "/tmp/file"
blob_object.download_to_filename(tmpdir)
return tmpdir
def drive_upload(event, context):
try:
if event["name"][-1] == "/":
print("Folder: {} is created in the bucket: {}".format(
event["name"],
event["bucket"]
))
return
# Downloading file from GCS
local_path = download_file_from_gcs(event["bucket"], event["name"])
# Uploading the file to drive
upload_to_drive(local_path, event["name"])
except Exception as e:
print(traceback.format_exc())
Also create requirements.txt file and paste following dependencies into it
google-api-python-client
google-cloud-storage
google-auth
Run the following command to deploy the function
gcloud functions deploy gcs-to-gdrive --region us-central1 --entry-point drive_upload --runtime python38 --service-account gcs-drive-function-sa@quiet-antler-350415.iam.gserviceaccount.com --trigger-resource gs://source-bucket-41ljczocug --trigger-event google.storage.object.finalize --set-env-vars GDRIVE_FOLDER_IDS=1hvU85TjN9_Qpefba-YKZoCxoDaL_FgCN
Replace folder ID with the desired folder ID in the above command.
Testing
Upload file in the bucket and check whether the file is uploaded in the drive.
Source GCS Bucket
GCS Source Bucket
Destination Drive Folder
Destination Drive Folder
If you find errors in the uploading file to drive, check the function logs.
If you like this post, do share it with your colleagues.
I am an Experienced Engineer with a strong IT background and a passion for technology exploration. My journey started in college, where I went into web development, AI/ML, and more. I've worked with GCP and AWS, holding certifications in both and being proficient in Kubernetes, Docker, and Terraform. This blog is where I write about my technical expertise.
Stay up to date