PHP file upload to a Google storage bucket

Download

bucket_upload_1.0.php.gz

Google setup & use

1- Create a storage bucket for the script to upload into

 

Go to the Google Cloud Console, click on “Storage”, “Browser”.

 

“Create Bucket”

 

Give it a name and click “Create”.

 

2- Create a service account for the script

Expand the “IAM & admin” section, click on “Service accounts”.

 

Click “Create Service Account”.

 

Give it a name, check “Furnish a new private key”, JSON, and click “Save”.

 

Save the JSON credentials file which you are prompted to download into a safe location.

3- Grant “Object Creator” permissions on the bucket to the service account

Go back to the storage bucket you created

 

Edit its permissions

 

The JSON credentials file you just downloaded contains the email for the service account you created, copy it.

 

And paste it into the “Add members” field, select the permission to be “Storage Object Creator”. This service account doesn’t need permissions for anything else than dumping files in there. Not even viewing them.

 

Optional: if you want the files uploaded by the script to be publicly viewable, add the permission “Storage Object Viewer” to the user “allUsers”. Accounts are all referred to by email in Google land, but there exist special keywords such as “allUsers”.

Done with the Google setup 🙂

4- Running the script

If you haven’t already, download the script at the top of this page. Decompress it and edit the config at the top.

$credentials_file_path is the full path to the JSON credentials file you got from Google when you created the service account. It should be a secure location.

$destination_bucket_name is the name of the bucket you created

$access_token_cache_file_path is a location where Google’s OAuth tokens are cached, it too should be a secure location.

Run the script with only 1 argument being the file you want to upload. The script can also be included and used outside of CLI, in that case simply call the upload( $filename ) function.

The script returns the URL to the file in the bucket.

VoilĂ :

Using Google's APIs with Python scripts

I was never able to find centralized, succinct and example based documentation for doing domain delegated API calls with Google. Hopefully here is exactly this documentation from all the pieces I gathered along the way.

Service Account Creation

  1. Go to https://console.developers.google.com/start and create a new project.
    Screen Shot 2016-03-15 at 10.11.31 AM
  2. Call it whatever you want
    Screen Shot 2016-03-15 at 10.11.48 AM
  3. Enable the right APIs that this project will use We’ll do drive API for the purpose of this testing
    Screen Shot 2016-03-15 at 10.15.06 AMScreen Shot 2016-03-15 at 10.16.40 AMScreen Shot 2016-03-15 at 10.16.48 AM
  4. Go to the “Credentials” screen
    Screen Shot 2016-03-15 at 10.18.08 AM
  5. Create a “Service Account Key”
    Screen Shot 2016-03-15 at 10.18.33 AM
  6. Make it a “New service account” and give it a nameScreen Shot 2016-03-15 at 10.19.17 AMScreen Shot 2016-03-15 at 10.20.09 AMScreen Shot 2016-03-15 at 10.23.45 AM
  7. Download that JSON file that follows.
    Screen Shot 2016-03-15 at 10.23.53 AM
    This file contains the credentials for the account you just created, treat it with care, anyone getting their hands on it can authenticate with the account. This is especially critical since we are about to grant domain delegation to the account we created. Any one with access to this file is essentially able to run any API call masquerading as anyone in your Google Apps domain. This is for all intents and purposes a root account.

Domain Delegation

  1. Back on the “Credentials” screen, click “Manage service accounts”
    Screen Shot 2016-03-15 at 10.26.43 AM
  2. Edit the service account you just created
    Screen Shot 2016-03-15 at 10.28.23 AM
  3. Check the “Enable Google Apps Domain-wide Delegation” checkbox and click “Save”.
    Screen Shot 2016-03-15 at 10.30.28 AM
    Google at this points needs a product name for the consent screen, so be it.
  4. At this point, if everything went well, when you go back to the “Credentials” screen, you will notice that Google create an “OAuth 2.0 client ID” that is paired with the service account you created.

Domain delegation continued, configuring API client access

Granting domain delegation to the service account as we just did isn’t enough, we now need to specify the scopes for which the account can request delegated access.

  1. Go to your Google Apps domain’s Admin console.
  2. Select the Security tabScreen Shot 2016-03-09 at 11.15.40 AM
  3. Click “Show more” -> “Advanced Settings” Screen Shot 2016-03-09 at 11.15.52 AM
  4. Click “Manage API client access Screen Shot 2016-03-09 at 11.16.08 AM
  5. In the “Client Name” field, use the “client_id” field from the json file you downloaded earlier. You can get it via the following command:
    cat ~/Downloads/*.json | grep client_id | cut -d '"' -f4

    In the “One or More API Scopes” field use the following scope:

    https://www.googleapis.com/auth/drive

    Screen Shot 2016-03-15 at 11.00.36 AM
    If you want to allow more scopes], comma separate them. This interface is very finicky, only enter URLs and don’t copy/paste the description that show up for previous entries. There also might be a few minutes delay between you granting a scope and its taking effect.

  6. Click “Authorize”, you should get a new entry that looks like this:
    Screen Shot 2016-03-15 at 11.01.51 AM
    If you need to find the URL for a scope, this link is helpful.

Scripting & OAuth 2.0 authentication

Okay! The account is all set up on the Google side of things, let’s write a Python script to use it. Here’s your starting point:

google_api_script.py

This scripts contains all the functions to get you started with making API calls to Google with Python. It isn’t the simplest form it could be presented in but it solves a few issues right off the bat:

  • All Google interactions are in the “google_api” class, this allows for efficient use of tokens. When “subing as” a user in your domain, the class will keep track of access tokens for users and only re-generate them when they expire.
  • Exponential back-off is baked-in and generalized to anything unusual gotten back from Google (based on observation).
  • SIGINT will get handled properly

Before running the script, you may need to:

sudo apt-get update && sudo apt-get install python-pycurl

Running the script is done as such:

./google_api_script.py /path/to/json/file/you/downloaded/earlier.json account.to.subas@your.apps.domain

It will simply run the “get about” Drive API call and print the result. This should allow you to verify that the call was indeed executed as the account you specified in the arguments.

Once you’ve ran this script once, the sky is the limit, all the Drive API calls can be added to it based on the get_about function.

Important note on scopes: the same way that you granted domain delegation to certain comma separated scopes in the Google Apps Admin Console earlier; this script needs to reflect the scopes that are being accessed and the same space separated list of scopes need to be part of your jwt claim set (line 78 of the script). So if you need to make calls against more than just drive, make sure to update scopes in both locations or your calls won’t work.

More scopes & more functions

Taking it one step further with the Google Enforcer. This is the project that lead me down the path of writing my own class to handle Google API calls. While it is not quite ready for public use, I’m publishing the project here as it is an excellent reference to making all kinds of other Google API calls; some doing POSTs, PUTs, DELETEs, some implementing paging, et cetera.

Download:
google_drive_permission_enforcer_1.0.tar.gz

The purpose of this project is to enforce on the fly permissions on a directory tree. There is a extravagant amount of gotchas to figure out to do this. If you are interested in implementing it with your organization, please leave a comment and I can either help or get it ready for public use depending on interest.

This project works towards the same end as AODocs, making Google Drive’s permission not completely insane as they are by default.

Here are the scopes I have enabled for domain delegation for this project.

Screen Shot 2016-03-15 at 4.55.25 PMProblems addressed by this project:

  • domain account “subbing as” other users AKA masquerading
  • a myriad of Google Drive API calls focused on file permissions
  • watching for changes
  • crawling through directory hierarchy
  • threading of processes to quickly set the right permissions
  • disable re-sharing of files
  • access token refreshing and handling
  • exponential back-off