Choose color scheme

About the Author

544 Posts By ben

  • Using Google’s APIs with Python scripts

    I was never able to find centralized, succinct and example based documentation for doing domain delegated API calls with Google. Hopefully here is exactly this documentation from all the pieces I gathered along the way.

    Service Account Creation

    1. Go to and create a new project.
      Screen Shot 2016-03-15 at 10.11.31 AM
    2. Call it whatever you want
      Screen Shot 2016-03-15 at 10.11.48 AM
    3. Enable the right APIs that this project will use We’ll do drive API for the purpose of this testing
      Screen Shot 2016-03-15 at 10.15.06 AMScreen Shot 2016-03-15 at 10.16.40 AMScreen Shot 2016-03-15 at 10.16.48 AM
    4. Go to the “Credentials” screen
      Screen Shot 2016-03-15 at 10.18.08 AM
    5. Create a “Service Account Key”
      Screen Shot 2016-03-15 at 10.18.33 AM
    6. Make it a “New service account” and give it a nameScreen Shot 2016-03-15 at 10.19.17 AMScreen Shot 2016-03-15 at 10.20.09 AMScreen Shot 2016-03-15 at 10.23.45 AM
    7. Download that JSON file that follows.
      Screen Shot 2016-03-15 at 10.23.53 AM
      This file contains the credentials for the account you just created, treat it with care, anyone getting their hands on it can authenticate with the account. This is especially critical since we are about to grant domain delegation to the account we created. Any one with access to this file is essentially able to run any API call masquerading as anyone in your Google Apps domain. This is for all intents and purposes a root account.

    Domain Delegation

    1. Back on the “Credentials” screen, click “Manage service accounts”
      Screen Shot 2016-03-15 at 10.26.43 AM
    2. Edit the service account you just created
      Screen Shot 2016-03-15 at 10.28.23 AM
    3. Check the “Enable Google Apps Domain-wide Delegation” checkbox and click “Save”.
      Screen Shot 2016-03-15 at 10.30.28 AM
      Google at this points needs a product name for the consent screen, so be it.
    4. At this point, if everything went well, when you go back to the “Credentials” screen, you will notice that Google create an “OAuth 2.0 client ID” that is paired with the service account you created.

    Domain delegation continued, configuring API client access

    Granting domain delegation to the service account as we just did isn’t enough, we now need to specify the scopes for which the account can request delegated access.

    1. Go to your Google Apps domain’s Admin console.
    2. Select the Security tabScreen Shot 2016-03-09 at 11.15.40 AM
    3. Click “Show more” -> “Advanced Settings” Screen Shot 2016-03-09 at 11.15.52 AM
    4. Click “Manage API client access Screen Shot 2016-03-09 at 11.16.08 AM
    5. In the “Client Name” field, use the “client_id” field from the json file you downloaded earlier. You can get it via the following command:
      cat ~/Downloads/*.json | grep client_id | cut -d '"' -f4

      In the “One or More API Scopes” field use the following scope:

      Screen Shot 2016-03-15 at 11.00.36 AM
      If you want to allow more scopes], comma separate them. This interface is very finicky, only enter URLs and don’t copy/paste the description that show up for previous entries. There also might be a few minutes delay between you granting a scope and its taking effect.

    6. Click “Authorize”, you should get a new entry that looks like this:
      Screen Shot 2016-03-15 at 11.01.51 AM
      If you need to find the URL for a scope, this link is helpful.

    Scripting & OAuth 2.0 authentication

    Okay! The account is all set up on the Google side of things, let’s write a Python script to use it. Here’s your starting point:

    This scripts contains all the functions to get you started with making API calls to Google with Python. It isn’t the simplest form it could be presented in but it solves a few issues right off the bat:

    • All Google interactions are in the “google_api” class, this allows for efficient use of tokens. When “subing as” a user in your domain, the class will keep track of access tokens for users and only re-generate them when they expire.
    • Exponential back-off is baked-in and generalized to anything unusual gotten back from Google (based on observation).
    • SIGINT will get handled properly

    Before running the script, you may need to:

    sudo apt-get update && sudo apt-get install python-pycurl

    Running the script is done as such:

    ./ /path/to/json/file/you/downloaded/earlier.json

    It will simply run the “get about” Drive API call and print the result. This should allow you to verify that the call was indeed executed as the account you specified in the arguments.

    Once you’ve ran this script once, the sky is the limit, all the Drive API calls can be added to it based on the get_about function.

    Important note on scopes: the same way that you granted domain delegation to certain comma separated scopes in the Google Apps Admin Console earlier; this script needs to reflect the scopes that are being accessed and the same space separated list of scopes need to be part of your jwt claim set (line 78 of the script). So if you need to make calls against more than just drive, make sure to update scopes in both locations or your calls won’t work.

    More scopes & more functions

    Taking it one step further with the Google Enforcer. This is the project that lead me down the path of writing my own class to handle Google API calls. While it is not quite ready for public use, I’m publishing the project here as it is an excellent reference to making all kinds of other Google API calls; some doing POSTs, PUTs, DELETEs, some implementing paging, et cetera.


    The purpose of this project is to enforce on the fly permissions on a directory tree. There is a extravagant amount of gotchas to figure out to do this. If you are interested in implementing it with your organization, please leave a comment and I can either help or get it ready for public use depending on interest.

    This project works towards the same end as AODocs, making Google Drive’s permission not completely insane as they are by default.

    Here are the scopes I have enabled for domain delegation for this project.

    Screen Shot 2016-03-15 at 4.55.25 PMProblems addressed by this project:

    • domain account “subbing as” other users AKA masquerading
    • a myriad of Google Drive API calls focused on file permissions
    • watching for changes
    • crawling through directory hierarchy
    • threading of processes to quickly set the right permissions
    • disable re-sharing of files
    • access token refreshing and handling
    • exponential back-off
  • From 380′ into the earth

    Not exactly gushing out but the fact that it reaches ground level will make a hand pump quite doable. The whole experience was quite a roller-coaster of emotions. I find myself staring in fascination at the water oozing out.IMG_3603

  • IPv6 link-local address to MAC address online converter

    The converter

    It can also be addressed directly via:
    for all your API needs.


    This converter was implemented per Dave Russell’s suggestion as a follow up to the MAC address to IPv6 link-local address online converter. If you are interested in the steps behind this conversion, they are simply a reverse of the original Mac->IPv6 converter.

    Please note that of the various IPv6 notations, the one this script will expect is fe80::xxxx:xxxx:xxxx:xxxx.

  • Getting a well drilled

    Unfortunately not a whole lot of water at 300ft. We’ll see where static is tomorrow and decide if we go deeper.IMG_3600

    Very impressive piece of machineryIMG_3597Not a very green process.

  • Swoosh

    It’s hard to tell what happened last night, let’s just say we felt like the roof was going to be ripped from the house. We felt every gust of wind move the house, I have never experienced anything like it. We slept downstairs with the windows covered that’s how bad it was. Everything was a mess outside the following morning including things I never thought could be brought down by the wind.

    No bees in it yet thankfully.


    Fuck you wind! Not the outside shower!IMG_3519

    I ended up outside in the middle of it to weight down a couple of things, it was surreal. Today at town meeting, I’ve talked with other people from our hill who experienced the same crazy night.

  • The little evaporator that could

    We fired our 2’x3′ Mason evaporator for our first boil ever. We boiled ~80 gallons in 12 hours. We’ve learned lots of lessons and will have to make several improvements to our current setup. It’s still very cool that we are now able to produce more than enough maple syrup to cover our family’s yearly consumption.FullSizeRender(2)


    The smell is great

    2016-02-29 21_51_57

    Boiling well into the night

    IMG_3491We still need to finish the syrup on our cookstove, filter it, and can it.

  • Remove all Exif data from JPEGs recursively

    Because I always spend 20 minutes googling it

    apt-get update && apt-get install libimage-exiftool-perl
    find /var/www -type f -iname *.jpg -exec exiftool -all= {} \;
  • Found a small spring

    springI’ll keep an eye on it through the Summer. It could prove very useful down the road.

  • Porcupine still around

    We also found tons of pine twigs on the ground. Maybe we’ll meet one day.IMG_3330