FreeBSD manual multipath script

I recently ran into an issue installing FreeBSD on a system that already had some disks & zpools. Because the disks were partitioned previously, automatic multipath was not an option as the last sector of all hard drives isn’t available to store an ID. The remaining option is to do manual multipath, and it needs to be done every time the system boots.

Here’s an rc script that will run early in the sequence and create a multipath “link” between drives based on their serial number.

/etc/rc.d/manual_multipath

#!/bin/sh

# PROVIDES: manual_multipath
# REQUIRE: sysctl
# BEFORE: hostid

. /etc/rc.subr

name="manual_multipath"
start_cmd="${name}_start"
stop_cmd=":"

manual_multipath_start()
{
        echo "> manual_multipath script started"
        echo "> linking drives with the same serial number with gmultipath"
        counter=0
        serials=""
        devices=`/usr/bin/find /dev -maxdepth 1 -regex '.*da[0-9]*' | /usr/bin/cut -d '/' -f 3`
        for device in $devices
        do
                echo $device
                serial=`camcontrol inquiry $device -S`
                substring=`echo "$serials" | /usr/bin/sed -n "s/|$serial|.*//p" | /usr/bin/wc -c`
                if [ $substring -eq 0 ]
                then
                        found_multi=0
                        arg1="$device"
                        arg2="$device"
                        for newdevice in $devices
                        do
                                newserial=`camcontrol inquiry $newdevice -S`
                                if [ "$device" != "$newdevice" -a "$serial" == "$newserial" ]
                                then
                                        echo "  same as $newdevice!"
                                        counter=`expr $counter + 1`
                                        found_multi=1
                                        arg1=$arg1"$newdevice"
                                        arg2=$arg2" $newdevice"
                                fi
                        done
                        if [ $found_multi -eq 1 ]
                        then
                                gmultipath create $arg1 $arg2
                        fi
                fi
                serials=$serials"|$serial|"
        done
        echo "> manual_multipath script finished, found $counter matches"
}

load_rc_config $name
run_rc_command "$1"

Don’t forget to “chmod 555 /etc/rc.d/manual_multipath”.

Lastly, when importing a zpool from the drives you just multipathed, make sure to specify where to look for devices or you might end up importing a mix of multipath and regular devices. Make sure to “zpool import -d /dev/multipath”.

I’m delving pretty deep into FreeBSD, time to grow an epic beard.

More Dr. Meter fun

Finger

Arm

Hair

Salt

Nutella (gross)

Peanut

Serrated knife blade

Ballpoint pen

Printed beer logo

Cloth

Melting snow

[flv:http://ben.akrin.com/wp-content/uploads/2013/02/temp.flv http://ben.akrin.com/wp-content/uploads/2013/02/Screen-Shot-2013-02-24-at-11.53.58-PM.png 640 426]

Google drive API file upload script

If you want to upload file to Google Drive, you will naturally gravitate towards the Google Drive API. While reading about the APIs that Google publishes for almost anything, you will learn about their SDK. The SDK provides you with easy functions for interfacing with the API for various programming languages.

The problem is that the more I used the SDK the more confused I was. The documentation is often unclear, not all bindings are implemented across all languages. But above all, the thing that made me dislike the SDK is the fact that uploading a file to Google Drive took 5 times the amount of memory than the file itself. Meaning that the datastructure they use and how they pass it around is SUPER LAME. Not a huge deal if you’re uploading a few doc files but definitely crappy for GB range files. It’s just not right and completes the pattern of “meh” surrounding the SDK.

What is clear & consistently documented is every single API call you can make. It was time to go straight to the API. Sure there is still quite a bit of poking involved in getting something working exactly right but my experience with the API has been a lot better. It has also helped me understand the mindset & design so figuring out new things is much faster.

How?

HTTP, every API call uses it. You can use any technology that you are familiar with to do your HTTP call but I have a penchant for PHP + cURL.

The script

It will read a file in chunks and upload them consecutively. Do to so it uses the API’s resumable uploads. As such it will never consume more RAM that the configurable chunk size.

download

It still needs a few improvements at the moment but it’s functional.

What it solves

  • authentication: getting a new access token from a refresh token as needed
  • curl custom HTTP requests with custom headers
  • file chunking
  • Google Drive API resumable upload
  • passing JSON encoded requests in body
  • exponential backoff

Doesn’t sound like much but it took a while to piece it all together.