Getting rid of Google Photos - Part 1

In my quest of de-google-ifying my day to day operations I wanted to replace Google Photos and take back the control of my own pictures.

Yes Google Photos is efficient, easy to use and just works™ but this comes to the price of handing over your pictures to a company who already know a tad too much about you and now get the extra access to where you were, when, and to whatever is present in the pictures.

As stated here it is also not as easy as querying an API to get back your pictures in full resolution with the original location:

Google Photos used to have an API to do this (the Picasa Web Albums API) but they removed it, replacing it with a new API that doesn't let you download your original photos. They instead let you download your photos with mangled EXIF, stripping location (and maybe recompressing the image bytes?).

Downloading the pictures

I've found two solutions to this, either via Google Takeout, or using a tool that download one by one all your pictures on Google Photos. The first is useful to get all of the pictures in one easy go. The second one is good if you want to just get the last pictures you've uploaded every few weeks as it keeps track of the last retrieved picture. The script for the second one unfortunately fails to extract the correct date if the picture itself does not contain any exif, instead it will default to today's date. It is not a problem if all the pictures you have on Google Photos are taken with your camera, it is otherwise an issue if you want to upload also pictures / screenshots sent to you via instant messaging apps as they usually strip the exif. Basically the tool does not take into account the date in the "details" section of Google Photos. Google Takeout in the other end also give a .json for each pictures with all the info they have (usually the upload date which if it's not the actual date the picture was taken it is at least the date you received it).

via Google Takeout

Ask for your Google Photos data in here and wait for the download link in your email (mine took 20 minutes). Once you receive the archive just follow the instruction in this repository and it should just work out of the box. I did had three pictures where the exif data couldn't be read as the date seemed to be invalid:

time data '2014:11:06 24:07:57' does not match format '%Y:%m:%d %H:%M:%S'

To fix that I installed exiftool and used grep to find which exif tag was wrongly formatted

exiftool -s FILE | grep 2014:11:06 24:07:57

And then just removed the tag altogether (whatever the tag is it needs to be written lowercase without spaces)

exiftool -overwrite_original -o -thefaultytag= FILE

via gphotos-cdp

1. Clone the following repo https://github.com/perkeep/gphotos-cdp

There's currently an issue regarding running the tool where I go a bit more in detail here

2. go run main.go and take note of the session dir

Session Dir: /var/folders/FOO/BAR/T/gphotos-cdpSOMENUMBER

3. Run the following and log in with your Google account

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --no-first-run \
  --password-store=basic \
  --use-mock-keychain \
  --user-data-dir= /var/folders/FOO/BAR/T/gphotos-cdp \
  https://photos.google.com/

4. go run main.go --dev in order to re-use the same user-data-dir instead of generating a new one

I left this running in the background but it does crash quite often so it needs to be restarted, thanksfully it starts again where it stopped. I had to download 8 years of pictures (8k+) and it took quite some hours but I still found it to be pretty fast. It should have generated a fair quantity of folders with random names but I wanted to organise it a bit so looked for a script that will allow me to do that

Sort the pictures

I found this one first but if I had to do it again I would probably use something more robuts and maintained, this look like it would have been a better choice.

The script is pretty straight forward and easy to use with quite some options. Unfortunately it hasn't been updated in years and I had to do a couple modifications for it to work for my use case

git clone git@github.com:andrewning/sortphotos.git

python3.8 setup.py install

I first tried to run it in test mode -t to see if everything work as expected but quickly saw the following error on pretty much all my pictures

sortphotos /Users/USERNAME/Downloads/gphotos-cdp /Users/USERNAME/Pictures -tr

Can't locate Image/ExifTool/Motorola.pm in @INC (you may need to install the Image::ExifTool::Motorola module) (@INC contains: /usr/local/lib/python3.8/site-packages/sortphotos-1.0-py3.8.egg/src/Image-ExifTool/lib /Library/Perl/5.18/darwin-thread-multi-2level /Library/Perl/5.18 /Network/Library/Perl/5.18/darwin-thread-multi-2level /Network/Library/Perl/5.18 /Library/Perl/Updates/5.18.4 /System/Library/Perl/5.18/darwin-thread-multi-2level /System/Library/Perl/5.18 /System/Library/Perl/Extras/5.18/darwin-thread-multi-2level /System/Library/Perl/Extras/5.18 .) at (eval 1059) line 2, <EXIFTOOL_FILE> chunk 3.
Can't find table Image::ExifTool::Motorola::Main
Unknown tag table Image::ExifTool::Motorola::Main at /usr/local/lib/python3.8/site-packages/sortphotos-1.0-py3.8.egg/src/Image-ExifTool/lib/Image/ExifTool/Exif.pm line 5946, <EXIFTOOL_FILE> chunk 3.

Indeed I've used a Motorola phone to take pictures most of the last few years and it seems the tool is missing the Motorola table to be able to read the exif

Looking at the open PR it seems someone updated the exif tables

git fetch origin pull/123/head:BRANCHNAME
git checkout BRANCHNAME
python3.8 setup.py install 
source ~/.zshr

This time all is good and it seems to not throw any error

So time to run the tool properly

sortphotos /Users/USERNAME/Downloads/gphotos-cdp /Users/USERNAME/Pictures/personal -crs --sort %Y/%m_%B --rename %Y_%m_%d_%H%M%S

Here I use a bunch of different options -c is to copy so I can actually keep the original photos I downloaded so if something is wrong I don't have to re-download everything and can start fresh easily. I decided to sort by year and also month, still keeping the month number so it is sorted by default and also I can eventually pick the month easily with a script later without relying on the month name. For the pictures I used the classic year_month_day_hoursecond which makes it easy to see quickly when a picture was taken and avoid too much name collisions (the tool append a number if there's a name collision anyway). -r is for recursive

I did discover though that I had a few pictures in folders by years that didn't make sense such as 0001, 0002... Turns out there's another bug highlighted here. Thanksfully it's only a one line change so updated the file and re-installed...

Just had a sanity check to see if it didn't skip any photos as the tool would just ignore a photo without warning if it can't process it I ran

find DIR_NAME -type f | wc -l

in both the downloaded picture folder and the new destination to verify I had the same number of pictures

After that it was working perfectly and I just had to transfer all those pictures to my NAS.

Next part will probably be about Photoprism