I’m interested in screenshots.  For me, they represent things I’m too lazy to write into my Evernote (not a big fan of typing stuff into there anyway), things I want to share with my friends and family, things I want to remind myself of, and things I just plain want to save (because I’m kind of a hoarder – that’s what happens when you don’t grow up with much).  A friend of mine tells me he uses screenshots to take pics of stuff he wants to share and laugh about with his friends.  I did some googling and came across this article in Wired, declaring the screenshot the “most important thing on the internet.”  “It’s photography for life on the screen” writes Clive Thompson, and it is clear that most of us are spending a lot more of our lives (around 4 hours a day!) interacting with our phones.

I want to count the number of screenshots I’ve taken over the years that are stored in Google Photos.  Doing so will allow me to also access each screenshot on its own, and by achieving this programmatically, I should be able to do interesting things – such as storing the screenshots in an array and filtering them by certain attributes discoverable by the Google Vision API.  Maybe I can even connect different people based on the content of their screenshots.

So, to start, I headed over to my Google Photos and to my luck, found that it automatically sorts photos by type.  I was in Mexico City for a business school course, and Google already recognizes the geotags.  It can also sort photos by type (selfie, screenshot, video, etc.)

Great, let’s see my screenshots then.

My uber driver was recommending restaurants, which I googled and captured (past tense of verb form of screenshot) to look at later.  But, I couldn’t help but notice that Google thinks I only took one screenshot yesterday (9/2) and none between July 22 and 9/2.  That can’t be right…what’s going on here?

For a sanity check, I selected screenshots from my phone’s Google Photos app, and took a screenshot of the output.  The left screenshot shows what Google thinks are screenshots.  The right shows what happens when you click those little upside down carrots on each date, which show the other photos uploaded to the cloud that day.  Clearly, there are other screenshots Google Photos is not picking up…dang.  Why?

This leads me to my first learning:

Google Photos tries, but is still incapable of accurately classifying screenshots from regular photos.

How can this be?  How hard can it be to tell whether an image is a screenshot or a live photo?  After a few minutes of Googling, I still didn’t have an answer.  Annoying.  I did another google search “do screenshots get uploaded to Google photos”, and the first two hits had to do with people not wanting their screenshots backed up and uploaded to Google photos.  For them, screenshots were ways for them to share something quickly with a friend, and they wanted them removed right after, rather than persisted to the cloud.  The community specialists in charge of the Google help forum replied that Google Photos backs up everything, including your screenshots.  But, the screenshots are saved to a device folder, and you can indicate that this device folder’s contents should not be backed up and synced to the cloud.  Naturally, I looked for this device folder on my phone, and found this:

I think I see the screenshots I’m expecting to see.  I checked this against Google Photos on my laptop and it checks out.  So now I’m led to the second conclusion:

On laptop: After selecting screenshots as a classification, Google Photos does not show all screenshots.

On mobile: After going into the screenshots device folder, all screenshots (until Jan 13 2017) are ostensibly visible.  I wonder what happened to the older ones, though I’m sure I had them archived to free up space on my phone (as they should be backed up anyway).

So, Google Photos on mobile does a great job classifying and storing my screenshots prior to syncing them to the cloud.  Instead of storing these photos in a folder in the cloud, it mixes them with all my synced photos, and then attempts to classify them again if I ask it to (the dropdown I referred to earlier).  This misses some screenshots.

Something else I learned – in terms of dimensions, the screenshots I took on my Pixel XL are exactly 1440 × 2560 pixels (if I do not alter them in any way).  My previous phone’s (Samsung Note 7, Google Nexus 5) will have different screen sizes and thus different dimensions.  Maybe I can use these dimensions to grab my screenshots later, but this is still kind of a headache – is this how Google determines a screenshot from regular photo one takes?

Well, I’ll need to investigate that a little later, but for the time being, let me return to my original goal: to count the number of screenshots I have backed up to my cloud, which go all the way back to 2014.  Now, Google Photo is smart – it’s not gonna load all these photos at once because that’d take a lot of time.  Imagine if everytime you visited your facebook app, every newsfeed item ever had to load, even if you didn’t scroll down.  Disaster. Instead, as a user scrolls down, images appear. This is called lazy loading.

Unfortunately, thanks to lazy loading, I can’t get an accurate count of my screenshots because only a certain number appear when I first get to the Google Photos screenshots page.  To overcome this, I had to zoom out as far as possible (30% in IE on a larger monitor), so that all photos are shown, and then use some nifty jQuery to select all screenshots, each of which contains at least one unique identifier (I used jsname=”NwW5ce”).  The actual query is

document.querySelectorAll(‘[jsname=”NwW5ce”]’)

which returns a Nodelist of length 1903, which I take to be 1903 “screenshots”.

Now, since I know that at least a few screenshots are missing (none from August 28th 2017 show up for some reason), and that a few of these images interestingly enough not screenshots, then I presume the total number of screenshots will end up being around 2000-2200.  Pretty upset that I can’t get all the screenshots I want, but will tackle this another day.

From the Nodelist, I’m going to need to convert it into an array, loop through each element (image), looking for its firstChild – an anchor element <a> with an href that points to the screenshot’s url.  Then, I just need to pass this to the Google vision API or store it in a database for future work.

Will get on this hopefully by the end of labor day weekend.