Off-loading Terabytes of Videos to YouTube
November 7th, 2016 | Back to Blog Listing


The YouTube Logo
When I first sat out to design and build kevinludlow.com, I did so under the impression that it wouldn't be long before every single thing that we had was digitally catalogued. I did so under the impression that it would become increasingly more difficult to protect oneself from having information digitally catalogued.

I'm not referring to police records and tax documents. I'm referring to that picture of you where you've had six too many to drink, your eyes are bloodshot, and there's still some puke running down your lip from when you last vomited. These are moments that we try dearly to make disappear, but they're not going to disappear. The second an iPhone or Android snaps a picture like that, not only is the content saved in an archive forever (yes, even if you delete it), but the META-data attached to the photo is frightening. There is a full story of exactly where you were, what you were doing, how long you were there for, and everything else in between included in that META-data.

I wouldn't exactly say that my attitude followed the old cliché of, "If you can't beat em, join em", but it was along those lines. The idea of my website was that if this is where we're headed, then perhaps it's just best to embrace those moments and own them as your own. So what if I had too many to drink one night. It happens. If anybody is going to own a picture of that moment, I'm going to. I'm going to be the one to make it publicly accessible for all to see. If someone should ever try to use it against me, I'm going to politely remind them that they got the image from my own personal archive. There's a power to this and I encourage everyone to embrace it.

That's the back story.

Along with that has come hundreds of hours of video footage that I've recorded over my lifetime. FINALLY, I have finished the segment of my site whereby I can archive all of my video footage not only on my own personal website, but on YouTube.

Google, like with most of their tools, has created a wonderful API for connecting with your YouTube channel and uploading videos to it. I've taken full advantage of this and in the past month or so have uploaded about 1,500 videos to their service as part of the initial experiment. Now that I finally have the tools to do it, I probably have 200-300 more hours of video that I will fully digitize and send to YouTube.

My process is simple. I start by placing all of my media content in the regular directory it falls on within my server. Think of it like a giant iPhoto gallery. It's just an enormous directory tree with about 90,000 unique memories in it (as of this writing). From there my servers run a series of automated processes that I've developed over the years. Photos are converted into different sizes and resolutions as needed by my software package. Videos are converted into OGG format (open-source video for the web) so that I can display them using HTML5 on this site. Thumbnails from the videos are automatically extracted and organized. Each entity that is created within my system is given a unique ID and catalogued within my database. There is a lookup table that will cross reference any of these unique IDs in the entire system with a specific file type (photo, video, etc.)

Focusing solely on the videos, at this point in my process there exist videos on my server that have thumbnails, that have been converted to a useful (and view-able) web format, but that lack any titles or description. I quickly scroll over the video, provide it with a title and a description, and so long as I'm okay archiving it within YouTube, I check a box and it gets marked for upload.

A standard CRON job on my server is routinely scanning the database for videos that have been approved, but not yet uploaded. Once it finds such a video, the YouTube API is invoked and the video gets uploaded to YouTube. All of the information about the video that has been populated in my database (such as the title and description) is sent over with the video for YouTube to archive. The date/time that the video goes over is logged in my database as is the unique video code that YouTube provides. From there I can reference any video in my personal catalogue by looking up the YouTube ID myself.

While I've not yet configured the videos as a YouTube plugin on the site, this is the obvious next step. Not only does it take cost off of my server, but YouTube's servers are likely significantly faster than mine.

That's the gist of how it works. I've wanted to do this for about 7 years now. I'm delighted to have finally finished the coding that makes it work.

Incidentally, if you go to my YouTube channel, you find thousands of videos, all of which originated on this site.