Off-loading Terabytes of Videos to YouTube
November 7th, 2016 | View Post


The YouTube Logo
When I first sat out to design and build kevinludlow.com, I did so under the impression that it wouldn't be long before every single thing that we had was digitally catalogued. I did so under the impression that it would become increasingly more difficult to protect oneself from having information digitally catalogued.

I'm not referring to police records and tax documents. I'm referring to that picture of you where you've had six too many to drink, your eyes are bloodshot, and there's still some puke running down your lip from when you last vomited. These are moments that we try dearly to make disappear, but they're not going to disappear. The second an iPhone or Android snaps a picture like that, not only is the content saved in an archive forever (yes, even if you delete it), but the META-data attached to the photo is frightening. There is a full story of exactly where you were, what you were doing, how long you were there for, and everything else in between included in that META-data.

I wouldn't exactly say that my attitude followed the old cliché of, "If you can't beat em, join em", but it was along those lines. The idea of my website was that if this is where we're headed, then perhaps it's just best to embrace those moments and own them as your own. So what if I had too many to drink one night. It happens. If anybody is going to own a picture of that moment, I'm going to. I'm going to be the one to make it publicly accessible for all to see. If someone should ever try to use it against me, I'm going to politely remind them that they got the image from my own personal archive. There's a power to this and I encourage everyone to embrace it.

That's the back story.

Along with that has come hundreds of hours of video footage that I've recorded over my lifetime. FINALLY, I have finished the segment of my site whereby I can archive all of my video footage not only on my own personal website, but on YouTube.

Google, like with most of their tools, has created a wonderful API for connecting with your YouTube channel and uploading videos to it. I've taken full advantage of this and in the past month or so have uploaded about 1,500 videos to their service as part of the initial experiment. Now that I finally have the tools to do it, I probably have 200-300 more hours of video that I will fully digitize and send to YouTube.

My process is simple. I start by placing all of my media content in the regular directory it falls on within my server. Think of it like a giant iPhoto gallery. It's just an enormous directory tree with about 90,000 unique memories in it (as of this writing). From there my servers run a series of automated processes that I've developed over the years. Photos are converted into different sizes and resolutions as needed by my software package. Videos are converted into OGG format (open-source video for the web) so that I can display them using HTML5 on this site. Thumbnails from the videos are automatically extracted and organized. Each entity that is created within my system is given a unique ID and catalogued within my database. There is a lookup table that will cross reference any of these unique IDs in the entire system with a specific file type (photo, video, etc.)

Focusing solely on the videos, at this point in my process there exist videos on my server that have thumbnails, that have been converted to a useful (and view-able) web format, but that lack any titles or description. I quickly scroll over the video, provide it with a title and a description, and so long as I'm okay archiving it within YouTube, I check a box and it gets marked for upload.

A standard CRON job on my server is routinely scanning the database for videos that have been approved, but not yet uploaded. Once it finds such a video, the YouTube API is invoked and the video gets uploaded to YouTube. All of the information about the video that has been populated in my database (such as the title and description) is sent over with the video for YouTube to archive. The date/time that the video goes over is logged in my database as is the unique video code that YouTube provides. From there I can reference any video in my personal catalogue by looking up the YouTube ID myself.

While I've not yet configured the videos as a YouTube plugin on the site, this is the obvious next step. Not only does it take cost off of my server, but YouTube's servers are likely significantly faster than mine.

That's the gist of how it works. I've wanted to do this for about 7 years now. I'm delighted to have finally finished the coding that makes it work.

Incidentally, if you go to my YouTube channel, you find thousands of videos, all of which originated on this site.

Configuring ImageMagick RAW Delegates with DCRAW and UFRAW-Batch
November 6th, 2015 | View Post

It's not very often that I take the time to make technical posts for all of the complexities that go into my personal website, but this one was so frustrating and time-consuming that I figured I would share my findings with the world. I've also been having to do a lot of ImageMagick work in my professional life lately and so I fully understand the frustration of not having good documentation on some of this.

The problem:

I have a bunch of .CR2 files in my photo gallery (RAW image files shot on a Cannon). I am using ImageMagick for a bunch of different processing components including resizing the images for my gallery. Unfortunately ImageMagick was failing due to UFRAW-batch not being found. The error would look something like this:

convert: delegate failed `"ufraw-batch" --silent --wb=camera --black-point=auto --exposure=auto --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i"' @ error/delegate.c/InvokeDelegate/1061.

After doing a ton of research and trying to hunt down ways to make ufraw-batch work with ImageMagick, I finally went down a different path of deciding to configure DCRAW as my RAW-file ImageMagick delegate. This method wound up working perfectly, but it does require some special configuring. I've detailed that process below.

Assumptions:

The following installation was done on an Amazon AMI instance (essentially a CentOS machine), with ImageMagick 6.7.8-9 2015-10-08 Q16 installed. It also assumes the user has ROOT access and that all of the steps are performed as ROOT.

Installing DCRAW:

  • If you haven't already, sudo to root:
    su root
  • Create a directory to work with somewhere in your home:
    mkdir ~/dcraw
    cd ~/dcraw/
  • Download DCRAW
    wget http://mirror.centos.org/centos/6/os/x86_64/Packages/dcraw-8.96-1.1.el6.x86_64.rpm
  • Install the RPM package
    rpm -Uvh dcraw-8.96-1.1.el6.x86_64.rpm


Installing DCRAW Dependencies (if necessary):

At this point you may find that you need to install the liblcms dependency. If this has happened then the installation of your RPM package will have failed. If DCRAW installed without any problems then skip to the libJpeg section below. If there was a dependency problem, take a look at the following:

  • Download liblcms:
    wget ftp://ftp.pbone.net/mirror/archive.fedoraproject.org/fedora/linux/releases/17/Fedora/x86_64/os/Packages/l/lcms-libs-1.19-5.fc17.x86_64.rpm
  • Install liblcms
    rpm -Uvh lcms-libs-1.19-5.fc17.x86_64.rpm
  • Attempt to install DCRAW again
    rpm -Uvh dcraw-8.96-1.1.el6.x86_64.rpm

Please note that there could certainly be other dependencies missing as well. This was simply a problem that I ran into with a pretty typical modern Linux installation.


Installing libJpeg (for cjpeg):

At this point DCRAW should be installed on your system. The best you'll really be able to do is to convert to either a TIFF file or get a raw stream of JPG. The problem with the RAW stream (assuming you're wanting to export to a jpeg as most people will want to do) is that it doesn't help you to actually compress the JPG stream into a usable file. If you try to simply pipe the STDOUT to a file (xxx.jpg for example), the image won't load as it won't be properly compressed for jpeg specs. You'll need to use cjpeg (or a similar tool) to accomplish this final piece.

  • Download cjpeg
    wget http://downloads.sourceforge.net/libjpeg-turbo/libjpeg-turbo-1.4.2.tar.gz
  • Unzip the gzip file and extract the tarball:
    gunzip libjpeg-turbo-1.4.2.tar.gz
    tar -xvf libjpeg-turbo-1.4.2.tar
  • Go into the newly extracted directory and install libjpeg:
    cd libjpeg-turbo-1.4.2
    ./configure
    make
    make install


Testing cjpeg:

At this point you should have DCRAW, any necessary dependencies, and cjpeg installed on your system. Incidentally, cjpeg will likely have been installed to /opt/libjpeg-turbo/bin/cjpeg. If you can't find it, do the following:

  • Update the mlocate database and find the jpeg tool:
    updatedb
    locate cjpeg

From here you'll have the fully qualified path of where the cjpeg executable lives. Mine was installed to /opt/libjpeg-turbo/bin/cjpeg (which I assume is a standard location for this program).


Configuring ImageMagick Delegates:

So the final part of this process is to configure the RAW delegate for ImageMagick so that instead of trying to use ufraw-batch it instead uses your newly installed dcraw. Be sure to backup your delegates.xml file before making any changes to the file! You can break your installation of ImageMagick if you're not careful!

  • Find your ImageMagick delegates file
    locate delegates.xml
  • Be sure to backup the delegates file first!
    cp /etc/ImageMagick/delegates.xml ~/
  • Open the file with your favorite editor:
    vim /etc/ImageMagick/delegates.xml
  • Search the document for ufraw-batch


Replace the ImageMagick Delegates:

That untouched line in the delegates.xml file should look something like this:

<delegate decode="dng:decode" command=""ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=%u.png" "%i""/>

You'll want to change it to look like this (using your path to cjpeg):

<delegate decode="dng:decode" command=""dcraw" -c -q 3 -H 5 -w "%i" | "/opt/libjpeg-turbo/bin/cjpeg" -quality 100 > "%o""/>
The most amazing server uptime!
October 26th, 2015 | View Post

I've been running a reasonably-sized farm of servers for almost a decade now. They were meticulously designed to meet the needs of my infrastructure and custom-secured to prevent intrusions. Given how long the boxes were up for I would have to say that they were pretty successful at their job.


The longest uptime I had at 5.97 years!


Slightly less at 4.35 years!


I had to install some software about 2 years ago and hence the mere 1.8 years of uptime.


The fourth server in my cabinet was reset with some regularity

Baba O'Riley with Ghost of Paul Revere
March 21st, 2015 | View Post

As part of my SXSW Broadmoor extravaganza with bands from all over the country, my headlining band (The Ghost of Paul Revere) played an unplugged version of The Who's "Baba O'Reily". It's definitely one of the better versions of the song I've heard and sounds great with the banjo and harmonica filling in the background pieces.

To anyone interested they'll be playing on April 23rd, 2015 at the Saxon Pub.

Workshopping "Tired of Trying" with The Ghost
March 21st, 2015 | View Post

As part of my 2015 music project I've been working on about 8 new songs (writing all of the music pieces and the lyrics). While I am collaborating with a dear friend of mine, I'm also trying to workshop the various songs with other people to get ideas on where to go with them.

With the Ghost of Paul Revere staying at my house for SX, we gave this affectionately titled song called "Tired of Trying" a little acoustic spin. Maybe the studio recording of it will have a banjo in the background?

We didn't rehearse any of it, but quickly went over the changes and improvised most of the pieces to it. This was also after a long day of substance abuse and was performed for two Portland girls I met along the way.

Acoustic Looping with a Violin Bow
March 17th, 2015 | View Post

I've recently been doing some experimenting with a violin bow after a friend of mine started teaching me some basics. Granted she is playing on an actual violin, but it translates to guitar fairly well.

The result was a harder sounding back beat with a nice wah-wah tick and a deep swell from the violin bow. I was basically just forming an E-chord and running the bow over the strings with heavy distortion to get the effect.

Newest Albums


Recent Posts
Off-loading Terabytes of Videos to YouTube
Configuring ImageMagick RAW Delegates with DCRAW and UFRAW-Batch
The most amazing server uptime!
Baba O'Riley with Ghost of Paul Revere
Workshopping "Tired of Trying" with The Ghost
Acoustic Looping with a Violin Bow
DADGAD Loop with LC2 and Telecaster
APD Brought the War on Drugs to my Front Door
3rd Challenge to Dawnna Dukes - State Income Tax
2nd Debate Challenge to Dawnna Dukes


Blog Archives
Recent Posts
November 2016
November 2015
October 2015
March 2015 ( 3 )
January 2015
October 2014
September 2014 ( 4 )
August 2014 ( 6 )
July 2014 ( 7 )
June 2014 ( 8 )
May 2014 ( 3 )
April 2014 ( 2 )
February 2014
January 2014 ( 4 )
December 2013 ( 4 )
November 2013 ( 2 )
September 2013
August 2013 ( 3 )
July 2013
June 2013
May 2013 ( 2 )
March 2013 ( 2 )
February 2013
January 2013
December 2012
November 2012 ( 4 )
October 2012 ( 2 )
September 2012 ( 4 )
August 2012
July 2012 ( 8 )
June 2012
May 2012 ( 6 )
April 2012 ( 7 )
March 2012 ( 4 )
February 2012 ( 5 )
January 2012 ( 4 )
December 2011 ( 5 )
November 2011 ( 2 )
October 2011
September 2011
August 2011 ( 5 )
July 2011 ( 6 )
June 2011 ( 2 )
May 2011 ( 3 )
April 2011 ( 3 )
March 2011 ( 2 )
February 2011
January 2011
December 2010 ( 2 )
November 2010 ( 2 )
September 2010
August 2010
June 2010
March 2010
February 2010 ( 3 )
November 2009
June 2009
May 2009 ( 3 )
April 2009
March 2009 ( 2 )
February 2009
December 2008
November 2008
October 2008
May 2008 ( 2 )
March 2008
January 2008
December 2007
July 2007 ( 2 )
June 2007 ( 2 )
May 2007
December 2006
October 2006 ( 3 )
July 2006
May 2006 ( 2 )
April 2006
December 2005
October 2005
September 2005 ( 5 )
August 2005
July 2005
June 2005 ( 3 )
May 2005
April 2005
December 2004
November 2004 ( 6 )
May 2004
February 2004 ( 3 )
January 2004
December 2003 ( 9 )
November 2003 ( 5 )
August 2003
July 2003 ( 15 )
June 2003
September 2002
August 2002
May 2002
April 2002
December 2001 ( 2 )
July 2001
April 2001 ( 3 )
February 2001 ( 5 )
November 2000
September 2000
May 2000 ( 2 )
March 2000 ( 2 )
December 1999
November 1999 ( 3 )
October 1999 ( 5 )
September 1999
August 1999
July 1999 ( 8 )
June 1999 ( 2 )
May 1999 ( 3 )
April 1999
March 1999
December 1998
November 1998 ( 2 )
October 1998 ( 3 )
September 1998
July 1998 ( 2 )
June 1998
April 1998
March 1998
November 1997
October 1997 ( 2 )
March 1997
February 1997
January 1997
November 1995
September 1995 ( 2 )
Complete Listing