Detecting and cropping letterboxing using FFmpeg and Imagemagick

At my place of work we have some old videos that were filmed in a 24:9 aspect ratio and then letterboxed using bright green bars to be 16:9 for consistency with the rest of our videos. When played, our video players would "zoom" in on the center of the video to hide the green bars.

Oh boy have we regretted that video production decision over the years.

It came to a head recently when we began migrating this legacy content to our new media players and content authoring systems, which do not and will never support these types of letterboxed videos. This was discovered by our content QA process post-migration, when they reported that all of the videos were playing back with neon green bars on the top and the bottom.

Given that users were getting access to this content next week, this issue very quickly became the highest priority ticket for our team, and I was assigned to work on it.

I immediately turned to our video encoding queue as a potential place to solve this issue, as all of our videos are processed through the queue, which means that any fix implemented in the queue would catch any legacy videos that had not been migrated to the new systems yet as they were migrated to the new systems, and they would be automatically fixed without any human intervention.

We were already running FFmpeg in the queue to scale/pad/resize videos with great success, and I know it was able to crop videos as well, so the hard part became detecting these videos because not every video has these green bars and needs to be cropped. It's a subset representing about 5% of our total content.

Checking the color of the pixels in the video was the easiest solution I could think of, so I immediately started looking up ways to detect the color of specific pixels in a video. The easiest way to do this was to use FFmpeg to generate a thumbnail, and then Imagemagick's convert command to crop a 1x1 piece of the thumbnail at specific coordinates, and then parse the srgb colors and compare them to our green bar color to detect whether or not a video needed to be cropped.

Next, we had to figure out how to crop the source video down. The part that made this a little more tricky is that the source videos are not known sizes either. They are arbitrarily sized, but it is known that the overall video will be 16:9 and the video content will be 24:9 inside, and centered vertically. Also complicating this is the fact that the green bar on top is actually slightly smaller than the green bar on the bottom of the video, which means if we crop exactly centered we will end up with a sliver of green at the bottom of the video that is output.

The above code uses FFmpeg's crop video filter, setting the target width, target height, and the the x and y coordinate to start the crop from originating in the top left corner of the video. This results in an output file that is 16:9, containing no green bars and all of the important content from the originating video.

We were able to run the resulting encoding script against all of our potentially affected source videos again, and successfully remove the green bars from every encoded video that we present to users!