Even in normal times, most organizations and developers struggle to manage the lifecycle of images and videos at scale. Websites with countless media assets, especially those that rely heavily on user-generated content, are nearly impossible to orchestrate effectively without huge teams of designers and developers operating in sync. That gets even harder once everyone is forced to work at home.
This disruption currently highlights one of the most promising developments of recent years for the developer community: automation, and more specifically, AI-enabled automation. While every company is forced to reevaluate their workflows and capabilities due to the disruption caused by the coronavirus outbreak, there’s no better time to explore the massive opportunity AI presents.
Although there are countless other applications, here are five prominent examples of how to apply AI to enable image and video management at scale.
1. Auto-cropping and resizing: Images
Cropping images for delivery across all channels, devices and browsers is a drain on team resources. It isn’t that hard to do manually, but it takes a lot of time to make sure it’s done right. The most important parts of an image have to remain visible and prominent, while ensuring optimal image size, aspect ratios and more.
As one way to automate the process, it has been somewhat common to use a mathematical pixel analysis that centralizes the image on the sharpest or densest pixel area. It is helpful, but it’s not foolproof. To replicate human cropping precision, you can’t use such restrictive rules. Rather, you need technology that can replicate the human evaluation process. Deep learning algorithms can detect the parts of an image that matter most to the viewer and the brand. GPU-based hardware clusters mimic the way their human ‘teachers’ identify the key parts of any image, regardless of subject or layout – and quickly start processing millions of crop requests on the fly.
2. Auto-cropping and resizing: Videos
Taking the complexity up a notch, AI makes video management a whole new ballgame. Displaying video correctly is very complex in large part because many videos are made with the horizontal aspect ratio even though they are consumed vertically on mobile phones. Social media’s square ratios add another tier of frustration, too.
Until now, there have really only been two options to address this. One requires the user to manually flip their devices, which causes friction and also means the video won’t fill their screens. Other sites lock the video by barricading it between black bars or adding a blurry version of the video behind it. On social media, the unique square aspect ratio means many videos are cut off around the edges.
The first rule of video cropping is to ensure that the most important features are displayed in any format, for every frame. For example, if you’re showing a one-hundred-meter dash with a photo finish, you’ll want the podium finishers to remain in the middle of the shot, not sliding off the right edge of the screen.
Fortunately, deep learning algorithms can quickly identify what matters most to the viewer and then creates a heat map it uses to intelligently crop the video. This allows it to ensure the most interesting subjects remain in central focus, maximizing the value to the viewer and ensuring your brand gets the intended value from sharing it.
3. Video previews
Another hurdle often presented by videos is that loading and displaying a number of video previews at once often restricts site performance and can be hard to manage manually. If there are only a few to be displayed on a site, designers can generally edit these previews manually. But while making video previews is an art on its own, creating even just a few seconds worth of a preview when you’re managing hundreds of videos is a massive headache. If that’s the case, you need automation.
Again, deep learning algorithms determine which segments in the original video would appeal to humans. Then they automatically select the most interesting pieces that fit into the time you’ve allotted to your previews.
4. Categorizing and tagging image and video content
One of the most classic “blessing and a curse” elements of the modern content environment is user-generated content (UGC). It’s great for providing authenticity and providing highly engaging content to leverage, but it’s hard to keep track of what’s available and how to match it to the right audiences. Manually tagging and sorting it all, for many companies, would slow the process down tremendously and limit their value.
But AI enables content-recognition tagging that can do that grunt work for you. Even better is that the companies building AI recognition technology have made it available to their partners who make it available to you as part of their management platforms. Now all you need to do is figure out how best to use it.
5. Background removal
Another use case for AI is automatically optimizing the non-subject parts of image backgrounds. This is particularly valuable for ecommerce product listings that want their high-quality product photos to pop on clean and sleek backgrounds. This requires the image background to be transparent, which naturally, requires intervention with the original asset. At scale, manually editing is too slow and laborious to make it possible for every item.
As you might have guessed by now, the AI defines the main objects in the image and distinguishes the foreground from background in order to understand the full context and composition of the scene. The AI then figures out which pixels to remove and replace with transparent displays. The process is highly refined, and AI can do this at a production-level quality similar to a human editor – but far faster.
AI applications abound
Of course, these are just a few of the examples of how AI is helping developers, creatives, marketers and brands escalate the scale and pace of their content publishing and consumer engagement. As moments of significant disruption such as the one we’re currently experiencing continue to reveal new appetites and opportunities to streamline workflows, we’ll surely see new applications emerge, and I’m excited for it.
