HTML5: Audio and Video Elements

Understanding audio and video file formats

There are plenty of different audio and video file formats. These files may include not just video but also audio and metadata—all in one file. These file types include:

.avi – A blast from the past, the Audio Video Interleave file format was invented by Microsoft. Does not support most modern audio and video codecs in use today.

.flv – Flash video. This used to be the only video file format Flash fully supported. Now it also includes support for .mp4.

.mp4 or .mpv – MPEG4 is based on Apple's QuickTime player and requires that software for playback.

How it works...

Each of the previously mentioned video file formats require a browser plugin or some sort of standalone software for playback. Next, we'll look at new open-source audio and video file formats that don't require plugins or special software and the browsers that support them.

H.264 has become of the most commonly used high definition video formats. Used on Blu-ray Discs as well as many Internet video streaming sites including Flash, iTunes Music Store, Silverlight, Vimeo, YouTube, cable television broadcasts, and real-time videoconferencing. In addition, there is a patent on H.264 is therefore, by definition, not open source. Browsers that support H.264 video file format include:

Google has now partially rejected the H.264 format and is leaning more toward its support of the new WebM video file format instead.

Ogg might be a funny sounding name, but its potential is very serious, I assure you. Ogg is really two things: Ogg Theora, which is a video file format; and Ogg Vorbis, which is an audio file format. Theora is really much more of a video file compression format than it is a playback file format, though it can be used that way also. It has no patents and is therefore considered open source.
Fun fact: According to Wikipedia, "Theora is named after Theora Jones, Edison Carter's controller on the Max Headroom television program."

Browsers that support the Ogg video file format include:

WebM is the newest entrant in the online video file format race. This open source audio/video file format development is sponsored by Google. A WebM file contains both an Ogg Vorbis audio stream as well as a VP8 video stream. It is fairly well supported by media players including Miro, Moovidia, VLC, Winamp, and more, including preliminary support by YouTube. The makers of Flash say it will support WebM in the future, as will Internet Explorer 9. Browsers that currently support WebM include:

There's more...

So far this may seem like a laundry list of audio and video file formats with spotty browser support at best. If you're starting to feel that way, you'd be right.

The truth is no one audio or video file format has emerged as the one true format to rule them all. Instead, we developers will often have to serve up the new audio and video files in multiple formats while letting the browser decide whichever one it's most comfortable and able to play. That's a drag for now but here's hoping in the future we settle on fewer formats with more consistent results.

Audio file formats

There are a number of audio file formats as well. Let's take a look at those.

AAC – Advanced Audio Coding files are better known as AACs. This audio file format was created by design to sound better than MP3s using the same bitrate. Apple uses this audio file format for its iTunes Music Store. Since the AAC audio file format supports DRM, Apple offers files in both protected and unprotected formats. There is an AAC patent, so by definition we can't exactly call this audio file format open source. All Apple hardware products, including their mobile iPhone and iPad devices as well as Flash, support the AAC audio file format. Browsers that support AAC include:

MP3 – MPEG-1 Audio Layer 3 files are better known as MP3s. Unless you've been hiding under a rock, you know MP3s are the most ubiquitous audio file format in use today. Capable of playing two channels of sound, these files can be encoded using a variety of bitrates up to 320. Generally, the higher the bitrate, the better the audio file sounds. That also means larger file sizes and therefore slower downloads. There is an MP3 patent, so by definition we can't exactly call this audio file format open source either. Browsers that support MP3 include:

Unlock access to the largest independent learning library in Tech for FREE!

Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.

Renews at $15.99/month. Cancel anytime

Ogg – We previously discussed the Ogg Theora video file format. Now, let's take a look at the Ogg Vorbis audio format. As mentioned before, there is no patent on Ogg files and are therefore considered open source.
Another fun fact: According to Wikipedia, "Vorbis is named after a Discworld character, Exquisitor Vorbis in Small Gods by Terry Pratchett."

File format agnosticism

We've spent a lot of time examining these various video and audio file formats. Each has its own plusses and minuses and are supported (or not) by various browsers. Some work better than others, some sound and look better than others. But here's the good news: The new HTML5 <video> and <audio> elements themselves are file-format agnostic! Those new elements don't care what kind of video or audio file you're referencing. Instead, they serve up whatever you specify and let each browser do whatever it's most comfortable doing.

Can we stop the madness one day?

The bottom line is that until one new HTML5 audio and one new HTML5 video file format emerges as the clear choice for all browsers and devices, audio and video files are going to have to be encoded more than once for playback.

Creating accessible audio and video

In this section we will pay attention to those people who rely on assistive technologies.

How to do it...

First, we'll start with Kroc Camen's "Video for Everybody" code chunk and examine how to make it accessibility friendly to ultimately look like this:

<div id="videowrapper">
<video controls height="360" width="640">
<source src="__VIDEO__.MP4" type="video/mp4" />
<source src="__VIDEO__.OGV" type="video/ogg" />
<object width="640" height="360" type="application/
x-shockwave-flash" data="__FLASH__.SWF">
<param name="movie" value="__FLASH__.SWF" />
<param name="flashvars" value="controlbar=over&
image=__POSTER__.JPG&file=__VIDEO__.MP4" />
<img src="__VIDEO__.JPG" width="640" height="360"
alt="__TITLE__" title="No video playback capabilities,
please download the video below" />
</object>
<track kind="captions" src="videocaptions.srt" srclang="en" />
<p>Final fallback content</p>
</video>
<div id="captions"></div>
<p><strong>Download Video:</strong>
Closed Format: <a href="__VIDEO__.MP4">"MP4"</a>
Open Format: <a href="__VIDEO__.OGV">"Ogg"</a>
</p>
</div>

How it works...

The first thing you'll notice is we've wrapped the new HTML5 video element in a wrapper div. While this is not strictly necessary semantically, it will give a nice "hook" to tie our CSS into.

<div id="videowrapper">

Much of the next chunk should be recognizable from the previous section. Nothing has changed here:

So far, we're still using the approach of serving the new HTML5 video element to those browsers capable of handling it and using Flash as our first fallback option. But what happens next if Flash isn't an option gets interesting:

<track kind="captions" src="videocaptions.srt" srclang="en" />

What the heck is that, you might be wondering.

"The track element allows authors to specify explicit external timed text tracks for media elements. It does not represent anything on its own." - W3C HTML5 specification

Here's our chance to use another new part of the HTML5 spec: the new <track> element. Now, we can reference the type of external file specified in the kind="captions". As you can guess, kind="captions" is for a caption file, whereas kind="descriptions" is for an audio description. Of course the src calls the specific file and srclang sets the source language for the new HTML5 track element. In this case, en represents English. Unfortunately, no browsers currently support the new track element.

Lastly, we allow one last bit of fallback content in case the user can't use the new HTML5 video element or Flash when we give them something purely text based.

<p>Final fallback content</p>

Now, even if the user can't see an image, they'll at least have some descriptive content served to them.

Next, we'll create a container div to house our text-based captions. So no browser currently supports closed captioning for the new HTML5 audio or video element, we'll have to leave room to include our own: