Codec shoot-out 2005 - Main round
Table of content:
1 : Introduction
2 : Test setup
3 : Usability
4 : Sources, bitrates and sizes
5 : Settings
6 : Encoding speed
7 : Test 1: Matrix 3 (normal bandwidth) -- ultra high bandwidth
8 : Conclusion
9 : Outlook
Welcome ladies and gentleman, to the main round of the 2005 codec comparison.
Just like last year, we have a new first timer: XviD AVC. Yes, you read right, the XviD team has whipped up an MPEG-4 AVC encoder which makes its testing debut right here. Because the codec is yet unreleased and still very young, I allowed a late submission, the build I tested only arrived 4 days after the deadline, a courtesy which will of course only be extended once.
I'll dispense with a lot of information that usually goes on the first page, because of the new mode introduced this year. You can read all about it in the article about this year's qualification round.
Just to quickly recapitulate, codecs that I had not previously tested, and codecs that did not perform too well in previous comparisons had to go through a qualification round. In the main round we find all qualificants, as well as codecs that directly qualified due to past results and / or my personal experience.
Let's have a look at the history of the codecs that are entered in this comparison:
Similar to the comparison in previous years, codec makers were given two weeks to provide suggested settings and binaries for the task at hand. Unlike the qualification phase, all participants in the main round replied to my request to provide parameters that provide optimal quality while not making you wait forever.
If you are wondering about 3ivX and RV10, I got in touch with the respective people but we concluded that the progress made in terms of visible quality would simply result in a re-test of essentialy what we saw last year. So I guess we'll wait for 3ivX's AVC encoder and RV11..
All codecs were tested in a 2 pass setup using the settings suggested by the developers.
In accordance with the rules for the main round, the codecs will be placed into the following three groups:
All testing was done on the following hardware:
AMD Athlon 64 X2 4600+
Shuttle SN25P barebone system
2x 512 MB PC3200 Kingston HyperX DDR RAM CL2
Gigabyte Geforce 7800GT GFX card
Philips 230W5B2 Display connected via DVI
To encode I used the following software:
DGIndex 1.4.5 & DGDecode
AviSynth 2.55 for frameserving
VirtualDub 1.6.11 for VfW encoding
Nandub 1.0 RC2 lumafix to multiplex AVI
Mp4box CVS dated December 16th to multiplex MP4 (except for XviD AVC, it requires a fix that was recently made to mp4box but was not readily available yet, so I used a mp4creator build dated December 20th to mux XviD AVC)
Usability and additional software
This year marks the introduction of another review category: usability. This is because the test field is very inhomogeneous and we have various means of encoding. Besides the traditional VfW codecs, more and more codecs offer a commandline utility or a DirectShow filter for encoding, or have a whole suite that does a lot more than just encoding. In this test, QuickTime is the most special tool because it's basically a player that can export a file it can play to another format.
Here is a quick table outlining which encoding methods are available
|Codec / Encoding method||VfW||CLI||DirectShow||Other|
( 1): not publicly available
( 2): via NeroDigital in 2006, filters restricted to the use of Nero's own tools
( 3): via Recode in 2006
( 4): Elecard MobileConverter
( 5): Mainconcept H.264 encoder
( 6): ffdshow
( 7): ffmpeg or mencoder
( 8): only when the filters are used via Recode
( 9): Recode
(10): Windows Media Encoder
Unlike the qualification round, encoding was problem free in the main round. All non VfW codecs were made available as a commandline tool supporting AviSynth input and I didn't have to tinker around with a lot of external tools to get things done.
The only note I have is regarding the VC-1 encoder. It uses the registry to store its settings, like VfW codecs do, but there is currently no GUI to make this configuration. I was given a separate HTML application that allowed to write export the proper registry keys so that they could be imported into the registry, but this made things a bit more complex as the encoder itself was available via the commandline.
|Codec / Playback method||VfW||DirectShow||Other|
(1): If the stream is muxed into an AVI and the fourCC is set to a value a
pre-installed MPEG-4 AVC VfW codec supports
(2): Ateme's own filters or the NeroDigital filters
(3): mplayer or VLC
When it comes to playback, DivX, HDX4, LMP4, x264 and XviD have the most out-of-the box functionality, but by using tools for other codecs, VfW and DirectShow is never a problem. Basically you can install ffdshow and anything goes except for the non MPEG-4 codecs. This is also a considerable improvement over what we've seen in the qualification.
Sources, Bitrates and Sizes
Movies I encoded:
Matrix Revolutions - Region1, NTSC, length: 2h09, 185917 frames (@23.976 fps). I will call this movie Matrix3 from here on
I used a 128kbit/s CBR MP3 audio track I created using BeSweet 1.5 beta 31 for all codecs using the AVI container. I used the same BeSweet and the latest Nero 6 AAC encoder DLLs to create a 128 kbit/s CBR HE AAC audio track for all codecs using the MP4 container. Microsoft uses the ASF container (WMV files are ASF files that contain just video and audio), so I encoded my audio to a 128 kbit/s WMA file for VC-1. As in previous years, the goal is to put Matrix3 onto a 700 MB CD. The audio file sizes were as follows:
The proper filesize would be 121'156 KB, and all encoders don't manage to hit the target size on the stop, but a 250 KB difference between the streams is nothing to worry about too much.
Because the makers of each ASP participant asked that their decoder be used, I switched the whole ASP category back to the AVI container, where using a different decoder is not so much of a problem. The only problem was HDX4 and LMP4, as ffdshow has both codecs in the same group, so either ffdshow decodes both, or none. Consequently I had to reconfigure ffdshow each time before reviewing either of the two codecs. I've asked the makers of HDX4 to offer a selection of which codecs the filter decodes, so that you an ensure it only decodes its own content, but it would be nice of ffdshow offered more flexibility in that area as well.
With MP3 being muxed into an AVI and the AAC audio stream into the MP4 container, this resulted in a video bitrate of 620 kbit/s for codecs using the MP4 container, and 626 kbit/s for codecs using the AVI container. Microsoft still owes me a formula for the ASF overhead, but for this test I used 620 kbit/s for the video as I was told the overhead was not much different from AVI. Please refer to the FAQ if you have any questions or doubts why this scenario was chosen.
As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got:
|DivX 6.1||717'030 KB|
|ND ASP||716'479 KB|
|XviD AVC||719'344 KB|
Note that 700MB equals 716'800KB. I've marked every size in orange if it is more than 2.5 MB above target size (2.5 MB is about the oversize a CD can have and you can still burn it without having to activate overburning), red if it's more than 10 MB off target (grounds for disqualification). The ones marked in bright yellow are undersized more than 2.5 MB (that points to a rate control issue as well but is less problematic than oversized files).
All codecs except VC-1 managed to be within the unproblematic size range. We can see that VP7 and XviD AVC could use a little more work to more accurately hit the target size, but I have no complaints about the rest of the participants. Note that when it comes to XviD AVC, when using mp4box, the output size is much closer to target, but so far I have no mp4box build that produces acceptable output. The encoder's output makes use of certain properties in raw AVC streams that make the raw file bigger.. the mp4box build dated December 16th cannot properly handle that and produces choppy output. A bugfix has already been submitted so as soon as I get a fixed build, I can update the size table.
Now about VC-1: Unfortunately, the pre-beta build I was given contained two bugs that were only discovered during the course of this comparison: It would not encode every frame, constantly missing 6 (but this may just be a reporting problem, I did not not verify that every frame is indeed there), and more importantly, somewhere in the 75'000 to 100'000 frame range, the encoder started missing the target bitrate. When encoding 100'000 frames, the bitrate was already off by 100kbit/s, when encoding the full movie, the bitrate was only half of what I asked for. I immediately informed Microsoft and performed various experiments (which lead me to figure out the the range of the breaking point), but I did not receive a fixed build or workaround in the time allotted for such a scenario (2 business days). As it makes no sense to compare two video streams at completely different bitrates, I chose to stop the evaluation of VC-1 at this point.
Codec settings (applied with respect to the default codec settings)
Since there is only one source in the main round, the choice of a source for this test was easy ;) As usual, I'm performing a two pass over the first 10'000 frames, note the time spent from the start of the first pass till the end of the second pass, and divide this by the number of frames encoded. The usual disclaimer also applies: even though 10'000 frame should be rather representative, I did only perform each measurement once and did not reset my PC in between. Looking at how major hardware sites on the web measure speed, I think my methodology still is a considerable improvement, but it in no way lives up to a scientific standard.
|DivX 6.1||69.69 fps||x|
|XviD AVC||21.19 fps||x(3)|
(1): Not used because it trades visible quality for a rather low gain in speed
(2): Non optimized codec, but because VirtualDub separates input reading (and thus decoding and resizing of the source) from encoding, it can still make good use of an SMP system. The speed gain seems to be dependent of the encoding speed, the CPU use of a faster codec (i.e. XviD) is considerably higher than of a slower codec (i.e. VP7)
(3): Not used because it's not fully ready yet
Note that since Ateme uses a proprietary mechanism to speed up the first pass, which gains in speed the longer the movie is, the numbers for the Ateme and ND ASP codec are taken from a full movie encoding session.
In addition for disqualification due to usability issues, I put another killer
requirement in the speed area. With 1 MB cache away from having the fastest
workstation CPU there is for video encoding these days, I decided to disqualify
all codecs that do not reach at least 10.0 FPS. This is not only cutting down
the total encoding time for the comparison, it is also to be considered that
if you have a slower CPU, encoding time will be much lower. And with HD content
gaining more importance, imagine encoding HD content on a 1 GHz box with a codec
that doesn't even do 10 fps on my high end machine.
I've used the following colorscheme for the speed table: red = disqualified due to a single digit FPS rate. Orange: below real-time speed (real-time means the codec encodes at least as many frames as the video has per second), and green for codecs that encode faster than 2x real-time. If you read the qualification, this table is much more to my liking. In fact, It took me only 2 days to encode all the content for the main round, not taking into account re-encoding (more about that in the reviewer notes in the conclusion). And AVC High Profile encoders going at almost 50 FPS shows that AVC doesn't necessarily indicate slow encoding speed, and coming from last year's winner, I expect that the codec did not simply trade quality for speed - otherwise it is unlikely to make the final round, considering what the best two participants showed (Elecard and XviD) in the qualification round.
Last but not least, if you've seen the speed table from the qualification,
please note that XviD did not use Turbo in the main round, which explains the
considerable speed difference.
Other important stuff
Here's the AviSynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DGIndex in Matrix.
With the exclusion of VC-1, the non MPEG-4 group only had one member (the qualification eliminated every other potential codec for this group), and since comparing VP7 against itself makes not much sense, I decided to merge the AVC and non MPEG-4 category, and thus have two groups of 5 codecs each. Only having two groups and reviewing them close together also allows to draw some direct conclusions between the two groups.
Because there is no fourCC in MP4, there is effectively only one playback filter for all MPEG-4 AVC codecs this year: the ateme decoder. It supports all MPEG-4 AVC high profile features including interlaced content (for which I have zero use and which should be banned altogether but that's another issue entirely).
In the two other categories, each codec could use its own filter, but this may change again in the future.
I enabled deblocking for MPEG-4 ASP content. There is no postprocessing for AVC content because AVC has a built-in deblocker (and those who have activated postprocessing for AVC content in ffdshow will be in for a really bad surprise.. it works but completely ruins your picture).
All files were reviewed using Media Player Classic as it can play back pretty much everything you can throw at it. Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player!
Now proceed to the test (low bandwidth JPEG version. This version loads 432 KB of images for the beginning, and the total image size for the matrix test is 3.66 MB). If you have a lot of bandwidth and / or don't mind waiting, there's a high bandwidth PNG version (loads 1.90 MB of images initially and if you want to see all the images, you have to download 16.7 MB). Also note that the high bandwidth version requires that you have enough browser cache left or the images will be reloaded each time you zoom in or out.
This document was last updated on December 26, 2005