Like Donald Knuth, I think much of the current multicore hype is overrated.
The machine I use today has dual processors. I get to use them both only when I'm running two independent jobs at the same time; that's nice, but it happens only a few minutes every week. If I had four processors, or eight, or more, I still wouldn't be any better off, considering the kind of work I do -- even though I'm using my computer almost every day during most of the day. So why should I be so happy about the future that hardware vendors promise? They think a magic bullet will come along to make multicores speed up my kind of work; I think it's a pipe dream. (No -- that's the wrong metaphor! "Pipelines" actually work for me, but threads don't. Maybe the word I want is "bubble.")
Despite that, I've acknowledged all along there are certain narrow tasks that the proliferation of CPU cores will make dramatically faster. And one of my very favorite tasks in that niche is encoding your DVD collection.
I bought my first DVD about 10 years ago. At the time, they were a technical marvel:
Today, those specs are rapidly becoming pedestrian in the face of high definition cable, broadcast, and Blu-Ray discs. A few of the video sharing websites offer something perilously close to DVD quality already.
I say the DVD is the new MP3. We're going to start tossing these things around like candy.
Unlike audio CDs, DVDs are already compressed digital data. You could extract the files from the DVD as-is, and play them back to your heart's content. No re-encoding required. But like The Six Million Dollar Man, we can rebuild them better than they were before. Video codecs have advanced tremendously since the heady days of MPEG-2. These new codecs take a lot more playback horsepower than MPEG-2, but offer comparable quality in about one-fourth the size. We can turn our digital DVDs into better digital DVDs through superior computer science.
But if you thought the playback performance demands of these new codecs were severe, wait until you see the encoding performance demands. It's only in the last year or two that typical CPUs could encode new-fangled MPEG-4 or VC1 at anything even approaching real time. But now, with extremely fast dual cores trickling all the way down to the mainstream, and quad-core CPUs carving out a decent share for themselves-- the average user could potentially rip and encode a typical DVD in less than 30 minutes. Per a recent H.264 benchmark dataset analysis, you can statistically expect to halve your encode time when going from 2 to 4 cores.. and almost do it again when you go from 4 to 8!
It helps, too, that there's great free software like Handbrake which makes it easy to harness that embarassment of desktop CPU power to encode your DVDs.
Yes, there are an intimidating number of knobs and dials to potentially tweak here, but what I like about handbrake is that I largely don't have to. There are some logical presets on the right (clipped out of the screenshot for size, unfortunately) which will get you headed in the right direction: AppleTV, iPod, Film, Xbox 360, and so forth. I do set a handful of variables, like the overall bitrate -- I prefer something between 900 and 1200 -- and whether I need to deinterlace the source. But I mostly let Handbrake use its "auto" defaults, and get excellent results. If you're curious about the details, there's a well-written description of many of the Handbrake settings.
I had the most luck with the H.264 codec, which is aggressively multithreaded. I achieved 30-40 fps on my modest new power efficient dual-core HTPC processor, and upwards of 100 fps on my overclocked 4 GHz dual-core. Fortunately, Handbrake supports a batch encode mode, so you can queue up a bunch of jobs to run overnight.
I was particularly excited to find that I can pass the digital audio directly through using the "AC3" setting for the audio encoder. That means, when playing these files back on a home theater PC, the digital audio arrives at your receiver in exactly the same way it would from a DVD, with a few minor adjustments in ffdshow. There is no audio degradation or conversion whatsoever! As something of an audiophile, I suffered mightily through many a re-encoded DVD that was downmixed to plain vanilla stereo over the years, so this is a huge step forward in my book.
So, in summary -- nearly the same digital video quality, exactly the same digital audio quality, all wrapped up in a single file less than a quarter of the original size of the DVD. Seriously, I get chills. It's geek nirvana. What's not to love about encoding your DVD collection? It's also a perfect use of those four CPU cores, which would otherwise lay idle 99% of the time.
Take, for example, one of my favorite movies, Idiocracy. I used Handbrake to convert this DVD from a set of files totalling 4.15 GB to a single 995 MB file, at almost no quality loss. See for yourself.
Still image captured from original DVD:
Still image captured from H.264 encoded video of DVD:
The above still was used in an earlier post, A World of Endless Advertising. I love being able to grab a movie file over the network and quickly get the exact still I need for a blog post. Yes, there is a tiny loss of fidelity -- particularly in the chair shadows on the bottom left, and the grain of the wall texture at the upper left. But I'm willing to live with that compromise if it means I don't have to pull ginormous 8 GB ISO images files over the network.
Why re-encode DVDs you already own? For convenience, mostly -- so you can watch them on your mobile devices, on your PCs, laptops, and anything else that vaguely resembles a computer in your home. Might as well put all those CPU horses to proper use. A re-encoded DVD is so much more flexible than those physical hunks of round, reflective plastic.
x264 has been getting a lot of visual improvements lately - the screenshot up there is definitely from before Variance AQ, which was specifically made to fix loss of detail in flat bits. That's really the only problem I can see there.
astrange on May 2, 2008 2:07 AMI too can't get enough cores. I did a Blu-Ray two days ago. I can fit it on a DVD at 1280x720 with a bitrate of about 6 Mbps (if memory serves). Superb image quality, no degradation AFAICT. Did it overnight on a dual-core @ 3.3 Ghz.
For fun, I figured I could do a DVD-9 at a full HD resolution of 1920x1080. The encode took me 26 hours... lol
Dennis already mentioned part of this, but...
Don Knuth (PBUH) said iI get to use them both only when I'm running two independent jobs at the same time; that's nice, but it happens only a few minutes every week./i
Which is, well, utter bollocks.
Sure, Dr. Knuth doesn't often do two "tasks" at once, directly, in the sense of doing two things with his active attention at once.
But even if he (unlike the vast majority of us) only uses a console with an editor and switches out to compile things, the Irest of the OS/i will run tasks while he's working all the time.
While he's spot on (of course) when speaking of individual programming tasks that parallelism doesn't help, the idea that an extra core isn't making the machine as a whole faster than not having it, seems unsupportable even in a plain-jane unix setup, though that's where the benefit will be least, and least obvious - though a more typical setup even then would have compiles going while one does other things, and it's nice to minimize the effect either has on the other.
(He says he's speaking of technical work, not recreation, but even in "technical work", many of us want multiple processes running at once, have environments that do background compilation* or optimization, are using those icky "web browser" things to look at Idocumentation/i, and even using our computers to play (efficiency-enhancing) music while we work.
And then there's those of us who are working on systems either for the web or with a DB backend or both, who need, just to work, to have a running webserver and/or SQL server, both of which will be competing for cycles with our actual work product.
*Yes, I'm talking to you, Visual Studio.)
Sigivald on May 2, 2008 2:09 AMActually the one thing (unless I'm missing it) I don't like about this concept is that you're limited to a single audio track. Sure, I can forego the Spanish, French, etc. tracks but I like the audio commentaries. Are there any good compression formats out there that have nice automated programs like Handbrake (DVD - Format) that can store multiple audio tracks? Because then I'd be all about having a Home Server with the ability to stream all this crap to a Media Center PC.
Schnapple on May 2, 2008 2:18 AMHow long before the movie industry wants to get in on that "format shifting" tax that the RIAA is pushing for in the UK? See here:
http://tinyurl.com/3ucb26
Aaron G said: Is there even a single case on the books where government lawyers have gone after a private citizen for transcoding? I'm not talking about the MPAA here, whose lawsuits generally don't involve the DMCA at all, not to mention that they only ever go after mass distributors.
You mean like when the DVD-CCA pressured Norway into arresting DVD Jon?
Robert R. on May 2, 2008 3:06 AMOne strange thing is that XBMC tends to be much better at playing XVid encodes of DVDs than it does playing the original DVDs. It's like there's too much data to keep up with, so skipping around is not instant.
Scott on May 2, 2008 3:37 AMI agree with Jon Raynor because it will *never* be possible to distribute your private key with the video.
alan on May 2, 2008 4:30 AMWith the cost of terabyte hard drives in a downward spiral, I find myself no longer encoding DVDs to DivX, xVid or H.264. I now just rip them to disks with DVDFab HD Decrypter, removing the FBI warning and foreign language(s) that I don't need. This has one major advantage for me compared to encoding... the chapter jumping features are intact. The other advantage, it takes just 10 to 15 minutes to rip a DVD compared to 2 to 3 hours to encode it to H.264 (at approximately 1/4th the size of the original).
cyclo on May 2, 2008 4:41 AMAh, but breaking the CSS to rip your dvd's is committing a serious crime there. How *dare* you steal from those poor filmmakers by not buying their movie a second time in a degraded format when you want to watch it on your computer?!
Handbrake is seriously awesome, though.
Mike on May 2, 2008 5:48 AMI'd just like to point out to those who don't know, we cannot "rebuild them better than they were before". This is due to the fact that H.264 involves *lossy* compression. What this means is that each time you compress the video, you lose some quality, the degree of loss depends on a few things though.
However, as it stands at the moment, there's no doubt that if you want the best compression out of your videos, H.264 is the best solution.
P.S. If I recall correctly, H.264 does allow for lossless encoding, however, the bitrates (and therefore storage space), are surplus to most people's requirements.
Haze on May 2, 2008 6:02 AMMaybe The actuel computers were meant for the future programming?
What do you think of the Parallel Extensions to the .Net framework Microsoft is working on?
That would give a "purpose" to the so called "hype" don't you think?
I mean.. even WoW benefits from the multicore now (Since 2.0.1)!
Mathieu Paul Le Clerc on May 2, 2008 6:08 AMWould there be any point in pre-scaling movies?
I know that you can't just create pixels out of thin air - but when you play a 720x480 dvd on a 1920x1080 screen, something somewhere is doing a lot of scaling of that video.
It seems as though it should be possible to use a higher quality but slower scaling algorithm ahead of time and end up with something that might actually, overall, look better than the original DVD scaled by your dvd player.
Obviously the bandwidth requirements go up as well, but HD space is cheap ;-) Any thoughts?
Mike on May 2, 2008 6:11 AMWhat do you think of the Parallel Extensions to the .Net framework Microsoft is working on?
I think the minute you touch the database or do any kind of significant I/O, the performance benefits will be miniscule. It is a small step in the right direction, of course.
Jeff Atwood on May 2, 2008 6:15 AMYou say: "Unlike audio CDs, DVDs are already compressed digital data.",
but remember that they are usually not originally digital data, in the
transfer from regular film (thats in the camera during filming) to the
actual Digital DVD, there can be a LOT of data lost.
I remember seeing the film "Once upon a Time in Mexico" on DVD and after
hearing that it was filmed on Digital camera, I ran it through an MPEG4
encode and the output was incredible! Even during fast motion scenes
there was no blur whatsoever.
Good post anyway.... You had me right up until the screenshot from
"Idiocracy" ... *Shakes Head* ... Shocking, just shocking movie.
It goes beyond geek nirvana, dvds and cds are dead.
http://www.furrygoat.com/2008/02/06/physical-media-is-dead/
First I read this: http://www.codinghorror.com/blog/archives/001103.html
And now I read this: "Like Donald Knuth, I think much of the current multicore hype is overrated."
I'm confused...!
Rawlyn on May 2, 2008 7:03 AMI used to routinely re-encode my DVD's, but the abundance of really cheap storage and gigabit ethernet has eliminated most of my desire to endure the re-encode process. I guess I don't need to be that flexible.
I routinely walk with 250+GB so bringing over a couple of deletable 8GB files is really no biggie and since I don't share, ever, transfer time is irrelevant.
Idiocracy is a great movie.
I found one really useful feature of quad processors. Running multiple virtual machines. Since I'm an OS X fetishist and a .NET developer being able to work in Windows and OS X simultaneously without having to reboot uncomplicates my life.
I'm still on the fence about re-encoding DVDs. I copy the entire DVD to the drive so my son can watch his shows. The little one loves to hit the eject button and it keeps me sane. I prefer keeping the menus; since I'm an anglophone in France being able to switch languages by remote is essential.
Joe on May 2, 2008 7:33 AMHate to break it to you, but based only on the two images you posted, there's a lot of quality loss. Maybe you can't see, then it's ok, you won't miss it either, but I can see it.
Mike on May 2, 2008 7:34 AMI pretty much watch all my moves/tv shows off my computer nowdays. Of course in the next few years BluRay players should come down in price and I'll be a slave to the disc yet again.
Kris on May 2, 2008 7:38 AMI don't like the lossiness; it's easily visible, and I bought an HD TV so I could get the clearest picture possible.
dnm on May 2, 2008 7:38 AMThe one thing I miss when watching transcoded videos is the concept of chapters or at minimum, decent fast-forward/rewind capabilities. Maybe I'm using the wrong players, but seeking through encoded video is not a seamless experience.
Derek on May 2, 2008 7:41 AMJeff,
Which codec do you have installed on your HTPC to play back your H.264 files in Windows Media Center? ffdshow?
Also, what program are you using to do your video screen grabs?
Brian DeHamer on May 2, 2008 7:41 AMSo what you are really saying is that a Developer should only have 2 Cores at work, but 4 cores at home.
Tim on May 2, 2008 7:47 AMYeah, ripping DVDs is basically the reason I just bought one of those new rev iMacs... that and teh nVidia board.
MattF on May 2, 2008 7:48 AMIf you have access to the master discs of the movies, you can easily beat the DVD quality when encoding with H.264. Now that'd be geek nirvana.
And you don't need to jump the horn about Handbrake. There are valid fair use possibilities, like transcoding your home videos.
everling on May 2, 2008 7:52 AMI kind of wish that the compression weren't lossy, though. I feel bad if I'm losing _any_ kind of quality---it means I should have just stuck with the original source, especially at only 4.7 GiB (or less if you only want the main movie, with one audio track).
I guess Blu Ray is the best of both worlds then? Because it's already H.264 encoded IIRC, so we're not going to improve on that much, and of course it's already at a delicious HD resolution. (Too bad portable players [understandably] don't like to downscale... but we'll get there eventually.)
Domenic Denicola on May 2, 2008 7:59 AM"How is any of this violating the law?" - Aaron G
"Unfortunately, the software to do so isn't legal to distribute in or import into the U.S., thanks to the Digital Millennium Copyright Act. Section 1201 of the law bans software designed for "circumventing a technological measure"--in this case, the CSS, a copy-protection algorithm in commercial DVDs.
"That's led to a bizarre legal result. Because of a twist in the law, the software to move DVDs onto a video iPod is illegal to sell but probably legal to use--if you can get it.
""You're permitted to do it, but nobody's permitted to help you," says Peter Jaszi, who teaches copyright law at American University in Washington D.C.. "And you're not permitted to help anyone else." (Although, Jaszi cautions, that's "not a perfect argument" because it relies on a legal theory that hasn't been tested in the courts.)"
http://www.news.com/Whys-it-so-hard-to-get-Buffy-on-my-iPod/2100-1041_3-6026753.html
For those commenting on the legality of this, quite frankly, if you're in the US, it's illegal because it's in violation of the DMCA, which prohibits the DeCSS portion of Handbrake to violate copy protection technology.
The reason why Handbrake is NOT is that it's not produced or distributed by anybody in a country which has such silly laws (the main guys are in the Netherlands IIRC). Therefore, Handbrake is legal at the point of production and distribution.
Jeff, however, by downloading it and using it in the US, has violated the DMCA. Even better, he's now publicly admitted it, and therefore the MPAA can sic their lawyers on him immediately.
Aside from that, great post! :-)
Kirk Wylie on May 2, 2008 8:07 AMThis sort of goes along with this topic, but if you are making backups of DVDs...so if it gets damaged you don't have to buy it again, I have a little sister who destroyed a few copies of The Incredibles and Monsters Inc...how long do the backups actually last on the burnt disc? And is there a way to store the DVD in a compressed manner with the menu and all?
Brandon on May 2, 2008 8:08 AMKnuth's words obviously carry significant weight (despite the fact that he made directly contrary indications further on), and when I saw his statement I knew that you would feel vindicated (even though he is arguing against dual cores, which you are fanatical about), but let's look a little closer -- only on rare occasion does Knuth see any benefit from multiple cores.
Now open up your task manager and sit watching your processing time all day long. Guess what -- the *overwhelming* percentage of time it's going to be sitting using less than 1% of its power.
So therefore we should all downgrade -- no need for multiple cores. No need for 3Ghz. A good Celeron 667 should be enough for anyone, because if it isn't saturated all day long, what's the point?
Ah, but look here -- you made the point directly: One of the very rare things that actually needs more than yesteryear's processor is very parallelizable, which is true for almost all intensive applications that actually require the power.
Dennis Forbes on May 2, 2008 8:08 AMThe PRO-IP act means there are going to be cops looking for exactly this as well, called United States Intellectual Property Enforcement Representatives (USIPER).
a on May 2, 2008 8:09 AMThe combination of Jeff's earlier posts on building your own HTPC and now about re-encoding DVDs using MPEG4 with the H.264 codec leads me to believe that he's going to soon have a large collection of movies ready to watch in his theater all via a remote.
I'm working on something very similar but will try to hack my AppleTV for the playback component instead of building another HTPC.
Jeff, let us know how it goes when the finished product is ready.
BradG on May 2, 2008 8:22 AMIt is illegal to defeat any copy protection scheme for content that you do know own the rights to. Purchasing a DVD doesn't give you that right, and backing it up is technically illegal. People used to say that theres no way that the RIAA / MPAA would ever go after someone for making a digital backup / copy of their own legally purchased music or movies, but they stopped saying that.
The RIAA has made the statement that people making a digital copy / backup of legally purchased content is still a pirated copy and they are actually actively prosecuting a man for his digital backups of legally purchased music right now, although he was file sharing, this is a seperate charge for his own digital copies of his own music.
So i'd say its a gray area indeed. Thank your senators for the DMCA!
http://www.washingtonpost.com/wp-dyn/content/article/2007/12/28/AR2007122800693.html
(originally posted in the windows home server forums on MSDC)
Jake Heidt on May 2, 2008 8:25 AMI am not from the US, but isn't any form of decryption to copy a DVD highly illegal in the US?
Would it be wise to post such information on your blog?! (namely publicly stating that you do said copying, and support it)..
Just dont want you to get a huge fine, I have seen news stories about how the DMCA has reacted in the past!
As to whether or not they are right/wrong, thats another debate!
Rob on May 2, 2008 8:31 AMSadly, soon it will be faster to download the whole movie again than it is to encode it. There are no Moore's Law for bandwith. Stream is the way to go.
Hoffmann on May 2, 2008 8:44 AMAt first, CSS is not an encryption scheme (as the name implies, it only scrambles to content, it doesn't encrypt it), so whatever law states that breaking an encryption is illegal (does that law apply to the NSA, too? Last time I checked they do that on a regular basis), this law cannot apply to "breaking" CSS. Well, in court the actual outcome would probably depend on your lawyer's abilities, so make sure, she's any good... ;)
At second, multi-cores are not overrated, they're just under-used.
Beware, this post is encrypted with Double-ROT13, reading it might be illegal in your state. :)
Vinzent Hoefler on May 2, 2008 9:00 AMThis falls under the "build it and they will come" category. Hardware always leads software and now that multi-core is becoming mainstream, sofware vendors will need to figure out a way to make use of all that extra horsepower. Those that can innovate on top of this new hardware paradigm will live to play another day. Those that can't will perish or become freeware.
Intel and Microsoft recently funded research centers at UC Berkeley and UIUC to tackle the problem of software parallelization. Unfortunately most coders these days don't know the difference between a SW thread and HW thread so any parallelization framework would need to be dead simple for the average joe coder to be able to use it.
"I think much of the current multicore hype is overrated" ...
You can't convince me there is no value in having 4 cores at roughly the same price as last year's dual core. Despite what Don Knuth says about running multiple tasks, there are loads of processes that Windows Vista loads up just to boot the desktop. If there are more cores to handle the OS tasks then you have more CPU time to run your own tasks. And if you can't think of a way to make use of the extra cores just download the FightAIDS@Home project from WorldCommunityGrid.org and put those extra cores to use for a good cause. Don Knuth also says the following: "Other people understand parallel machines much better than I do; programmers should listen to them, not me, for guidance on how to deal with simultaneity." So even he admits he is not an authority on this subject.
I remember going to a WinHEC conference in 2003 or 2004 and Microsoft were touting some SW technology they were implementing at the time in the OS scheduler in order to reduce the hisses and pops one get while playing back media on Media Center. I think that project quietly died because they realized this problem is mostly solved by having more cores to run things on and intelligently using those cores to minimize interrupts of user tasks. Offloading to GPU/HW MPEG encoders etc also helps. This is my conjecture only but seems plausible.
Bottom line: 4 cores IS ACTUALLY better than 2 and not just because of simple arithmetic. In real world usage 4 cores is simply better and doesn't cost much more. I would also argue quad-core is far more relevant to the desktop (vs. Server) user than 64-bit was. But AMD were able to milk that advantage over Intel for all it was worth in 2004. If I had been given the choice in 2004, I would rather have had a quad-core 32-bit CPU than a single core 64-bit CPU.
Jeff, if you want to pick on hardware excess, the multi-GPU market is far more wasteful of computing cycles than multi-CPU. You double your cost, your power consumption and the amount of expansion real-estate in your PC for maybe a 20% gain in 3D performance? Tell me how that's a good trade off. Now if I could use my GPU to encode to DiVX ... hmmm
Generally I download my copies from rapidshare (which I hold original ones - beileve or not :) ) instead of encoding. For me bandwidth is less important then my CPU and hassle, and most of the professional movie rippers doing their jobs so much better than me.
John Ther on May 2, 2008 9:02 AMA lot of people in the USA still believe in the old copyright law's notion of "fair use" that allowed you to make backup copies of things. With the DMCA "fair use" has been effectively removed.
Kuerwen on May 2, 2008 9:05 AMSuper is another great encoder that can actually handle more formats then Handbrake can: http://www.erightsoft.com/SUPER.html
MarketGarden on May 2, 2008 9:39 AMWhat do you think of the Parallel Extensions to the .Net framework Microsoft is working on?
"I think the minute you touch the database or do any kind of significant I/O, the performance benefits will be miniscule. It is a small step in the right direction, of course."
Not everyone is touching a database. And if you can fit your dataset in RAM then parallelization will be more relevant. It isn't possible to solve what is fundamentally a hardware issue (I/O bottlenecks) in software. Parallel .NET is trying to address the software bottleneck that multicore CPUs have exposed. There are some problems that just aren't amenable to parallelization but for those that are (raytracing, codecs, medical imaging, etc) there will be very scalable performance gains.
Louis on May 2, 2008 9:46 AMI have re-encoded my huge collection of DVDs (500+) to mpeg4/ac3 and love it. There is very little quality loss if you do it right. I picked up a 52" DLP TV and I had to re-do some becuase of encoding at too low of a quality.
I started doing this on a AMD X2 3800+ and it would take upwards of 4-6 hours to complete one movie. I moved to a Intel e6400 and it knocked 1-2 hours off the encoding time. I then moved up to a q6600 and now I can do most movies in 2 hours or less.
It is great having my collection of movies and TV series at the touch of my fingers.
Chris W on May 2, 2008 9:53 AMI've experimented with Handbrake and CloneCDMobile.
At the moment I am resorting to ripping two copies of every DVD...one for viewing on PS3 via HPMediaVault NAS, and another for viewing on iPod touch.
Has anyone found a good codec combination that will allow ripping of one file that looks good on iPod and 40inch home theatre?
goap on May 2, 2008 10:02 AMThere's A LOT of color differentiation between the two video clips you show. In particular, I immediately noticed that there's a pink border around the adverts on the screen in the first clip, and all of that is replaced with blue in the second.
I noticed that before i noticed the hand being raised in the second, and change of wording in one of the advertisements in the second picture definitely show that these are two seperate clips taken at two different points in the film.
I only bring this up because you can't REALLY tell the difference between these two if they aren't from more exactly matched frames in the movie, and I would have dismissed your encoding evaluation as being pretty screwy...
The Postindustrialist on May 2, 2008 10:09 AMIdiocracy is a great movie!
I also see a lot more degradation in the reencoded still than you've mentioned, even in the text on the tv screen and especially in the black levels. Also, I'd disagree that pulling 8GB over the network is the headache. It's the 8Gb of storage per DVD that bothers me.
What do you do with all the bonus DVD content? Rip it as well or forget it ever existed?
Caleb on May 2, 2008 10:24 AMIt's hard to compare the relative quality of the two frames, as they're both encoded with JPEG -- which happens to be notoriously bad with red-on-black (Ow! My Balls!).
Having said that, the second image is definitely of significantly lower quality, but not enough to stop one from enjoying the movie, I would think.
Eam on May 2, 2008 10:29 AMLooks like the transcoded video has a pretty gigantic artifact right in the middle - in the shape of a human arm. Does that happen to all your movies, Jeff?
MarcT on May 2, 2008 10:30 AMYou're crazy. With storage and bandwidth becoming cheaper all the time, you're really going to regret this loss of quality. Look at all the people who ripped their CD collections to MP3 at mediocre bitrates and are now lamenting the fact that they didn't go FLAC to begin with. Video is going to go the same way. And as the fidelity of display devices also goes up, the quality loss is going to become more and more evident to you over time.
I think the one place where this is truly useful is to transcode to some low-resolution device - an iPhone or whatever. You're not going to notice the quality loss there. (Audio is the same - you won't notice the difference listening to your iPod on cheap earbuds while you ride the subway, but it would be obvious at home with a decent set of speakers.) If I had to watch a movie with compression artifacts on top of other compression artifacts like your example it would drive me batty.
Steve on May 2, 2008 10:33 AMYou should really check out the HandBrakeCLI tool. Especially on windows, it's far more powerful than the UI and almost as easy to use (given the fact that you're already comfortable in the command line). I have a few scripts which wrap around HandBrakeCLI and allow me to dispatch encoding processes a season at a time once the VOBs themselves have been ripped using MacTheRipper or DVDShrink. These scripts also allow me to devote multiple computers to the same rip, providing a truly linear speed boost.
Daniel Spiewak on May 2, 2008 11:14 AMI love that movie.
^^ SQL injection thwarted!
Mike on May 2, 2008 11:45 AMI can feel your mentioned concerns in DVD encoding and decoding.
Once I had to encode an already decoded film to make a present for my uncle. The film was decoded with a slower frame-rate, so the uncle had to watch the good film with a lot of patience.
I think it is easier to implement the formats into each existing player than re-encoding each existing film.
Cute Jasminka on May 2, 2008 11:49 AMHas anyone found a good codec combination that will allow ripping of one file that looks good on iPod and 40inch home theatre?
I don't think there is such a setting -- at some level, quality always means larger file size for video. Plus what is good for a 40-inch display would be MASSIVE overkill for a 3 inch screen.
With the proliferation of fast quad (and eventually octa) cores, you can rapidly re-encode from the original DVD to an appropriate size and quality for the target platform.
I started doing this on a AMD X2 3800+ and it would take upwards of 4-6 hours to complete one movie. I moved to a Intel e6400 and it knocked 1-2 hours off the encoding time. I then moved up to a q6600 and now I can do most movies in 2 hours or less.
That's what I'm talking about -- the quiet revolution in halving your encode times opens up new possibilities.
Look at all the people who ripped their CD collections to MP3 at mediocre bitrates and are now lamenting the fact that they didn't go FLAC to begin with
I still have the original DVDs so I can re-encode any time I want. That said, the entire goal of this exercise is to keep 99% of DVD video quality, which I think is achievable, while reducing file size by 1/4 or 1/8. DVD quality (720x480, MPEG-2) is a constant; if a DVD doesn't look good upscaled on a large TV, re-encoding it isn't going to change that fact.
Also, I found a few additional helpful links that I updated the article with:
Friendly Description of Handbrake Settings
http://www.modmini.com/theatre/howto/dvdjukebox/conversion.php
Handbrake Surround Sound Guide
http://trac.handbrake.fr/wiki/SurroundSoundGuide
@Dennis Forbes: I have a dual core. When I open task manager, I see 100%. It has been at 100% since I bought it. No, it's not being used by hackers. Right now it's solving the NQueens problem. Although I'll go back to protein folding in a few days.
I recently have gone through the process of trans-coding my Scrubs and Family Guy DVDs into H.264 to playback on my iPod. While I found it was quite easy to get settings of for a good quality RIP of Scrubs, I've yet to find some to get the same quality with Family Guy. I would have thought when this process started that the opposite was going to be true. I guess all the codecs really don't like doing Cell-shading.
As for Handbrake, I actually had to use DVD Decrypter to first remove the CSS from the DVDs before re-encoding them.
Evan M. on May 2, 2008 12:46 PMAbout a month ago I switch to using my Xbox 360 as a media center and I'll never go back.
It's so much more convenient to have my library on a server than on a shelf. My only beef is I don't know how to set up thumbnails/description text for the movies to make browsing through the Xbox360 interface better.
engtech on May 2, 2008 1:02 PM@engtch - put your preview .jpg in the same folder, with the same filename. (i.e. trainspotting.avi trainspotting.jpg). The 360 will do the rest.
notsteve on May 2, 2008 1:14 PMYou left out the biggest reason to rip DVD's in my book: getting rid of those annoying delays and forced viewing of FBI warnings, corporate logos, and ads. I will personally never buy a DVD again, they are just too obnoxious. And I REFUSE to buy a product that threatens to put me in jail if I step out of line every time I watch it.
One Clifford DVD for my daughter has 45 seconds of mandatory viewing, 4 (F-O-U-R) corporate logo animations, followed by 5 previews. Each preview had to be skipped manually, which due to DVD slowness took about 5 seconds each. So to start this damn thing required basically 2 minutes of my time. Ripped to AVI and problem solved.
spemless on May 3, 2008 3:31 AMWhile I respect, admire, and even like Knuth he doesn't write real software. His opinion isn't relevant outside his domain of expertise.
I have serious doubts he has a clue about how real OSs work and how they are used. But then again that isn't a failing on his part, it just means the scope of the universe where his opinions matter is as limited as anyone's.
For a client machine (i.e. desktop) multicore machines solve a real problem with real software. While Knuth is correct that we got them for other reasons entirely, their true purpose is badly behaving software that would consume a single CPU and render machines unresponsive. A lot of this crapware involves gratuitous polling because it comes from the Unix tradition of software where async programming is seen as a black art.
Please DON'T write multithreaded software just to soak up my multicore resources. Back off! They're MINE damnit, not yours to squander.
old fart on May 3, 2008 5:22 AMThough SUPER's site may look downright-suspicious, I can't find any evidence of adware or spyware either on the site or bundled with the program. Personally, I use it quite a bit for conversions of videos to iPod.
Andrew on May 3, 2008 5:26 AMOk, no HTML.
I'm a Spam bot on May 3, 2008 5:55 AMThe big bonus i see of duel-core is that a run-away process can only use one so it is much rarer that one bad process will crash the whole system, or leave it completely unresponsive, or too sluggish to fix. The wierd side-effect of this is that you can have a semi-frozen in windows xp where it much be core internals crashing where TM and window management is totally gone much flash applets continue to run fine--even though you cannot interact with them.
Also-even with only one core you rarely see it fully optimized (except when your computer has bad processes, and then its optimized doing gibberish) so this is really only a change that ligitimate processor-intensive tasks must accomadate
shawn on May 3, 2008 8:14 AM"The big bonus i see of duel-core ..."
"Duel-Core" ... that's a nice one.
Vinzent Hoefler on May 3, 2008 9:09 AMdid anyone else realize that the advertisement on the upper left hand corner of the tv screen in the screenshot says 'butt-fuckers'?
ADD child on May 3, 2008 9:21 AM What do you think of the Parallel Extensions to the .Net framework Microsoft is working on?
I think the minute you touch the database or do any kind of
significant I/O, the performance benefits will be miniscule. It is
a small step in the right direction, of course.
What a peculiar reply.
"What do you think of my new carburettors?"
"Well, the moment you touch the brakes or do any kind of significant maneuvering, the effects will be miniscule".
Why didn't you just say "I won't need them".
Those of us who crunch numbers and whose IO is mostly (say) DirectX are drooling over these extensions. We're hoping they'll do for threaded development what Java/.NET did for memory management. (I'm a happy ex-C++ developer who'd be happy if he never saw new/delete/malloc ever again). I hope they make mutex obsolete.
David Dawkins on May 3, 2008 10:02 AMThe single most important reason why we are going to benefit from multiple cores is increase in responsiveness that increases usability. That is, if operating systems start to use those cores for that. Then there would be on excuse for program to respond slower that human who uses it.
Another benefit would be the ability to create some freaking sweet features that have barely been thought of because doing them in real time has been impossible. How about 3D graphics creating program that shows scenes as rendered and can include chances as fast as you can make them, just like 2D programs can now. Or how about vector art program that allows more natural editing by more intelligently matching user input to vector graphics primitives.
The problem with these features is how intelligent developers are needed and the amount of research and development is needed to come up with those features (so that Clippy is not accidentally reinvented).
I understood that this kind of activity was illegal in the land of the free, and e.g. automated "media centers" that does this kind of thing have had problems or have bought licenses for their stuff.
No answer from Jeff yet, maybe he's afraid to give legal advice on this...
While storage cost has dropped considerably, there is still a strong argument for re-encoding DVDs that is more than just about file-size.
There are many DVDs out there that have been poorly encoded, suffer interlacing effects when play on the PC, or are badly letterboxed with huge gaps on the sides.
Also, coming from a PAL country (Australia), films suffer the 24-25 fps speedup.
Of course the DMCA is stupid but that's the way it is. And nobody much cares to change things there. Handbrake is probably illegal and linking to it and talking about it also.
But I thought it only mentioned breaking encryptions, and as you well know ASCII isn't meant to be encryption (even if it might be comparable to CSS in complexity...). Encoding/decoding is not encrypting.
I think some clarifications are needed...
• H.264 typically achieves half the bitrate of MPEG-2 (DVD) for same quality, not 1/4 or 1/8.
But there are catches to know:
-H.264 achieves much better on high bitrates and high resolutions.
-H.264 (as most video formats) aims for perceived moving image quality. Still images may show defects not relevant during playback, it is a bad way to compare encodes (even with the same codec).
-Audio is also a significant part of total bitrate.
-The usual encode wiseness: Source material, codec, encoding settings and playback platform can make a lot of difference. Choose what suits you.
• DVD is not original material !
-It uses itself lossy compression wich induce artifacts.
-Source material may have been digitally filtered to better accommodate those artifacts.
Both of which can seriously impede re-encode efficiency.
More on the intent of the article, I totally understand the commodity of re-encoding DVDs.
Not only do you skip search-open box-insert-load-eject-frown at scratches-put back, but you can ablate abusive mandatory viewings and bad menus. These are major nuisances.
Jeff,
I really think your characterization of the new content as having "almost no quality loss" or "99% of DVD video quality" is quite exaggerated. The big red text in the second image looks like crap, the colors have changed, the wall is a blocky mess; the general appearance of the image has shifted (in my eyes) from "DVD" to "streamed video." And I suspect it is even worse when there is a lot of motion on the screen.
It's fine if you're willing to accept significant quality loss in exchange for more significant hard disk savings. But, please, don't call those changes "tiny."
Chris Hundt on May 5, 2008 1:44 PMI read Knuth's Art of Programming years ago and completed many exercises. It was a great learning tool. Then I started reading his blog and realized that his interests are not in the machines themselves but in the mathematical arcana. He also seemed to procrastrinate a lot; it has been 20 years since he announced additional volumes in the "Art" series. Seriously, 20 years is a lot of time to get anything done, especially for a retired professor who refuses to read email, has a secretary, has a cushy spot at Stanford and has time to play an organ in his spare time. It seems to me he is not writing most of the time. Just pointing the bleeding obvious here. And oh yes. It is so great that he spelled his name in Chinese in his blog. That's totally awesome and impressive.
As of multicore PCs, main problem is that nobody really knows how to program them effectively. That does not mean we do not need them. It would be awesome to have software (and an OS) that never freezes (or twitches) when switching between 4-5 applications, however CPU intensive they are.
I am going to list my "Art" somewhere for sale, even though I do not generally sell my books for any reason. These, however, are so obsolete and irrelevant that I can no longer justify the bookshelf space for them.
SamG on May 6, 2008 2:19 AMIf I buy a dvd and I want to encode it for use on MY PC then since I bought the dbd in the first place I should be allowed to copy it to my computer so long as I don't distribute it.
I'm kinda unclear on the law but should it not be applied the same way software is?
I didn't go to law school so someone help me out here please.
Thanks...
Mac on May 7, 2008 7:21 AMWow, this article touches on two things that have really disappointed me recently: my Mac Pro running Handbrake and H264 encoding quality!
When re-encoding my DVDs, the 8-core (!) Mac takes about 20 minutes to do the job. That's pretty long. CPU usage averages at around 40% during this time which to me could mean that either the DVD drive's speed sucks badly and/or there is so much data contention in the system that the encoder can't even keep the cores busy.
Anyway, is there substancial quality loss due to re-encoding? Hell, yes! It's not enough to make me care on an aesthetic level, but my inner geek was kinda appalled when I saw ugly blocks and dithering effects (especially in near-unicolored image areas). The DVD was 4.7GB, the encoded H264 was 1.1GB - but that wasn't enough to prevent serious degradation. So... how good is H264, really?
Udo on May 7, 2008 11:21 AMWell breaking the CSS encryption on the DVD is illegal, and besides you probably just have a license to the content on the dvd in the first place... Sad but true, but what are you going to do about it?
MPAA on May 8, 2008 2:52 AMMaybe someone can help me out here.
I want to rip my DVDs, but not for the standard reason; I personally have no problem popping in the disc and waiting through the advertisements and warnings. It's of little inconvenience to me.
I want to rip my DVDs so I can use video editing software (like Adobe Premiere) to make compilations for fun. Something like a compilation of Jackie Chan fight scenes. That sort of idea.
The problem I'm running into is when I use Handbrake and try to encode to the AVI format it freezes partway through the second phase, regardless of the settings I choose or the source. I'm running a quad core and 8GB of RAM on Vista x64. When I encode to something else like MP4 it goes through fine and the end result is wicked, but problem is Adobe Premiere (and to a lesser extent, Windows Movie Maker) doesn't support MP4 files. Nor do they support MKV files.
My question is has anyone been successful in encoding (using Handbrake) to the AVI format. I use DVD Decrypter to decrypt the VOB files then use those as the source.
Thanks in advance.
Brent on May 12, 2008 5:25 AMWhen I bought my Kaypro-II (in a dodgy-looking shipping office in Miami - the secrraty was lovely but certainly wasn't Devin Devasques) in 1984? it came with a bundle of compilers and interpreters. I don't think they were all Basic; I sent it all to to rubbish heap a dozen years ago. I guess that was my first contribution to the bilg money pile.
Does anyone remember CP/M compilers/interpreters other than Basic?
Nat on May 12, 2008 7:13 AMwe are specialized DVD seller, and have many discounting commodities everyday
http://www.dvdonlineorder.com
http://hosted.ap.org/dynamic/stories/H/HOLLYWOOD_REALNETWORKS?SITE=WWJAMSECTION=HOMETEMPLATE=DEFAULT
How long until a federal judge asks Jeff Atwood to remove this post linking to Handbrake software?
We aren't free. This isn't a free nation.
Jay R. Wren on October 7, 2008 8:23 AMI have the same problem as Brent (May 12). Whenever I attempt to encode to .avi, the final stage (whether it's 1/1 or 2/2) freezes at 10.24%. This has occurred with two separate titles of exactly the same size, which leads me to guess that it's not file specific but may be size or memory specific.
Ian on October 10, 2008 7:36 AMIf you are interested in learning how to exploit multicore processors, like Quad core processors, to speed up image or video processing using, for example, open source AForge.Net, you will find a great example using AForge.Net and multithreading to improve the performance in working with images and with multicore CPUs, in the book C# 2008 and 2005 Threaded Programming: Beginners Guide, by Gaston Hillar, Packt Publishing - www.packtpub.com
It includes many exercises related to image management with multicore support. Highly recommended if you want to improve performance and UI responsiveness.
You can download the code from Packt's website. http://www.packtpub.com/beginners-guide-for-C-sharp-2008-and-2005-threaded-programming/book
There is also an article in Packt's website: http://www.packtpub.com/article/simplifying-parallelism-complexity-c-sharp
I bought the book last week and it helped me a lot in my image processing needs. Now, I can split an image in many parts using the code from the book and I can make it exploit my Core 2 Quad. My boss is impressed!
You probably shouldn't be putting information on how to violate the law (at least in the country you live in) on your blog.
anon on February 6, 2010 10:24 PMCDs are digital too. On an audio CD data is stored as two-channel 16-bit PCM encoding at a 44.1 kHz sampling rate per channel. You can extract exactly this format and listen to it without any loss in quality on your PC.
Slightly off topic: I have developed a strong dislike for anything like DVDs or even all that HD stuff. As a consumer I would like to be able to watch the stuff I bought when and whereever I want, but without a licensed player you have to use illegal software to do so. And if you want to copy it to your harddisk for streaming or backup purposes there is no way to do so without breaking some "copy protection" (which makes it illegal again).
But then, I am not too much into movies anyway, so it doesn't bother me much. And for music, I hope that the CD will stay around the primary medium for some time.
Simon on February 6, 2010 10:24 PMHow is any of this violating the law? If it were illegal, vendors wouldn't be allowed to distribute software like Handbrake in the first place.
There's no click-through license agreement with a DVD. It's a physical product that you own and have the right to do whatever you want with. Copyright laws only come into play when other parties are involved.
If you're distributing copies of the re-encoded video, then yes, you're breaking the law. But I didn't see Jeff suggesting that.
Aaron G on February 6, 2010 10:24 PMI love how everybody points to the DMCA to say that it's technically illegal, even though there's no specific language in there that makes it so. There are a few parts that might be open to interpretation, but that's about it.
Is there even a single case on the books where government lawyers have gone after a private citizen for transcoding? I'm not talking about the MPAA here, whose lawsuits generally don't involve the DMCA at all, not to mention that they only ever go after mass distributors.
Seems that many people have a hard time understanding the difference between criminal law, which is a government matter, and copyright law, which is a civil matter.
Aaron G on February 6, 2010 10:24 PMDisks and DVDs will dissappear when bandwidth increases so that it only takes a few seconds to download a movie.
When we reach that point, you will still pay for the movie, but it will be encrypted with your digital signature when you download it. All devices that you own will have the same digital signature, your own. You give the movie to someone else, they can have the file, just not play it because the signature doesn't match.
The problem with today's media is it has no such signature and thus can be copied for legitimate or illegitimate means, thus the laws (just or unjust) to protect it.
Jon Raynor on February 6, 2010 10:24 PMSUPER is also linked with adware and spyware. I don't even have to make the observation -- the hard sell on the erightsoft website makes this abundantly clear to anyone.
Simon Wright on February 6, 2010 10:24 PMMaking DVD's smaller? I've been going the opposite direction, up-scaling with FFDshow to make DVDs bigger in real-time.
I just wrote a quick intro over at:
http://blog.clinttorres.com/index.php/2008/05/03/re-encoding-dvds-for-the-home-theater/
The DMCA is too broadly written and will eventually be overturned when someone points out to the courts that under its provisos the entire internet is "illegal" in the U.S.
The content of this very site is copyrighted in human readable symbolic form but in order to get to each and every one of our computer screens it is "encrypted" into a binary "code" (ASCII is one example) which controls access to said copyrighted work and that upon arrival has to be "decoded" into human readable symbols on our screen.
Nowhere on this site are we expressly "authorized" to decrypt said binary transmision into human readable symbols by the author.
The internet violates the DMCA by its very existence.
But that's just it MPAA, the DMCA is so broad it doesn't care about "intent" or whether a mechanism is any good at encypherment.
The U.S. courts have shown no inclination to distinguish between weak, silly, trivial or strong encyphering when ruling on the copyright holder's right to control under the DMCA.
Mathematically there is no distinction between the human readable symbols "encoded" in ASCII and a simple (binary) numeric substitution encyphering.
digital digital versatile discs?
John Ferguson on February 6, 2010 10:24 PMThe comments to this entry are closed.
|
|
Traffic Stats |