July 5, 2007
In a previous post, I examined the difference between bilinear and bicubic image resizing techniques. Those are the two options available in most graphics programs for resizing an image.
After some experimentation, I came up with these rules of thumb:
- When making an image smaller, use bicubic, which has a natural sharpening effect. You want to emphasize the data that remains in the new, smaller image after discarding all that extra detail from the original image.
- When making an image larger, use bilinear, which has a natural smoothing effect. You want to blend over the interpolated fake detail in the new, larger image that never existed in the original image.
Of course, there are plenty of conditions that might make you want to choose one method over the other, but I think these are reasonable guidelines to start with.
What I didn't realize when I wrote the original article is that there are other, more advanced resizing algorithms available. Some are specific to particular kinds of images, such as the 2xSAI algorithm which works on pixel art. Compare this shot of Mario vs. Wario using pixel resizing, and the same shot using 2xSAI resizing. It's a dramatic difference, especially since traditional bilinear and bicubic upsizing methods degenerate into a giant blur on pixel art.
Supposedly, one of the best image resizing algorithms on the market is Genuine Fractals. The web site boasts that you can use its fractal-based resizing algorithm to "enlarge your images over 1000% with no loss in image quality". It's probably pure marketing hyperbole, but I was still intrigued. Bilinear and Bicubic are decent, but there has to be room for improvement in there somewhere. I downloaded a trial version of the tool (which requires Photoshop Elements, or Photoshop CS) and gave it a shot.
I took the the reference Lena image and blew it up 500%.
Here's a closeup of the results using Bicubic Sharper:
Here's the same closeup using Genuine Fractals:
Bicubic wouldn't normally be my choice here, but I chose it because it's technically the most advanced method, and it produces the results closest to the effect that the fractal resizing delivers. Still, the fractal algorithm comes out way ahead; you can't see any pixel resize artifacts in the enlarged image, and the edges are sharp and well defined. It does start to bear an unfortunate resemblance to a watercolor drawing filter, but arbitrarily resizing images to 5 times their original size will always involve tradeoffs of some kind.
Bicubic and bilinear are well understood image resizing algorithms, and they're "good enough" for most image resizing chores. That's why they are provided out of the box in almost all graphics applications and graphics libraries. There's an outstanding article on CodeProject which digs into advanced image resizing algorithms with actual C# code for some spline and fractal resizing algorithms. But before you begin resizing images, consider whether you need those advanced algorithms.
Reducing images is a completely safe and rational operation. You're simply reducing precision and resolution by discarding information. Make the image as small as you want, and you have complete fidelity-- within the bounds of the number of pixels you've allowed. You'll get good results no matter which algorithm you pick. (Well, unless you pick the naive Pixel Resize or Nearest Neighbor algorithms.)
Enlarging images is risky. Beyond a certain point, enlarging images is a fool's errand; you can't magically synthesize an infinite number of new pixels out of thin air. And interpolated pixels are never as good as real pixels. That's why it's more than a little artificial to upsize the 512x512 Lena image by 500%. It'd be smarter to find a higher resolution scan or picture of whatever you need* than it would be to upsize it in software.
But when you can't avoid enlarging an image, that's when it pays to know the tradeoffs between bicubic, bilinear, and more advanced resizing algorithms. At least arm yourself with enough knowledge to pick the best of the bad options you have.
* e.g., if I really needed the Lena image that large, I'm better off hunting down old copies of Playboy and scanning them myself. Or at least that's what I tell my wife...
Posted by Jeff Atwood
I think you may have these backwards:
* When making an image smaller, use bicubic, which has a natural sharpening effect. You want to emphasize the data that remains in the new, smaller image after discarding all that extra detail from the original image.
* When making an image larger, use bilinear, which has a natural smoothing effect. You want to blend over the interpolated fake detail in the new, larger image that never existed in the original image.
Bicubic definitely smooths more than bilinear. It interpolates from a 16x16 neighborhood, whereas bilinear interpolates from a 4x4.
Regarding enlarging: the big problem is information density. By definition, interpolated pixels cannot add information to the image. The information entropy of the interpolated image (the minimum number of bits necessary to encode it) will always be exactly the same or less than the original. Things like Genuine Fractals do better (by visual inspection, or provably with certain assumptions, for a single class of images) by making strong inferences about the new pixels based on local geometry.
It'd be more correct to say that you can't magically synthesize any pixels, period.
(I'm being pedantic, but that's only because I do research in this area. Oh, and someone might find my pedantry useful - at least, that's what I tell myself.)
The two image resizing kernels most famous for being "soft" and "sharp" respectively are Gaussian and Lanczos.
Some common software like IrfanView should implement those; for some reason, Photoshop doesn't.
You can use something like this to try out different options:
Just one thing: why would anyone want to upscale an image? By doing that you increase the file size (as well as processing power required to display the imagem) and decrease the quality. I just don't see the point.
Of course downscaling is much more useful. I always do that before publishing pictures I took.
Here's a nice explanation of the tradeoffs between aliasing, blurring, and edge halo in non-adaptive interpolation:
"Even the most advanced non-adaptive interpolators always have to increase or decrease one of the above artifacts [aliasing, blurring, edge halo] at the expense of the other two-- therefore at least one will be visible."
what about GREYCstoration? it does pretty good inpainting, I even got rid of the flag in the Windows 95 logo, once! (easy because it's just clouds, which look GOOD when blurred)
A little off-topic, but there's a serious downside to attempting to smooth upscaled pixel art images: they become smooth. As a retrogamer, I just can't live without being able to see every pixel exactly as intended. "Nearest neighbor" used by increments of exactly 100% is the only algorithm I ever use when I want to enlarge some sprites.
Yep, HQ?x (hiend3d.com/hq3x.html) is clearly the most sophisticated algorithm for upscaling low-res images. Though it's more suited for cartoon-like scenes with less colors - think Super Mario World instead of Donkey Kong Country.
Personally, I prefer good ol' interpolation.
You'd want to upscale an image before printing. Most printers, even very good, industrial photographic and poster printers, don't do a great job. Upscaling requires subjective judgment, which a computer can't do. You *know* that a bit of a photograph is supposed to be a whisker, and you know what one is supposed to look like. You can tweak it until the upscaled whisker looks right, but the default interpolation a printer uses hasn't got a clue.
My senior project in college (undergrad) was to implement fractal image compression and study the differences between it and other compression techniques. It was a great project that I really enjoyed - and taught me a lot about image compression in general. (That was the idea, I suppose.) Thanks for the post, Jeff! Now I'm going to have to go dig out my final paper and re-read it. :)
Good summary, Jeff.
Bicubic = sharpening, bilinear = smoothing just isn't a good rule of thumb. It looks like the second kitty image in the post you linked to was interpolated using a bicubic interpolator with a Catmull-Rom kernel. If it had been a B-spline bicubic interpolator, for example, it would have been much smoother than the result from the bilinear. The Catmull-Rom kernel is actually sort of an odd duck as cubic kernels go, as it yields a C^1 continuous surface, making the result rather sharp, whereas most (maybe all) other popular cubic kernels yield a C^2 continuous surface.
It really comes down to the fact that weighted averaging over a 16x16 area will generally yield smoother results than weighted averaging over a 4x4 area, except in special cases.
Photoshop has three different bicubic interpolators, and I would guess that Catmull-Rom is one of them. B-spline is probably another, and I have no idea what the third is. (They may even be using BC-splines, which is a generalization of many popular splines, with hand-picked assignments of B and C - see http://www.cg.tuwien.ac.at/~theussl/DA/node11.html.) The thing is, which kernel is used for a "bicubic" interpolator will vary from photo package to photo package, which is what really kills the rule of thumb.
I'll dig around for a page that demonstrates the difference between Catmull-Rom bicubic interpolation and B-spline bicubic interpolation. If I can't find one, maybe I'll make one.
The real question I have is: why don't Firefox or IE implement the better resizing algorithms? It's sad that client-resized images on the web look terrible, even they really don't have to.
I agree with lord trousers - after resizing images up + down for nearly a decade, bicubic is usually best for up + bilinear for down, whether it's Paint shop pro, photoshop, or coding in dot net.
To answer Ivy Mike, I'm guessing the browsers go for speed.
I once read that expanding an image, using photoshop, but only increasing it by 10% at a time produces dramatically better results, and indeed it does.
In fact, I just tried it on the reference image, and I believe it does produce better results than either method listed above: http://compucatedsolutions.com/Lenna.png. And that is actually 500%, with a size of 2589 x 2589. You can grab the action I used here (http://compucatedsolutions.com/imageupsize.atn), just keep tapping F10 till you get to the desired size.
I have used this to blow up legal size documents to poster size for print, and it has always worked like a charm.
I guess you forgot to mention scale2x and hq3x for pixel art, these are the generally best accepted algorithms for upscaling pixel art imagery, as they don't suffer from blurriness.
for any other high-color image, it could be worth to mention Lanczos Interpolation, a great resampling algorithm that clearly surpasses Bicubic for any given image.
To be nitpicky, not always "Reducing images is a completely safe and rational operation". When you are working with icon sizes, photographic images almost don't work and pixel art is mangled slightly beyond recognition. It's that pixel-grid from the cleartype discussions, again.
kind of related, but do you ever notice how hollywood goes nuts with image resizing, mostly from security cameras? they'll take a crappy image, and tell the IT guy "hey, can you sharpen that up?" then in 5 seconds it's a crystal clear image of their suspect.
i think the only movie i've seen get this right is "revenge" with kevin costner. in this movie, it took them days to enhance/reconstruct an image.
the worst one was probably "enemy of the state" with gene hackman and will smith where they magically create a 3d model of a handbag to see a large bulge in the side that isn't even facing the camera.
Yes, upscaling for printing may be necessary, but today many consumer cameras have enough resolution to print on an A4 sheet with good resolution. And still most people only print 4" x 6" pictures. At that size, you only need to upscale if you take pictures with less than 2 megapixels.
So the average person won't ever need to upscale an image. I have never had to do that, and I have worked for a few years with computer graphics, and as an amateur photographer. I only see the point if you are working with really large images, but if you do you probably have a professional camera with REALLY sharp lenses, and won't need much more resolution either.
I think we can agree that we would prefer to downscale instead of upscale given the option. However, there are times when a large image is not available and upscaling is the only choice. You'll never know when you'll need that one image that just happens to be larger than the largest that's available.
Availability is the big question. I've traded emails with a guy from a commercial research group working on better upscaling for DVD - HDTV conversion. Your typical consumer doesn't have the luxury of working from the original in this case.
The font is damn blurry here...
If you're interested in quality, lord trausers is right... you're looking in entirely the right place. Try the open source tool ImageMagick ( http://www.imagemagick.org/script/index.php ) which supports Gaussian and Lanczos, among a zillion others.
There's a lot of information theory (specifically sampling theory) which goes into this, and I'm highly skeptical that fractals buy you anything. I know a little about fractals for image compression (generally, they just mean some variation on wavelets) and, seeing as Lanczos comes straight out of information theory, I find it hard to believe that fractals would do any better.
Come to think of it, the Genuine Fractals image looks no better-- and in some ways worse-- than the video scaling built into Mac OS X. The fractal one seems to be doing erronious line/edge detection. Note how the hair to the right of her face doesn't curve smoothly; the "watercolor effect" Jeff describes.
Hi Jeff. Another contender in the image resizing arena is Alien Skin's BlowUp, just as the previous commenter mentioned. It's here:
My testing shows it to be marginally better or at least as good as Genuine Fractals, though it's relatively a newcomer and thus not as popular.
I helped make a resizing program called Blow Up. In my opinion it's always more attractive than bicubic or bilinear. In some cases it does better than Genuine Fractals, such as the Mario example. In other cases it's about as good at Genuine Fractals. Here are examples from this article.
We use a polygonal mesh approach rather than fractal compression.
I didn't want to post this because I don't want to seem like I'm using Coding Horror for advertising. But Jeff Atwood encouraged me to post, so here you go.
What I'd like to know is, it's possible to upscale like we see on TV series?
Everytime I see a low res image of video/camera upscaled by 10 times. And It looks almost perfect. :-)
I realize this is nitpicking, but why refer to "Genuine Fractals" as an algorithm? Theirs is a product, the exact algorithm behind which they're probably trying to patent (if it isn't already patented-- I seem to recall from a data compression class many years ago that fractal image compression is patented). The two methods you highlight, one could conceivably research and implement; with GF you can only buy their product.
There's an even newer contender out there. It's free to use for desktop resolutions. See for yourself: http://reshade.com.
It seams to work better than any of the other commercial products for resizes of up to 300%.
Thank you, very interesint information. We use CXImage library in one of our products - http://www.softorbits.com/batch_picture_resize/ it supports 8 types of picture resizing algorithms including BSpline, lanzcos, hermite, bicubic (soft and hard). We will try to implement fractal algorithm for more qualitative resizing.
Thank you for your information.
We have a very cheap product called Image Resize
which use the so used Bilinear, Bicubic (sharp and smooth), but we have
also implemented the Biquadratic method (sharp and smooth).
The latest is fastest than bicubic with no deterioration.
But the best to conserve the details is real the Bicubic sharp.
Is a PC the fastest way to resize images? Is there a FPGA, GPGPU or other specialized hardware that could beat a SOTA x86 PC?
Anyone know of any comparisons such?
Open source Clone of Topaz Moment.
Hello People. You all seem very knowledgeable here about about upsizing ang resampling and I wondered if you could give me a few pointers. I have deciced to make the move to an open source O/S (kubuntu), however an recent favourite program of mine Topaz Moment is Windowz only. I know of Wine and virtualisation, but since I'm a software developer I want to try and create an open source clone of Topaz Moment to give something back. The info about it says it obtains its fabulous results by sampling adjacent frames to one you intend to grab. I wonder if some of you guys could give me a few pointers about this technique and links to any possible algorithm implementations
Thanks in advance T
Thank you for the great article. I found a great tool for resizing and processing of image .This tool provides image resizing maintaining quality. It is very easy to use and you can test it online using their free online resizer. I wish people would talk more about it because it can solve a lot of problems concerning resizing and quality of the picture. I hope it will be useful for you too as it was for me.You can find it at http:reshade.com
i lik you mario so i lik yer games ad i for got this dear mario
Jo jo: I'm not sure what you want to ask, but an FPGA can always be faster for this kind of specific algorithm.
I find the best Method in Photoshop is to Change your mode to 32 bit, then downscale with bilinear. This also prevents the Gamma Error http://www.4p8.com/eric.brasseur/gamma.html?
Once you're done resizing down - change your mode to what you had it last, be it 16 bit or 8 bit.