"If you're worried the person on the receiving end of the archive won't have a RAR client, you can create a self-extracting executable..."

Please see the attached self-extracting RAR format Anna-Kournikova pictures, which I'm sure you will find of interest.

Sorry, if people send me self-extracting compressed files, I don't run them.

I note that none of the compression formats used by the free 7-zip archiver (7zip.org) are discussed here. As 7-zip will unpack (though not create) RAR, that's my tool of choice these days.

Yes, the self-extracting scenario is not so good for email. That's when you opt to produce a ZIP file from WinRAR.

7zip is nice but it's unbelievably, godawfully, geologic-time slow on anything larger than, say, a breadbasket. It does very poorly on this particular dataset, but 7zip does generally offer somewhat better compression rates than RAR-- but at the cost of literally 3x-5x more time. Same problem I documented in the post, really.

I am all for WinRAR, too, having bought a couple of licenses from them a long ago.
The only problem with RAR is that if you don't know the receiver you have to use zip....

Uf, almost forgot. Another downside of RAR is that they don't offer compression library to use it with your apps.

I use WinRAR extensively because I love its shell integration; however, I wouldn't tout it as a universal format to switch to from ZIP.

RARLab publishes the data structure and source code for the decompressor, but does not for the compressor; in fact, it expressly states in the license that the developer documentation/source may not be used to recreate the compressor. They have been fairly consistent about enforcing this as well; they want to ensure a revenue stream up from RAR. (Which is their right, but it also makes me less likely to use it longterm.)

I'm all for moving away from ZIP, but I'd prefer that it move towards another format that is more open. Unfortunately, data compression is currently a patent minefield, especially if you want to implement arithmetic coding, so nobody wants to work on it.

I'm clearly being very thick today. Can you explain how to read your graph? How do the blue and red lines interact, if at all? The axis along the bottom isn't labelled. Is that the software used? I can see the kink in the red line between 60% and 70% (eye-balling puts it at around 63%), but nothing at about 73%.

Actually for that matter can you explain the Google spreadsheet? What's the Y/N column for example? And the first column?

"Except in cases where backwards compatibility trumps all other concerns, we should abandon the archaic ZIP format-- and switch to the power and flexibility of WinRAR."

Unless you're not a fan of vendor lock-in. Me, I think I'll stick with zip and bzipped tarballs.

If I send someone a ZIP they can open it ..
If I send someone a self extracting ZIP or RAR then it will be blocked or stripped ...
If I send someone a RAR they will not be able to open it (and it might get stripped as a "unknown" format) ...

So I send ZIPs ...

Same reason I send PDF's, and not another format ....

Interesting post. I redid the graph to use the X and Y axes for time and ratio respectively, I find it clearer that way.

After all your praise of RAR I find it funny that you post the Excel spreadsheet as a ZIP though ;-)

http://www.isotton.com/sandbox/graph.png

As a followup: the red lines are the means, the blue lines are the medians.

(The mean is the sum divided by the number of elements, the median is the value which divides the elements into two equal sized sets).

So the algorithms in the top left part of the blue cross are the ones which are "better than most". Someone else could label them...

Interestingly, Microsoft's CAB archives are symmetric - at least half the time spent on a typical Windows install is your CPU chugging away at decompressing the installation media. I guess that was the only way they could get the OS to fit on one CD, back before WinRK's ROLZ3 existed. The decompression is single-threaded as well, so multicore doesn't get your XP installed any faster. (This may have changed with Vista, I don't know.)

7z is an excellent choice if you're not on Windows and don't want to deal with Wine, obviously, but its GUI and integration is pretty lacking. I have no issue with its speed, as long as PPMD isn't chosen. Winrar is practically the default upgrade from the built-in Zip support, though, it's so well-known and so good.

(I like WinRK 3 quite a bit, it was going good places with a nice GUI, but then development died and I think the beta version expired months ago. =\ Sad. SBC is neat, but no GUI - WinSBC barely counts - and it's single threaded and also dead. WinRAR murders its efficiency in SMP mode.)

I am using 7zip both on Windows and Linux. It's better than WinRAR IMO.

Wow! Solid archives! Where did they come up with that idea? What innovation!

Hmmm....the multiple file compression benchmark isn't that comprehensive though - take a look at the following:

"Considering the fact it's supposed to be a 'real-world' test I will not look at the best possible (command-line) switch combination to use for optimal compression, but only test a limited set as 'normal users' would do. For 7-zip this means for example I will use the GUI and select the Ultra compression method, WinRar will be tested with max dictionary size, solid archiving etc."

So, for GUI programs he's allowed to tweak them as much as they possibly can be, but command-line programs have to use default options? Come on! All compression programs give you a compression/speed trade-off and stick the default near the middle; selecting "top compression" for all the GUI tools but not for the command-line tools is just unfair to them. And that's not even taking into account using a GUI tool that can drive bzip2 with pretty clickies. Pah!

7Zip produces smaller zip files than WinRAR does. Tested and it's the only thing keeping me using 7Zip as it's slow with nearly everything other than it's own format.

Please, I urge readers to ignore this advice. WinRAR is a non-free application and a non-free format. In simple terms this means that you cannot copy, modify, study or freely use applications that create RAR archives.

If there is a bug in the WinRAR implementation or a feature you would like to add you will be stuck. If you would like to write your own version of the software or improve it in anyway, you will be stuck.

Even to write an application that unpacks RAR archives you will have to licence your code under an unRAR licence which is not Free Software compatible.

If the makes of WinRAR ever decided to abandon the project you will all be stuck with a stale collection of RAR archives that you will only be able to unpack, never repack.

I recommend you use GNU Zip[1], the free compression utility along with the tar[2] archiver.

[1] http://en.wikipedia.org/wiki/Gzip
[2] http://en.wikipedia.org/wiki/Tar_%28file_format%29

And WinRar came from USSR...

Try WinUHA, it's a great free option! (sometimes better than WinRAR!)

Noah: Are you serious? Don't use WinRAR simply because it's not free? Because you can't add your own features? I'm sure there's a Bileblog article written specifically about you somewhere.

If the makers of WinRAR ever decide to abandon the project, we'll be stuck with the last released version. Hopefully by then it will be able to re-pack RAR files. I'm sure it'll be a popular feature.

Your committment to support independent software developers is praiseworthy.
I completely agree, and bought my WinRAR license some months ago as well (best tool I ever used... 7zip is not only slower, but has also - IMHO - an ugly interface).

First better JPEG, now better ZIP compression. I think it must be time for Jeff to get a bigger disk or faster internet connection or something!? :-) Storage space is nearly free at this point. No, correction, since Gmail, it is free!

Bandwidth is still far from free

I'd have to agree Noah Slater, it's a proprietary and closed format. 7-Zip is your friend.

Jeff - I've only recently started following your blog, and I'm very impressed. I'm a developer with an interest in data analysis, and I appreciate very much you sharing your excel file. I love seeing what other people do with their data, and as I was reading your article I was preparing to ask you for a copy of the xls. Seeing it available at the end was a great pleasure - thanks so much for realizing that there are others who may want to tinker with the data themselves - not necessarily to refute your claims, but just to get our hands dirty ourselves.
Keep up the great work!

Even if some tool had 10x the performance of 7-Zip, i will continue to use 7-Zip. I don't want my data locked in a proprietary format.

For what it's worth: same archive tarred and gzipped - 72K. Tarred and bzip2-compressed: 54K. Open formats, libraries available for just about any system, free tools abound, and can be opened and read by WinRAR among many, many others.

I'll point out that a lot of times, the material being compressed might vary your choice in compression algorithm. For instance, tar-bz2 of the csharp snippets seems to still do a bit better than 7zip:
53K ms-csharp-snippets.tbz
59K ms-csharp-snippets.7z
But as the linked survey shows, bzip does less well on other material.

I'm also a big fan of 7-zip which supports quite a few compression formats and it's own 7z format is fantastic. That and the fact that it's open source...well, the price is right. And in this case, you get way more than you pay for.

$ tar cjf csharp.tar.bz2 ms-csharp-files/
$ ls -l csharp.tar.bz2
-rw-r--r-- 1 user user 54786 2007-02-23 09:17 csharp.tar.bz2

That is all.

Eam, yes I am entirely serious. Insult me all you like, but I'm not the only one who thinks like this.

http://www.google.co.uk/search?q=free+software+movement

Why bother praising WinRAR for it's excellent compression when the algorithm is not freely available? Keeping such advances from the rest of society does not benefit anyone and should be called out as such, not praised.

"Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman

I personally like Power Archiver (http://www.powerarchiver.com/) it has integrated shell support as well as complete built-in support for 7-ZIP, ZIP, CAB, LHA (LZH), TAR, TAR.GZ, TAR.BZ2, BH, RAR, ARJ, ARC, ACE, ZOO, GZ, and BZIP2 file formats. (http://www.powerarchiver.com/features/list.php). Another feature I love about it is that it can unpack ISO CD-Images. I mostly use the 7-ZIP compressions because it seems to pack my files tighter than ZIP, RAR, or GZIP. Try it out you will not be disappointed. I haven't and I have used it for 4 years now.

Lck - "7zip is not only slower, but has also - IMHO - an ugly interface."

I agree! I only use it because I use PortableApps on my thumbdrive and it seems to work great for lugging around all of my school-related code snippets and such. Being a novice coder, my code seems to bloat more than anything else. I really like 7Zip, but don't care much for its interface.

Interesting post. I remember back in the day doing lots of comparisons between different formats.. .zip, .arj, .lzh, etc (remember those?)

These days I use on Windows systems I use 7-Zip, and on *nix I just use GNU Tar/GZip

- Corey

------
blog: www.goldb.org/goldblog

> Another feature I love about it is that it can unpack ISO CD-Images.

Note that the Winrar client is perfecly able to unpack ISO images, as well as 7z archives for that matter.

When talking about compression it's not necessarily the best compression the thing that matters the most. It's also about availability, licensing, and such.

This issue is rather similar to the one that you discussed on video quality before (http://www.codinghorror.com/blog/archives/000755.html)

Comments from people like Noah Slater (who is, no doubt, a very nice person, no harm intended) help convince me more and more than this Open Source cult is nothing more than creeping Socialism.

That can't be good for *anybody*.

Sorry, Noah. Can't go there.

"Insult me all you like, but I'm not the only one who thinks like this."
I know, and it's mildly annoying.

"Why bother praising WinRAR for it's excellent compression when the algorithm is not freely available?"
Because it's cheaply available.

"Keeping such advances from the rest of society does not benefit anyone and should be called out as such, not praised."
I agree with you up to a point: I'd also like to see the details of the algorithm published. However, when I'm paying for WinRAR, I'm paying the authors for the excellent work they've done, which should be praised, and not "called out".

"Creativity can be a social contribution, but only in so far as society is free to use the results." - R. Stallman
You're killing me, boo

Another "Yay, WinRAR" post here, although the reasons I like it aren't because of the .rar format. .zip really works just fine for me, and it's a bit of a defacto standard.

No, the reason I love WinRAR is because of the spartan way it integrates into the Windows shell. I don't want a notification area *cough*systemtray*cough* icon for my decompression utility. Seriously, how often do I zip and unzip files? It's comparitively rare. WinZIP tries to be this helpful decompression buddy but it just comes off as annoying -- like Clippy. WinRAR does the job when called for and quits when it's done. bam. No fuss. Done. I only ever enter the GUI when I'm building complex or complicated archives.

Ok, I'm starting to rant, and before I get started on Acrobat Reader, I'm gonna go ahead and quit.

hehe! best compression ratio is by far KGB ARCHIVER (if you can spare the CPU and the time and the 2 Gb RAM used for EXTREME COMPRESSION.....)

What I think about both algorithms.

Zip. Almost obiquitious compression format. Multi-platform well documented. It's integrated on the shell of the mainstream operative system. So you can be sure that Joe can open the archive.

Rar. Proprietary closed format. Legally you can't use a lib to create .rar files. You need to install extra software to handle this format.

Well in the Unix world things are different.

Personally I'm sticking with the gzip and the bzip2 formats.

Don't know 7z but looks like a great format.

Also, PocketRAR (http://www.win-rar.com/pocketrar.html) is freeware. It's the best compression/decompression program for PocketPCs, and handles both RAR and ZIP format.

I have WinRAR installed but only use it to decompress files since the format is proprietary and closed. Typically I just stick with simple zip since it is ubiquitous. I have used 7-zip in the past (but only for standard zip archiving) and found it a bit slow, maybe I will give it another try using it's native published format (7z). I have also used Power Archiver (http://www.powerarchiver.com/) and like it quote a bit, I didn't know it had built in support for 7z! I will have to see how it compared in the tests.

> If I send someone a ZIP they can open it ..
> If I send someone a self extracting ZIP or RAR then it will be blocked > or stripped ...
> If I send someone a RAR they will not be able to open it (and it might > get stripped as a "unknown" format) ...

Actually, we have several clients (major NY financial firms) who are also unable to send/receive .zip files without them being stripped. So we end doing a lot of .zip_ or .zip.txt... stupid mail policies.

You've made the error of judging a service by the provider.

Your code snippets, for example, will be compressed ONCE, but downloaded and decompressed MANY TIMES. Take the download time, and add to it the decompression time, and you have a more accurate number.

An archiver that was for taking backups -- emergency archives that will hopefully never be used -- should be judged for compression time, since decompression is a rare event, so for that, RAR is a great choice.

That 2% compression for 5x time, it depends on whether than means you ship one CD or two, 2DVDs or 3 DVDs, and the logistics that a company has around part numbers, packaging, support ("No, Mr Wilson, you need to start with the other disk first"), etc.

I have to agree with Noah Slater and all the others who insist on using a free compression algorithm.

Archives (that's valid for all file formats, but archives is what we're talking about here) are bound to stay here for a long time - maybe even decades. So saving your data in a proprietary format is the most stupid thing you can possibly do. What will you do if you want to repack a RAR in 10 years - for whatever reason? Winrar might not exist any more, and you might not have a Windows box available for the task.

What will you do *now* if you want to pack or unpack a RAR on your Sparc, Alpha or Itanium server? The answer is you can't.

File formats should be open, cross-platform and there should be at least one major open source implementation for them before I consider using them for my work. This has nothing to do with "socialism" as some seem to think; it has to do with me wanting to be able to access my data now and in the future.

I think it is great if someone writes a commercial, closed source Zip program which is twice as fast as anything else - I can buy it and use it if I want, but it will give me a standard Zip file I can work with using dozens of programs on dozens of platforms. Or I can even implement my own if I feel like. *That's* the point of open file formats.

RAR's a good compression format, but I've never liked WinRAR. Thankfully, my zip utility of choice (IZarc) works fine with them, but it'll be tar/gz, tar/bz2, or classic ZIP for a while yet.

Noah, the existence and popularity of Winrar has not hampered the growth of Bzip2 and 7zip, so I'm not sure what your point is. We'll always be able to extract rar archives long after the format is abandoned, on any platform (as we can for many other long-dead formats), because the source code for unrar is freely available. Whether the license requires external plugins is irrelevant. Then you can repack them into 7z or whatever the flavor of the decade is by then.

Odd that you recommend gzip for archival, though, when its main niche is ultra-highspeed compression and total backwards compatibility.

I don't think it's stupid to base all of your software decisions on 'open and free', but many people don't have that luxury and would rather pay for closed source software up until a truly better open-source equivalent comes along. Admittedly, Power Archiver is a pretty decent analogue to Winrar, but it's just as Windows-only and closed-source.

Er, Aaron, you might want to take a look at:
http://www.rarlab.com/rar_add.htm

WinRar also allows you to put a recovery record (up to 10%) in the Rar file, which allows you to repair the archive if slightly corrupted.

I gotta agree with Aaron here. These are archives; they should be in an open format. Ten years from now your data's going to be locked up and you won't have the key. Maybe you'll have the WinRar executable, but a lot of good that'll do you on new hardware, OS, etc.

"What will you do *now* if you want to pack or unpack a RAR on your Sparc, Alpha or Itanium server? The answer is you can't."

Just because you say so?

I've had to use RARs on an ancient OpenVMS server before, and didn't have a bit of trouble.

http://www.rarlab.com/rar_add.htm

stick with 7-zip. the fact thats its open source is reason enough to choose it over winrar.

Phil Katz, RIP :)

Another vote for ZIP until RAR is implemented on a bunch of platforms. You can still get PKZip for Windows if you don't like WinZip, among a bunch of other free Zip clients.

The RAR guys should recognize that it is in their own interests to release the RAR format so that others can write to their format. Their superior compression would actually stand a chance at the kind of ubiquitous adoption that ZIP has.

WinACE seems to have won this compression shootout. At least for the MOV file. The .tiff and .wav file compression size was comparable to .rar

http://news.softpedia.com/news/WinAce-VS-WinRAR-VS-WinZip-17365.shtml

winRAR supports writing and opening .7z files.

According to Wikipedias comparison page. http://en.wikipedia.org/wiki/Comparison_of_file_archivers
7-zip wins almost across the board.

Edit...

Another vote for ZIP until RAR is implemented by **more than one vendor**.

Not to mention that the best compression ratios often come from RARing .zip files.

Sorry to disappoint mrprogguy, I am a proponent of the Free Software movement. The Open Software movement is something entirely different.

Using the words cult and socialism clearly show a lack of understanding on your part and you would do well to do some research into the issues at hand.

Eam, just because the authors have chosen this particular economic model for their business does not mean we should support it. There are many other viable ways to profit from software development. If the author released the algorithm and source code under a free (as in speech) licence that would truly be something to show appreciation for, be it in thanks or money.

Just because something is cheep doesn't mean we should praise the distributor. Free (as in libre) software is not the issue at hand - if it was, you would chose one of the free (as in libre) alternatives.

Foxyshadis...

> Noah, the existence and popularity of Winrar has not hampered the growth of Bzip2 and 7zip, so I'm not sure what your point is.

My point was that it is unwise and unethical to use non-free formats/software such as this.

> Whether the license requires external plugins is irrelevant.

Untrue. If I wanted to package a free (as in speech) operating system such as Debian I could not include such tools and hence your RAR files are useless.

> ... but many people don't have that luxury...

Sorry, come again? Many people don't have the luxury of being able to download/get a free (as in speech and beer) GNU/Linux CD and replace Windows? Are you for real?

+1 for WinRAR, but as others have mentioned, it is no replacement for ZIP when sending files to people who might not have WinRAR or who might not have the computer acumen to know what the heck a .rar file is or how to decompress it or don't trust running an EXE sent via email.

I use WinRAR to make/decompress ZIPs for externally-bound archives, but use the maximum RAR compression when making local backups.

Jeff started the blog with Today is "Support Your Favorite Small Software Vendor Day" why must EVERY thing dealing with computer be FREE. I make a living writing software and it is my ideas that I've put into my code. WinRAR is a good product with good stability and I don't think that people should abandon software companies simply because they would like to stay in business. I'm all for paying a reasonable price for a stable product.

Dennis, there is a difference between free as in libre and free as in speech. You cannot tell me that there is no money to be made by distributing free (as in speech) software.

Red Hat? SuSE?

Noah, Linux is free (as in speech) only because Linus decided to make it that way, he could have and still could enact his intellectual rights copyright if he so chooses. An make people license the kernel. I like open software and its concepts but I also think that people deserve to get paid for code that the write if they so choose, and that good, low cost, software should be supported.

You mention Red Hat, I think they violate the spirit of open software by charging for the management tools they've writen that wouldn't have a place to run if Linus and countless other people hadn't be generous with their "free speach".

Lck - "7zip is not only slower, but has also - IMHO - an ugly interface."

True, but I still use it as its the only one with shell support for WinXP64.

WinRAR is free anyway... trial period never expires.

hi Noah Slater,

could you post a reasonable business plan for writing and releasing software like winrar? (software that everyone understands)

thanks

>Noah, Linux is free (as in speech) only because Linus decided to make it that way, he could have and still could enact his intellectual rights copyright if he so chooses. An make people license the kernel.

Your point being? The Linux kernel isn't the be-all and end-all of all free (though Linus would prefer the term open source) software.

My point was that Linus could do what we wants regarding the kernel but it wouldn't make a shred of difference. He could continue to develop his non-free version of the kernal but the rest of the free software and open source communities would thrive around the free version of the kernel as Linus cannot revoke the GPL licence on previously released version.

That is the core of my point. The Linux kernel will be around longer than me, I doubt I can say the same for WinRAR.

Which sounds like the best option from a pragmatic, future proof, perspective?

>I like open software and its concepts but I also think that people deserve to get paid for code that the write if they so choose, and that good, low cost, software should be supported.

I agree completely. I am a professional programmer. However, not everyone chooses to make money solely by distributing non-free binary blobs.

hacktick, sure thing. Type "Noah Slater" into Google to get my email and we can discuss my consultancy fees.

Noah Slater: "Sorry to disappoint mrprogguy, I am a proponent of the Free Software movement. The Open Software movement is something entirely different.

Using the words cult and socialism clearly show a lack of understanding on your part and you would do well to do some research into the issues at hand."

Wrong. The Free Software Movement is socialist (or at least communist). "Everything belongs to everyone."

Free software is great, for those developers who want to give away their work. It's *not* an entitlement, as you seem to think. I write software for a living, both at my full-time job and on my own time. I choose to pay my bills, have a roof over my head, eat at least once a day, and being paid for my skills does that.

And Red Hat charges money for *supporting* free software.

I do believe you're at the wrong site. Try http://www.slashdot.org instead. :-)

Randolpho: The WinZip notification area icon can be turned off. I do that every time I install WinZip on a machine.

Aaron Isotton: Future access isn't an issue. In this day and age of the Internet, you'll always be able to find a way to get to the data. The .ZIP format started out as a proprietary format a long time ago.

7zip has sped up considerably in their last few releases. I think they took advantage of multi-threading for multiple processors. My Dual Core 1.8's compressed a 1 GB archive into a 680 or so MB .7z file in about ten minutes. Other programs I've used for that usually take about a half an hour or so. Maybe it just seems faster because I walk out of the room when I compress large archives, because I'm used to the bad old days of ancient speeds. Slowness, thy name is StuffIt 2.0...

Noah,

"My point was that Linus could do what we wants regarding the kernel but it wouldn't make a shred of difference. He could continue to develop his non-free version of the kernal but the rest of the free software and open source communities would thrive around the free version of the kernel as Linus cannot revoke the GPL licence on previously released version."

Once again, you're wrong. Linus couldn't revoke the license for the kernel, but the license requires that he have approval of changes to the kernel itself. All he would have to do is stop allowing changes to the kernel except in the version he was selling. That would prevent changes to support new processors, memory, etc.

And you say you're a "professional programmer". How exactly do you pay your bills, if programming is your profession and you don't believe in charging money for software? Rich parents? Public welfare? Panhandling on the street corner? Or just have a job writing code for internal use at your company? Gee... I guess that means you're charging money for your software - you're not doing it for your boss for free, are you?

KenW, you are horribly mistaken. The Free Software movement is founded on copyright law to enforce the four freedoms [1].

I assure you, just because you find something licensed under the GPL DOES NOT mean you can take it and treat it like your own.

"Everything belongs to everyone."

I challenge you to find this phrase or something even remotely similar on www.fsf.org or www.gnu.org.

As a side issue, you have hit on one of the most profitable areas of software development. Support costs. I suspect Microsoft make a large (if not THE largest) % of there revenue from business support contracts.

[1] http://en.wikipedia.org/wiki/Free_software_definition

> Once again, you're wrong. Linus couldn't revoke the license for the kernel, but the license requires that he have approval of changes to the kernel itself.

Incorrect. The kernel is licensed under the GPL which freely allowed me to do whatever I like as long as my modified version is also distributed under the GPL.

> All he would have to do is stop allowing changes to the kernel except in the version he was selling. That would prevent changes to support new processors, memory, etc.

Completely incorrect. See above.

>And you say you're a "professional programmer". How exactly do you pay your bills, if programming is your profession and you don't believe in charging money for software?

I develop web sites and intranets for private clients. As my software is not publicly distributed the licensing is a moot point.

Ok, my two cents worth here:

I see free and open source software as a great benefit as it allows us to do a lot more together than as individuals. Free and open source software is inherently smaller and more secure because many people look at it and say "Oh, this method doesn't work very well, this is faster and smaller." Go Linus!

However, sometimes people need proprietary software (Like WinRAR) because it has an easy interface, does it's job well, and allows them to do their work quickly and efficiently. Many open source applications do not yet have smooth streamlined interfaces that are easy for your poor grey-haired mother to understand. True, there are many out there that do, but when my mother can right click on her files, pick add to archive, and click ok, that's a lot easier for her to remember than having to write "tar cjf csharp.tar.bz2 ms-csharp-files/". For codes like you and me, this is easy, but most people still don't understand computers.

"Not many people have the luxury"... for me, that is speaking of the knowledge we have of these crazy machines. How many people truly have the luxury of going in and trying to learn how to do a mail merge, or tar and gzip a batch of files? Most common users need things simple, and that keeps companies like Microsoft, Apple, and Rarsoft in business. They have proprietary software that has been tested and recieved feedback not only on functionality, but ease of use and understandability (if that's even a word). Likewise, I don't do any front side design without getting input from at least 3 people. That way, not only can I make it a powerful application, but others can understand and use it fairly easily to get the data they need.

To sum my statements up: free and open source is great, but not for everyone (yet). For those who need simple, proprietary is often the main way since they can't find much free and open source that works as well or as easily. Try teaching a lawyer how to backup his files, and you'll see why I just set them up to put a black cd in a drive, double click a batch file, and let the batch file take care of everything.

"right click on [your] files, pick add to archive, and click ok"

That's exactly how you do it on the Gnome desktop.

Have you tried Ubuntu recently? Desktop GNU/Linux has come a LONG way.

Ok, here is an argument for RAR: Unicode support for file names!

Try compressing then uncompressing a folder structure containing files/folders with a mixture of Japanese, Korean, Russian, Greek names and tell me what your preferred tool does.

Noah: True. Unfortunately, my desktop is dead, and backing everything up on my laptop would require too many cd's (no dvd burner on the laptop), so I have to wait for my desktop to get fixed (hopefully i've got it finally). As soon as I can, I plan on switching to a recent flavor of Linux (last one I tried was Red Hat 9 in virtual PC and loved it, but the box died right after). But there's still the underlying command line for a lot of things. My only questions about Ubuntu are how easy is it to install new drivers, and how easy is it to install new third-party applications? Some linux apps still require extra packages to be downloaded and manually installed, and some things still require command line usage. Even with Red Hat 9, I wasn't able to install all my test applications (made it a virtual version of what I did in Windows), and some that I was able to install took a lot of experimenting to get the settings right. Red Hat 9 may be too old for this example, but unfortunately, I can't try anything newer yet.

So do the free software alternatives...

nslater@mahora: ~ $ mkdir ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ
nslater@mahora: ~ $ touch ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ/⠊⠗⠕⠝⠍⠕⠝⠛⠻⠹
nslater@mahora: ~ $ tar -cvvf ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ.tar ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ | gzip -9 > ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ.tar.gz
nslater@mahora: ~ $ rm ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ
nslater@mahora: ~ $ tar -xvzf ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ.tar.gz
nslater@mahora: ~ $ ls -l ᚩᚱᚦᚹᛖᚪᚱᛞᚢᛗ
total 0
-rw-r--r-- 1 nslater nslater 0 2007-02-23 17:34 ⠊⠗⠕⠝⠍⠕⠝⠛⠻⠹

Ugh, I used RAR for a while and while it is interesting for comparison, I'm not really interested in using yet another proprietary tool for general purpose archiving. I lost interest in how many percent one compression tool wins over another under what circumstances some time in the 90s (remember ARJ?), and the issue is moot to me. WinZip's latest version stopped working, so I just switched to 7Zip...I can extract and create Zip and BZip2 files just fine. As far as I'm concerned bz2 should be the future.

Tom, Ubuntu is a great choice for your desktop. I have been running it for 2 years now and have not had to compile a single package.

Everything your will probably* need is handled via a GUI installer!

* 2 years and not a single thing I wanted didn't already exist in the Ubuntu repository.

Noah,

What does -xvzf mean for the tar command?
What does ls -l mean?

Double click, ooh, there's the files. That's what people want.
If they have to do a command line, dir is a lot easier to remember than the -l flag for ls. And my poor mother doesn't have to try to remember the man command to get information, nor does she have to write down instructions like "open command line. type ls -l. type tar -xvzf". That's too much. My point is, free software (like 7zip) often has complex and confusing interfaces. I don't use 7zip because I don't understand it and don't have the time to learn all the compression types and tweaks to get the files compressed the way I need them. WinRaR lets me pick the name, pick maximum compression, and I'm done. And my poor mother likes it because she can just right click, select add to name.rar, and she's done. What about your poor mother?

KenW wrote:

> How exactly do you pay your bills, if programming is your profession
> and you don't believe in charging money for software?

I think you are missing the point. I don't think many of the posters have a problem with WinRar charging money for their product. The problem many posters have is that WinRar requires you to license their code if you wish to support their compressed format in your software. This is one reason I would go with Power Archiver over WinRar, though it is not free, it does support compressing into 7 zip.

Tom, maybe you missed my previous comment.

On most modern GNU/Linux desktops you can right click, save to archive just like Windows. There is ABSOLUTELY NO DIFFERENCE there.

I was making an explicit example of how the free software gzip and tar utilities are Unicode aware.

A normal Ubuntu user doesn't have to ever use the command line. Are you seriously berating GNU/Linux for simply HAVING one?

You know... like Apple's OS X?

It's going to be an uphill battle getting everyone on WinRar. Heck, we all know that OGG, AAC, and WMF are all better formats than MP3, but MP3 rules the roost.

IMHO this is similar to the Betamax VHS battle. Technically, one is better than the other. But when choosing a format, Adoption is definitely a non-trivial measurement.

I think the site you point to ought to include adoption rate into the formula somehow.

Any C# libraries for compressing code to WinRar?

p.s. Before people flame me about Betamax vs VHS, I do know that whether Betamax was really technically superior is in dispute. I am just referring to perception. My understanding is that Betamax did suffer from requiring two tapes to hold a movie.

I'm not berating it for having a command line. If GUI's in Linux have caught up to Windows, Great. Your plug in for Ubuntu does strengthen my interest in trying that over other flavors of Linux. And now it looks like we've gone way off base of the original article with this. So to go back, Free and Open Source software and formats are great (one thing I agree with you wholeheartedly on), but not necessarily for everyone. Let them spend the $20 on WinRAR if they want to. It's going to be around a long time, RARSoft has already taken care of that with releasing the format and everyone has integrated that into freeware and open source decompression programs. Every archiver I've used in the last few years has not only been able to decompress it's own format, but that of every other major (and most minor) formats as well.

Tom,

7-Zip, with its warts is the best bet going. It has better commpression, and because it is open source, I can make an api for direct manipulation from my apps.

I personally punished Winrar for cripling their shareware version and purchased Winzip for not. I see that has changed on both ends.

-Doug

Tom - If your mother is using WinRAR or even WinZip to compress her files, all power to her. Most people's mothers don't have any conception of file size nor what it means. They attach 6MB JPEGs from their digital cameras to emails and send them to people on dialup (of course compressing wouldn't help here, but resizing would.)

Which is all true but kind of irrelevant. GUI-based file compression does exist on linux - file-roller under GNOME for example. If you're capable of using the commandline you'll probably stick with that method as it's just more efficient. If you're happier in a GUI app that option exists.

Getting back to the article I think the fundamental point that most of the people advocating free software over proprietory is that the file format for a compression program ought to be available and it ought to be legal to freely implement a decompressor. That way no matter what happens to the vendor you can theoretically always get your data back. And it's the data that we care about here - could be legal documents or a voice recording of a relative who died unexpectedly or anything. So as well as the theoretical advantage of the published format you'd probably like to see at least one good open source implementation too so that you know practically you'll be able to get your stuff back.

Yes, you could use a proprietory format and then switch to the free format when you compress anything "important" but that's just going to be annoying.

Have you really tried some of the latest versions of 7-Zip? It's not particularly slow on any dataset that I use it on. In fact, when I use it to compress a gigabyte of log files every day, it’s faster than RAR and it compresses much better; the algorithm used is PPMd. Its LZMA algorithm with a somewhat large dictionary compresses common data so much better than RAR that it’s a no-brainer choice to me.

I think a better choice would be 7zip. There's no way I would choose a format like rar, when there are plenty of choices that are open source and just plain better.

Concerning the self-extracting exe via email there's a simple solution: create the RAR-extractor and zip it up to email someone. Gets through the email scanner and allows you (hopefully) to still benefit from sending less bits across the wire.

"Eam, just because the authors have chosen this particular economic model for their business does not mean we should support it. There are many other viable ways to profit from software development. If the author released the algorithm and source code under a free (as in speech) licence that would truly be something to show appreciation for, be it in thanks or money."

Noah, you seem to be arguing with an unsettling glint of religion in your eyes. I find Tom's arguments a lot easier to stomach as he's not suggesting we "call out" anyone just for selling software.

Like he said, open-source has its place, but people will still pay for convenience. I don't think anyone is going to seriously argue that 7zip has a better interface than WinRAR.

"include adoption rate into the formula somehow"

Let's not confuse the application Jeff purchased, WinRAR, with the RAR format.

There's nothing wrong with buying WinRAR and most modern file compression applications support RAR (except ones that are trying to push their own format over all others). WinRAR supports 7z I believe as well as the other formats mentioned in the study.

So picking a well performing compression format has little to do with the application you use to compress your files.

Two things to note about the study Jeff linked to. 1) The study used an older version of 7-zip and didn't use the stable version but opted to use the alpha version. 2) The study forced all of the applications to create TAR files. Which means you may get better performance/compression if you use the applications native/preferred format (e.g. RAR for WinRAR, ZIP for WinZIP, 7z for 7-zip, etc...). The Wikipedia study I linked to earlier compared file compression formats rather than applications.

One of the cornerstones of science is repeatable experiments. There is a lot of information missing from the study Jeff linked to that prevents us from being able to replicate and test the results.

Is WinRAR's Linux version any good? The biggest need I'd have for this kind of thing is taking a huge number of files from a Linux computer, and transferring them to a Windows computer, and vice-versa. Doing that over SMB or SFTP would just take too long, so I use zip right now. Thinking of switching to 7zip on the windows end and using .7z or bz or gz formats though...

And it's an automated process, so I need a good command line tool on both sides..

> What will you do *now* if you want to pack or unpack a RAR on your
> Sparc, Alpha or Itanium server? The answer is you can't.

Hilarious that you chose three dead-or-dying platforms to berate a definitely-not-dead archive format.

Regardless of how much better RAR may be as an application or file format I would not switch to it simply based on the fact that most of the people I know would not be able to open it and would come to me to help them. Which I most cetainly do not want to do.

Allan Clark had a great point: "Your code snippets, for example, will be compressed ONCE, but downloaded and decompressed MANY TIMES. Take the download time, and add to it the decompression time, and you have a more accurate number."

The point is to think about what you're compressing and why, rather than just blindly picking one based on raw speed, or GUI (ooh, look, aqua buttons!!!). If you're backing up files on your own system and plan to extract as often as you compress then by all means pick fast and shiny. If you're distributing something large then take the one time hit on compression time and save the GB's of bandwidth.

Whatever you use, anything's better than the built in Windows Compressed Files support when it comes to speed. 7-zip's compression to standard ZIP format runs around 100 times faster that compressed folders on my machine.

It sounds like one of the arguments against ZIP is that the OS's built-in support for handling ZIP files sucks. But this is really a Windows problem and not a file-format problem. So problem #1 with RAR is that it's not built-in to OSes while ZIP is.

The other problem is libraries. I can generate ZIP files from many languages, without paying a fee to anyone. ZIP support is for instance, part of the core JDK. Not so for RAR. I don't understand why RAR is a win for developers.

But whatever. Encode your data in RAR and make it harder for your recipients to get to it if you want to. Perpetuate the stereotype of the oblivious pocket-protector geek type if you must. Do your part to ensure that the IT group of every firm will be located in its basement. Or do your part to play well with the rest of the world and use formats they use.

This isn't a comment about competing products (except in terms of price), but I use the free Izarc application for my zip needs (which, granted, are not terribly demanding). It's got pretty decent support for file formats and a familiar interface. I'm guessing it's probably slower than WinZip and WinRAR for larger files, but I don't compress/decompress large files all that often. I think it's a solid app. I do have one complaint, though: it, like other apps, uses what I'd guess are non-true type fonts, so they aren't compatible with ClearType. That's a MAJOR pet peeve of mine, although not a deal-breaker for these kinds of apps.

I'm also a big fan of WinRAR, although I have yet to buy a license (I guess one just gets used to clicking on the 'nag' dialog).

I recommend that software publishers strongly consider 7zip/LZMA for packaging, however. The reason is that RAR seems to be limited to a 4MB dictionary, whereas 7zip/LZMA will use whatever size you want. For Paint.NET, I distribute a "full" installer that bundles the .NET Framework. Not only that, but I bundle both the 32-bit and 64-bit versions just to make download choice ("Do I have 64-bit? What is a 32-bit?") and installation easier ("Here, just click this and *go*!"). With some clever packaging arrangement* and a 32MB dictionary size I am able to reduce a 90MB installer down to 50MB, whereas both ZIP and RAR can only muster about 75MB.

The only downside is that the 90MB->50MB compression task takes a minute and a half on a QX6700. NSIS doesn't (yet?) have the multithreading optimizations that 7zip has incorporated into its LZMA implementation.

In the end it's a classical time vs. space tradeoff. I'm very willing to trade 1 minute of my CPU time for reduced download times across hundreds of thousands of users :)(Oh and it takes 300MB of memory while compressing -- no biggie, I've got 4GB!) But if I'm sending some Word docs to a friend, I'm not going to say "Oh give me 10 minutes so that my computer can find the optimal compression layout."

* The x64 .NET installer includes the x86 .NET Framework. If you unpack the installers and place the files in the right order then the compressor just goes, "Oh I've already seen this", and the entire x86 installer fileset basically gets included for free.

Love the commie free software nuts. Yep, doesn't matter how good something is, don't touch it if you have to pay for it or you can't find enough information to build it yourself!

(Boooring...)

Jeff, you know that MS Office 2007 documents are zips, right?

"I was making an explicit example of how the free software gzip and tar utilities are Unicode aware."

Can you please point me to a Windows port of tar/gzip that is able to do that?
In the Linux/UNIX world this support is added by using UTF-8, which is not the case with Windows.

And please do not start preaching the superiority of Linux + UTF-8, because that has other problems, of which I am well aware of.
Try this in your experiment: compress the files on one system, then unpack-it on another system that did LANG=ja_JP.SHIFT_JIS before.
Oups!

I have been using winRAR for a long time, which makes me think I should actually purchase a lisence. I don't see a point for anything else. When I deal with EU's who can't use the .exe because of e-mail, I'll have them just install winRAR, and they end up buying because they love it. It's such a cleaner install too compared to WinZip.

> stick with 7-zip. the fact thats its open source is reason enough to choose it over winrar.

That's fine, but how exactly is this a benefit to the user? Because WinRAR could one day disappear forever? That seems unlikely to the point of ridiculousness. Sure, I *could* get hit by a meteor tomorrow, but I don't think it's reasonable to use that risk as a justification for any of my behavior.

> WinRAR is free anyway... trial period never expires.

C'mon, that's cheesy. If you use something consistently over a long period of time, you should give something back. And WinRAR is so reasonably priced.

> rather than just blindly picking one based on raw speed, or GUI

I'm picking WinRAR because it offers the best efficiency ratio of compression size to compression time. This is backed up by the data from a number of compression studies. I'm too impatient to wait 5x - 10x longer for 2% to 4% more compression.

But I totally agree, as Rick Brewster illustrated, there are times when this can make sense. Just not many that a typical user would run into.

> As far as I'm concerned bz2 should be the future

WinRAR can decompress bz2, but you're right: it would be nice if WinRAR could create bz2 archives as well. More choice is always better.

Jeff,

Sorry to spoil the party, but winrar just isn't as spectacular as you make it out to be. Firstly cross platform availability *is* an issue to all serious software developers who want their applications to run on all platforms and licensing *is* an issue to those who work in companies where purchasing procedures are very cumbersome.

Secondly winrar really isn't the wonder you make it out to be - the free alternatives do just as well.

zip: 229K
winrar: 73K
7zip (on the page you linked to): 59K

free systems:
gzip: 71K
bzip2: 53K

By the way, I'd like to point out that winrar bears an uncanncy resemblence to Hackles' pigzip:
http://www.hackles.org/cgi-bin/archives.pl?request=314

> Have you really tried some of the latest versions of 7-Zip? It's not particularly slow on any dataset that I use it on. In fact, when I use it to compress a gigabyte of log files every day, it’s faster than RAR and it compresses much better.

Indeed, you're absolutely right. Here are my results on my work rig, which is a dual core Athlon 64 X2 4800+. We are comparing WinRAR 3.70 beta 2, and 7-ZIP 4.44 beta.

Compressing 2 files:
587 MB VHD file
11 KB VMC file

Extraction:
WinRAR - 39 sec
7-ZIP - 40 sec

Compression:
WinRAR - 3:09, 135 MB
7ZIP - 3:03, 125 MB

Those are very, very different results than what I saw in summer 2005 (ZIP and RAR are about the same speed).

http://www.codinghorror.com/blog/archives/000313.html

It does look like the latest versions of 7-ZIP have improved performance *dramatically*. Follow-up blog entry here:

http://www.codinghorror.com/blog/archives/000798.html

Nice advice Jeff. I just purchased a WinRAR licence as I've been using it for a while too. I wasn't too sure at first but the carping of the Free Software extremists made me decide to go out and support something proprietary today!

For those complaining about 7zips interface... in my day we used arj from the command line and we spanned our archives in 1.44 chunks and we liked it!

Scott Said:

> WinRAR supports 7z I believe as well as the other formats mentioned
> in the study.

According to WinRAR's comparison page (http://www.win-rar.com/winzipcomparison.html), WinRAR only supports compressing to RAR and ZIP formats. It does have more support for extracting files though.

ZIP sucks as it lacks options (lack of solid, or varied algorithms)

RAR is great as it has features allowing to specify compression details (ie, enable disable sharkin/ppm compression, or go down to set up its dictionary size), as well as practical features (such as delayed reads, that save hd from trashing)

7ZIP seems to be more geared towards people that truly like to experiment with their parameters. Sure, if you know what you are doing its amazing. (ie : ''7za a -t7z -mx=9 -m0=LZMA:a=2:d=55m:fb=255:lc=8:lp=4:pb=4:mf=pat4h '' is a very simple line)

PAQ series is the one that true compression fans go for. Its headed by comp-sci people and fueled by a challange and fame. http://cs.fit.edu/~mmahoney/compression/

Some of the previous posters were worried about whether or not to unrar a self-extracting file. It seems to me that this is a problem that has been solved by digital signatures and PKI (http://en.wikipedia.org/wiki/Public_key_infrastructure). Once the source of the file is trusted, then the only consideration should be for the efficiency of the algorithm. Though, in my own opinion, I have to fall in on the side of F/OSS movement. Tar and bzip2 are so nearly ubiquitous as to be a good choice.

Defacto standard, cross platform, widely implemented... or some bizarre format that most end users can't read, isn't ported to all operating systems and is heavily encumbered.

Yes, RAR has technical advantages, but you better *need* them to take all the pain that goes with it. I refuse to even attempt to open RAR files anymore.

"I wasn't too sure at first but the carping of the Free Software extremists made me decide to go out and support something proprietary today!"

No kidding. If I weren't already a long-time WinRAR customer I'd buy a license just to spite these GNU fools...

Also, WinRAR is a good choice because it's excellent across the board. You get a great compression ratio AND fast compression AND good Windows integration AND a powerful command line AND lots of useful options AND a very low price compared to WinZip -- not just one or two of these things.

Jeff,

re: solid archive - should it be "inter-file" redundancies? NOT intra-file ...

tar + bzip2, EOD.

RAR annoys me because it is not included with Windows like ZIP is.

first, the person who says "RAR annyos me because it is not included with windows" is a fucking idiot. second, yes winrar is indeed the best... we've been using it since around 2001

Cliff, I'm sorry my little comment negatively affected your opinion of the intelligence of a TOTAL STRANGER that typed ONE LINE into a text box. This is a big, dangerous world, and people should protect you from opinions that you don't agree with.

I just think compressing files is as important as editing text files, calculating simple sums or displaying web pages. I just think it is annoying to have to go and download WinRAR when I need to unpack a compressed file. No need to call me a fucking idiot.

I became a WinRAR user when I started learning Japanese and realised that I couldn't use unicode filenames with WinZIP.

If you send someone a rar and they cant open it.... then chances are they dont DESERVE! to open it =P. winrar r0x0r my s0x0r!

I have worked on one of the big realtime data warehousing projects and we were persisting large amount of data in database from one daemon while another daemon supposed to read it.
At some point we tried to use ZIP compressed datastream, and it worked just fine for the
compression. However, decompressing data with zip takes at least 6-10x time of normal reading data.
So we had to look into another algorithms to do it.
I guess all I am saying is before make decision you need to find match between compression
algorithm, content of your data and process.

RAR is a poor choice, for a number of reasons:

1. It is in extricably bound to one supplier.
2. No source for using it in other applications.

If you want a good compression for archives, try tar+bzip2. The compression is slower but the files are ultrasmall. tar+gzip is the fastest combination, smaller than zip but fast enough to be usable.

@Adam

"So, for GUI programs he's allowed to tweak them as much as they possibly can be, but command-line programs have to use default options? Come on!"

No I only said I will use the GUI when available and select the 'best' option there (which are sometimes not the best anyway). Command line programs are also tweaked for best compression but not all combinations are tested as it will take ages for programs like 7-zip (command-line mode) and EPM to find the best switches.

So, I try to find the best possible compression for both GUI and command line programs but not at the cost of hours of tweaking. Maybe the text on the site is a bit unclear. will try to fix that.

BTW I not only test the 'best' modes but also the 'normal' modes (of the command-line and gui programs).

Werner

Very informative post Jeff.

If compression time matters more in comparison with compression size, the best archiver is not WinRAR. Consider for instance that you have to send the file over a network and you can send 1MB per second. In this case you want to minimize compression size+2^20*compression time, and the top 10 winners are (in order):

Program Switches Efficiency (smaller is better)
PKZIP 2.50 (none) 137569935
THOR 0.93a exx 140586704
THOR 0.93a (none) 141152880
AIN 2.32 (none) 141791336
PKZIP 2.50 -exx 143382475
GZIP 1.2.4 (none) 144803092
WinXP (Builtin) (none) 145697399
ALZip 6.32 (normal) 145758694
WINZIP 10.0 Normal 148619279
ESP 1.92 (none) 148719009

I think that THOR 0.95 with -e3 is the best!

Hi,
Seem to finally have found some people knowing
about RAR ...

My problem concerns a huge file (about 1 TB).
A test-file with similar content from 100 MB
was compressed with RAR at a factor +1000.

This opens possibilities to really work with
the actual file, for it would be compressed to
something around 1 GB, if the ratio remains
1000.

But I need to work IN that file, yes reading
and ... writing too.

My life is hell since then.
Please bring me back in heaven.

Please don't take this advice. RAR is a niche format not natively supported by any platform. Stick with zip, or maybe try gzip or bzip2 on your tarball. With any of the above, you'll get native support in at least 2 of the 3 major platforms.

Chaps, your kidding discussion is not worth a penny -- even for newbies!

http://www.maximumcompression.com
http://www.uclc.info

I have both of them.So I sometimes uses .zip,another uses .rar acording to the files I received/download.

"I wasn't too sure at first but the carping of the Free Software extremists made me decide to go out and support something proprietary today!"

Good on you! I'll buy a bottle of wine with those $20 instead... we can share it if you want. Want red or white?

I will stick with using my free copy of Linux and my free copy of PeaZip for opening RARs. I'm not a socialist, and I'm not an extremist, just a student who thinks that there are better ways to spend his money. I was happy to buy a Photoshop licence, for example, as there is not a REAL good free alternative on Linux (GIMP is getting better but it's not there yet).
But WinRAR... seriously, it's just an archive manager. I personally don't care if it takes me a little bit more to decompress a file also because, unless you're dealing with gigantic archives (and >95% of the users surely are not), you won't notice any difference. And I don't care if I can spare a few kb on my archive. Actually, today's hard drives are big and cheap enough to keep everything uncompressed and live happy.

Free software is also about beeing free to choose. You are free to pay WinRAR, I'm free not to do so if I think that's better for me.

As for the metter of "how will I decompress my archives in 20 years?"
Are you guys really planning to keep 20 years worth of backups of your data? How often do you open backups of files say... 5 years old? Not very often, probably never I would say. In 20 years the problem simply won't be there.