The traditional method of distributing large files is to put them on a central server. Each client then downloads the file directly from the server. It's a gratifyingly simple approach, but it doesn't scale. For every download, the server consumes bandwidth equal to the size of the file. You probably don't have enough bandwidth to serve a large file to a large audience, and even if you did, your bandwidth bill would go through the roof. The larger the file, the larger the audience, the worse your bandwidth problem gets. It's a popularity tax.
With BitTorrent, you also start by placing your large file on a central server. But once the downloading begins, something magical happens: as clients download the file, they share whatever parts of the file they have with each other. Clients can opportunistically connect with any other client to obtain multiple parts of the file at once. And it scales perfectly: as file size and audience size increases, the bandwidth of the BitTorrent distribution network also increases. Your server does less and less work with each connected client. It's an elegant, egalitarian way of sharing large files with large audiences.
BitTorrent radically shifts the economics of distribution. It's one of the most miraculous ideas ever conceived on the internet. As far as I'm concerned, there should be a Nobel prize for computing, and the inventor of BitTorrent should be its first recipient.
There's a great Processing visualization of BitTorrent in action which explains it far better than I can. The original visualization is not only down semi-permanently, but also written for an ancient version of Processing. I grabbed a cached copy of the code and updated it for the latest version of Processing.
This meager little animated GIF doesn't do the highly dynamic, real-time nature of the visualization justice. I highly recommend downloading Processing and downloading the updated bittorrent visualization code, so you can see the process from start to finish on your own machine. It's beautiful.
But as as wonderful and clever as BitTorrent is, it isn't perfect. As an avid BitTorrent user, I've noticed the following problems:
The efficiency of BitTorrent is predicated on popularity. The more people downloading, the larger the distribution network gets. But if what you want is obscure or unpopular-- part of the long tail-- BitTorrent is painfully, brutally slow. With only a handful of clients sharing the workload, you're better off using traditional distribution methods.
Download work is shared by the clients, but how do the clients locate each other? Traditionally this is done through a centralized server "tracker", or list of peers. This means BitTorrent is vulnerable to attacks on the centralized server. Once the server is out of commission, the clients have no way of locating each other, and the whole distribution network grinds to a halt. There are alternatives which allow clients to share the list of peers amongst themselves, such as distributed hash tables, but centralized tracking is more efficient.
Also, in order to even begin a BitTorrent download, you must first know where to obtain a .torrent file. It's a chicken-and-egg problem which also implies the existence of a centralized server out there somewhere.
The BitTorrent distribution network is predicated on clients sharing pieces of the file during the download period. But if the download period is small, the opportunity window for sharing is also small; at any given time only a few users will be downloading. This is another scenario where you're unlikely to find any peers, so you're better off with traditional distribution methods.
There's no rule that says clients must share bandwidth while they're downloading. Although most BitTorrent clients default to uploading the maximum amount a user's upstream connection allows, it's possible to dial the upload rate down to nothing if you're greedy. And some users may have their firewalls configured in such a way that they can't upload data, even if they wanted to. There's no way to punish bad peers for not sharing, or reward good peers for sharing more.
Furthermore, every torrent needs a "seed"-- a peer with 100% of the file downloaded-- connected at all times. If there is no seed, no matter how many peers you have, none of the peers will never be able to download the entire file. It's considered a courtesy to stay connected if you have 100% of the file downloaded and no other seeds are available. But again, this is a convention, not a requirement. It's entirely possible for a torrent to "die" when there are no seeds available.
The BitTorrent model is innovative, but it isn't suitable for every distribution task. The centralized server model is superior in most cases. But centralized distribution is a tool for the rich. Only highly profitable organizations can afford massive amounts of bandwidth. BitTorrent, in comparison, is highly democratic. BitTorrent gives the people whatever they want, whenever they want it-- by collectively leveraging the tiny trickle of upstream bandwidth doled out by most internet service providers.
But just because it's democratic doesn't mean BitTorrent has to be synonymous with intellectual piracy. BitTorrent has legitimate uses, such as distributing World of Warcraft patches. And Amazon's S3 directly supports the torrent protocol.
BitTorrent, in short, puts distribution choices back in the hands of the people. And that's why everybody loves BitTorrent. Everyone, that is, except the MPAA and RIAA.
Posted by Jeff Atwood View blog reactions
« Beyond JPEG Because They All Suck »
If I remember correctly BT clients watch the upload rates of other peers and allow them more bandwidth so you are rewarded for uploading by having more download. This only really matters in a famine situation, when you have a feast situation and theres plenty to go around then everyone gets as much as they want.
Also even if the last seed disconnects from the swarm as long all the pieces are available somewhere the torrent will live on. The rarest pieces are prioritised by the clients so that there is the least chance that the torrent will die out.
Another part of how it all works that I like to visualise in my head is when a bad peer joins the swarm and starts dishing out bad data. The other peers see this bad peer and cut it off, effectively self healing the swarm!
It truly is a work of art and makes me glad I'm a developer and can understand the beauty of it.
AndyJ on February 20, 2007 03:18 AMIf it was a *legitimate* protocol it would sit on top of HTTP(S) outgoing on port 80. Or so says my work's firewall rules anyway.
Alastair Rankine on February 20, 2007 03:56 AMSo in that case Email is not *legitimate* ... it's not on port 80...
ports 25/110/143 etc ...
(oh an neither is HTTPS ...port 443)
Jaster on February 20, 2007 04:09 AMYes, BitTorrent is great.
BitTorrent does have a mechanism to encourage altruism: other peers are more likely to send you data if you send them data. That said, this mechanism is still flawed -- I access the internet over an ADSL line, and I've discovered that I can increase my download rate by ~decreasing~ my upload rate. If I let my uplink saturate, ack packets for the download get hung up, which drops my download rate. With a properly tuned Azureus, I can usually finish downloads with a share ratio of 0.3 or so.
There are interesting questions about what altruism is -- for instance, some people say you should always have a share ratio above 1.0. That's crazy -- it's impossible for everyone to have a share ratio of 1.0 if a torrent is stable in size. A torrent that has 80 seeds and 5 peers and lets me finish with an 0.05 share ratio doesn't need me as a seed. My participation in a torrent that has 2 seeds (me and somebody else) and 5 peers that I finished with a share ratio of 1.7 really would improve performance for the peers.
So far as centralization vs decentralization, I think BT is decentralized enough -- unlike Napster or Kazaa, there isn't one organization that you can sue to stop it. The MPAA and RIAA can play whac-a-mole with web sites that host .torrent files, but people can set up new ones faster than they get torn down.
Google already provides an excellent discovery mechanism -- type in the name of what you want + torrent, and you'll probably get it.
Sailor Moon on February 20, 2007 04:12 AM> BitTorrent is unsuitable for small files, even if they are extremely popular.
Depends of your definition of "small" though, BT is used for files in the 5Mb scale. The main issue is that the smaller the file the faster problem 1 gets felt. On the other hand, problem 4 becomes much less of an issue with small files: since you don't get lot, you don't have to share much for everything to work out, whereas big files (in the 100s of Mb) requires much higher absolute sizes of sharing (even if equivalent ratios).
> There's no way to punish bad peers for not sharing, or reward good peers for sharing more.
Actually there is, trackers and clients keep track of the data received and sent by clients, and the more you send the more you may get. Non-altruistic clients are usually choked out of the swarm, or only get a trickle of the total bandwidth
> Furthermore, every torrent needs a "seed"-- a peer with 100% of the file downloaded-- connected at all times.
No, having seeds is the best because they share without taking anything (they add to the total bandwidth of the swarm without taking from it) and you're ensured that your download will complete, but having multiple peers with part of the file is enough _as long as they have different parts and you can recreate the whole file from their parts_ which is usually the point that fails (and everyone ends up with 99.8% of the data) unless you have a few clients with high completion ratios (>90%) when the last seed quits, at which point there's a quite large chance that they don't have overlapping missing data.
> If it was a *legitimate* protocol it would sit on top of HTTP(S) outgoing on port 80.
So your work's firewall doesn't consider IMAP, SMTP, FTP or SSH to be legitimate protocols? Whoa, talk about harsh
Masklinn on February 20, 2007 04:38 AM+1 to Masklinn
hotdogs on February 20, 2007 04:43 AM> That said, this mechanism is still flawed -- I access the internet over an ADSL line, and I've discovered that I can increase my download rate by ~decreasing~ my upload rate. If I let my uplink saturate, ack packets for the download get hung up, which drops my download rate. With a properly tuned Azureus, I can usually finish downloads with a share ratio of 0.3 or so.
That's an issue of saturating your bandwith, it's not an issue with the BT protocol per se, it has more to do with TCP on which BT is built. Which is why BT saturating your uplink can even choke your web surfing.
That's why you should always limit your upload bandwith to 50% to 75% of your max upload bandwitch _at most_.
> for instance, some people say you should always have a share ratio above 1.0.
I usually shoot for a ratio of 3
> That's crazy -- it's impossible for everyone to have a share ratio of 1.0 if a torrent is stable in size. A torrent that has 80 seeds and 5 peers and lets me finish with an 0.05 share ratio doesn't need me as a seed.
Of course it doesn't, yet if it doesn't need you it won't use you.
Modern clients such as Azureus or uTorrent allow you to configure the torrents on which you want to seed based on various factors including but not limited to the seeds/peers ratio and the absolute number of seeds. And to cycle between your various completed torrents.
This means that you can just leave your completed torrent in your client and not bother with it, if in 2 or 3 weeks most of the seeds are gone you may still be there and help people get the files they want/need, if the torrent doesn't seem to need you your client will just switch it out and seed something else in your queue.
Masklinn on February 20, 2007 04:46 AMAnother item: Even if your firewall is configured so others can't connect to you, you can still connect to other clients and upload data to them. Back when I had a cable modem, and a 10.10 address, I could still share data, even as a seed, IIRC. It just wasn't as efficient since others couldn't connect to me.
Mike Swaim on February 20, 2007 05:39 AMI, too, love BitTorrent, but there's a problem for me: it crashes my router. I can usually download files okay, depending on filesize and popularity, but after I close the program I must go to my router settings and update the port forwarding to drop the forwarding for the range I use for BitTorrent. If I don't, the sheer amount of connection requests will crash my router or, at the very least, make my network unstable.
I realize this is a hardware problem, but it's only evident when I'm using BitTorrent. It's not the software's fault--it is doing what it's supposed to do. This is just an unfortunate byproduct.
Mark on February 20, 2007 06:04 AMMark: I had the same problem with DD-WRT until I found this page: http://www.dd-wrt.com/wiki/index.php/Router_Slowdown
The problem was that the TCP and UDP connection timeout periods were so long that the ip_conntrack table would get full. Maybe your router has settings to reduce the connection timeout length as well.
jon on February 20, 2007 06:33 AMDoes BitTorrent really matter to people who are only using legitimate content? I honestly don't know the answer. I've never used BitTorrent or any of the other peer to peer networks. I've simply never had a need to. Is there a legitimate use for these networks other than sharing stolen software or multimedia files?
Matt on February 20, 2007 06:41 AMGood timing to write on this, since Miguel just wrote about some updates to MonoTorrent:
http://tirania.org/blog/archive/2007/Feb-17.html
@Matt: I've only used BitTorrent a few times, mostly to grab very large ISO files and videos (archived MIT lectures, etc) that I have trouble downloading directly. It's been a blessing during those instances!
On a similar note, Larkware posted this yesterday:
http://www.mono-project.com/Bitsharp
greyboy on February 20, 2007 06:54 AM@Matt:
Yes, there are. You can start here: http://www.legaltorrents.com/index.htm
Tom on February 20, 2007 07:04 AM> Is there a legitimate use for these networks
Jeff mentioned World of Warcraft patches, many linux distributions offer Bittorent link, likewise, TA Spring installers and stuff used to be avaible on BT (I don't know if they still are, haven't played for some time), Sub Pop Records offers some files via Bittorent (http://www.bittorrent.com/users/subpoprecords/) on top of the "regular" downloads available from their website.
> other than sharing stolen software or multimedia files?
1. That probably wouldn't be a "legitimate" use
2. We disagree on the meaning of the term "stolen", but oh well...
3. Part of the "illegitimate" multimedia file distribution lies in a gray area: british TV series downloaded by americans or chineses who couldn't have access to them legitimately, fansubbed japanese animation watched by westerners who'd have no way to access most of them legitimately (because most of the series aren't distributed in western countries, and because most of the westerners don't speak japanese), ...
So while most of the uses of bittorent are illegitimate, there is a legitimate use for the networks indeed.
Finally, about part 3 (especially on the subject of japanese animation), it should be noted that the "popularity" of series' fansubs can be used as a market study on the potentials of an official distribution of the serie.
Masklinn on February 20, 2007 07:06 AMAlong with the legitimacy of the downloads, is BitTorrent secure? Is there any mechanism to ensure the integrity (i.e., non-modified nature) of the downloads/uploads? Is there some mechanisms to keep my machine secure once I open the firewall to BitTorrent? How do I ensure that I have a legitimate (i.e., non-malware) torrent downloady-thing (client software?)?
David A. Lessnau on February 20, 2007 07:31 AMyou don't need a seed, as long as the peers have all the pieces amongst them.
nordsieck on February 20, 2007 07:41 AM>BitTorrent is a terrible Long Tail client.
That's only half right. Most people haven't caught on yet, but BT is great as a *supplement* to existing protocols. That is, don't just provide an HTTP or FTP link, but provide an alternate torrent link. When the torrent dries up, you still have the direct download to fall back on. Even better, work has been done to make PHP (and other-language) scripts that act as both seeding clients and trackers, so that you don't need anything more than a standard web server. (Because, really, that's what the HTTP Partial-Content spec is all about. Same for FTP ranges.)
Similarly, there are RSS clients that are smart enough to use the torrent file if it is provided, but will fall back to the direct download if it is not. I'm pretty sure there was even work done to provide an (unofficial, of course) HTML attribute for anchor elements that would specify a torrent to try before the regular link. You'd need an extension to handle it, but it's not like building an extension is rocket science. Effectively, you've recreated the opposite of the old LOWSRC attribute. "Try this high-bandwidth option first, then fall back to the old stuff later."
I've spent more time than I can retell trying to convince podcasters to provide two feeds: one with direct-link enclosures, and one with torrent file enclosures. A large majority of the people smart enough to subscribe to a podcast are also smart enough to recognize the benefits of BT. Give them a simple page that shows how to use their BT client to subscribe to an RSS feed (uT and Az can do this), and you have faster podcast downloads with smaller bandwidth bills. And really, BT is *perfect* for podcasts, as the flash mob you get with BT is exactly what you get with a podcast. Similarly for vlogging, screencasting, etc.
The most common argument is "but I can't track it, and I can't advertise with it". This is totally untrue. You can track BT downloads just as well as you can track HTTP downloads. (Yes, there are edge cases where you can get in via DHT without talking to the tracker, but a stealth client in the swarm will catch those malfeasant folks.) And really, you can advertise in a torrent file RSS feed the exact same as in a direct-download RSS feed.
Rick O on February 20, 2007 07:43 AMhttp://www.publicdomaintorrents.com is one.
I'd love to see some BT implementations that make it work well for small bits of content... E.g. a plugin to Firefox that is a BT client that can download images, and a server-side component that would be the tracker and also embed the tracking info into the HTML document so the client doesn't need to do a seperate request for that.
Second Life should probably also adopt a similar technique if it wants to actually scale.
Reed Hedges on February 20, 2007 08:10 AMThis is a great blog, but I was disappointed to see lots of inaccurate information in this post. Most of which has already been pointed out above so I will refrain from re-posting it.
Hmm... why do I love Bit Torrent?
1. Super slow downloads. I can get stuff through bearshare twice as fast.
2. 10 minute tracker connections. P2P stuff like Kazaa and Bearshare are connected in 30 seconds, but Bit Torrent takes 10 minutes before it will start looking for other peers to download from. Now that's efficiency.
3. Stability. I've tried various Bit Torrent clients (and client versions) and with the exception of the WoW updater, all have caused my firewall, antivirus, or both to crash. Then the Bit Torrent clients crash themselves. On Windows XP SP2, that's a lot of fun.
4. Security: see #3.
5. Availability. I've let the max upload and download run on my machine and had plenty of space, but no one is ever on sharing the stuff I want. The host says there's 40 seeders and 20 leechers on right now, why can't I get anything? I don't care about porn, or music, I just want to get the stuff I want.
Yeah, I love Bit Torrent... NOT
Tom on February 20, 2007 08:22 AMYou only need to provide a traditional download link because many people don't have bittorrent clients. You can get rid of your ftp server, and just seed your own file.
Eventually, browsers will have bittorrent capabilities, and you will be able to right-click and save, and not even notice the difference (unless no one is seeding.)
mynameishere on February 20, 2007 08:26 AMAbout the connection speed. I have also suffer with many torrents, mostly the 'unpopular' ones, and I came back to emule, which I stopped using some years ago because it was really crappy and, voila! It has become a fast reliable way of sharing. If you have never given it a try, do it now, if you have and dropped it, give it a second chance as I did.
Jorge on February 20, 2007 08:27 AMMaybe it's been pointed out already but bittorrent was not first with this kind of "sharing your bandwith", so he doesn't deserv any reward for that. Although he was probably the first to base a protocol for distributed file sharing on top of HTTP which was what made it explode.
All the other solution that predated BT was too complicated and required central servers.
PL on February 20, 2007 08:58 AMEven NASA agrees :
"I thought P2P and Filesharing were illegal!
This is a common misconception. BitTorrent, and peer-to-peer (P2P) are protocols, like HTTP and EMail. It is true that they can be used to share files illegally, but the same is true of HTTP. Our use here is legitimate, however, so you should have no need to be concerned."
From the Visible Earth FAQ, located here: http://visibleearth.nasa.gov/faq.php
Manu on February 20, 2007 09:00 AMmynameishere: Opera has bittorrent capabilities, I don't notice the diference :)
Manu on February 20, 2007 09:11 AM@mynameishere - "You only need to provide a traditional download link because many people don't have bittorrent clients."
BlogTorrent - http://www.blogtorrent.com/ - makes this easier by providing a nice user experience in the case the user doesn't have a BT client. It downloads a lightweight BT client (if you don't already have one) which then automatically pulls down the torrent file. The end result feels just like downloading and Adobe project (with its downloader / installer).
Apparently they've got a newer version out now called Broadcast Machine.
___
As for legal uses, I'd think the Monoppix project was a good poster child. We had tens of thousands of downloads of a 400MB ISO, which would have been very expensive for a free community project. We set up a simple BT server (later updated to include server side seeding), and were able to host on a $5/month server plan.
Server side seeding made a big difference in end user experience, since there was always a share ratio of at least 1.0.
Part of the problem I ran into when I was setting this up was finding a reliable, legal tracker. From some of the links above it looks like that may have improved since then.
Jon Galloway on February 20, 2007 09:48 AM
To Tom: hummmm.... with the right trackers BT starts almost inmediatly, and when it comes to proper legal downloads (Linux Iso's and so) it's amazing how fast it goes. About the rest of yopur complains... a) you don't have tried a good client and b) you don't understand what's about.
To Sailor Moon: I use to leave the most uncommon torrents for a few weeks before deleting them from the incoming folder. I like to keep the share ratio over 2.5, for that I have just left the PC at home with no downloads but loads of seeds, just for the spirit of sharing. Some people were nice to you, be nice to the following in the chain!!
Love it... it's fast, lightweight and does what it says in the tin.
argatxa on February 20, 2007 10:16 AMThanks for the background on BitTorrent. I really exicted by the idea of leveraging this stuff with .NET.
Jonathan Allen
InfoQ
> I was disappointed to see lots of inaccurate information in this post
How so? Others have clarified with additional information, but I don't believe anything I said was inaccurate.
BTW, if you're running Windows, the best client by far is uTorrent (Micro Torrent). It's ultra small, has a very clean UI, and it's extremely efficient:
Jeff Atwood on February 20, 2007 10:31 AMAlso, if you want to experiment with a torrent for the first time, try out the South by Southwest (SXSW) artist showcase torrent. 739 MP3s from 739 artists.
http://2007.sxsw.com/blogs/plat.php/2007/02/19/don_t_miss_the_sxsw_toolbox
AND it's legal. ;)
Jeff Atwood on February 20, 2007 10:45 AMYou might also mention the distribution of various Linux operating systems as a highly utilized, perfectly "legal" use of the BitTorrent protocol.
WesleyTech.com on February 20, 2007 10:55 AM@argatxa
"To Tom: hummmm.... with the right trackers BT starts almost inmediatly,"
so what are the right trackers? I've been to over a dozen websites and checked trackers on all of them"
"and when it comes to proper legal downloads (Linux Iso's and so) it's amazing how fast it goes."
Is that why I couldn't get Red Hat 9 disc 3?
" About the rest of yopur complains... a) you don't have tried a good client"
Huh? Double check your grammar please, and what is a good client. As I stated before, I have tried multiple clients and multiple versions of them over the last couple of years. I gave up on Bit Torrent a month ago after three popular clients failed. The first caused an error in Zone Alarm, the second caused Avast to shut down, and the third just crashed the entire system.
" and b) you don't understand what's about."
Care to enlighten me?
Tom on February 20, 2007 11:10 AMOn the Mac I use xTorrent, very nice (yet beta). xtorrentp2p.com
I like the protocol, although it's true that ADSL lines tend to discourage sharing due to the HUGE difference between the downstream and upstream (in my case 2MBytes vs 284Mbits)
Martin Marconcini on February 20, 2007 11:48 AMPart of the problem with BitTorrent is the standard 'two active torrents' client distributed as a default. With the default client, even when someone wants to serve up fifteen or twenty torrents while downloading one, they can't. Even extremely large torrents which should have tons of seeds like the Ubuntu CD ISO's have a limited number. My guess is that noone cares about serving the Ubuntu CD's as a torrent when they only have two slots for it. If the client and broader range of 'good torrent karma' was important in the grander scale of torrent distribution, it might be more effective.
I'm not saying that Azureus isn't better, but a large number of the clients I see in the torrent peer list aren't using Azureus. With proper settings, other clients fix the long tail and small file problems; the small file problem is really just the same problem as the long tail one.
Dylan Brams on February 20, 2007 12:52 PM> BitTorrent is a terrible Long Tail client. The efficiency of BitTorrent is predicated on popularity.
That isn't really true, the BitTorrent client itself is fine. It's just that most hosts serve files via BitTorrent but do not keep a seed open at all times on their servers. Instead, they just let the clients "figure it out between themselves", effectively giving their download a limited lifespan (when it's popularity dies). This "problem" has nothing to do with BitTorrent, but only how people choose to use it.
Having the host keep a seed open is becoming a popular alternative to regular HTTP downloads for more robust/efficient downloads. Because essentially, clients will still download from the server (so the swarm never dies), but as other clients download, they add to the available bandwidth, on top of what a standard client-server link would provide. I believe this is how WoW handles patch updates.
For another example of this setup, try downloading the SICP lecture videos via BitTorrent at http://swiss.csail.mit.edu/classes/6.001/abelson-sussman-lectures/
Not many people are usually downloading those videos, but you will see that one of your peers will always be from swiss.csail.mit.edu, and uploading at the same speed as the regular HTTP download. It's just the host permanently seeding the file.
Of course HTTP is still required for people who don't have a BitTorrent client, not because of the popularity issue you bring up. That's really just a side-effect of people posting torrents with no permanent seed.
Damien on February 20, 2007 12:53 PMThe problem I have with BT is my ISP. I thought I would be generous and maintain a ratio of 5 but I didn't realize that my ISP watches upload volumes and they throttled my upload speed for all traffic.
Now I keep my upload speed throttled to half of my ISP throttle speed and I kill my torrent as soon as I complete (I am now a complete leech). Hopefully after a month of low uploads I can convince my ISP that someone else was using my wireless router and get my throttle lifted.
Most ISPs appear to be very antagonistic towards BT.
ed on February 20, 2007 12:55 PMHave you tried enabling encryption in your BitTorrent client? It's a scorched earth solution, but many clients support encryption now.
Jeff Atwood on February 20, 2007 01:30 PMEncryption doesn't matter. My ISP monitors the aggregate upload volume. Thay don't care what I am trying to do. If I upload too much, they throttle ALL upload bandwidth.
Even normal web browsing is sometimes affected by the throttle (it's pretty slow).
Maybe their VOIP port might not be throttled. Maybe if I signed up for their VOIP service they might have to lift the throttle. They have not notified me of the throttle, maybe if I call customer support and pretend I don't know what is going on, I could convince them to lift the throttle. ("Oh yeah, I have a wireless router for my laptop. No, I did not turn on WEP. Gee, why would I need to do that?")
Bob Cringely has posted several articles claiming that P2P should actually be the bandwidth savior instead of the bandwidth demon that ISPs make it out to be ( http://www.pbs.org/cringely/ ). But unfortunately the ISPs have not learned that yet. His latest article hints that maybe Apple will shove it down their throats.
ed on February 20, 2007 01:54 PMI don't understand why companies don't simply provide a .torrent link, and then put a massive-bandwidth seeder to start it off. Ensure that the seeder's always up, and then it works like a centralized link but with all the benefits of reduced bandwidth. Has all the benefits of both approaches. Why isn't this being done?
Malderi on February 20, 2007 01:54 PMMalderi -- I'd imagine because many many many companies are very stupid, still, when it comes to the internet. Too many companies simply do not understand the underlying architecture to make informed decisions like that. Some, however, do exactly what you're suggesting. Not enough, though. Not nearly enough.
Jae on February 20, 2007 03:06 PM"Is there any mechanism to ensure the integrity (i.e., non-modified nature) of the downloads/uploads?" - David A. Lessnau
Yes, there's several layers to this:
1) The .torrent file itself contains an overall integrity check (SHA-1 or MD5 sum, I think) for each file in the torrent.
2) For each piece/chunk you transfer, the sending peer notifies you of the intended hash of the piece.
--
To prevent "poisoning" of the torrent (i.e, someone deliberately sending false data) - clients record those who consistently send bad data and refuse to communicate with them.
--
To prevent someone consistently leeching and giving nothing in return, (most) clients operate on a "tit-for-tat" arrangement. i.e if you send me data, I'll send you data more regularly/faster.
If you don't return any data, you will still get some data, but at a slower rate.
This also stops someone with access to high-bandwidth connections "soaking up" all the available bandwidth (i.e to prevent/slow down the distribution of the torrents)
"you don't need a seed, as long as the peers have all the pieces amongst them.- nordsieck.
This needs reiterating - a lot of people seem to think that you always need a seed. If, between all peers, there is at least one copy of all pieces in the torrent, the seed can disconnect and the remaining clients will share completed pieces. It operates slower (you lose a peer which had all of the data), but will recover given time.
"Stability. I've tried various Bit Torrent clients (and client versions) and with the exception of the WoW updater, all have caused my firewall, antivirus, or both to crash. Then the Bit Torrent clients crash themselves" - Tom.
This is a sign of an incorrectly configured system, client, and/or network - and possibly failing hardware too.
Some routers don't handle having many open connections, some don't clear their NAT tables (for instance, the default Linksys WRT54G router firmware won't flush NAT tables - you need to power-cycle it). Some routers can't handle high volumes of traffic from many sources at once. You can mitigate this by reducing the number of torrents open, capping traffic to less than the full line-speed of your connection, and altering your Bittorrent client to keep less connections open at once.
If your antivirus software is locking up, then I suggest making these changes:
1) Some Bittorrent clients (such as uTorrent and Azureus) allow you to configure an "incomplete" folder. Configure your Bittorrent client to download to a seperate folder (eg: "X:\Bittorrent-Incomplete\"), and when completed - get it to move to your normal use folder. eg: "X:\Bittorrent\"
2) Remove the Bittorrent-Incomplete folder from your Antivirus scanner's watch list.
This will stop your AV software trying to lock and scan files that change constantly. (Which can in turn cause your torrent client to lock up/crash when it needs to modify the file again)
3) If your firewall is crashing due to bittorrent, then get another firewall, or configure it properly (i.e allow all incoming connections to your BT client). If you are using a modern software firewall on your PC, then it can open those ports automaticly when you load your Bittorrent Client.
Regarding the super slow downloads/availability: This sounds like either a problem with your network configuration, or using poorly shared torrents.
If you are using a NAT router or similar, you need to either enable UPnP, or manually forward certain ports - then configure your BT client to use those ports for incoming connections.
> As far as I'm concerned, there should be a Nobel prize for computing, and the inventor of BitTorrent should be its first recipient.
The Turing Award (http://en.wikipedia.org/wiki/Turing_Award) is generally considered the equivalent. I don't know how serious you meant that statement, but I'd assume BT probably has some less popular prior art and you certainly have to do a little more to justify being in such prominent company.
Thomas on February 20, 2007 03:43 PMTo be fair, commenters on reddit pointed out that the BT protocol does define "a tit-for-tat-ish algorithm to ensure that [the client] gets a consistent download rate":
http://www.bittorrent.org/protocol.html
However, I still maintain that torrents *need* seeds to be viable. Relying on a 100% perfect distribution of data from an arbitrary number of peers seems improbable to me. I've participated in many, many torrents where there were lots of peers, but nobody had 100% of the data.. the plaintive cry of the torrent is always, always "need a seed!"
> I don't understand why companies don't simply provide a .torrent link, and then put a massive-bandwidth seeder to start it off.
Totally agree, but I think before that can happen, we need mass penetration of a "don't make me think" torrent client. Perhaps something in the web browser, like Opera, or as a native part of the OS itself.
Jeff Atwood on February 20, 2007 04:01 PMI haven't actually tested this, but it seems you don't actually need someone with 100% of the file downloaded. If two people each have half of the file, and they have exact opposite halves, it should be capable of combining them into one whole.
Of course, the odds of a situation like that are a little ridiculous.
Psy on February 20, 2007 06:24 PM>Totally agree, but I think before that can happen, we need mass penetration of a "don't make me think" torrent client. Perhaps something in the web browser, like Opera, or as a native part of the OS itself.
I'm actually somewhat surprised that Firefox hasn't included it into the base OS. Perhaps I should check for plugins? That'd be a big thing for Firefox to have, a Bittorrent client built in... simply click the torrent and Firefox will take care of it for you.
Any Firefox coders in the house? :-)
Malderi on February 20, 2007 09:06 PMOne other place torrents are useful is for distributing virtual machines. Similar to ISO, they tend to be large and are very useful.
Down here ADSL2 is only just taking off, so most people have upload rates under 512kb/s. Sucky. But at least very few ISPs cap uploads or even count them. For all it's crapness, mine at least does the throttle if you download too much but never touches my upload rate... but at 512k, they don't really need to.
Moz on February 21, 2007 12:35 AMsome tips when using the bittorrent protocol to download:
1) if you are behind a router in a LAN, you might need to
activate port forwarding to your PC, and if you are firewalled
you might need to allow that traffic in.
Bittorrent performs poorly if you are not directly reachable.
2) experiment a lot with your client configuration to gain
performance. The most configurable client is probably azureus.
3) generally, offering more upload bandwith increases your download
speed, but only up to a certain point, where it starts to hurt
instead. Find the best download/upload ratio experimentally.
4) if you are downloading material that might incur in legal problems
in some countries, be sure to filter out ip ranges of "bad"
trackers. Search "blocklist".
4a) watch for fake torrents/malware.
5) be a good citizen and seed.
> I'm actually somewhat surprised that Firefox hasn't included it into the base OS. Perhaps I should check for plugins? That'd be a big thing for Firefox to have, a Bittorrent client built in... simply click the torrent and Firefox will take care of it for you.
That's exactly what Opera 9 does actually (even though the bt client part of Opera is much slower than, say, uTorrent).
And there is a project for a Firefox Bittorent plugin, but it's in very early alpha stages (http://moztorrent.mozdev.org/)
Masklinn on February 21, 2007 04:13 AMFollowup on previous post: seems like there is also http://firepuddle.mozdev.org/, and from the Support Bittorent bugzilla page (https://bugzilla.mozilla.org/show_bug.cgi?id=bittorrent), it seems that both projects are in fact MIA.
masklinn on February 21, 2007 04:15 AMThat animated GIF could have benefitted from BitTorrent at that size! :)
Peter Bridger on February 21, 2007 04:48 AMI normally find your posts somewhat informative and interesting.
Gotta say, this one does not deliver. This is sooooo old news I have no idea why you only *just now* got around to posting it. If you're starving for material, talk about the philosophy of science and how it relates to computer science... that's a subject I'd LOVE to hear about.
Rafajafar on February 21, 2007 07:00 AMAnybody familiar with Participatory Culture Foundation's "Democracy?" It contains a generic Bittorrent client in addition to HTTP/FTP and is geared towards the "masses." I believe that they are trying to take things in the direction that some here have suggested it really needs to be going. Easy seeding of content and easy access for most non-technical users.
Anon-withal on February 21, 2007 07:35 AM"BitTorrent is a terrible Long Tail client"
Not really. As far as public trackers are concerned, I can see that. But if you apply a community layer over BitTorrent exchanges, the Long Tail thrives. That's what happens in private trackers. Just take a look at OiNK or Karagarga, and you'll notice a very, very long tail.
Jonesy on February 21, 2007 09:59 AM"I'm actually somewhat surprised that Firefox hasn't included it into the base OS. Perhaps I should check for plugins?"
Installing a plugin doesn't really qualifies, in my opinion, as "don't make me think"...
manu on February 21, 2007 02:23 PMWow.I can't believe people talk all the time about bittorrent
as the new incredibly intelligent idea.
REally, eMule was out there with same thing such as torrent
(digest file) decentralized and centralized network, integrated
search engine rewarning system and i don't know how many other things.
Is scalable as well.But its kind of slow and if you are behind nat its pretty bad.But its only a matter of something else.
@Matt >> Is there a legitimate use for these networks
VMware Virtual Appliances [1] are usually distributed via BitTorrent. They are by nature simply too big to be effectively downloadable by FTP if more than even a small number of people want to download one simultaneously.
@Masklinn >> Finally, about part 3 (especially on the subject of japanese animation), it should be noted that the "popularity" of series' fansubs can be used as a market study on the potentials of an official distribution of the serie.
Exactly! They should be embracing this usage, not wanting to kill it. Why the MPAA and RIAA et. al. doesn't get this speaks to their lack of ability to actually think. It must be that the brain rots once people start using lawyers instead of innovating...
@Rick O >> BT is great as a *supplement* to existing protocols....That is, don't just provide an HTTP or FTP link, but provide an alternate torrent link.
I'm not used BT much, but in downloading some VMware appliances I found that BT would take about 2-3 days whereas if one was offered via FTP I could get in a few hours. I don't understand the protocol at a technical level so this suggestion is really a question: wouldn't it make sense if BT were evolved such that it would start seeding from an FTP(-like) protocol but then move to the Torrent automatically so that there would only need to be one link? I guess why I'm asking is having multiple download links seems to be an either/or situation.
@Rick O >> The most common argument is "but I can't track it, and I can't advertise with it". This is totally untrue.
Ya just gotta those kinda people... ;-0
@Tom >> 10 minute tracker connections. P2P stuff like Kazaa and Bearshare are connected in 30 seconds...
I've seen Kazaa and Bearshare take an eternity... when the file isn't available via them! ;-P
@PL >> Maybe it's been pointed out already but bittorrent was not first with this kind of "sharing your bandwith", so he doesn't deserv any reward for that.
Don't know what kind of *reward* you are referring to, unless you just mean "kudos", but what BT does deserve acknowlgment for was that it did so in a way that worked for enough people to get widespread adoption. In my book, that is what deserves kudos, and all else is secondary; great works in a vacuum are simply wasted.
Martin Marconcini >> I like the protocol, although it's true that ADSL lines tend to discourage sharing due to the HUGE difference between the downstream and upstream
Damn those telecom providers!!! ;-)
@Damien >> Having the host keep a seed open is becoming a popular alternative to regular HTTP downloads for more robust/efficient downloads.
As a follow up on my comment/questio to Rick O, it would seem that the best approach would be paired BT+HTTP/FTP links and then clients that would negotiate those protocols to switch to BT as soon as it was seeded. Or maybe you are just saying that having the server always seed solves the need for that?
@ed >> Bob Cringely has posted several articles claiming that P2P should actually be the bandwidth savior instead of the bandwidth demon that ISPs make it out to be. But unfortunately the ISPs have not learned that yet.
Well, ISPs are acting just like most people act (especially these days): "What in it for me, at this moment?" Sadly when people/companies need to be altruistic to achieve an eventual greater collective good, they are usually shorted and won't go there. :-(
@Jeff >> Totally agree, but I think before that can happen, we need mass penetration of a "don't make me think" torrent client. Perhaps something in the web browser, like Opera, or as a native part of the OS itself.
Here's a thought: Can BitTorrent possible be implemented in JavaScript, or is that asking too much?
@Rafajafar >> I normally find your posts somewhat informative and interesting. Gotta say, this one does not deliver. This is sooooo old news I have no idea why you only *just now* got around to posting it.
Why would you post such a negative comment? Do you feel the world revolves around you? Clearly you are not familiar with the technology adoption lifecycle [2]; new technologies are not adopted all at once by all people, and obviously from the number of comments this is still a viable topic especially related to the misunderstand torrent publishers have about not seeding.
@Anon-withal >> Anybody familiar with Participatory Culture Foundation's "Democracy?" It contains a generic Bittorrent client in addition to HTTP/FTP and is geared towards the "masses."
Ah..sounds like what I was asking about above.
-Mike
[1] http://www.vmware.com/appliances
[2] http://ist-socrates.berkeley.edu/~fmb/articles/lifecycle/
Hello Y'All
I am relatively new to serious use of computing- such as topics discussed here- Bt Micro torrent AZ, et al, and various uses of these for education of self, interests hobbies etc. I will check back regularly, because I feel with all the postings and links here, I may be able to : 1) get Democracy Player up and running properly on my Ubuntu Edgy OS, which seems a bit of a challenge- 2) get Myth TV running as well,- it's a MYSQL issue- tutorial mysqlsetup needed- and- I am sure I had other software items- oh- I have 3) attempted to get bt set up properly as well but- apparently unless, like many here have stated, you have access to an open seeder, the cycle will crash when no data is available. So locating good available files for whatever one is looking for is critical.
I would like feedback at wkpickett0154atcomcastdotnet on the following- what torrent client is an overall "10" as far as user ratings? and any other comments ref my posting here as well is welcomed.
Thanks for all who post here - it- your post- helps educate the rest of us who may not be so tech savvy- and I hope that those who are, will be more explanatory going forward so everyone gets the benefit of your technical knowledge/application experience. In other words, spell out, describe more in detail, make it simple- MIS/KISS :-))
William Pickett
Albuquerque
Insanely detailed mathematical analysis of upload/download ratios.
A Mathematical Analysis of Why Ratios Are Bad, and a Better Solution to the Problem of Unwanted Leechers
http://thoughtyard.com/twiki/bin/view/Main/TheProblemWithRatios
Question: if trackers are so good at controlling this, then why is this analysis necessary? Answer: I stand by my original statement; BitTorrent largely relies on client altruism.
Jeff Atwood on February 26, 2007 10:29 PMMost of the problems with the bandwith / availablity compromise can *thankfully* be avoided by running a tracker and forced seed on one of your own machines instead of a direct-download (HTTP/FTP) server. i.e. Your live copy guarantees a full file is available, while the brunt of the download is spread between the peers.
Shawn on February 27, 2007 05:50 PMPeople should check out Metalink. It isn't a new P2P network, but it lets you download at your full capacity usually, by using multiple sources.
http://en.wikipedia.org/wiki/Metalink
http://www.geospatialsemanticweb.com/2007/02/25/metalink-unifies-internet-downloads
http://slashdot.org/article.pl?sid=07/02/25/144209
Ant Bryan on February 28, 2007 02:07 PMBittorrent users SHOULD be alturistic because it's only good karma.
Kyle Korleski on March 1, 2007 07:02 AMWhy is BitTorrent, though, overly complicated to use, suitable only for nerdy types who have the time and willingness to spend hours tweaking the awkward network settings, messing with piles of file types and, generally do all these steps neccesary to download a simple file?
Nesh Johnson on October 25, 2007 02:16 PMRather, why Bittorrent as such - sucks?
Nesh Johnson on October 25, 2007 02:17 PM@Jeff: That mathematical analysis isn't "needed", it's just something someone decided to do. And the analysis itself is somewhat flawed: point 2, about deadlocking, assumes all finite cases boil down to the two-peer situation. This is untrue, as any student of dynamical systems can tell you. Just consider three peers, A B and C, where A downloads from B at exactly the same rate he uploads to C, etc.
Steve on February 26, 2008 10:54 AM| Content (c) 2008 Jeff Atwood. Logo image used with permission of the author. (c) 1993 Steven C. McConnell. All Rights Reserved. |