You're right about the general treatment of uploads, it's a real pain.

As for video in particular, Imageshack (I think) has a toolbar to compress prior to the upload and it saves a ton of time. Saves zero time for other services, but it does make a huge difference if you don't care about which service you use.

Sorry to double-comment but it is Imageshack that does it, it's explained here: http://news.imageshack.us/blog/?p=16

LiveVideo also has a desktop uploader. How are you going to resume a file upload if the target server does not save the bytes for an incomplete upload?

The most general uploading tool that I know of is Firefox Universal Uploader:

https://addons.mozilla.org/en-US/firefox/addon/4724

I have no idea how well it works, but something like this should be easy to extend. In fact, I predict that it's only a matter of time before this becomes part of the Firefox core.

http://www.smugmug.com

SmugMug nails the upload process. I drag my photos onto the page, at which point I see thumbnails and checkboxes for each. When I press the upload button, a modal dialog with three upload streams appears. I see a progress bar and thumbnail for each stream. There is also an overall progress bar and a cancel button. I can't navigate away from the page until the upload finishes or I have canceled. Furthermore, when I cancel, the photos that did upload are already there.

Granted, this is an ActiveX control, but it is that best uploader I've ever seen.

Google and YouTube are terrible because they are monopolies, and don't have any incentive right now to innovate. The best uploaders, like the best software, will continue to come from those fighting to stay alive.

mozilla DOES provide the upload feedback. the little progress indicator in the status bar shows how much of the file is transferred at the moment.
MSIE has a similar progress bar but i'm not sure if it shows the upload status correctly, but it guess it should.

so what's the fuss all about?

In terms of http servers supporting restartable uploads one option is the upload support added to BITS - Background Intelligent Transfer Service.

http://msdn2.microsoft.com/en-us/library/aa362828.aspx

I don't believe you're right in saying that web browsers have "failed us" in terms of uploads. Uploading was never an intent of web browsing and only a consequence of user-generated content and (forgive me) Web 2.0.

The real problem is our infrastructure. Even our download speeds are horrible compared to other countries like Japan. If our download speeds are like this, we can't do anything about our uploading.

Not really what you are looking for, but SWFUpload allows developers to write good upload scenarios. Browser support would be better of course...

> Uploading was never an intent of web browsing

Nor was Gmail, or dozens of other things.

> mozilla DOES provide the upload feedback. the little progress indicator in the status bar shows how much of the file is transferred at the moment.

This doesn't appear to work. I just initiated an upload of a 41 mb video to Google Video with Firefox. The progress indicator in the status bar immediately shoots to about 65%-- and sits there, as far as I can tell.

I do think this is a two part problem. First, our ISPs have not kept up with the modern web. Upload speeds should be faster than they generally are in the US.

Second, browsers (and perhaps HTTP) should have better upload functionality. I could see HTTP having a built in protocol for using some sort of "upload token", generated on the web server, for the end user's browser to use to re-establish disconnected uploads. This way web servers like IIS and Apache could store partial uploads with the token (taking the need to program this increasingly common functionality away from the PHP or ASP.Net programmer). This would also provide web browsers like Firefox, IE, Safari and Opera a common protocol to auto-magically restart disconnects uploads. Perhaps this is a protocol that should sit *on top* of HTTP, but it would be nice to see something standardized across all clients and servers.

This may be a little off topic since its specific to ASP.NET

For asp.net apps there is an awesome free and open source upload control/httpmodule
http://www.brettle.com/neatupload
It handles large uploads gracefully with a nice progress bar. It also overcomes the default upload limitations where the entire file is loaded into server memory before you can save it somewhere on the server.

I use it any project where I need to upload files.

Cheers,

Joe

I see what you are getting at, but I see a caveat also:

When you download a file, and your internet dies, you can resume the transfer because you have part of the file stored on your drive. With uploaded content, its the youtube or whatever site that stores the download, and for them to have some kind of resume feature, they would have to store every full video, and every cancelled video (for some arbitrary amount of time).

However, I think a solution would be to just open up a separate browser window, which mimics the firefox downloader functionality, but is entirely developed by the video site. This way, your browsing is not at all affected by the transfer, and you can continue on your merry way, even closing that page down

Stop whining and do something. The Firefox code is out there. If it scratches your itch, fix it.

Welcome to the beauty of open source.

I guess you could always wait for IE 8, and hope it is in there.

Obviously not for video, but the new Flickr (http://www.flickr.com/photos/upload/) upload tool handles multiple files very easily.

I've uploaded large batches (75-100 pics) that total 150-200MB with ease.

It has a very simple step-by-step process and provides excellent feedback along the way.

Worth checking out for sure.

If I could grep the whole internet for "stop whining" and get a nickel for each hit, I'd, well, I wouldn't be able to carry all the nickels home.

It's true, the state of file uploads sucks. There's a reason though. HTTP was made for downloads. You send it a request, and it sends you a document. Request->document. So simple. That's what a webserver does, take requests for and serve documents. And that's what a browser does: make requests for and then receive and display documents.

So, when you do an upload, you're kind of faking it. You're just putting a BIG package of data in your request for a document. Files are literally encoded in POST data. It's not like the webserver asked for a document from YOU.

However, this kind of fakery is necessary. The web long ago became a two-way medium, and it's pretty sad that browsers don't handle post uploads with the same elegance (or at least similar) to downloads.

As for people who say "stop whining, there's a firefox extension"... that's great for me, but this is a DEVELOPMENT blog, and Jeff is talking about how developers can make it better FOR THEIR USERS.

Let's face it, the upload user experience for large files is horrible. I personally think what youtube does is great, and I think it's silly that Gmail file uploads don't have the same flash-based interface (since they have flash already for the in-browser mp3 player). They take 20MB uploads to email now, this would be pretty useful.

On a recent project where I needed to allow large uploads, I did pretty well with this hack:
http://digitarald.de/project/fancyupload/
(It's javascript + a tiny bit of flash)

Perhaps Opera could blaze the trail as it has on so many other important browser features?

Well... Opera actually displays the amount of uploaded bytes and the current transfer rate, so uploading is at least clearly displayed - even if it's still slow and painful.

I HATE your new fonts for this blog.

As far as the browser is concerned, an upload is simply a HTTP Post.

It doesn't really know, or care that your uploading files, just that its sending a big POST message to a server.

Consequently, if you don't use a technology like Flash or Silverlight, the only way give a progress report on your upload is to write a client script that calls the server every second ago and gets the progress from the server. This information is not accessible by client script to the browser.

And then there are very few servers that can create a filestream and let you poll it periodically for its size. I know I've seen an implementation done using Python and Apache, and also one using ASP.NET.

Either way, having to poll for your upload status is a horrible way to do it, and could be resolved if browsers would expose more of the internals of a form post, as well as the total size of the post.

Either way HTTP posting a file in general is not elegant and prone to timing out. Its amazing that Google works as good as it does.

What I really would like to see is a native File object exposed through javascript. This object would open a connection much like a XMLHTTPRequest object, yet would allow for asynchronous polling of its status.

Just snail mail the video! NetFlicks does it. Probably get there sooner!

well, yes, it sucks, except of when you live in sweden where I had about 500kilobytes per second upstream with a normal home user connection... but back in switzerland the upstream really is sadly crippled :(

John, I've considered sneakernet before..

The Economics of Bandwidth
http://www.codinghorror.com/blog/archives/000783.html

Call me naive, but isn't this what FTP is for? It sure seems that an FTP extension/enhancement to the browser would be the ticket.

well, yes, it sucks, except of when you live in sweden where I had about 500kilobytes per second upstream with a normal home user connection... but back in switzerland the upstream really is sadly crippled :(

Dudes, all I can say is http://digitarald.de/playground/uplooad.html :)

Greg

Call me naive, but isn't this what FTP is for? It sure seems that an FTP extension/enhancement to the browser would be the ticket.

Reporting stats from the browser can always be better. I totally agree. A standard way to implement resumable uploads is a long way off.

In order for a POST to continue an upload, the browser needs to know not to encode and send the beginning of a file. The multipart/form-data post that browsers support do not allow this. I could tell a user that they only send a partial file, but I wouldn't be able to request the second half because there is no way for them to send just that part.

Also, there is a security risk in allowing a server to specify which file from a users machine it would like partial data from. The server would have to rely on the user to verify that they want to send the rest of the file and hope they selected the correct file.

I image the best solution would be if a browser needed to send a file, the protocol could first send a hash of the file to be uploaded so the server could determine if it only needed some of the data, then report back which section it needed and let the browser send only that part. This would involve two POST requests and a change to the protocol.

The HTTP protocol and HTML form spec would have to be expanded to allow for resumable uploads. Current solutions are going to have to use AJAX, Flash or some other plugin or downloadable control, which means browser and os compatibility issues.

I agree that uploading has been neglected and that browsers can and should do more, i guess the reason is that many people dont use it a great deal if ever. Also its going to take along time before upload speeds increase to current download speeds, that would require fiber-to-the-door, or alike, since current tech (ADSL) is only capable of about 1Mbps upstream, altough most get half that (at leaar from my experience here in the uk)

Amazon has a pretty decent upload UI on its video reviews creation page. It provides bytes uploaded so far, a percentage, and gives a time remaining estimate. It looks like its pure JS.

http://www.amazon.com/gp/customer-reviews/create-review

(click the "Video Review" radio button, near the middle of the page)

I know I'm nitpiking, but people keep saying POST. The proper operation for file upload is PUT. Of course, PUT is not supported by modern browsers, which I think may be contributing to the lack of support for resumable uploads. http://cafe.elharo.com/web/why-rest-failed/

>I know I'm nitpiking, but people keep saying POST. The proper operation >for file upload is PUT. Of course, PUT is not supported by modern >browsers, which I think may be contributing to the lack of support for >resumable uploads. http://cafe.elharo.com/web/why-rest-failed/

File Upload in a browser is done with a POST. As explained above, its basically a hack.

> Most people have highly asymmetric internet connections: massive download bandwidth...

I wish :-(

I agree with Jim. The answer is probably FTP which, unlike HTTP, is specifically designed for two-way file transfer. Incidentally, all major browsers double as (simple) FTP clients. The question is how to integrate FTP file transfers with HTTP requests in a seamless manner. In any event, it would probably be easier than the current kludgey solutions, as long as all vendors would agree on a solution.

Eh, not according to Tim Berners Lee, who has written how upload was and should have been an integral part of the web browsing experience, going forth, from the early browser he tooled around with…

::I don't believe you're right in saying that web browsers have "failed us" in terms of uploads. Uploading was never an intent of web browsing and only a consequence of user-generated content and (forgive me) Web 2.0.

Hey Jeff,

We recently posted a Silverlight proof of concept app that resizes photos in the browser on the client before uploading:

http://fluxtools.net/

It's only a matter of time before an effort is made to do the same for video!

Cheers,
- Adam

Opera has an upload progress indicator. It isn't perfect, but it's something. You can expect it to be cloned in Firefox in 2009 or 2010.

While down here with my dial up connection, _downloads_ fail all the time. I have to use getright/wget for any thing larger than a couple of hundred kbs, or I end up with a failed download. Upload is _less_ painful, if anything.

Storing partial uploads on the server shouldn't be a concern since they are designated to store the entire file anyway. Now it's just a question of file identity.

But then again, it's obviously no surprise that, as has been said before, the hypertext transfer protocol isn't a particularily useful file transfer protocol.

But it can be extended to be one: a lot of storage sites do provide WebDAV access.

And of cause, there is HTTP PUT.

Pownce (http://www.pownce.com) has a pretty nice upload system. It gives real time status with a progress bar. However unless you go pro you are limited to 10MB uploads. I believe it is still invite only so if you would like an invite just ask.

I have to echo people from above. I started using the internet back in the Gopher/JANet days.

For uploads ftp is the only real way.

I alwyas reencode my videos before uploading.

My digital camera makes 848x480 or 640x480, 30 FPS in Quicktime format. Considering how bad Quicktime is, the video is almost 2 MB per second.

What I always do is resize the video to 320x240, and reencode it with a better codec.

Check out http://www.mediafire.com/ for a decent upload status bar. It's the best I've seen.

And what about online games? I takes as much download as upload to game online.

"My digital camera makes 848x480 or 640x480, 30 FPS in Quicktime format. Considering how bad Quicktime is, the video is almost 2 MB per second."

Corrected version:

My digital camera makes 848x480 or 640x480, 30 FPS in Quicktime format. Considering how bad the camera's encoder is is, the video is almost 2 MB per second.

Quicktime is a file format and media wrapper, with overhead measured in bytes or kilobytes, not megabytes. It is not a codec. It supports many of the best codecs (H.264, which is what youtube will convert your uploaded video too in any case, or even WMV of FLV codecs if you buy a plugin). This is in contrast to, say, WMV, which I believe only allows the WMV family of codecs. Most likely, your camera is using a ten-cent video encoding chip and writing out the equivalent of MJPEG inside a .mov wrapper. That being said, the priorities for a camera encoder chip are:

1. Get the video saved with as little quality loss as possible
2. Keep the cost down, because the quality/size of the encoded video is not generally a selling point and is easily "fixed in post" (ie, on the user's computer). It doesn't make sense to raise the cost of a consumer camera by $1 to provide better quality/size as more consumers will pass up the camera over that dollar difference than will buy the camera due to the encoding difference.

To be able to provide adequate support for HTTP upoads, the HTTP protocol needs to be changed. Browsers do not send the full size of the upload from beforehand, therefore servers do not provision for this and do not know how to handle it either.

Something else I'd love is the ability to select multiple files from the "Select File" dialog. It's really annoying doing the whole Browse, select, Browse, Select when you're uploading a lot of files

Why not switch it from HTML forms to C++ applications?

Instead of relying on the web browser and squabbling over different methods, why not just an application to slice-and-dice your video file then upload the chunks (doing this automatically). The server can then join everything up again.

Make it simple, divisible by 20, giving you 5% chunks. 100MB = 20 files of 5MB.

The "warez" scene uses this method, typically slicing everything up into 14.7MB RAR files. If something goes wrong with the connection, one is corrupt, it's quick and easy to fix.

Just a thought.

Okay

facebook.com resizes and compresses images before sending
- they use flash

http://digitarald.de/playground/uplooad.html
- allows you to upload many files at once AND gives you a progess indicator. The only thing it doesn't do is resume broken uploads.
- it uses flash

see a pattern?

Flash isn't broken. !! :)

The problem with uploads is that HTTP was not meant for doing uploads. As was mentioned in this article. I share all my family pictures and videos using dropshots (www.dropshots.com). They have an application that you does the uploads, that is completely separate from your browser. Sure it's yet another application to install, but it makes everything much easier. Just select the pictures and videos you want to upload from windows explorer (the file manager, not IE) and drop then on the upload window. Everything is done for you. The videos are converted to the proper format on your machine, so uploads are small, and it displays a nice little progress bar. I haven't examined the protocol, but I assume it isn't using HTTP to send the videos.

Mr. HTTP - I think you're wrong (or at least I'm not 100% sure what you meant when you said "Browsers do not send the full size of the upload from beforehand". I'm not sure what you mean by "beforehand" since HTTP is a stateless protocol. In vanilla HTML files are uploaded from the browser by a HTTP POST. All major browsers include a "Content-Length" header as part of the POST request. Although this is the length of all the fields in the post (for example if you had some checkboxes/radio buttons/text boxes on the form that was POST'ed the size of this data will also be included in the Content-Length). For most file upload cases the size of the file being uploaded dwarfs any other fields in the post, so you can take it as being very close to the size of the file being uploaded.

My real beef with file upload as a developer is that there is no "nice" way on the client to restrict the size of the file upload in vanilla HTML. none of the major browsers honour RFC 1867 which says "If the INPUT tag includes the attribute MAXLENGTH, the user agent should consider its value to represent the maximum Content-Length (in bytes) which the server will accept for transferred files." I guess they can be forgiven since the HTML 3.2 spec says that the maxlength attribute can be used to restrict the length of the file NAME!!! (how helpful is that). This (AFAIK) is why most major sites that deal with file upload use flash, because the flash file upload widget gives access to the size of the upload via script, and thus you can prevent users trying to upload their entire DVD collection to your site by mistake. The following site has a nice analysis of the file input HTML widget http://www.cs.tut.fi/~jkorpela/forms/file.html at least now we've developed decent approaches to styling the file inputs http://www.quirksmode.org/dom/inputfile.html

I googled a bit about the HTTP PUT method. Miminalistic perhaps, but it does provide upfront Content-Length and other headers. Seems HTTP PUT is one of smugmug's preferred upload methods, citing efficiency over the MIME encoding that HTTP POST requires. While a quick glance shows that recent version of Firefox and IE no longer support PUT natively, that doesn't seem to be a big issue in this AJAX and OSS world.

As mentioned by other commenters, I believe the problem, primarily, to be that originally the people providing content to a site, and the people visiting the site, were very distinct subsets of of people with no overlap. Sure, that's not true today- But every site which allows uploads has their own way of doing it, and it would be unreasonable to ask the web browser to abstract out each individual site's uploading methodology.

I'm hoping firefox (or IE- as much fun as elitism can be, lets face it, it's a major player in the browser space) implement some sort of drop-box for files, say, as a side window- and then, via a little javascript, the website could tell the browser "if a user drops a file in here, this is what we'd like you to do with it". Flock has "2.0 support", as buzzwordy as that may sound- but they just took a few of the more popular services and abstracted the API's into the browser. I'm saying, if this was standardized, we wouldn't need flock. Drag & drop a video into the drop-box when you're logged into youtube. Wait a bit for it to finish uploading (sadly, this isn't a solution for the upload bandwidth problem. Living in a college dorm or paying out the nose are pretty much your two options there), and be taken to a special page where you can add some metadata.

Also, as a side note, AOL came out with a photoservice recently called BlueString, backended by their "xdrive" backup service. I haven't tried it yet, but it seems like an interesting way to go, dropping files into a mapped drive on your desktop and seeing them appear on the website...

It's always been a joke among my friends when we need to send videos to each other that's NOT in some miserable quality provided by YouTube: It'll get there faster by snail mail.

While on that topic, considering how we're gradually moving to HD videos, I always thought YouTube was something of a step backwards and poor upload mechanisms of the web today is to be blamed largely for that.

Look what I found:
https://bugzilla.mozilla.org/show_bug.cgi?id=255014

What about a torrent solution? Wouldn't it be nice if you could upload just few kb to tell the server where to get the file and then forget about bandwidth caps, network glitches or even system crashes? even if you're the only seeder it's still the most convenient way.

"Wow, this POST is over 5 MB, so I'll show the upload progress dialog." Seems like the easiest solution from the Firefox side. They don't have to treat all POSTs the same.

In fact, GETs are already treated differently. If it's a "large" file, it goes in the download manager, and if it's small, it doesn't.

Spot on Jeff

The main 'pain' with uploading is the limited speed. Hopefully this will improve over time (you can already get 1-2meg uploads with some ADSL providers).

The upload status can be resolved by using some form of applet, for example, I use a Java applet which also supports drag and drop. Facebook and MSN use various applets that are good.

You can also get upload status with normal post/put methods however, for various technical reasons, this requires you to patch Apache.

I upload files to a server all the time via FTP and it supports resume as well.

Surely this can be utilised by browsers as well.

I uploaded a video to MSN's SoapBox last night and the experience was much better than anything you described. While the service isn't as popular as YouTube, I noticed 2 things:

* the quality of the resulting video was much better than YouTube's
* the upload experience was actually pleasant. During the upload, I was allowed to view other views and even upload multiple files!

If MSN can do it, I am sure YouTube and others can too. BTW, I wasn't using Internet Explorer so there was no ActiveX components involved.

I just love that your screenshot of IE's download dialog is downloading Firefox.

I totally disagree that 'HTTP was never meant for uploads' - this is observation is completely baseless. HTTP is underdeveloped for uploads via the browser, which is what Jeff's post is all about. Because it's underdeveloped we all invent little nonstandard hacks to get around this basic need that is currently unfulfilled.

We could easily implement better (and stricter) standards for HTML and browsers to support better uploading capabilities. Unless you're doing something like re-encoding video or resizing pictures, chances are that you don't (or shouldn't) need a specialized uploader app. What we're talking about here is meeting the needs of 80% easily and let the other 20% do their own things.

I also have to make a rebuttal against all of the 'just use flash/whatever' crowd. Flash is not a standard part of HTTP/HTML. Things need to be folded into the standard so that compliant clients can be built without relying on 3rd party proprietary applications. Flash isn't free nor is Flash typically found in a web developers skillset. I don't want to hire someone just to do a flash-based upload applet and then have to support that as well.

---

A personal peeve is how in ASP.NET you have to set both IIS and a .config setting for the max upload size with the unfortunate effect that you can't (or couldn't) restrict it to just one page, but it affects the entire web application. Per-page restrictions are necessary to prevent DOS-style attacks.

I also hate how you can't just drag and drop the 'to be uploaded' files onto browsers. They unfortunately want to display the file in the browser instead of putting the path into the file input control. Flickr without the uploadr is painful at best, and usability is reduced.

Sounds like a great candidate for a Firefox plugin.

I get thoroughly fed up at the way providers such as comcast hawk their service and encourage us to share our photos and videos etc. but don't provide adequate upstream bandwidth to make it usable and reliable.

I agree that the browsers could do a much better job of providing upload support for progress and restart, but there wont be much incentive until the ISPs open up the pipe.

I use a dynamic iframe which posts to my upload PHP, a server-side CGI which monitors the upload in realtime, and some ajaxy stuff to show progress. All links on the page should have a different target, and the window.onbeforeunload event should be trapped to allow the user to confirm they want to leave the page.

It's actually pretty complicated to pull this off (for eg, iframe behaviour across browsers is a total PITA) but there's no excuse not to add at to HTTP/javscript standards least some manner of feedback to the form file upload field so people don't have to struggle so much with this crap.

There's a whole other problem, too -- getting old film TO digital format so it CAN be uploaded. Disclosure - I work for www.imemories.com. This solves the problem of upload for Baby Boomer types and those who don't have bandwidth.

I total agree. And this is why we build all upload forms with Telerik's upload/progress bar control. It's pretty great. Uses AJAX and you can program it to provide as much, or as little feedback as you want. Check it out at telerik's product demos at telerik.com

The IE download example is presented downloading the Firefox installation file! Very subtle indeed!

The conclusion of the article calls for solutions via browser plugins "such as Silverlight and AIR."

A minor knit, but AIR is not a browser plugin. If you're looking for Adobe's "sophisticated browser plugin", you already have it - it's Flash.

AIR is to .NET as Flash is to Silverlight. Roughly.

>>> mozilla DOES provide the upload feedback. the little progress
>> indicator in the status bar shows how much of the file is transferred
>> at the moment.
>
>This doesn't appear to work. I just initiated an upload of a 41 mb
>video to Google Video with Firefox. The progress indicator in the
>status bar immediately shoots to about 65%-- and sits there, as far
>as I can tell.

i think this is because google video uses some javascript hack to monitor the upload progress, effectively disabling the browser's own indicator. i can recall it worked fine for "plain" upload forms without javascript.
so the course of events was: it worked well, javascript broke it (by using out of band requests), now the browsers should find a way to fix it again...

While reading the article i thought - what about ajax?

And Gregory Magarshak already posted a link above on how to realize this with mootools (javascript framework). Works like a charme and is the better way then using flash imho.

the polarizer

I definitely agree with you Jeff, that's why on my website(www.tripntale.com) to be launched within a few days, my main goal is to let users upload by select multiple files at once and show real progress while the upload is in progress.

sorry, I forgot to mention that I'm using both Flash control and Ajax to show uploading progress

Using HTTP for file transfer (and email and everything else under the sun) seems like a classic case of a developer armed with a hammer treating every problem as a nail.

What happened to the protocol stack? Tunneling everything over HTTP is a big step backwards - TCP provides much better tunnels, it has flow control, error checking etc etc. POP, IMAP and FTP are all much better suited to their respective tasks. HTTP sucks for anything requiring state information.

This is where web 2.0 has got us. It seems to stem from the public perception in the mid 90's that the web and the internet are the same thing ("teh interweb"). Perhaps it's just the default mental model most people have of computers. WYSIWYG. The GUI is the program, (anything else is just too confusing). We have folders instead of directories that we "paste" files between.

It's bad enough when it' Jo Public's mental model of the net, it's unforgivable when programmers view the internet the same way.

Our team has developed a web site that allows file uploads along with a fairly accurate progress indicator and estimated time of upload. Send me an email at mehus***NOS_PAM***ain[at]yahoo[dot]com in case you would like a preview...

I'd like to point out that opera does have full transfer feedback, up and down. While it does not appear to calculate the total upload/download, so it cannot give you a percentage, it does give upload speed and progress in byte-units. this is a general setting for all webpages, and the interface covers both up and down. When I open a page, I can see it downloading and, by the same token, I can see how much has been uploaded when a web form requests a file. This is extremely useful for email attachments and any other web-based uploading. I don't however, use youtube or any other such video site unless I want to spread the video to the general public. FTP (or SFTP rather) is still a far superior method of uploading files to a server, and BitTorrent is without a doubt the best way to upload to multiple computers. And yes, you can use BT to upload to a server network, not justt peer clients.

Here's an example screen shot showing the interface on a regular page load. Uploading is the same, the upload is lumped in with the loading of the recieving page.

http://img456.imageshack.us/img456/8277/operaawesometh9.png

It's also extremely useful for embedded content as you can see it's load progress in terms of speed and size, if not percentage.

While there is a percentage progress bar there, it is rather innacurate as the browser's information on the page is updated during loading.

I've been using http://transferbigfiles.com lately. Not sure what their upload progress bar actually is, but it's a nice enough service for sending big files to one or more people quickly.

What an amazingly timely post, Jeff, for me at least. I'm currently dealing with this very situation in a web application I'm writing that needs to handle large (multi-gig) file uploads in a user-friendly way.

The problem, as others have mentioned, is that the current version of the much hacked HTTP protocol doesn't support any sort of file upload other than the whole damn thing. That's how a file upload works -- the browser sends a POST request with the entire file base-64 encoded as part of the request. There's no way to get *real* feedback from the server because the server doesn't *do* anything until it gets the whole POST and if the timeout isn't long enough or the maximum acceptable file length isn't large enough, the user ends up with a failed transmission after having waited for several minutes/hours. There's no way to get around it without hackery.

Some others have suggested FTP. Yes, FTP is designed for file transfers, but remember that it's designed *specifically* for file transfers, and for no other purpose, and if you go the FTP route, you open up several cans of worms. Web applications are built on top of HTML/HTTP, with cookie or URL-embedded session based authentication. If you want to include FTP you have to figure out a way to include that authentication in the FTP session, a very non-trivial task. Even then, you're stuck with whatever FTP client the browser provides, meaning the entire website layout disappears, creating a big usability nightmare.

There are third-party hacks out there that do a decent job of providing feedback... typically they take the form of an AJAX component that sends the file in the background, with a server-side component that monitors the progress of file upload tied to a web service that the AJAX uploader component calls repeatedly to obtain a percent uploaded.

Although this helps solve the usability issue, it also opens the server up to denial of service attacks, as it requires very large maximum file upload and very long timeout settings for the server.

Alas, there isn't a web application silver bullet. The best way to maintain usability and security in a pure web application would be to split the file into chunks the web server can handle (as suggested by transcriber) and reassemble them server-side. Unfortunately (for entirely different security reasons), browsers don't typically allow client-side scripts to perform the necessary file I/O for splitting the file.

This can be gotten around with an active X control or a plugin, but that route brings up user-pain issues (another thing to install, etc.).

Still... having to install an active x control (or silverlight, or whatever) may be the past of *least* pain, especially given the feedback benefit.

And now I need to post a retraction. It turns out that IIS 7 / Windows Server 2008 will make FTP authentication integration with a web application much more trivial:

http://weblogs.asp.net/scottgu/archive/2007/09/27/iis-7-0-hits-rc0-lots-of-cool-new-iis7-extensions-also-now-available.aspx

Of course, there's still the whole separate client issue, but it's a big jump in the right direction.

I just had a discussion with some colleagues about this. They just got an upload progress bar setup for their folder sharing site [1]. They said it involved "a deal with the devil" and "a world of darkness and terror" from a code perspective (.NET shop), but that once it was setup it works really well. Just my (well, their) two cents.

-Erik

[1] - http://folders.freshlogicstudios.com/

I'm currently using FileUp from SoftArtisan in a project where we have 3GB+ uploads & downloads. To get all of the functionality you need to purchase the Enterprise version.

http://fileup.softartisans.com/

It's true. I use my own written software:
.NET clickonce or a Java applet or an ActiveX to upload. Real responses but you have to install software..

It would have been funnier if the Firefox downloader was downloading Opera. Then again it's also an indictment of our society that the second download in the Firefox box is AVG (not sure about the first, but is the third one Adaware?).

>The most general uploading tool that I know of is Firefox Universal Uploader:
>
>https://addons.mozilla.org/en-US/firefox/addon/4724
>
>I have no idea how well it works,

The UI is painfully, *painfully* slow. It runs like molasses. If you don't mind being tied to a specific upload type (e.g. photo sites), there are some good site-specific upload managers like Fotofox, which will not only manage uploads but also handles tagging, folders on the remote site, and so on, all on the local machine rather than having to do it via POSTs to the remote site.

I agree absolutely with your article. I do not understand why Firefox developers do not take the chance to enhance their product offering a decent file upload interface. I think browsers should offer a wider palette of tools, making upload forms in .js or flash looks like running around an obstacle in fancy ways instead of finding a simple solution. Enhancing the browser's user interface for file upload would give instant benefit to all traditional websites as well!

I've developed a really cool uplading bar that doesnt use all these refreshes and tricks. Supports really large files and you can pull down a whole host of variables from the server while a file is uploading. Best of all its AJAX-free and doesnt use refreshes in popups. Uses simple JavaScript call. I do not have an online demo right now. But it can be found in AudioIsland and VideoIsland web products: http://www.giantisland.com.

As far as HTTP, the browsers, and the Web as a whole...yes, it is NOT designed for large file uploads. No matter what people may say in this block, uploading a file to a server is still a "hack". Not only is it prone to error via the client-server POST, but connections can be dropped, variable client bandwidth issues can delay uploads, server connection timeouts can occur, server script timeouts can occur, sessions can expire, browser connection management can fail, etc etc. The list goes on and on.

Thats also why we have FTP for this...its designed for file transfers.

As far as improving the process, yes, the servers, vendors and browsers could all contact the W3C and tell them we want a new recommendation for all this in the next release of the latest markup rec. But that doesnt address the central problem of client-to-server connections using large file sizes, security, connection issues and other things. At best, what we can hope for is better integration of TCP/IP with some new ways of handling these types of transfers. Until we have some giant pipe and better compression scheme for picking up these types of things, its always going to be a "hack", no matter who the vendor is.

Good news is....like my solution and many on this blog, there are some "great workarounds"!

FTP does not work well with a proxy server. There are about 9 different methods to connect over a proxy server because there is no standard like a RFC document.
The FTP protocol is too old and too unflexible; HTTP1.1 or WebDAV is the better solution for uploading files to a webserver.