diff --git a/README.md b/README.md index 5016a50..3a019f4 100644 --- a/README.md +++ b/README.md @@ -10,10 +10,11 @@ Mule-upload * VERY resilient against upload interruptions. Even if your internet connection goes down, you accidentally close the browser or you want to continue the upload tomorrow, your upload progress is saved. Hell, it even works if you switch browsers or wifi connections! * HTML5 - uses the `File`, `FileList`, and `Blob` objects * Speed - it uses multiple workers for (potentially) four time increase in upload speed. E.g. on my computer I got 2.5-3 MB/s vs. < 1MB/s using only one worker. There is a tradeoff between upload speed and CPU consumption though. +* Multiple file uploads supported #### What people think of it: -> We use Mule Uploader to archive audio in our Rails/AngularJS application www.popuparchive.org. I tried many projects that integrate with S3 in various ways before using this. By using the multipart upload API, multiple threads, and resumable uploads, it met our essential needs for handling large media files, and without requiring a specific UI or DOM elements. It also came with no dependencies on jQuery or other libraries, making it easy to add to our AngularJS front-end. +> We use Mule Uploader to archive audio in our Rails/AngularJS application www.popuparchive.org. I tried many projects that integrate with S3 in various ways before using this. By using the multipart upload API, multiple threads, and resumable uploads, it met our essential needs for handling large media files, and without requiring a specific UI or DOM elements. It also came with no dependencies on jQuery or other libraries, making it easy to add to our AngularJS front-end. > > -- Andrew Kuklewicz, Tech Director prx.org, Lead Developer www.popuparchive.org. @@ -82,13 +83,50 @@ In order to use this library, you need the following: } ``` -5. You need a backend to sign your REST requests (a Flask + SQLAlchemy one is available at example_backend.py). +5. You need a backend to sign your REST requests (a Flask + SQLAlchemy one is available at example_backend.py). Here are code samples for creating the signing key: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html 6. For detailed instructions about how each of the ajax actions should respond, read the source code; there are two actions: * `signing_key` - returns a signature for authentication -- http://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html . Also returns key/upload\_id/chunks if the file upload can be resumed. Should also return a backup\_key to be used in case that the first one is not usable. * `chunk_loaded` - (optional) notifies the server that a chunk has been uploaded; this is needed for browser-refresh resume (the backend will store the chunks in a database, and give the user the file key + upload id + chunks uploaded for the file to be uploaded) +7. Setup minimal html code: + ``` + + ``` + or if you want to allow multiple files to be uploaded at once: + ``` + + ``` +8. Setup minimal javascript code: + * for single file upload: + ``` + var settings = { + file_input: document.getElementById("file"), + access_key: YOUR_AWS_ACCESS_KEY, + content_type: "application/octet-stream", + bucket: YOUR_AWS_BUCKET, + region: YOUR_AWS_REGION, + key: YOUR_S3_FILE_PATH, + ajax_base: YOUR_BACKEND_URL + }; + var upload = mule_upload(settings); + ``` + * for multiple file uploads: + ``` + var settings = { + file_input: document.getElementById("file"), + access_key: YOUR_AWS_ACCESS_KEY, + bucket: YOUR_AWS_BUCKET, + region: YOUR_AWS_REGION, + + content_disposition: false, // if false - don't add Content-Disposition:attachment header for the uploaded file + key_prefix: "some-subdir/", // "subdir" for the uploaded files, used if "key" setting is empty or not set, each file will be uploaded under it's own name.ext + ajax_base: YOUR_BACKEND_URL + }; + var upload = mule_upload(settings); + ``` + If you'd want example backends in other languages/with other frameworks, let me know. @@ -119,4 +157,4 @@ Due to the new technology used by this library, it's only compatible with the fo * Updated Chrome * Updated Firefox * Safari 6+ -* not sure about IE +* IE 11+ diff --git a/mule-uploader.js b/mule-uploader.js index f9ff8e2..bb8c05c 100644 --- a/mule-uploader.js +++ b/mule-uploader.js @@ -172,7 +172,12 @@ // the S3 object key; I recommend to generate this dynamically (e.g. // a random string) to avoid unwanted overwrites. - settings.key = settings.key || "the_key"; + // if key is empty - file.name used as a key + settings.key = settings.key || ""; + settings._key = settings.key; //remember original settings key for multi-file upload case + + // "sub-directory" for the uploaded files, used if settings.key is empty + settings.key_prefix = settings.key_prefix || ""; // the Amazon S3 bucket where you'll store the uploads settings.bucket = settings.bucket; @@ -183,8 +188,12 @@ // the Mime-Type of the content. You must match this with the backend value // or you'll get an Invalid Signature error. If unsure about the // mime type, use application/octet-stream - settings.content_type = settings.content_type || "application/octet-stream"; + // if no content_type defined - file.type will be used + settings.content_type = settings.content_type || ""; + //if content_disposition is explicitly set to false - don't add it to requests + //otherwise all uploaded files will be uploaded as an "attachment" + settings.content_disposition = (settings.content_disposition===false) ? false : true; // acl can be set to: // private @@ -202,6 +211,7 @@ settings.on_select = settings.on_select || function() {}; settings.on_error = settings.on_error || function() {}; settings.on_complete = settings.on_complete || function() {}; + settings.on_complete_all = settings.on_complete_all || function() {}; settings.on_init = settings.on_init || function() {}; settings.on_start = settings.on_start || function() {}; settings.on_chunk_uploaded = settings.on_chunk_uploaded || function() {}; @@ -220,7 +230,7 @@ u.set_state("waiting"); if (u.input) { - u.input.onchange = function(e, force) { + u.input.onchange = function(e) { if(!u.settings.autostart) { return true; } @@ -230,10 +240,21 @@ return false; } - // the uploader doesn't support multiple uploads at this time, - // so we get the first file - var file = e.target.files[0]; - u.upload_file(file, force); + //support for multiple files upload + u.files = e.target.files; + u.current_file_index = -1; + u.files_uploaded = 0; + u.files_total = u.files.length; + //remember total filesize for use in total progress + u.files_total_size = 0; + u.files_total_uploaded_size = 0; + for (var i = 0; i < u.files_total; i++) { + u.files_total_size+=u.files[i].size; + }; + + //proceed with first file + u.upload_file(); + return true; }; } @@ -246,29 +267,51 @@ Uploader.prototype.start = function() { if(this.input && this.input.files && this.input.files.length > 0) { - return this.upload_file(this.input.files[0], false); + return this.upload_file(false); } else { - alert("No file selected"); + alert("No file(s) selected"); } }; - Uploader.prototype.upload_file = function(file, force) { + //if force=true - don't proceed next file, instead force re-upload current file + Uploader.prototype.upload_file = function(force) { var u = this; - // the `onchange` event may be triggered multiple times, so we - // must ensure that the callback is only executed the first time - // also make sure the file is not already set. - if(u.get_state() != "waiting") { - return false; + if (!force){ + u.current_file_index = u.current_file_index+1; } - if (file) { - u.file = file; + u.set_state("waiting"); + + //check if all files processed + if (u.current_file_index >= u.files.length){ + u.settings.on_complete_all.call(u); + return; } + //reset for new file + u.settings.on_progress.call(u, 0, 0); + u.upload_id = null; + u._progress = null; + u._total_progress = null; + u._loaded_chunks = null; + u._uploading_chunks = null; + u._chunks = null; + + var file = u.files[ u.current_file_index ]; + u.file = file; + if (!u.file) { return false; } + //set key for current file upload + if (u.settings._key=='') { + u.settings.key = u.settings.key_prefix + file.name; + }else{ + u.settings.key = u.settings._key; + } + if (u.settings.content_type=='') u.settings.content_type = file.type; + // we use the lastModifiedDate, the file name and size to uniquely // identify a file. There may be false positives and negatives, // but the chance for a false positive is basically zero @@ -280,8 +323,11 @@ alert( ["The maximum allowed file size is ", (u.settings.max_size / GB), - "GB. Please select another file."].join('') + "GB. Please select another file.", + "\nFile: ", + file.name].join('') ); + u.upload_file();//upload next file return false; } @@ -307,22 +353,39 @@ alert( ["This file format is not accepted. ", "Please use a file with an extension like ", - u.settings.accepted_extensions].join('') + u.settings.accepted_extensions, + "\nFile: ", + file.name].join('') ); + u.upload_file();//upload next file return false; } } - // initialize the file upload - u.settings.on_select.call(u, file); + file.is_force = force; + // trigger the on_select event callback + if ( u.settings.on_select.call(u, file, u.on_selected)=== false ){ + // if on_select returns exactly false - just wait for on_selected callback; + }else{ + //otherwise - just proceed with upload + u.on_selected(file); + }; + + }; + + //called after on_select event + Uploader.prototype.on_selected = function (file) { + var u = this; + // initialize the file upload var args = utils.extend_object(u.settings.extra_params || {}, { filename: file.name, filesize: file.size, - last_modified: file.lastModifiedDate.valueOf() + last_modified: file.lastModifiedDate.valueOf(), + contenttype: u.settings.content_type }); - if(force) { + if(file.is_force) { args.force = true; } @@ -344,7 +407,10 @@ u.settings.backup_key = u.settings.key; if(!u.upload_id) { - AmazonXHR.init(json, u.settings.key, file, function(e) { + AmazonXHR.init(json, u.settings.key, file, u.settings, function(e) { + if(e.target.status / 100 != 2) { + return u.settings.on_error(e); + } var xml = e.target.responseXML; // get the given upload id @@ -354,7 +420,7 @@ }); } else { // resume a previus upload - if(!force) { + if(!file.is_force) { // get the uploaded parts from S3 AmazonXHR.list(u.auth, u.file, u.settings.key, u.upload_id, u.settings.chunk_size, function(parts) { for(var i=0; iMIME_TYPE = getenv_default('MIME_TYPE', 'the_mime_type'); $this->BUCKET = getenv_default('BUCKET', 'my_bucket'); $this->AWS_SECRET = getenv_default('AWS_SECRET', 'the_secret_access_key'); $this->AWS_ACCESS_KEY = getenv_default('AWS_ACCESS_KEY', 'the_public_access_key'); @@ -162,6 +160,7 @@ public function upload_action($action) { $filename = $_GET['filename']; $filesize = $_GET['filesize']; $last_modified = $_GET['last_modified']; + $contenttype = $_GET['contenttype']; $data = array( "date" => $date->format('c'), @@ -170,7 +169,7 @@ public function upload_action($action) { "region" => $this->REGION, "bucket" => $this->BUCKET, "backup_key" => strval(rand(0, 10000000000)), - "content_type" => $this->MIME_TYPE + "content_type" => $contenttype ); $fresh_upload = isset($_GET['force']); @@ -199,7 +198,11 @@ public function upload_action($action) { if(!isset($_GET['action'])) { $key = (string)rand(1, 1000000); - include 'index_template.php'; + if ( isset($_GET['multi']) ){ + include 'index_multi_template.php'; + }else{ + include 'index_template.php'; + } } else { echo $backend->upload_action($_GET['action']); } diff --git a/php_backend/index_multi_template.php b/php_backend/index_multi_template.php new file mode 100644 index 0000000..eb07649 --- /dev/null +++ b/php_backend/index_multi_template.php @@ -0,0 +1,215 @@ + + + + Mule Uploader + + + + + + + + + +
+
+

Mule Uploader multi-file example

+

+ Mule Uploader gets your file from your computer to Amazon S3, no matter what. + Wireless just went down? A badger ate the power cord? + You have to go now and need to resume the upload when you get back? Mule Uploader has you covered. +

+

+ Why use this? Because having a 3GB file interrupt at 87% isn't pretty, not for you and especially not + for your users. +

+

+ Go ahead, select one or multiple files. It will start uploading automatically. +

+
+ + +
+ Current file: +
+
+
+
+
+
+ Total progress: +
+
+
+
+
+ +
+ +
+

+ New feature! – we've upgraded to Amazon Signature V4. This release is still experimental, so if you notice a bug please report it to me and / or add a pull request. There is also a new dependency on Web Workers. The old uploader version will work for all Amazon regions added prior to Jan 30 2014, as per Amazon's documentation. Also, the backend is much simpler, needing only 2 endpoints instead of 6. +

+

+ For more information, visit the GitHub page. + (you can see the source code for this demo there, and you get free cookies!) +

+

If have any questions about this, drop me a line at + + + +

+
+
+
+ Patterns thanks to subtlepatterns.com +
+ + diff --git a/php_backend/index_template.php b/php_backend/index_template.php index 51de0e2..8e025b0 100644 --- a/php_backend/index_template.php +++ b/php_backend/index_template.php @@ -59,7 +59,7 @@ file_input: document.getElementById("file"), access_key: "AWS_ACCESS_KEY ?>", - content_type: "MIME_TYPE ?>", + content_type: "application/octet-stream", bucket: "BUCKET ?>", region: "REGION ?>", key: "", @@ -150,6 +150,10 @@ For more information, visit the GitHub page. (you can see the source code for this demo there, and you get free cookies!)

+

+ New feature! Multiple files upload now supported, demo page here. +

+

If have any questions about this, drop me a line at