Skip to content

Multi file upload support #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
osalabs wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Multi file upload support #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef72c6c
multi-upload support enabled
osalabs Jun 8, 2015
3dc787b
fixed: uploader won't work second time without page refresh, added te…
osalabs Jun 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 41 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ Mule-upload
* VERY resilient against upload interruptions. Even if your internet connection goes down, you accidentally close the browser or you want to continue the upload tomorrow, your upload progress is saved. Hell, it even works if you switch browsers or wifi connections!
* HTML5 - uses the `File`, `FileList`, and `Blob` objects
* Speed - it uses multiple workers for (potentially) four time increase in upload speed. E.g. on my computer I got 2.5-3 MB/s vs. < 1MB/s using only one worker. There is a tradeoff between upload speed and CPU consumption though.
* Multiple file uploads supported

#### What people think of it:

> We use Mule Uploader to archive audio in our Rails/AngularJS application www.popuparchive.org. I tried many projects that integrate with S3 in various ways before using this. By using the multipart upload API, multiple threads, and resumable uploads, it met our essential needs for handling large media files, and without requiring a specific UI or DOM elements. It also came with no dependencies on jQuery or other libraries, making it easy to add to our AngularJS front-end.
> We use Mule Uploader to archive audio in our Rails/AngularJS application www.popuparchive.org. I tried many projects that integrate with S3 in various ways before using this. By using the multipart upload API, multiple threads, and resumable uploads, it met our essential needs for handling large media files, and without requiring a specific UI or DOM elements. It also came with no dependencies on jQuery or other libraries, making it easy to add to our AngularJS front-end.
>
> -- Andrew Kuklewicz, Tech Director prx.org, Lead Developer www.popuparchive.org.

Expand Down Expand Up @@ -82,13 +83,50 @@ In order to use this library, you need the following:
}
```

5. You need a backend to sign your REST requests (a Flask + SQLAlchemy one is available at example_backend.py).
5. You need a backend to sign your REST requests (a Flask + SQLAlchemy one is available at example_backend.py).
Here are code samples for creating the signing key: http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html

6. For detailed instructions about how each of the ajax actions should respond, read the source code; there are two actions:
* `signing_key` - returns a signature for authentication -- http://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html . Also returns key/upload\_id/chunks if the file upload can be resumed. Should also return a backup\_key to be used in case that the first one is not usable.
* `chunk_loaded` - (optional) notifies the server that a chunk has been uploaded; this is needed for browser-refresh resume (the backend will store the chunks in a database, and give the user the file key + upload id + chunks uploaded for the file to be uploaded)

7. Setup minimal html code:
```
<input type="file" id="file"/>
```
or if you want to allow multiple files to be uploaded at once:
```
<input type="file" id="file" multiple/>
```
8. Setup minimal javascript code:
* for single file upload:
```
var settings = {
file_input: document.getElementById("file"),
access_key: YOUR_AWS_ACCESS_KEY,
content_type: "application/octet-stream",
bucket: YOUR_AWS_BUCKET,
region: YOUR_AWS_REGION,
key: YOUR_S3_FILE_PATH,
ajax_base: YOUR_BACKEND_URL
};
var upload = mule_upload(settings);
```
* for multiple file uploads:
```
var settings = {
file_input: document.getElementById("file"),
access_key: YOUR_AWS_ACCESS_KEY,
bucket: YOUR_AWS_BUCKET,
region: YOUR_AWS_REGION,

content_disposition: false, // if false - don't add Content-Disposition:attachment header for the uploaded file
key_prefix: "some-subdir/", // "subdir" for the uploaded files, used if "key" setting is empty or not set, each file will be uploaded under it's own name.ext
ajax_base: YOUR_BACKEND_URL
};
var upload = mule_upload(settings);
```


If you'd want example backends in other languages/with other frameworks, let me know.

Expand Down Expand Up @@ -119,4 +157,4 @@ Due to the new technology used by this library, it's only compatible with the fo
* Updated Chrome
* Updated Firefox
* Safari 6+
* not sure about IE
* IE 11+
142 changes: 110 additions & 32 deletions mule-uploader.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,12 @@

// the S3 object key; I recommend to generate this dynamically (e.g.
// a random string) to avoid unwanted overwrites.
settings.key = settings.key || "the_key";
// if key is empty - file.name used as a key
settings.key = settings.key || "";
settings._key = settings.key; //remember original settings key for multi-file upload case

// "sub-directory" for the uploaded files, used if settings.key is empty
settings.key_prefix = settings.key_prefix || "";

// the Amazon S3 bucket where you'll store the uploads
settings.bucket = settings.bucket;
Expand All @@ -183,8 +188,12 @@
// the Mime-Type of the content. You must match this with the backend value
// or you'll get an Invalid Signature error. If unsure about the
// mime type, use application/octet-stream
settings.content_type = settings.content_type || "application/octet-stream";
// if no content_type defined - file.type will be used
settings.content_type = settings.content_type || "";

//if content_disposition is explicitly set to false - don't add it to requests
//otherwise all uploaded files will be uploaded as an "attachment"
settings.content_disposition = (settings.content_disposition===false) ? false : true;

// acl can be set to:
// private
Expand All @@ -202,6 +211,7 @@
settings.on_select = settings.on_select || function() {};
settings.on_error = settings.on_error || function() {};
settings.on_complete = settings.on_complete || function() {};
settings.on_complete_all = settings.on_complete_all || function() {};
settings.on_init = settings.on_init || function() {};
settings.on_start = settings.on_start || function() {};
settings.on_chunk_uploaded = settings.on_chunk_uploaded || function() {};
Expand All @@ -220,7 +230,7 @@
u.set_state("waiting");

if (u.input) {
u.input.onchange = function(e, force) {
u.input.onchange = function(e) {
if(!u.settings.autostart) {
return true;
}
Expand All @@ -230,10 +240,21 @@
return false;
}

// the uploader doesn't support multiple uploads at this time,
// so we get the first file
var file = e.target.files[0];
u.upload_file(file, force);
//support for multiple files upload
u.files = e.target.files;
u.current_file_index = -1;
u.files_uploaded = 0;
u.files_total = u.files.length;
//remember total filesize for use in total progress
u.files_total_size = 0;
u.files_total_uploaded_size = 0;
for (var i = 0; i < u.files_total; i++) {
u.files_total_size+=u.files[i].size;
};

//proceed with first file
u.upload_file();

return true;
};
}
Expand All @@ -246,29 +267,51 @@

Uploader.prototype.start = function() {
if(this.input && this.input.files && this.input.files.length > 0) {
return this.upload_file(this.input.files[0], false);
return this.upload_file(false);
} else {
alert("No file selected");
alert("No file(s) selected");
}
};

Uploader.prototype.upload_file = function(file, force) {
//if force=true - don't proceed next file, instead force re-upload current file
Uploader.prototype.upload_file = function(force) {
var u = this;
// the `onchange` event may be triggered multiple times, so we
// must ensure that the callback is only executed the first time
// also make sure the file is not already set.
if(u.get_state() != "waiting") {
return false;
if (!force){
u.current_file_index = u.current_file_index+1;
}

if (file) {
u.file = file;
u.set_state("waiting");

//check if all files processed
if (u.current_file_index >= u.files.length){
u.settings.on_complete_all.call(u);
return;
}

//reset for new file
u.settings.on_progress.call(u, 0, 0);
u.upload_id = null;
u._progress = null;
u._total_progress = null;
u._loaded_chunks = null;
u._uploading_chunks = null;
u._chunks = null;

var file = u.files[ u.current_file_index ];
u.file = file;

if (!u.file) {
return false;
}

//set key for current file upload
if (u.settings._key=='') {
u.settings.key = u.settings.key_prefix + file.name;
}else{
u.settings.key = u.settings._key;
}
if (u.settings.content_type=='') u.settings.content_type = file.type;

// we use the lastModifiedDate, the file name and size to uniquely
// identify a file. There may be false positives and negatives,
// but the chance for a false positive is basically zero
Expand All @@ -280,8 +323,11 @@
alert(
["The maximum allowed file size is ",
(u.settings.max_size / GB),
"GB. Please select another file."].join('')
"GB. Please select another file.",
"\nFile: ",
file.name].join('')
);
u.upload_file();//upload next file
return false;
}

Expand All @@ -307,22 +353,39 @@
alert(
["This file format is not accepted. ",
"Please use a file with an extension like ",
u.settings.accepted_extensions].join('')
u.settings.accepted_extensions,
"\nFile: ",
file.name].join('')
);
u.upload_file();//upload next file
return false;
}
}

// initialize the file upload
u.settings.on_select.call(u, file);
file.is_force = force;
// trigger the on_select event callback
if ( u.settings.on_select.call(u, file, u.on_selected)=== false ){
// if on_select returns exactly false - just wait for on_selected callback;
}else{
//otherwise - just proceed with upload
u.on_selected(file);
};

};

//called after on_select event
Uploader.prototype.on_selected = function (file) {
var u = this;

// initialize the file upload
var args = utils.extend_object(u.settings.extra_params || {}, {
filename: file.name,
filesize: file.size,
last_modified: file.lastModifiedDate.valueOf()
last_modified: file.lastModifiedDate.valueOf(),
contenttype: u.settings.content_type
});

if(force) {
if(file.is_force) {
args.force = true;
}

Expand All @@ -344,7 +407,10 @@
u.settings.backup_key = u.settings.key;

if(!u.upload_id) {
AmazonXHR.init(json, u.settings.key, file, function(e) {
AmazonXHR.init(json, u.settings.key, file, u.settings, function(e) {
if(e.target.status / 100 != 2) {
return u.settings.on_error(e);
}
var xml = e.target.responseXML;

// get the given upload id
Expand All @@ -354,7 +420,7 @@
});
} else {
// resume a previus upload
if(!force) {
if(!file.is_force) {
// get the uploaded parts from S3
AmazonXHR.list(u.auth, u.file, u.settings.key, u.upload_id, u.settings.chunk_size, function(parts) {
for(var i=0; i<parts.length; i++) {
Expand All @@ -375,7 +441,7 @@
u._uploading_chunks = null;
u._chunks = null;
u.settings.key = u.settings.backup_key;
u.upload_file(file, true); // force reload
u.upload_file(true); // force reload
});
} else {
// force-start the upload
Expand Down Expand Up @@ -562,8 +628,13 @@
// trigger a final progress event callback, with 100%
u.settings.on_progress.call(u, u.file.size, u.file.size);

u.files_total_uploaded_size+=u.file.size;

// also trigger the complete event callback
u.settings.on_complete.call(u);
u.settings.on_complete.call(u, u.file);

//upload next file if any
u.upload_file();
}, function() {
// we have a genuine error
log("Error: ");
Expand Down Expand Up @@ -650,8 +721,14 @@
u.set_state("finished");
u.settings.on_progress.call(u, u.file.size, u.file.size); // it's 100% done

u.files_total_uploaded_size+=u.file.size;

// trigger the complete event callback
u.settings.on_complete.call(u);
u.settings.on_complete.call(u, u.file);

//upload next file if any
u.upload_file();

} else if(e.target.status == 400 &&
e.target.responseText.indexOf("EntityTooSmall") !== -1) {
// an "EntityTooSmall" error means that we missed a chunk
Expand All @@ -665,7 +742,7 @@
// 404 = NoSuchUpload = check if already finished
// if so, start a new upload
u.cancel(function() {
u.upload_file(u.file, true);
u.upload_file(true);
});
} else {
u.check_already_uploaded(function() {
Expand Down Expand Up @@ -712,7 +789,7 @@
var key = u.settings.key;
var upload_id = u.upload_id;
var url = u.settings.ajax_base + '/chunk_loaded/';

var args = utils.extend_object(u.settings.extra_params || {}, {
chunk: chunk,
key: key,
Expand Down Expand Up @@ -958,6 +1035,7 @@
Uploader.prototype.on_select = function(f) { this.settings.on_select = f; };
Uploader.prototype.on_error = function(f) { this.settings.on_error = f; };
Uploader.prototype.on_complete = function(f) { this.settings.on_complete = f; };
Uploader.prototype.on_complete_all = function(f) { this.settings.on_complete_all = f; };
Uploader.prototype.on_init = function(f) { this.settings.on_init = f; };
Uploader.prototype.on_start = function(f) { this.settings.on_start = f; };
Uploader.prototype.on_chunk_uploaded = function(f) { this.settings.on_chunk_uploaded = f; };
Expand Down Expand Up @@ -1085,7 +1163,7 @@
state_change_callback: readystate_callback
})).send(xhr_callback);
};
AmazonXHR.init = function(auth, key, file, callback) {
AmazonXHR.init = function(auth, key, file, u_settings, callback) {
return new AmazonXHR({
auth: auth,
key: key,
Expand All @@ -1095,7 +1173,7 @@
},
headers: {
"x-amz-acl": "public-read",
"Content-Disposition": "attachment; filename=" + file.name,
"Content-Disposition": u_settings.content_disposition ? "attachment; filename=" + u.file.name : '',
"Content-Type": auth.content_type || "application/octet-stream"
},
payload: "",
Expand Down
1 change: 0 additions & 1 deletion php_backend/.htaccess
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ SetEnv AWS_ACCESS_KEY [aws_access_key]
SetEnv AWS_SECRET [aws_secret]
SetEnv AWS_REGION us-east-1
SetEnv BUCKET mule-uploader-demo
SetEnv MIME_TYPE application/octet-stream

RewriteEngine On
RewriteRule ^$ example_backend.php
Expand Down
Loading