S3 multipart upload

[rgeyer]

Hey all,

I found myself needing a multithreaded multi part upload to S3 for some big files I was moving around. I found that only Fog had any existing support, and I didn't want yet another gem to install with my ServerTemplates so I decided to add the functionality to right_aws.

I wrote a few tests which do not fully cover my method, but I have used the method pretty extensively already and it's reasonably stable. It also appears as though the existing tests don't cover all of the existing code. Besides, I am a bit of a Ruby n00b and I couldn't get the entire suite of existing tests to pass, so I sorta gave up. ;-)

I posted in the RightScale forums about this as well; https://forums.rightscale.com/showthread.php?t=774

Hope this is useful for someone!

-Ryan J. Geyer-

[rgeyer]

I just had an opportunity to try this out from a windows box and discovered that it doesn't exactly play nice there. Looks like I have some additional work to do for windows.

It worked great on my OSX box, and linux though. shrug

-Ryan J. Geyer-

[konstantin-dzreev]

Hi Ryan,

Your fix may not work correctly because right_aws is not thread-safe. We are looking into this issue now...

Konstantin.

[ryansch]

I was just wishing this existed and here it is! Thanks again Ryan.

[rgeyer]

No problem ryansch. As I'm sure you read above, this seems to work fine in *nix environments, but not so much in Windows.

I actually intend to refactor this code a bit such that right_aws is only providing the raw AWS method calls for S3 multi part. Then consuming applications can build their own multi-threaded approach to simultaneous uploading of chunks. I'll likely leave the existing store_object_multipart method in the file, but either comment it out, or move it up the chain to the S3 class which abstracts the S3Interface.

[titas]

Hi guys, any news on this?