Support multipart uploads

[bitspittle]

Currently, Kobweb servers don't have a way to handle multipart form submissions.

In ktor, this is handled with a special multipart handler:

call.receiveMultipart(formFieldLimit = 1024 * 1024 * 100).forEachPart { part ->
   when (part) {
      is PartData.FormItem -> { ... }
      is PartData.FileItem -> { ... }
      else -> { ...}
    }
    part.dispose()
}

A few key features that need to be supported:

• Abstracting over the ktor API with a Kobweb class
• The ability to process one part at a time (using a suspend function).
• Recommendations for how to handle the PartData when branches but in a Kobweb way
• The ability to configure the multipart size. (If not explicitly set, defaults to 50MB)
• Disposing the parts
  • (I think we can do this on behalf of the user -- why doesn't the ktor API do it though?)

The only data that we read early enough to be able to configure ktor has to live in the conf.yaml, I think, so something like this:

server:
   multipart:
      - api: "upload"
        limit: 100mb

And finally the handler itself, which can probably be seamlessly added to the @Api endpoint concept in Kobweb by adding a multipart property to the ApiContext class:

@Api
fun upload(ctx: ApiContext) {
   ctx.multipart.forEachPart { /* how to handle formitem and fileitem etc types here?? */ }
}

But I still need to investigate a little more.

---

Questions:

• What should happen if you call ctx.multipart for a route that didn't upload one? Should the field be nullable and only non-null if a multipart value is set? Should it just no-op? Should it throw an exception?
• Should I try to surface all the ktor PartData classes but with Kobweb wrappers? Or would it be better to be simple, where each part is just the raw binary data for that part?

[bitspittle]

I'm imagining a class that looks something like this:

import io.ktor.http.content.MultiPartData

class Multipart private constructor(private val produce: () -> MultiPartData) {
   suspend fun forEachPart(handlePart: MultipartScope.() -> Unit) {
      val sourceMultipart = produce()
      sourceMultipart.forEachPart { part ->
         val scope = MultipartScope(part)
         scope.handlePart()
      }
   }
}

Then, in Kobweb's ktor code, something like:

val limit = findMultipartLimitFromApi(...)
val request = MutableRequest(
   ...
   multipart = Multipart { call.receiveMultipart(limit) }
)

[FlyingPig525]

For the question relating to ktor's PartData, you could probably have a generic class that contains the name of the part and all other headers in a list.

[bitspittle]

@FlyingPig525 Please consider dropping some concrete code, to make sure I'm not missing something that I don't see as well as you do.

My understanding with multipart data is that each part can be a different type from each other, so for robustness you may want to do a type check per part. In that case, I think a sealed class models the behavior better than a generic class does (as sealed classes encapsulate the idea of a predefined, limited list of types).

That said, I think the ktor API approach is not exactly what I would have done. Actually writing down the code I'm thinking of may prove me wrong, but I feel they combined two orthogonal concepts into one -- metadata and data input. My current plan is to still use a sealed class approach but to break these concepts up into two different classes.

[FlyingPig525]

I was thinking something upon the lines of:

data class PartData(val name: String, val contentType: String, val headers: List<String>, val data: ByteArray)

I'm not exactly sure if multipart content types are the same as HTTP content types, or if there is a more restrictive subset of them, so I'm not too sure about this approach, but this is probably what I would go with if I were the one implementing this. I'll probably go look at how ktor does it with their PartData types, as I haven't really used their multipart api.

[bitspittle]

If you're curious, here are the four part types provided by ktor: https://api.ktor.io/ktor-http/io.ktor.http.content/-part-data/index.html

Unfortunately, not really well documented.

But you can see there's some common properties across all of them (contentDisposition, contentType, headers, name) and some type-specific ones ("value" for form data, a "channel" provider, for file item, a chunkable buffer for binary-item, and a "channel" provider as well for binary-item-channel)

My plan is still to introduce a class MultipartScope (as prototyped in the example earlier), and that will store the common properties applicable to all types. But I'm also thinking about separating the data provider and metadata classes into separate concepts (unlike ktor, which bundles them together).

(Actually, the only part type which includes extra metadata at this point in the FileItem part, so I'm still thinking about my approach in case I can simplify it further)

Anyway, I'm thinking that the final user's code might look something like this:

ctx.req.body.multipart().forEachPart { dataProvider ->
   // Here, `this` is `MultipartScope`, and `headers`, `contentType`, etc. is in scope)
   val fileName = (this.metadata as? FileMetadata)?.originalFileName

   when (dataProvider) {
      is ChannelDataProvider -> ...
      is StringDataProvider -> ...
      is BinaryStreamDataProvider -> 
   }
}

[FlyingPig525]

I see, that looks much better than what I was thinking. I somehow completely missed the entire MultipartScope stuff.