Missing `Lambda.invokeWithResponseStream` in the generated `lambda-2015-03-31.lua`

[DiscreteTom]

Even in the latest release (1.5.0-1) the Lambda.invokeWithResponseStream is missing.
It is provided in the aws-js-sdk, see https://github.com/aws/aws-sdk-js/blob/d5f58a06ed7e551658293f7fb555543945ed783b/apis/lambda-2015-03-31.normal.json#L1133

[DiscreteTom]

Looks like the generated files are based on the aws-js-sdk v2.751.0.

However the Lambda.invokeWithResponseStream is added in v2.1353.0.

[hbagdi]

I don't think this SDK is auto-generated; it is written by hand.
What APIs are you interested in?

cc @Tieske @windmgc

[ttyS0e]

I think the SDK can be re-generated to add the missing REST shapes for request phase, however I don't think that response streaming will work immediately because it currently tries to "single shot" all requests using resty REST client which buffers the whole response to memory.

We would need to:

1. Add the ability to return the RES table to the caller
2. The caller (Kong plugin, technically) then needs to buffer each chunk back to the client. This is totally possible by ngx.say-ing each chunk back to the API caller/client.

[windmgc]

So two different issues for us now:

• SDK needs to be re-generated with latest API schema defs
• Currently lua-resty-aws does not support streaming response.

@DiscreteTom Could you elaborate your use case with invokeWithResponseStream?

[DiscreteTom]

Thanks for the update. I'm implementing streaming response for kong's lambda plugin.

I've modified the lib locally and tested it with kong, and the streaming response is indeed buffered (if I test it correctly).

So I'm planning to rewrite some parts of the lambda plugin to make it support response streaming. This lib will stil be used for sigv4 signing or other stuff.

[ttyS0e]

SDK is updated and I see the new shape, I'm just waiting for push permission so I can open the PR. I will test and prove the theory in a minute - if correct (it buffers the whole response), I think I can easily make it return the res object to the caller, optionally.

Problem obviously is then that the AWS Lambda plugin still needs updating to stream the chunked response to the Kong client.

-===-

New shape:

    "InvokeWithResponseStream": {
      "name": "InvokeWithResponseStream",
      "http": {
        "method": "POST",
        "requestUri": "/2021-11-15/functions/{FunctionName}/response-streaming-invocations"
      },
      "input": {
        "shape": "InvokeWithResponseStreamRequest"
      },
      "output": {
        "shape": "InvokeWithResponseStreamResponse"
      },

So yeah @DiscreteTom the best way is if I modify the request executor to return you the whole res table, instead of just the buffered string.

That way, you can use the body_reader to stream directly back to the client in the Lambda Plugin. You basically just connect the "aws response" and "client request handle" streams together and let them run.

I've done this before on other things, if you need help.

[ttyS0e]

@DiscreteTom Sorry I haven't refreshed the SDK yet, problems with GitHub permissions.

I should be able to do today.

In the meantime, here is the AWS Stream parser, in Lua format, that you can use to parse each Lambda chunk:

https://github.com/Kong/kong/blob/d875bacf80d3ff2657b33d7cb8863142ca1083a2/kong/tools/aws_stream.lua

You would use it like normal Lua package in the plugins:

local dump = function(...)
  local t = { n = select("#", ...), ...}
  if t.n == 1 and type(t[1]) == "table" then t = t[1] end
  print(require("pl.pretty").write(t))
end

local chunk_in_hex = "000000760000005296d5fade0b3a6576656e742d7479706507000c6d65737361676553746172740d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22726f6c65223a22617373697374616e74227d6fa8c599000001110000005706f176a90b3a6576656e742d74797065070011636f6e74656e74426c6f636b44656c74610d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22636f6e74656e74426c6f636b496e646578223a302c2264656c7461223a7b2274657874223a2250692028cf80292069732061206d617468656d61746963616c20636f6e7374616e74207468617420726570726573656e7473207468652063697263756d666572656e63652d746f2d6469616d6574657220726174696f206f66206120636972636c652e204974277320617070726f78696d6174656c7920332e31343135392e227d7d7d4e31940000007d00000056e6680fd60b3a6576656e742d74797065070010636f6e74656e74426c6f636b53746f700d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b22636f6e74656e74426c6f636b496e646578223a307db32e4d340000007a00000051ca2c46650b3a6576656e742d7479706507000b6d65737361676553746f700d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b2273746f70526561736f6e223a22656e645f7475726e227d5fbf09fc000000b90000004ee991d99b0b3a6576656e742d747970650700086d657461646174610d3a636f6e74656e742d747970650700106170706c69636174696f6e2f6a736f6e0d3a6d6573736167652d747970650700056576656e747b226d657472696373223a7b226c6174656e63794d73223a313530367d2c227573616765223a7b22696e707574546f6b656e73223a382c226f7574707574546f6b656e73223a33392c22746f74616c546f6b656e73223a34377d7d5ad2dd7b"
local _STREAM = require("kong.tools.aws_stream")
local parser, err = _STREAM:new(chunk_in_hex, true)

if err then
  print("ERROR: ", err)
  return
end

while true do
  local msg = parser:next_message()

  if not msg then
    break
  end

  print(dump(msg))
end

I will be adding this to this SDK so that it can be called directly, and each chunk can be parsed into an array of messages for the client response.

[DiscreteTom]

@ttyS0e I made some local changes to bypass the dependency issue so I can already test it locally for now. Thanks for the stream parser, it is very helpful!

[DiscreteTom]

I created a draft PR on kong: Kong/kong#13171

The response streaming is working now. I use two files (aws_stream.lua and execute.lua) as a workaround to support the feature for now. You can reference these for the next release of resty-aws lib.

When the resty-aws library is ready I will remove these two files, modify the handler.lua to adopt this lib and finish the PR.

[ttyS0e]

Nice, this is excellent! Thanks.

I will get this into the SDK and produce a new tag for you, I just need gateway team approval and such.