Skip to content

fix(CubeProxy): disable buffering for envd streaming endpoints #647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zyl1121 wants to merge 1 commit into
base: master
Choose a base branch
from
+115 −2
Open

fix(CubeProxy): disable buffering for envd streaming endpoints #647

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CubeProxy/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ RUN sed -i 's#https\?://dl-cdn.alpinelinux.org#http://mirrors.tencent.com#g' /et

WORKDIR /usr/local/openresty/nginx/
COPY lua/ /usr/local/openresty/nginx/lua/
COPY conf/includes/ /usr/local/openresty/nginx/conf/includes/
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
COPY rotate_nginx_log.sh /usr/local/openresty/nginx/sbin/rotate_nginx_log.sh
COPY root /etc/crontabs/root
Expand Down
24 changes: 24 additions & 0 deletions CubeProxy/conf/includes/envd_streaming_host_route.inc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
proxy_buffering off;
# Streaming envd errors must pass through verbatim; intercepting them here
# would replace protocol-native error frames with nginx error pages and break
# client-side parsing.
proxy_intercept_errors off;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

proxy_send_timeout 7206s;
proxy_read_timeout 7206s;
proxy_connect_timeout 3s;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

rewrite_by_lua_file lua/rewrite_phase.lua;

proxy_pass http://backend;

header_filter_by_lua_file lua/header_filter_phase.lua;

log_by_lua_file lua/log_phase.lua;
28 changes: 28 additions & 0 deletions CubeProxy/conf/includes/envd_streaming_path_route.inc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
proxy_buffering off;
# Streaming envd errors must pass through verbatim; intercepting them here
# would replace protocol-native error frames with nginx error pages and break
# client-side parsing.
proxy_intercept_errors off;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

proxy_send_timeout 7206s;
proxy_read_timeout 7206s;
proxy_connect_timeout 3s;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Prefix /sandbox/$ins_id/$container_port;

proxy_redirect ~^/(.*)$ /sandbox/$ins_id/$container_port/$1;
proxy_cookie_path / /sandbox/$ins_id/$container_port/;

rewrite_by_lua_file lua/path_rewrite_phase.lua;

proxy_pass http://backend;

header_filter_by_lua_file lua/header_filter_phase.lua;

log_by_lua_file lua/log_phase.lua;
64 changes: 62 additions & 2 deletions CubeProxy/nginx.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,23 @@ http {
proxy_pass http://$cube_sidecar_addr/internal/resume?$args;
}

# Path-based server-streaming envd endpoints. Keep buffering enabled
# elsewhere under /sandbox/ and disable it only for these verified
# response streams.
location ~ ^/sandbox/[^/]+/\d+/(?:process\.Process/(?:Start|Connect)|filesystem\.Filesystem/WatchDir)$ {

@chenhengqi chenhengqi Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need SendInput here?

@zyl1121 zyl1121 Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need SendInput here?

I tested this on a real cluster as a follow-up, and I don't think we need a separate unbuffered SendInput route. SendInput itself is still a unary/input-side call, the real-time output is delivered through the already-unbuffered Start / Connect streaming paths.

In the PTY repro, with the current patch, pty.send_stdin() returned in about 10ms and the PTY marker was streamed back in about 15ms. After removing the unbuffering change for Start / Connect, the serial PTY flow was blocked earlier: pty.create() was buffered until near the connection timeout, and the later connect() / send_stdin() calls failed with NotFoundException. This points to the streaming Start / Connect path as the failure point, not SendInput.

I also attached the repro script I used. After exporting the corresponding SDK environment variables for the target cluster, it can be run directly, for example:

export E2B_API_URL="<your-cube-api-url>"
export E2B_API_KEY="e2b_000000"
export CUBE_TEMPLATE_ID="<your-template-id>"
export SSL_CERT_FILE="/root/.local/share/mkcert/rootCA.pem"
python demo-pty.py

The script I used: demo-pty.py

Comment thread
zyl1121 marked this conversation as resolved.
Comment thread
zyl1121 marked this conversation as resolved.
Comment thread
zyl1121 marked this conversation as resolved.
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
Comment thread
zyl1121 marked this conversation as resolved.
Comment thread
zyl1121 marked this conversation as resolved.
set $backend_ip "";
set $backend_port "";
set $access_time "";
set $host_proxy_port 8081;

include /usr/local/openresty/nginx/conf/includes/envd_streaming_path_route.inc;
}

# Path-based routing: /sandbox/<sandbox-id>/<container-port>/<rest>
# Lets clients reach a sandbox via a plain IP+port without wildcard DNS or TLS.
location ^~ /sandbox/ {
location /sandbox/ {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
set $backend_ip "";
Expand Down Expand Up @@ -169,6 +183,25 @@ http {
log_by_lua_file lua/log_phase.lua;
}

# Verified server-streaming envd endpoints used by clients today.
# Keep buffering enabled elsewhere and disable it only for these
# response streams; otherwise nginx batches early frames until timeout
# or stream completion and breaks immediate-return semantics.
# Current unbuffered streaming endpoints:
# - process.Process/Start
# - process.Process/Connect
# - filesystem.Filesystem/WatchDir
location ~ ^/(?:process\.Process/(?:Start|Connect)|filesystem\.Filesystem/WatchDir)$ {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
set $backend_ip "";
set $backend_port "";
set $access_time "";
set $host_proxy_port 8081;

include /usr/local/openresty/nginx/conf/includes/envd_streaming_host_route.inc;
}

location / {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
Expand Down Expand Up @@ -226,9 +259,23 @@ http {
proxy_pass http://$cube_sidecar_addr/internal/resume?$args;
}

# Path-based server-streaming envd endpoints. Keep buffering enabled
# elsewhere under /sandbox/ and disable it only for these verified
# response streams.
location ~ ^/sandbox/[^/]+/\d+/(?:process\.Process/(?:Start|Connect)|filesystem\.Filesystem/WatchDir)$ {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
set $backend_ip "";
set $backend_port "";
set $access_time "";
set $host_proxy_port 8080;

include /usr/local/openresty/nginx/conf/includes/envd_streaming_path_route.inc;
}

# Path-based routing: /sandbox/<sandbox-id>/<container-port>/<rest>
# Lets clients reach a sandbox via a plain IP+port without wildcard DNS or TLS.
location ^~ /sandbox/ {
location /sandbox/ {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
set $backend_ip "";
Expand Down Expand Up @@ -262,6 +309,19 @@ http {
log_by_lua_file lua/log_phase.lua;
}

# Verified server-streaming envd endpoints. Must not be buffered; see
# the 8081 server block for the scope rationale and endpoint list.
location ~ ^/(?:process\.Process/(?:Start|Connect)|filesystem\.Filesystem/WatchDir)$ {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
set $backend_ip "";
set $backend_port "";
set $access_time "";
set $host_proxy_port 8080;

include /usr/local/openresty/nginx/conf/includes/envd_streaming_host_route.inc;
}

location / {
include /usr/local/openresty/nginx/conf/global/global.conf;
set $garyscale_test "none";
Expand Down
Loading