feat(entities): log console.error when realtime broadcast is oversize

[theunreal]

Why

Realtime entity broadcasts have a transport size limit. Records with very large fields (e.g. base64-encoded media, long markdown) can exceed the limit and get dropped — subscribers see stale state until refresh.

The Base44 backend now slims oversize broadcasts to fit under the limit and signals it with _oversize: true on the event data. Without this SDK change, customer apps that read the slimmed fields would get an empty string with no signal that anything happened — devs writing the app have no idea their >10 KB field was just blanked at the wire.

What

In entities.X.subscribe(callback):

1. Parse the realtime event as before.
2. If event.type !== "delete" and event.data._oversize, log a single console.error naming the entity, id, and the recommended fetch call.
3. The callback fires with the slimmed payload as-is — caller decides whether to ignore the event, refetch with entities.X.get(id), or render the partial data.

Why not auto-refetch? An earlier draft of this PR did transparent HTTP refetch behind the scenes (with debouncing across concurrent subscribers). PM pushback: a popular entity in a fleet/logistics app could have tens of subscribed components per browser, hundreds of users online, and updates every few seconds. Auto-refetching on every oversize broadcast — even with per-browser debounce — multiplies backend HTTP load by the active-user count of the affected app. Surfacing the issue to developers and letting them decide is the safer default.

Wire contract is preserved: the server always emits data as an object containing at minimum id, so older SDK clients that don't know about _oversize continue to receive a structurally valid event. They render the empty/stub fields without the warning — same UX as a brand-new install today, just no debug signal.

Tests

tests/unit/entities-subscribe.test.ts — 3 new tests under oversize broadcast handling, all 9 pass:

• logs a console.error and passes the stub through when data._oversize is true
• does NOT log on delete events even if _oversize is set
• does NOT log when _oversize is absent

Full suite: 156 / 156 pass. npm run lint and npm run build clean.

Backward compatibility

• Existing subscribers: no behavior change for events without _oversize. Pure passthrough.
• Older SDK versions: receive the slimmed payload as-is — render empty big fields until refresh, no console signal. Same UX as today's drop-and-stale, slightly better since at least the small fields are fresh.
• Server without slimming: _oversize flag never appears, so this code path is dormant.

Versioning

Recommend a minor bump (0.8.x → 0.9.0) since this is additive observability behavior.

---

Reopened from #165 to run from base repo (fork PRs lack secrets/OIDC).

[claude]

Claude finished @theunreal's task —— View job

---

Code Review for PR #167: Auto-refetch when realtime broadcast is truncated

I've reviewed the implementation and tests for the auto-refetch feature. Overall, this is a well-designed solution to handle oversized realtime broadcasts. Here's my feedback:

✅ Code Quality & Best Practices

Strengths:

• Clean, modular implementation that integrates seamlessly into the existing architecture
• Good separation of concerns with the refetchTruncated function
• Clear comments explaining the rationale and behavior
• Follows TypeScript best practices with proper typing

Minor suggestions:

• Consider extracting the magic number 5_000 (5 seconds) in entities.ts:107 to a named constant for better maintainability

✅ Bug Analysis

The implementation appears solid with no critical bugs identified. Key defensive programming practices observed:

• Proper error handling with fallback to stub payload on HTTP failure
• Guards against missing data fields with optional chaining
• Clean promise handling to avoid unhandled rejections

✅ Performance Considerations

Excellent optimizations:

• Smart debouncing mechanism prevents duplicate HTTP calls for concurrent subscribers
• Cache key design ${entityName}:${id}:${timestamp} ensures distinct updates aren't collapsed
• 5-second grace period allows late subscribers to benefit from cached results
• Promise-based approach ensures efficient resource utilization

Potential enhancement:

• Consider making the cache cleanup timeout configurable for different use cases

✅ Security Assessment

No security concerns identified:

• No credentials or sensitive data exposed
• HTTP calls use existing axios instance with proper authentication
• No user input directly interpolated into URLs
• Error messages don't leak sensitive information

✅ Test Coverage

Comprehensive test suite covering:

• Basic refetch behavior when _oversize is true
• Correct handling of delete events (no refetch)
• Normal events without _oversize flag
• Error recovery with fallback to stub payload
• Debouncing mechanism for concurrent requests

Test quality highlights:

• Excellent use of mocks and spies
• Async handling with proper waiting
• Edge cases well covered
• Clear test descriptions

🔍 Additional Observations

1. Backward compatibility: The implementation is properly designed to be non-breaking:

   • Older SDKs ignore the _oversize flag
   • New behavior only activates when flag is present

2. Wire protocol preservation: The server contract is maintained with minimal required fields

3. Error handling: The warning logs provide good visibility while maintaining service continuity

📋 Summary

This PR implements a robust solution for handling oversized realtime broadcasts. The code is production-ready with excellent test coverage and thoughtful design decisions. The debouncing mechanism is particularly well-implemented, avoiding unnecessary network calls while ensuring data freshness.

The only minor suggestion is to extract the cache timeout constant for better configurability, but this is not blocking.

Great work on this implementation! 👍

[github-actions]

🚀 Package Preview Available!

---

Install this PR's preview build with npm:

npm i @base44-preview/sdk@0.8.26-pr.167.fdb48d8

Prefer not to change any import paths? Install using npm alias so your code still imports @base44/sdk:

npm i "@base44/sdk@npm:@base44-preview/sdk@0.8.26-pr.167.fdb48d8"

Or add it to your package.json dependencies:

{
  "dependencies": {
    "@base44/sdk": "npm:@base44-preview/sdk@0.8.26-pr.167.fdb48d8"
  }
}

• 📦 Preview Package: @base44-preview/sdk@0.8.26-pr.167.fdb48d8
• 🔗 View this commit on GitHub

---

_{Preview published to npm registry — try new features instantly!}

[cowchimp]

sweet 🍬

[claude]

Claude encountered an error —— View job

---

Code Review for PR #167: Auto-refetch when realtime broadcast is truncated

I've reviewed the implementation and tests for handling oversized realtime broadcasts. Here's my comprehensive feedback:

✅ Code Quality & Best Practices

Strengths:

• Clean, well-documented code with clear explanations of the behavior
• Follows TypeScript best practices with proper typing
• Good error handling that maintains backward compatibility
• Clear console error messages that guide developers on how to handle oversized data

Observations on the evolution:

• The PR originally implemented auto-refetch functionality but was later refactored to use console.error instead
• This change from automatic HTTP refetch to developer notification is a good architectural decision as it:
  • Avoids uncapped HTTP fan-out that could increase backend load
  • Gives developers control over when to fetch full data
  • Maintains transparency about what's happening

✅ Potential Bugs or Issues

No critical bugs identified. The implementation is solid with proper safeguards:

• Guards against missing data with optional chaining ((event.data as any)?._oversize)
• Correctly skips logging for delete events where records no longer exist
• Maintains event flow even when oversized data is detected
• Error handling in callback execution prevents crashes

✅ Performance Considerations

Current approach (console.error) is performant:

• No additional HTTP calls are made automatically
• Minimal overhead - just a console.error log when _oversize is detected
• No memory leaks or accumulating state
• Synchronous operation maintains realtime performance

Previous approach (auto-refetch) had good optimizations:

• The debouncing mechanism with Map-based cache was well-designed
• 5-second cache cleanup prevented memory accumulation
• Promise reuse for concurrent subscribers was clever

The shift to console.error eliminates any performance concerns about HTTP fan-out.

✅ Security Concerns

No security issues identified:

• No credential exposure or sensitive data leaks
• Error messages don't reveal internal implementation details
• The _oversize flag is properly validated before use
• No user input is directly interpolated into strings

✅ Test Coverage

Excellent test coverage:

• Tests for the happy path (normal messages without _oversize)
• Tests for oversized data detection and console.error logging
• Tests for delete events with _oversize flag (should not log)
• Tests for error handling in callbacks
• All edge cases are covered

Test quality:

• Proper use of spies for console methods
• Clean mock setup for socket and axios
• Tests are focused and well-named
• Good isolation between test cases

🔍 Additional Observations

1. Wire Protocol: The implementation maintains backward compatibility perfectly - older SDKs will receive the stub payload and continue working

2. Developer Experience: The console.error message is helpful and actionable, telling developers exactly how to fetch the full record

3. Commit History: The evolution from auto-refetch to console.error shows thoughtful consideration of production implications

4. Code Comments: Excellent documentation in the code explaining the rationale and behavior (lines 199-213 in entities.ts)

📋 Summary

This is a well-implemented solution that balances functionality with performance concerns. The final approach of using console.error to notify developers about oversized broadcasts is pragmatic and gives applications control over how to handle these scenarios. The code is production-ready with comprehensive test coverage.

The only minor suggestion would be to consider adding a configuration option in the future to allow developers to customize the behavior (e.g., provide a callback for handling oversized events), but this is not necessary for the current implementation.

Great work on this PR! 👍

[claude]

Claude finished @theunreal's task —— View job

---

Code Review for PR #167: log console.error when realtime broadcast is oversize

I've thoroughly reviewed the implementation and tests for handling oversized realtime broadcasts. Here's my comprehensive feedback:

✅ Code Quality & Best Practices

Strengths:

• Clean, well-documented implementation with clear comments explaining the rationale (lines 199-213 in entities.ts)
• Follows TypeScript best practices with proper typing
• Good separation of concerns - the check is isolated within the subscription handler
• Maintains backward compatibility by not breaking existing behavior

Observations:

• The evolution from auto-refetch to console.error shows thoughtful architectural decision-making
• The PR description clearly documents the rationale for avoiding automatic HTTP refetch

✅ Potential Bugs or Issues

No critical bugs identified. The implementation is solid with proper safeguards:

• Optional chaining on (event.data as any)?._oversize prevents crashes if data is missing
• Correctly excludes delete events where the record no longer exists
• Error handling in the callback (try-catch block) ensures resilience
• The event still propagates to the callback even when oversized, maintaining data flow

✅ Performance Considerations

Current implementation is performant:

• No additional HTTP calls - just a console.error log
• Minimal overhead with a simple boolean check
• No memory accumulation or state management
• Synchronous operation maintains realtime performance
• The decision to avoid auto-refetch prevents potential HTTP fan-out that could overwhelm the backend

✅ Security Concerns

No security issues identified:

• No credential exposure or sensitive data leaks
• Error message provides helpful developer guidance without revealing internal details
• The _oversize flag is safely accessed with optional chaining
• No user input is interpolated into strings

✅ Test Coverage

Excellent test coverage in the new test suite:

• ✓ Tests console.error logging when _oversize is true
• ✓ Tests that delete events don't trigger logging even with _oversize
• ✓ Tests normal events without _oversize work as before
• ✓ Verifies the stub payload is still passed to the callback
• ✓ All test cases properly mock console methods and clean up

The tests are well-structured, focused, and cover all edge cases.

🔍 Additional Observations

1. Developer Experience: The console.error message is actionable, telling developers exactly how to fetch the full record
2. Wire Protocol: Maintains compatibility - older SDKs receive the stub and continue functioning
3. Documentation: The inline comments clearly explain the feature and its purpose

📋 Summary

This is a well-implemented, pragmatic solution that balances functionality with performance concerns. The shift from auto-refetch to console.error notification gives applications control over handling oversized broadcasts while preventing backend overload. The code is production-ready with comprehensive test coverage.

Minor suggestion for future enhancement: Consider adding a configuration option to allow developers to provide a custom handler for oversized events, but this is not necessary for the current implementation.

Excellent work on this PR! 👍

Tasks

• Review implementation in entities.ts
• Review test coverage in entities-subscribe.test.ts
• Check code quality and best practices
• Analyze potential bugs or issues
• Assess performance considerations
• Review security concerns
• Provide constructive feedback