Implemented event overlay as test example#612
Implemented event overlay as test example#612Sameersribot wants to merge 1 commit intonottelabs:mainfrom
Conversation
WalkthroughTwo changes are introduced: First, a visual highlight feature is added to the session module with three class-level constants defining highlight duration (450ms), fade duration (220ms), and padding (6 pixels). An asynchronous helper method injects a DOM overlay, animates it, and cleans up after execution. The feature is configurable via config flags and includes error handling with verbose logging. Second, a new test runner script is added that bootstraps local packages, adjusts import paths, and demonstrates an agent executing a search task with a Gemini model. Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/examples/Interaction-overlay.py (1)
54-62: Consider uncommenting the response summary or removing the assignment.The
responsevariable is assigned but never used because the summary print statements are commented out. Since this is a test example, uncommenting the summary would provide useful feedback to users about the agent execution results.Apply this diff to uncomment the summary:
response = agent.run(task="go to google and search for am and select the second dropdown suggestion") + status_icon = "✅" if response.success else "❌" + print("\n=== Agent Run Summary ===") + print(f"{status_icon} Success: {response.success}") + print(f"⌛ Duration: {response.duration_in_s:.1f}s | Steps: {len(response.steps)}") + print(f"🧠 Answer: {response.answer}") print(f"\n(session.py in use: {inspect.getfile(notte.Session)})") - # status_icon = "✅" if response.success else "❌" - # print("\n=== Agent Run Summary ===") - # print(f"{status_icon} Success: {response.success}") - # print(f"⌛ Duration: {response.duration_in_s:.1f}s | Steps: {len(response.steps)}") - # print(f"🧠 Answer: {response.answer}") - # print(f"\n(session.py in use: {inspect.getfile(notte.Session)})")Alternatively, if the summary isn't needed, remove the
response =assignment and just callagent.run(...)directly.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/notte-browser/src/notte_browser/session.py(3 hunks)tests/examples/Interaction-overlay.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
packages/notte-browser/src/notte_browser/session.py (3)
packages/notte-core/src/notte_core/actions.py (3)
BaseAction(59-137)InteractionAction(898-984)selectors(929-934)packages/notte-browser/src/notte_browser/dom/locate.py (1)
locate_element(24-50)packages/notte-browser/src/notte_browser/window.py (2)
page(189-197)page(211-213)
tests/examples/Interaction-overlay.py (1)
packages/notte-sdk/src/notte_sdk/client.py (1)
Session(77-78)
🪛 Ruff (0.14.2)
packages/notte-browser/src/notte_browser/session.py
370-370: Do not catch blind exception: Exception
(BLE001)
456-456: Do not catch blind exception: Exception
(BLE001)
tests/examples/Interaction-overlay.py
54-54: Local variable response is assigned to but never used
Remove assignment to unused variable response
(F841)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: tests
🔇 Additional comments (4)
tests/examples/Interaction-overlay.py (1)
12-41: LGTM! Bootstrap logic correctly prioritizes local packages.The implementation properly walks the directory tree to locate the repository root, constructs paths for all notte packages, and forces re-import from local sources. This approach effectively resolves the environment path conflicts mentioned in the PR objectives.
packages/notte-browser/src/notte_browser/session.py (3)
77-79: LGTM! Well-chosen constants for visual feedback timing.The highlight duration (450ms), fade timing (220ms), and padding (6px) provide a good balance between visibility and unobtrusiveness for debugging feedback.
360-471: Well-implemented visual highlight feature with appropriate error handling.The implementation correctly:
- Guards against non-applicable actions (non-interaction, disabled feature, missing selectors)
- Locates the target element using existing infrastructure
- Injects safe, hardcoded JavaScript to create the DOM overlay
- Handles failures gracefully without disrupting execution
- Provides verbose logging for debugging
The broad exception catching at lines 370 and 456 is acceptable here since this is a non-critical visual feature that should never break the main execution flow. The defensive approach aligns with the PR objective of "safe error handling."
Minor observation: Line 466 uses
min(0.12, self.EXECUTION_HIGHLIGHT_DURATION_MS / 1000)which will always evaluate to 0.12 given the current constant of 450ms. If the intent is to cap the sleep duration, this is correct; otherwise, consider clarifying the logic.
528-528: LGTM! Correct placement for execution highlighting.The highlight is triggered after action resolution but before execution, allowing the visual feedback to appear at the right moment. The async/await usage is correct.
|
hey, thanks for the PR! will edit it slightly and merge |
ah great! |
2 participants
🔍 Summary
This PR introduces a visual overlay feature that highlights browser interactions with an orange box during agent execution.
The goal is to enhance debugging visibility and real-time feedback while observing browser actions like clicks, form submissions, and element interactions.
✨ Key Changes
File Modified: session.py
Added logic to trigger orange overlay rendering when specific browser events occur.
Adjusted screenshot capture flow to ensure the overlay appears in the latest frame before saving.
Added safe error handling for cases where the browser target might close during overlay capture.
Improved debug logging for overlay and screenshot states.
🧠 Implementation Details
🧪 2. Changed
sessions.py in notte-browser package
Uses the Playwright page overlay mechanism to draw an orange border/box on interacted elements.
Runs asynchronously within the event loop, ensuring minimal blocking.
Automatically disabled in headless=True mode for cleaner CI runs.
🧪 2. Interaction-overlay.py
New file created to test overlay functionality and bypass environment path conflicts.
Implements _bootstrap_local_packages() to force import of local Notte modules (instead of installed wheels).
Dynamically rebuilds sys.path to point to local packages:
📸 Example Output
When an interaction occurs (e.g., element click or text input), an orange box overlay will momentarily highlight the target element, improving visibility during automated browsing.
🚀 Impact
Enhances visual debugging and traceability of browser automation steps.
Improves developer experience when analyzing agent trajectories.
Provides clearer visual cues during demonstrations or video captures.
Run python notte/tests/examples/Interaction-overlay.py to test the example.
Summary by CodeRabbit
Release Notes
New Features
Tests