Add youtube#213
Conversation
…ter info for Twitch events
…iption statuses with Twitch API
… webhook notifications
…r EventSub webhooks
… periodic execution
…tion-actions task
… database item management
…nature validation
…n with message lengths
…s and adjust page filtering logic
…when webhooks are enabled
…tatements and log messages
…n_config function
…e subscription verification
…lify method calls
…ce code readability
…ssage_matches_action function
…me_max calculation
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive YouTube integration to the automation platform, enabling users to trigger automations based on YouTube activity and perform actions on videos. The implementation includes webhook-based real-time notifications via YouTube PubSubHubbub, polling fallback, and three reaction types.
Key changes:
- Added YouTube service with 3 actions (new video, channel stats, search videos) and 3 reactions (post comment, add to playlist, rate video)
- Implemented YouTube PubSubHubbub webhook support for real-time notifications
- Extended Google OAuth scopes to include YouTube API access
- Created reusable Google webhook infrastructure supporting Gmail, Calendar, and YouTube
Reviewed Changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| mobile/lib/utils/service_token_mapper.dart | Maps YouTube service to Google OAuth token |
| frontend/src/pages/ServiceDetail.tsx | Adds YouTube to Google OAuth services and displays webhook banner |
| frontend/src/pages/Debug.tsx | Simplifies external event trigger message |
| frontend/src/components/GoogleWebhookBanner.tsx | New informational banner for Google webhook services |
| backend/users/oauth_views.py | Auto-triggers Google webhook setup after OAuth connection |
| backend/scripts/test_google_webhooks.py | New test script for webhook infrastructure validation |
| backend/automations/webhooks.py | Adds YouTube PubSubHubbub verification and event ID extraction |
| backend/automations/validators.py | Defines YouTube action/reaction schemas and compatibility rules |
| backend/automations/urls.py | Registers dedicated webhook endpoints for Google services |
| backend/automations/tasks.py | Implements YouTube polling, webhook management, and reaction execution |
| backend/automations/serializers.py | Adds YouTube logo and OAuth mapping |
| backend/automations/models.py | New GoogleWebhookWatch model for tracking push notification subscriptions |
| backend/automations/migrations/0012_google_webhook_watch.py | Database migration for GoogleWebhookWatch model |
| backend/automations/management/commands/init_services.py | Registers YouTube service with actions and reactions |
| backend/automations/helpers/youtube_helper.py | YouTube Data API v3 helper functions |
| backend/automations/helpers/google_webhook_helper.py | Google webhook creation and management utilities |
| backend/automations/helpers/gmail_helper.py | Adds history API support for Gmail webhooks |
| backend/automations/helpers/calendar_helper.py | Adds calendar_id parameter for multi-calendar support |
| backend/automations/google_webhook_views.py | New dedicated webhook receivers for Gmail, Calendar, YouTube |
| backend/area_project/settings/base.py | Adds YouTube OAuth scopes and webhook configuration |
| backend/area_project/celery.py | Configures periodic tasks for webhook renewal |
| .env.example | Minor formatting fix |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| if video_id: | ||
| # Include published timestamp for uniqueness | ||
| return f"youtube_video_{video_id}_{published}" |
There was a problem hiding this comment.
The event ID generation includes the full published timestamp which can contain special characters (colons, plus signs) that may cause issues in IDs. The timestamp should be sanitized or hashed to ensure the event ID is URL-safe and database-friendly. Consider using published.replace(':', '').replace('+', '') or hashing the timestamp.
| if video_id: | |
| # Include published timestamp for uniqueness | |
| return f"youtube_video_{video_id}_{published}" | |
| # Sanitize published timestamp to remove problematic characters | |
| sanitized_published = published.replace(':', '').replace('+', '') | |
| if video_id: | |
| # Include sanitized published timestamp for uniqueness | |
| return f"youtube_video_{video_id}_{sanitized_published}" |
| return HttpResponse("OK", status=200) | ||
|
|
||
| # Process history changes | ||
| for history_item in history: |
There was a problem hiding this comment.
The code iterates over history directly but get_history() returns a dict with a 'history' key containing the list of changes. This should be for history_item in history.get('history', []): to properly access the history items.
| for history_item in history: | |
| for history_item in history.get('history', []): |
| """ | ||
| try: | ||
| # Parse XML (YouTube Atom feeds are trusted sources from youtube.com) | ||
| root = ET.fromstring(xml_string) # noqa: S314 # nosec B314 |
There was a problem hiding this comment.
While the nosec comment indicates awareness of the security concern, using ET.fromstring() on untrusted XML can still be vulnerable to XML entity expansion attacks. Consider using defusedxml.ElementTree.fromstring() instead, which is specifically designed to prevent XML attacks, even from sources that might seem trusted.
| watch_request = { | ||
| "labelIds": ["INBOX"], | ||
| "topicName": webhook_url, | ||
| } |
There was a problem hiding this comment.
The Gmail watch API expects 'topicName' to be a Cloud Pub/Sub topic (e.g., 'projects/myproject/topics/mytopic'), not an HTTP webhook URL. For direct HTTPS push endpoints, Gmail requires domain verification and the watch should be created without topicName or use the appropriate Pub/Sub integration. This configuration will likely fail when called.
| "description": "Comment text (supports variables: {video_title}, {channel_name})", | ||
| }, | ||
| }, | ||
| "required": ["comment_text"], |
There was a problem hiding this comment.
The youtube_post_comment reaction requires comment_text but not video_id, even though video_id is essential for posting a comment. While the code attempts to get video_id from trigger_data as a fallback, this should be explicitly required to prevent runtime errors when used with non-YouTube triggers. Consider adding 'video_id' to the required list or making the default value validation more robust.
| "required": ["comment_text"], | |
| "required": ["video_id", "comment_text"], |
| video_id = reaction_config.get("video_id") or trigger_data.get("video_id") | ||
| comment_text = reaction_config.get("comment_text") | ||
|
|
||
| if not video_id or not comment_text: | ||
| raise ValueError( | ||
| "Video ID and comment text required for youtube_post_comment" | ||
| ) |
There was a problem hiding this comment.
This validation pattern is repeated identically across all three YouTube reactions (youtube_post_comment, youtube_add_to_playlist, youtube_rate_video). Consider extracting this into a helper function like get_video_id_from_config_or_trigger(reaction_config, trigger_data) to reduce code duplication and ensure consistent behavior.
| ] | ||
| constraints = [ | ||
| models.UniqueConstraint( | ||
| fields=["user", "service", "resource_uri"], |
There was a problem hiding this comment.
The unique constraint includes resource_uri which has blank=True, default=''. This means multiple Gmail watches for the same user will all have an empty string for resource_uri, causing unique constraint violations. Consider either making resource_uri non-nullable with a meaningful default, or excluding it from the constraint and adding logic to prevent duplicate watches.
| fields=["user", "service", "resource_uri"], | |
| fields=["user", "service"], |
| @@ -0,0 +1,299 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
The script uses #!/usr/bin/env python3 shebang but is documented to run with python scripts/test_google_webhooks.py. The shebang suggests it should be executable and run directly as ./scripts/test_google_webhooks.py. Either make the file executable (chmod +x) and update documentation, or remove the shebang line if it's meant to be run via python interpreter.
| #!/usr/bin/env python3 |
| user = areas[0].owner | ||
|
|
||
| # Save watch to database | ||
| watch, created = GoogleWebhookWatch.objects.update_or_create( |
There was a problem hiding this comment.
Variable watch is not used.
| watch, created = GoogleWebhookWatch.objects.update_or_create( | |
| GoogleWebhookWatch.objects.update_or_create( |
| user = areas[0].owner | ||
|
|
||
| # Save watch to database | ||
| watch, created = GoogleWebhookWatch.objects.update_or_create( |
There was a problem hiding this comment.
Variable created is not used.
| watch, created = GoogleWebhookWatch.objects.update_or_create( | |
| GoogleWebhookWatch.objects.update_or_create( |
This pull request introduces comprehensive support for Google push notifications (webhooks) for Gmail, Google Calendar, and YouTube, moving away from polling for these services. It adds new helper utilities for managing Google "watches," updates the Celery beat schedule to reflect webhook usage, and expands OAuth scopes and configuration options to support these features. Additionally, it includes improvements to Gmail and Calendar helpers for more flexible API usage.
Key changes:
Google Webhook (Push Notification) Support
google_webhook_helper.pywith functions to create, stop, and renew push notification "watches" for Gmail, Calendar, and YouTube, including PubSubHubbub support for YouTube. This enables the backend to receive real-time updates from Google services instead of relying on polling.settings/base.pyto enable/disable webhooks for Gmail, Calendar, and YouTube, specify webhook URLs, and set watch renewal intervals.OAuth and API Enhancements
settings/base.pyto include YouTube read-only and force-ssl access, required for YouTube webhooks and API interactions.BACKEND_URLconfiguration for webhook callback endpoints.Helper Improvements
list_upcoming_eventsfunction to accept acalendar_idparameter, supporting non-primary calendars. [1] [2] [3]get_historyfunction to the Gmail helper for retrieving changes since a specific history ID, supporting efficient processing of Gmail push notifications.Other Changes
These changes collectively modernize the backend's integration with Google services, improving efficiency and responsiveness by leveraging push notifications instead of periodic polling.