Skip to content

Enhance content moderation and admin panel features#109

Open
joan-code6 wants to merge 15 commits into
hackclub:mainfrom
joan-code6:main
Open

Enhance content moderation and admin panel features#109
joan-code6 wants to merge 15 commits into
hackclub:mainfrom
joan-code6:main

Conversation

@joan-code6

Copy link
Copy Markdown

Hey mahad,
i have created a violation system that will help you dealing with nsfw and co.
The OpenAI api is free (but needs a active billing acc).
I also added a small Admin panel were you can manually ban unban and review violations.

For users with apps were everyone can use the AI they can activate the "strict" mode in the violations page.
This means every review will be reviewed before being send to the AI. If in that case a violation should get detected it wont count towards the generall violation limit and just return with a clear error (403 forbidden i think but dont take my word).

It follows this concept i made in slack:

Every second (or manually configured) request gets selected for review, these requests will go through but will trigger a async job to classify the text as NSFW etc and if true it tries to block the output immediately and flags the user.
After a user has been flagged all of his requests will be selected for review.
After 2 Violations within a week requests will wait till the request was classified
After 5 violations per month the user gets blocked and can only be unblocked by manual review of you

For users who know that theire application will be abused for such things they get a option in hai dashboard to trigger review each time and delay the requests to be reviewed first. If violating content should be found during that process it wont count towards the system and only notifiy the app its coming from so they can ban / punish the user how ever they want

also the user should see all violations in his dashboard (edited)

Here are some screenshots from my local dev env

image image image image image image

Joan Code and others added 15 commits May 8, 2026 11:08
…rade

- Fix sample rate: normal users now only reviewed when sampled, not always
- Fix status downgrade: users now correctly return to 'normal' when violations age out
- Add migration for missing database columns (review_status, violation counts)
- Add dedicated /banned page with violation history and appeal info

- Redirect banned users to /banned instead of throwing 403

- Fix unban to properly clear isBanned flag

- Add moderation stats dashboard, status filters, and quick actions to admin

- Add ban/unban buttons with confirmation dialogs on user detail page

- Improve admin navigation with tabs and breadcrumbs
@joan-code6

Copy link
Copy Markdown
Author

btw the contact mahad on slack thingy redierects to slack.hackclub.com which allows user which dont have slack to still reach you / join slack and get a tutorial

@joan-code6

Copy link
Copy Markdown
Author

@skyfallwastaken 🧐

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant