You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
β οΈ Sensitive handling note
This issue intentionally uses IAM user names and key suffixes only. Do not paste full access keys, secret keys, bearer tokens, or private candidate data into comments on this issue.
Background
A destructive S3 lifecycle rule named PressureDelete3Days was discovered enabled on the permanentstudenthub-uploads bucket, scoped to the entire bucket (not just temp files). This rule has now been disabled, but files deleted before that point may be unrecoverable as bucket versioning was not enabled at the time.
Additionally, several IAM service users (railway-s3-access, n8n-s3-access, mediaconverter) appear to have had bucket-admin level permissions they should never have had β including the ability to modify lifecycle rules, CORS, bucket policies, replication, and logging. These are the root-cause risk, not just the exposed keys themselves.
CORS has been added to the temp upload bucket (studenthub-public-anyone-can-upload-24hr-expiry) and new Civil ID uploads are working again. The permanent bucket lifecycle rule is disabled. What remains is key rotation, code patches, backend fixes, and hardening.
Upload Architecture (for contributor context)
Browser / Candidate app
-> GET /aws/config
-> PUT file directly to temp S3 bucket (studenthub-public-anyone-can-upload-24hr-expiry)
-> POST filename to backend
-> Backend copies file to permanent S3 bucket (studenthub-uploads)
-> DB stores filename only
-> Frontend builds permanent image URL from bucket URL + prefix + filename
Upload type
Storage path
Civil ID front/back
Temp S3 β studenthub-uploads/photos/<filename>
Resume
Temp S3 β studenthub-uploads/candidate-resume/
Video
Temp S3 β studenthub-uploads + MediaConvert job
Personal photo
Temp S3 β Cloudinary (older backend flow)
β Already Completed
CORS added to studenthub-public-anyone-can-upload-24hr-expiry β browser direct-to-S3 uploads work again
PressureDelete3Days lifecycle rule disabled on studenthub-uploads
Permanent bucket path confirmed: Civil ID files expected at photos/<filename>
Old inactive keys FZMN (textract-access) and 4T67K (public-environment-s3-access) confirmed with no active references β ready for deletion
Patch DELETE /v1/account/remove-civil-photo-front and /remove-civil-photo-back to tolerate missing S3 objects β delete photos/<filename> best-effort only, clear DB field, and mark verification needed (no 500)
Patch POST /v1/account/update-civil-id-expiry-date to validate civil_id and civil_expiry_date, wrap save in try/catch, return operation:error instead of raw 500
Fix Candidate::deleteFile("civil-id") prefix β change from candidate-civil-id/ to photos/ (wrong prefix causing failed deletes)
Fix Candidate::deleteFile("civil-id") β stop adding errors to the unrelated candidate_resume field
Patch Candidate::updateCivilId() to: copy new file first β verify destination β delete old file after success only β log failures
Add S3ResourceManager::fileExists() headObject wrapper if missing
Add route/candidate_id/filename logging for copy/delete failures
π΄ Phase 3 β Frontend Error Handling Fixes
Candidate app: Handle remove-civil-photo API errors locally β show toast/message, prevent global 500 route trap
Candidate app: Allow back navigation from /civil-id?fromProfile=1 to profile even after partial failure
Staff app: Add error handler to loadCandidateDetail() so candidate profile does not become a blank page
Staff app: Fix back Civil ID template condition β check candidate_civil_photo_back, not the front field
Staff app: Show "Civil ID image unavailable β request re-upload" when image returns 403 or is missing
π΄ Phase 4 β Patch Hardcoded Keys Out of Code (ODY2X & WCUM)
Do not delete these keys until code is patched, deployed, and tested.
ODY2X β temp S3 backend credentials, still hardcoded in common/config/main.php
Patch common/config/main.phptemporaryBucketResourceManager to use env vars: AWS_TEMP_BUCKET_KEY / AWS_TEMP_BUCKET_SECRET
Search and patch any microservice or PlugN/Wallet references
WCUM β permanent S3 credentials (railway-s3-access), still in prod Railway configs and microservices
Patch environments/prod-railway/common/config/main-local.phpresourceManager to use env vars: AWS_PERMANENT_S3_ACCESS_KEY_ID / AWS_PERMANENT_S3_SECRET_ACCESS_KEY
Patch PlugN, Wallet, Yeastar voicemail, and dev-server configs that still reference WCUM
Remove bucket-admin permissions from railway-s3-access immediately β restrict to object-level GetObject, PutObject, PutObjectAcl, CopyObject, DeleteObject, ListBucket on studenthub-uploads only
π΄ Phase 5 β Rotate/Delete ODY2X & WCUM
After Phase 4 deploy and tests pass:
Deactivate ODY2X β retest all upload flows β delete ODY2X
Treat railway-s3-access, n8n-s3-access, and mediaconverter as over-permissioned at minimum and potentially compromised until CloudTrail confirms known source IPs, user agents, and expected automation.
Suspicious event types to search for (none of these should have been callable by app service users):
Event
Why it matters
PutBucketLifecycleConfiguration
Can delete permanent S3 objects automatically β likely explains missing Civil ID files
DeleteBucketCors / PutBucketCors
Can break or loosen browser uploads
DeleteBucketPolicy / PutBucketPolicy
Can expose or block S3 data
PutBucketReplicationConfiguration
Can disable backups
PutBucketLogging
Can alter audit logging
PutPublicAccessBlock / DeletePublicAccessBlock
Can change public-read behavior
Export CloudTrail events for Apr 17β19 (and surrounding suspicious period) for railway-s3-access, n8n-s3-access, mediaconverter
For each event record: eventTime, eventName, userName, accessKeyId suffix, sourceIPAddress, userAgent, bucketName, region, errorCode
Confirm whether the lifecycle script touched only studenthub-uploads or also PlugN/Wallet buckets
Ask team who owns n8n-s3-access and which n8n workflow or server used it
π‘ Phase 8 β Missing Civil ID Data Audit & Recovery
Old DB records reference photos/<filename> objects that no longer exist in S3 (likely deleted by the lifecycle rule).
Run the following query to export all affected Civil ID filenames:
SELECT candidate_id, 'front'AS side, candidate_civil_photo_front AS filename,
CONCAT('photos/', candidate_civil_photo_front) AS expected_s3_key, candidate_updated_at
FROM candidate
WHERE candidate_civil_photo_front IS NOT NULLAND candidate_civil_photo_front <>''UNION ALLSELECT candidate_id, 'back'AS side, candidate_civil_photo_back AS filename,
CONCAT('photos/', candidate_civil_photo_back) AS expected_s3_key, candidate_updated_at
FROM candidate
WHERE candidate_civil_photo_back IS NOT NULLAND candidate_civil_photo_back <>''ORDER BY candidate_updated_at DESC;
For each filename, check studenthub-uploads/photos/<filename> and legacy path studenthub-uploads/candidate-civil-id/<filename>
If found under legacy prefix, copy to photos/<filename>
If still in temp bucket within lifecycle window, copy to permanent
If missing everywhere β flag candidate for re-upload (do not mass-clear DB fields until audit is complete)
π’ Phase 9 β Bucket Hardening & Guardrails
Enable versioning on studenthub-uploads to protect future accidental deletions
Audit all studenthub-* buckets plus PlugN/Wallet buckets for lifecycle rules, CORS, policies, ACLs, public access settings
Set up EventBridge/CloudWatch alerts for: PutBucketLifecycleConfiguration, DeleteBucketCors, PutBucketCors, DeleteBucketPolicy, PutBucketPolicy, PutPublicAccessBlock, PutBucketReplicationConfiguration
Route bucket-admin change alerts to Slack/Discord #tech and email
Run IAM Access Analyzer for all prod buckets and service users
Enable GitHub secret scanning + push protection across BAWES-Universe repos
Remove committed .env files containing AWS keys from repos (and history where practical)
Move secrets to Railway env vars or AWS Secrets Manager β no full keys in Discord or GitHub issues
Add owner / service / environment tags to every IAM user and key
Create a service-user permissions boundary that explicitly denies all bucket-admin actions for service IAM users
Schedule monthly IAM access key review
Least-Privilege Permissions Model (Reference)
User type
Allowed
Must NOT allow
Temp browser upload user
s3:PutObject, s3:PutObjectAcl, s3:GetObject, s3:AbortMultipartUpload, s3:ListMultipartUploadParts on temp bucket only
Any access to studenthub-uploads; any bucket-admin actions
Permanent S3 app user
s3:GetObject, s3:PutObject, s3:PutObjectAcl, s3:CopyObject, s3:DeleteObject, s3:ListBucket on studenthub-uploads only
This issue is eligible for an Algora bounty. Contributors are welcome to pick up individual phases. Comment below to claim a phase before starting work.
AWS S3 Security Remediation & Civil ID Upload Fix
Background
A destructive S3 lifecycle rule named
PressureDelete3Dayswas discovered enabled on the permanentstudenthub-uploadsbucket, scoped to the entire bucket (not just temp files). This rule has now been disabled, but files deleted before that point may be unrecoverable as bucket versioning was not enabled at the time.Additionally, several IAM service users (
railway-s3-access,n8n-s3-access,mediaconverter) appear to have had bucket-admin level permissions they should never have had β including the ability to modify lifecycle rules, CORS, bucket policies, replication, and logging. These are the root-cause risk, not just the exposed keys themselves.CORS has been added to the temp upload bucket (
studenthub-public-anyone-can-upload-24hr-expiry) and new Civil ID uploads are working again. The permanent bucket lifecycle rule is disabled. What remains is key rotation, code patches, backend fixes, and hardening.Upload Architecture (for contributor context)
studenthub-uploads/photos/<filename>studenthub-uploads/candidate-resume/studenthub-uploads+ MediaConvert jobβ Already Completed
studenthub-public-anyone-can-upload-24hr-expiryβ browser direct-to-S3 uploads work againPressureDelete3Dayslifecycle rule disabled onstudenthub-uploadsphotos/<filename>FZMN(textract-access) and4T67K(public-environment-s3-access) confirmed with no active references β ready for deletionπ΄ Phase 1 β Delete Safe Inactive Keys (Ready Now)
These keys are already inactive and have no confirmed active references. Screenshot IAM evidence before deleting.
FZMNfrom IAM usertextract-access(replacement Textract key already active in Railway)4T67Kfrom IAM userpublic-environment-s3-access(replacement temp key active, used by/aws/config)π΄ Phase 2 β Fix Civil ID Edit/Remove 500 Errors (Backend β High Priority)
Files to patch:
candidate/modules/v1/controllers/AccountController.phpcommon/models/Candidate.phpcommon/components/S3ResourceManager.phpRequired fixes:
DELETE /v1/account/remove-civil-photo-frontand/remove-civil-photo-backto tolerate missing S3 objects β deletephotos/<filename>best-effort only, clear DB field, and mark verification needed (no 500)POST /v1/account/update-civil-id-expiry-dateto validatecivil_idandcivil_expiry_date, wrap save in try/catch, returnoperation:errorinstead of raw 500Candidate::deleteFile("civil-id")prefix β change fromcandidate-civil-id/tophotos/(wrong prefix causing failed deletes)Candidate::deleteFile("civil-id")β stop adding errors to the unrelatedcandidate_resumefieldCandidate::updateCivilId()to: copy new file first β verify destination β delete old file after success only β log failuresS3ResourceManager::fileExists()headObject wrapper if missingroute/candidate_id/filenamelogging for copy/delete failuresπ΄ Phase 3 β Frontend Error Handling Fixes
/civil-id?fromProfile=1to profile even after partial failureloadCandidateDetail()so candidate profile does not become a blank pagecandidate_civil_photo_back, not the front fieldπ΄ Phase 4 β Patch Hardcoded Keys Out of Code (ODY2X & WCUM)
ODY2X β temp S3 backend credentials, still hardcoded in
common/config/main.phpcommon/config/main.phptemporaryBucketResourceManagerto use env vars:AWS_TEMP_BUCKET_KEY/AWS_TEMP_BUCKET_SECRETWCUM β permanent S3 credentials (
railway-s3-access), still in prod Railway configs and microservicesenvironments/prod-railway/common/config/main-local.phpresourceManagerto use env vars:AWS_PERMANENT_S3_ACCESS_KEY_ID/AWS_PERMANENT_S3_SECRET_ACCESS_KEYAWS_PERMANENT_S3_ACCESS_KEY_ID,AWS_PERMANENT_S3_SECRET_ACCESS_KEY,AWS_PERMANENT_S3_REGION=eu-west-2,AWS_PERMANENT_S3_BUCKET=studenthub-uploadsrailway-s3-accessimmediately β restrict to object-levelGetObject,PutObject,PutObjectAcl,CopyObject,DeleteObject,ListBucketonstudenthub-uploadsonlyπ΄ Phase 5 β Rotate/Delete ODY2X & WCUM
After Phase 4 deploy and tests pass:
ODY2Xβ retest all upload flows β deleteODY2XWCUMβ retest permanent copy flows β deleteWCUMπ Phase 6 β Investigate DOBNJ, n8n-s3-access & Extra Keys
DOBNJ β SQS/EventManager/Yeastar-related, still in committed configs
DOBNJn8n-s3-access β High suspicion, associated with bucket lifecycle/CORS/policy changes
Extra exposed keys to review:
55KF,OFLT(MediaConvert),XW5I(SES SMTP),TMEZ(process-id-request S3)π‘ Phase 7 β CloudTrail Investigation (Compromise Assessment)
Suspicious event types to search for (none of these should have been callable by app service users):
PutBucketLifecycleConfigurationDeleteBucketCors/PutBucketCorsDeleteBucketPolicy/PutBucketPolicyPutBucketReplicationConfigurationPutBucketLoggingPutPublicAccessBlock/DeletePublicAccessBlockrailway-s3-access,n8n-s3-access,mediaconvertereventTime,eventName,userName,accessKeyIdsuffix,sourceIPAddress,userAgent,bucketName,region,errorCodestudenthub-uploadsor also PlugN/Wallet bucketsn8n-s3-accessand which n8n workflow or server used itπ‘ Phase 8 β Missing Civil ID Data Audit & Recovery
Old DB records reference
photos/<filename>objects that no longer exist in S3 (likely deleted by the lifecycle rule).studenthub-uploads/photos/<filename>and legacy pathstudenthub-uploads/candidate-civil-id/<filename>photos/<filename>π’ Phase 9 β Bucket Hardening & Guardrails
studenthub-uploadsto protect future accidental deletionsstudenthub-*buckets plus PlugN/Wallet buckets for lifecycle rules, CORS, policies, ACLs, public access settingsPutBucketLifecycleConfiguration,DeleteBucketCors,PutBucketCors,DeleteBucketPolicy,PutBucketPolicy,PutPublicAccessBlock,PutBucketReplicationConfiguration#techand email.envfiles containing AWS keys from repos (and history where practical)owner/service/environmenttags to every IAM user and keyLeast-Privilege Permissions Model (Reference)
s3:PutObject,s3:PutObjectAcl,s3:GetObject,s3:AbortMultipartUpload,s3:ListMultipartUploadPartson temp bucket onlystudenthub-uploads; any bucket-admin actionss3:GetObject,s3:PutObject,s3:PutObjectAcl,s3:CopyObject,s3:DeleteObject,s3:ListBucketonstudenthub-uploadsonlytextract:DetectDocumentText+ minimal S3 readsqs:SendMessage/ReceiveMessage/DeleteMessage/GetQueueAttributeson exact queue ARNsPost-Fix Test Checklist
studenthub-uploads/photos/; expiry and ID updatestudenthub-uploads/photos/AWS Support Evidence Package (to request restriction removal)
FZMNand4T67Kinactive/deleted and replacements activeODY2X/WCUM/DOBNJand extra keys from reposstudenthub-uploadslifecycle rule disabled screenshot