test: add k6 load/stress/spike/soak scripts for AI problem generation… - #48
Conversation
📝 WalkthroughWalkthroughProblem-AI API 부하 검증을 위한 공통 k6 설정과 인증·페이로드 헬퍼를 추가하고, 부하·소크·스파이크·스트레스 시나리오 및 각 실행 결과 JSON을 구성합니다. ChangesProblem-AI 성능 테스트
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)공통 테스트 실행 흐름sequenceDiagram
participant k6Setup
participant AuthEndpoint
participant ProblemAiApi
participant Metrics
k6Setup->>AuthEndpoint: 로그인 및 accessToken 획득
k6Setup->>AuthEndpoint: XSRF-TOKEN 쿠키 조회
k6Setup-->>Metrics: authToken, csrfToken 전달
Metrics->>ProblemAiApi: 인증·CSRF 헤더와 페이로드로 POST
ProblemAiApi-->>Metrics: 201 응답 및 처리시간 기록
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
k6/problem-ai/spike-test.js (1)
21-31: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win[중요] 복구 구간의 성공률을 별도 지표로 측정하세요.
현재
error_rate와 체크는 스파이크 전·중·후 요청을 모두 합산합니다. 따라서 결과 JSON의 전체 성공률만으로는 마지막 30초에 실제로 복구되었는지 검증할 수 없습니다. 복구 구간 전용Rate/Trend와 임계값을 추가해 배포 판단 근거를 분리하세요.Also applies to: 60-65
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@k6/problem-ai/spike-test.js` around lines 21 - 31, Add a recovery-only Rate or Trend metric alongside error_rate and record only requests from the final 30-second recovery stage, using the existing request/check flow rather than aggregating all stages. Add a dedicated threshold for this metric in options.thresholds so deployment validation separately enforces recovery success, while preserving the existing overall error_rate threshold.k6/problem-ai/load-test.js (1)
18-30: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueAI 문제 생성 지연시간 지표에 SLO 임계값을 함께 적용하세요.
thresholds의http_req_duration는 k6의 표준 HTTP 요청 지표라 로그인(setup())까지 포함할 수 있으며, 현재ai_generation_latency에 기준 값은 있지만 임계값이 없어 결과 판정에서 누락됩니다.ai_generation_latency: ['p(95)<15000']를 추가하고, 필요하면 기존 기준 유지 여부를 판단하세요.
k6/problem-ai/load-test.js#L18-L30:http_req_duration추가 기준만 유지되도록 설정하거나ai_generation_latency기준으로 명확히 갱신하세요.k6/problem-ai/soak-test.js#L19-L31:http_req_duration추가 기준만 유지되도록 설정하거나ai_generation_latency기준으로 명확히 갱신하세요.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@k6/problem-ai/load-test.js` around lines 18 - 30, The thresholds in k6/problem-ai/load-test.js lines 18-30 omit the custom ai_generation_latency SLO; add an ai_generation_latency p95 threshold of 15000 ms and retain or replace http_req_duration consistently with the intended metric. Apply the same threshold update in k6/problem-ai/soak-test.js lines 19-31, using the existing aiLatency Trend metric.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@k6/problem-ai/config.js`:
- Around line 17-23: Remove the hardcoded credentials from ADMIN_USER and source
both its email/password and BASE_URL exclusively from the corresponding __ENV
variables, without fallback defaults. Rotate or revoke the exposed password
separately, and preserve the existing exported configuration symbols for
consumers.
- Around line 50-57: Update setup() authentication failure handling around
isLoginSuccess and the CSRF acquisition flow: do not return the mock JWT or null
when initial authentication or CSRF retrieval fails; instead fail the load test
immediately using the existing setup error mechanism. Ensure requests proceed
only when both valid authentication and CSRF credentials are available.
In `@k6/problem-ai/load-result.json`:
- Around line 163-164: Remove the exposed setup_data.authToken and
setup_data.csrfToken fields from k6/problem-ai/load-result.json lines 163-164,
k6/problem-ai/soak-result.json lines 163-164, k6/problem-ai/spike-result.json
lines 129-130, and k6/problem-ai/stress-result.json lines 167-168; also revoke
the exposed tokens and associated sessions.
---
Nitpick comments:
In `@k6/problem-ai/load-test.js`:
- Around line 18-30: The thresholds in k6/problem-ai/load-test.js lines 18-30
omit the custom ai_generation_latency SLO; add an ai_generation_latency p95
threshold of 15000 ms and retain or replace http_req_duration consistently with
the intended metric. Apply the same threshold update in
k6/problem-ai/soak-test.js lines 19-31, using the existing aiLatency Trend
metric.
In `@k6/problem-ai/spike-test.js`:
- Around line 21-31: Add a recovery-only Rate or Trend metric alongside
error_rate and record only requests from the final 30-second recovery stage,
using the existing request/check flow rather than aggregating all stages. Add a
dedicated threshold for this metric in options.thresholds so deployment
validation separately enforces recovery success, while preserving the existing
overall error_rate threshold.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4c337003-6743-4b34-bd94-af44715f8911
📒 Files selected for processing (9)
k6/problem-ai/config.jsk6/problem-ai/load-result.jsonk6/problem-ai/load-test.jsk6/problem-ai/soak-result.jsonk6/problem-ai/soak-test.jsk6/problem-ai/spike-result.jsonk6/problem-ai/spike-test.jsk6/problem-ai/stress-result.jsonk6/problem-ai/stress-test.js
| export const BASE_URL = 'http://momogo-alb-1906718718.ap-northeast-2.elb.amazonaws.com'; | ||
|
|
||
| // 2. [테스트 계정] ADMIN 권한 + 공간(Space) 소유 계정 | ||
| export const ADMIN_USER = { | ||
| email: 'jun@test.com', | ||
| password: '7964a23!', // TODO: 실제 비밀번호로 교체해서 로컬에서만 사용 (커밋 금지) | ||
| }; |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== repo files matching config.js / problem-ai =="
git ls-files | rg '(^|/)config\.js$|problem-ai|^k6/' | sed -n '1,120p'
echo
echo "== k6/problem-ai/config.js =="
if [ -f k6/problem-ai/config.js ]; then
nl -ba k6/problem-ai/config.js | sed -n '1,120p'
else
echo "missing"
fi
echo
echo "== search for BASE_URL, ADMIN_USER, Problem-AI, bearer, Authorization headers =="
rg -n "BASE_URL|ADMIN_USER|Authorization|Bearer|PROBLEM_AI|momogo-alb|jun@test\.com|7964a23!" . -S --glob '!node_modules' --glob '!dist' --glob '!build' | sed -n '1,200p'Repository: MoMoGo-QuizPlatform/MoMoGo
Length of output: 742
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== git diff status/stat =="
git diff --stat || true
echo
echo "== focused file inspection with ast/rg if available =="
if command -v ast-grep >/dev/null 2>&1; then
ast-grep outline k6/problem-ai/config.js --view expanded || true
fi
echo
echo "== imports/usages from config and k6 =="
for f in $(git ls-files 'k6/problem-ai/*.js' 'k6/problem-ai/**/*.js' | sed -n '1,100p'); do
echo "--- $f"
nl -ba "$f" | sed -n '1,160p' | rg -n "BASE_URL|ADMIN_USER|problem-ai/config|loadCookie|Bearer|Authorization|http request|check\\(|open|post\\(|get\\(|setHeaders|__ENV" -C 2 || true
doneRepository: MoMoGo-QuizPlatform/MoMoGo
Length of output: 1057
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for p in [Path("k6/problem-ai/config.js"), Path("k6/problem-ai/load-test.js"), Path("k6/problem-ai/soak-test.js"), Path("k6/problem-ai/spike-test.js"), Path("k6/problem-ai/stress-test.js")]:
if not p.exists():
continue
lines = p.read_text().splitlines()
print(f"--- {p} ({len(lines)} lines)")
for i, line in enumerate(lines, 1):
if any(k in line for k in ("BASE_URL", "ADMIN_USER", "obtainAuthToken", "obtainCsrfToken", "Authorization", "Bearer", "Authorization:", "k6/http", "check(","login", "csrf")):
start=max(1,i-3); end=min(len(lines),i+8)
for j in range(start,end+1):
print(f"{j:4}: {lines[j-1]}")
print()
PY
echo "== search for hardcoded credential patterns =="
rg -n "7964a23!|jun@test\.com|Authorization:\s*['\"']Bearer|Bearer|Authorization: Bearer|Bearer|PROBLEM_AI|BASE_URL|momogo-alb" . -S --glob '!node_modules' --glob '!dist' --glob '!build' || trueRepository: MoMoGo-QuizPlatform/MoMoGo
Length of output: 34324
기본값 없는 비밀값 구성만 요청합니다.
k6/problem-ai/config.js의 테스트 계정 비밀번호가 저장소에 직접 들어갔습니다. 커밋이 완료된 상태이므로 이 암호는 폐기/교체해 주세요. 동시에 BASE_URL과 ADMIN_USER도 기본값 없이 __ENV.*에서 주입하도록 분리하는 것이 좋습니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@k6/problem-ai/config.js` around lines 17 - 23, Remove the hardcoded
credentials from ADMIN_USER and source both its email/password and BASE_URL
exclusively from the corresponding __ENV variables, without fallback defaults.
Rotate or revoke the exposed password separately, and preserve the existing
exported configuration symbols for consumers.
| if (!isLoginSuccess) { | ||
| console.error(`[Setup 에러] 로그인 실패 (Status: ${loginRes.status}, Body: ${loginRes.body}).`); | ||
| console.error(`=> DB에 ${ADMIN_USER.email} 계정이 존재하는지, ADMIN 권한인지 확인해 주세요!`); | ||
| return 'mock-jwt-access-token-sample'; | ||
| } | ||
|
|
||
| const token = loginRes.json('accessToken') || loginRes.headers['Authorization'] || 'mock-jwt-access-token-sample'; | ||
| const cleanToken = token.replace('Bearer ', ''); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)k6/problem-ai/config\.js$|k6/problem-ai|config\.js' || true
echo
echo "Outline:"
ast-grep outline k6/problem-ai/config.js || true
echo
nl -ba k6/problem-ai/config.js | sed -n '1,130p'Repository: MoMoGo-QuizPlatform/MoMoGo
Length of output: 957
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "config.js relevant sections:"
awk 'NR>=1 && NR<=110 {printf "%6d: %s\n", NR, $0}' k6/problem-ai/config.js
echo
echo "load-test.js relevant sections:"
awk 'NR>=1 && NR<=220 {printf "%6d: %s\n", NR, $0}' k6/problem-ai/load-test.js
echo
echo "Search for obtainAuthToken/obtainCsrfToken usages:"
rg -n "obtainAuthToken|obtainCsrfToken|csrf|accessToken|mock-jwt|CSRF" k6/problem-ai -SRepository: MoMoGo-QuizPlatform/MoMoGo
Length of output: 11386
🌐 Web query:
k6 setup exception non-zero exit if exception thrown
💡 Result:
In k6, the behavior of an exception depends on where it occurs and how you handle it. By default, uncaught exceptions during the test execution phase (in the default function) abort the current iteration but do not necessarily terminate the entire test run with a non-zero exit code [1][2]. To ensure your test process exits with a non-zero code when an error occurs, you can use the following approaches: 1. Use exec.test.abort: If you want to stop the entire test execution immediately and signal a failure, use the k6/execution module [3]. import exec from 'k6/execution'; // When you encounter an error exec.test.abort('Aborting test due to critical error'); This will cause k6 to stop and exit with a non-zero status [4]. 2. Use Soft Assertions: If you are using the k6/testing library, soft assertions can mark a test as failed while allowing it to continue execution. If any soft assertions fail, k6 will exit with code 110 upon completion [5]. import { expect } from 'https://jslib.k6.io/k6-testing/0.6.1/index.js'; export default function { expect.soft(response.status).toBe(200); } 3. Exceptions in the Init Context: If an exception is thrown in the init context (outside the default function), k6 will immediately abort the entire test execution with a non-zero exit code [3]. 4. Understanding Exit Codes: k6 uses specific exit codes for different scenarios [4]. A ScriptException typically corresponds to code 107 [4]. If you are looking to distinguish between different failure types, note that these codes are internal to k6 and serve as the standard mechanism for reporting how the process finished [4]. Summary: - fail only aborts the current iteration [2]. - exec.test.abort terminates the entire test run [3]. - Soft assertions (from k6/testing) will cause an exit code of 110 [5]. - Uncaught exceptions generally do not guarantee a non-zero exit code for the entire test process, so explicit abortion is recommended for reliable CI/CD integration [6][7].
Citations:
- 1: https://grafana.com/docs/k6/latest/javascript-api/jslib/k6chaijs/error-handling/
- 2: https://grafana.com/docs/k6/latest/javascript-api/k6/fail/?pg=5-tips-to-write-better-browser-tests-for-performance-testing-and-synthetic-monitoring&plcmt=in-text
- 3: https://community.grafana.com/t/abort-the-test-in-k6/99618/4
- 4: https://github.com/grafana/k6/blob/master/errext/exitcodes/codes.go
- 5: https://grafana.com/docs/k6/next/javascript-api/jslib/testing/expect/
- 6: Return a non 0 exit code when script execution fails grafana/k6#735
- 7: [Feature Request]: Stop execution when Javascript error encountered grafana/k6#2207
[중요] 초기 인증 또는 CSRF 획득 실패 시 부하 테스트를 중단하세요.
setup()에서 로그인 실패면 mock-jwt-access-token-sample을, CSRF 쿠키가 없으면 null을 zurück주고 요청이 계속됩니다. main 요청에서는 401/403이 집계되어 성능 오류로 표시되고, LLM 지연 시간까지 부하 결과에 섞여 결과가 무효화될 수 있습니다. 인증 기준이 충족되지 않으면 실행을 실패 처리하세요.
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 51-51: Avoid logging sensitive data
Context: console.error(=> DB에 ${ADMIN_USER.email} 계정이 존재하는지, ADMIN 권한인지 확인해 주세요!)
Note: [CWE-532] Insertion of Sensitive Information into Log File.
(log-sensitive-data)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@k6/problem-ai/config.js` around lines 50 - 57, Update setup() authentication
failure handling around isLoginSuccess and the CSRF acquisition flow: do not
return the mock JWT or null when initial authentication or CSRF retrieval fails;
instead fail the load test immediately using the existing setup error mechanism.
Ensure requests proceed only when both valid authentication and CSRF credentials
are available.
| "authToken": "eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJqdW5AdGVzdC5jb20iLCJyb2xlcyI6WyJST0xFX0FETUlOIl0sIm5hbWUiOiJqdW4iLCJ1c2VyRW1haWwiOiJqdW5AdGVzdC5jb20iLCJ0eXBlIjoiYWNjZXNzIiwiZXhwIjoxNzg1MjIxMTczLCJ1c2VySWQiOiJjYmE4YTFkZi1hZTYzLTRlYmMtYWI5Ni1kNjc1ZGQxMjljNjMiLCJpYXQiOjE3ODUyMTkzNzMsImp0aSI6IjIzODMwZDg0LWFkZmQtNDg2Ni1iNmZmLTIyNGM1OTYzOTU0MSJ9.AI070Smmkh9iEhirwSCgCyHs7vY6I6i6FKezWwE5RA8", | ||
| "csrfToken": "bb86b3a1-9911-4131-99f3-64c75d44f20b" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | 🏗️ Heavy lift
[차단] 실행 결과에 포함된 인증 정보를 즉시 제거하고 폐기하세요.
커밋된 결과 파일에 JWT와 CSRF 토큰이 그대로 포함되어 있습니다. 결과 저장 전 setup_data를 제거하고, 이미 노출된 토큰과 연계 세션은 폐기하세요.
k6/problem-ai/load-result.json#L163-L164:setup_data.authToken및setup_data.csrfToken을 제거하세요.k6/problem-ai/soak-result.json#L163-L164:setup_data.authToken및setup_data.csrfToken을 제거하세요.k6/problem-ai/spike-result.json#L129-L130:setup_data.authToken및setup_data.csrfToken을 제거하세요.k6/problem-ai/stress-result.json#L167-L168:setup_data.authToken및setup_data.csrfToken을 제거하세요.
🧰 Tools
🪛 Betterleaks (1.7.0)
[high] 163-163: Uncovered a JSON Web Token, which may lead to unauthorized access to web applications and sensitive user data.
(jwt)
📍 Affects 4 files
k6/problem-ai/load-result.json#L163-L164(this comment)k6/problem-ai/soak-result.json#L163-L164k6/problem-ai/spike-result.json#L129-L130k6/problem-ai/stress-result.json#L167-L168
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@k6/problem-ai/load-result.json` around lines 163 - 164, Remove the exposed
setup_data.authToken and setup_data.csrfToken fields from
k6/problem-ai/load-result.json lines 163-164, k6/problem-ai/soak-result.json
lines 163-164, k6/problem-ai/spike-result.json lines 129-130, and
k6/problem-ai/stress-result.json lines 167-168; also revoke the exposed tokens
and associated sessions.
Source: Linters/SAST tools
idktomorrow
left a comment
idktomorrow
left a comment
There was a problem hiding this comment.
테스트 코드 작성하시느라 고생 많으셨습니다 ! ❤️
3 participants
목적
POST /api/spaces/{spaceId}/problems/ai)는 동기 LLM 호출 구조(캐시/큐 없음)라테스트 구성 (k6/problem-ai)
결과 요약
체크리스트
참고 사항
관련 이슈
Summary by CodeRabbit
새 기능
테스트 결과