Evidence Library

Checkable QA evidence

One defect report, one coverage matrix and the runs behind them, all from my public Razorpay payment-testing repository (Test Mode, an independent project). Every ID links to the exact file on GitHub at commit da9e7a2.

Defect report Coverage matrix Test runs

Defect report: DEF-002

Source: docs/DEFECTS.md (opens in a new tab) in the repository. Order and payment IDs are left out.

Webhook signature validation can never succeed: the global JSON body parser consumes the body before the webhook route

Environment
Express 4.x app (4.22.2 in the lockfile) on Node.js 22.5 or later, with SQLite; Razorpay Test Mode; webhook secret configured. Found on 14 Sep 2026 during real Test Mode execution (Phase 3.3).
Severity / Priority
Severity: High. Priority: not recorded in the repository's defect log.
Component
src/app.js (middleware order) and src/routes/webhooks.js
Status
Resolved 14 Sep 2026 (Phase 3.3.1); last recorded CI run passed on commit da9e7a2

Exact reproduction steps (as recorded)

  1. Compute hmac_sha256(rawBody, RAZORPAY_WEBHOOK_SECRET) exactly as validateWebhookSignature() does.
  2. Send that exact raw body and the correct signature to POST /api/webhooks/razorpay, once with fetch() and once with curl --data-binary using a byte-identical file, to rule out client-side body changes.
  3. Observe the response.
Expected

HTTP 200 with { success: true, message: 'Webhook received and validated' }.

Actual

HTTP 401 Webhook signature validation failed, every time, even with a provably correct signature over the provably correct bytes.

Root cause

app.use(express.json()) was mounted globally before the webhook router. It consumed and parsed the request body, so the route's own express.raw() received an already-parsed object instead of a Buffer. req.body.toString('utf-8') on a plain object returns the literal string "[object Object]", and the HMAC was computed over that instead of the real payload. The comparison was correct; it was hashing the wrong bytes. Fixed by mounting the webhook router before the global parser:

app.use('/api/webhooks', webhooksRouter);

// Middleware
app.use(express.json());
The fixed mount order, with the repository's own comment. src/app.js, lines 16–19 @ da9e7a2 (opens in a new tab)

The handler that depends on it, which hashes the raw body and only then parses it:

router.post('/razorpay', express.raw({ type: 'application/json' }), (req, res) => {
  try {
    const signature = req.headers['x-razorpay-signature'];
    const eventId = req.headers['x-razorpay-event-id'];

    if (!signature) {
      console.error('❌ Webhook rejected: Missing X-Razorpay-Signature header');
      return res.status(401).json({
        error: 'Unauthorized',
        message: 'X-Razorpay-Signature header is required',
      });
    }

    if (!eventId) {
      console.warn('⚠️  Webhook received without X-Razorpay-Event-Id header');
      return res.status(400).json({
        error: 'Bad Request',
        message: 'X-Razorpay-Event-Id header is required',
      });
    }

    // Check for duplicate webhook event before validation
    if (isWebhookEventProcessed(eventId)) {
      console.log(`ℹ️  Webhook event already processed (idempotent): ${eventId}`);
      return res.status(200).json({
        success: true,
        eventId,
        message: 'Webhook already processed',
      });
    }

    // Validate signature using raw body
    const rawBody = req.body.toString('utf-8');
    const isValid = validateWebhookSignature(rawBody, signature);
Webhook handler, excerpt. src/routes/webhooks.js, lines 55–88 @ da9e7a2 (opens in a new tab)

Impact

No correctly signed webhook could be accepted. It failed safe, not open: a forged webhook could not get through either, so this was a functional defect and not a security hole. It also went unnoticed because the tests that only assert rejection (a 401 was already the expected answer) never exercised the accept path.

Regression tests

New: WEB-FIX-001 (valid signature over the exact raw body is accepted and moves the order from created to paid), WEB-FIX-002 (modified payload with the original signature is rejected and the order is untouched), WEB-FIX-003 (invalid signature rejected), WEB-FIX-004 (duplicate event ID handled idempotently).
Previously failing, now passing: WEB-001, WEB-005, WEB-006, WEB-007. All are in tests/security/webhook-validation.spec.js (opens in a new tab) and run in CI.

Reproduced again on 15 Oct 2026

To check the report rather than repeat it, I cloned the repository at da9e7a2, installed it, and ran the webhook spec with the CI placeholder configuration. With the fix in place, all 11 tests passed. With the fix reverted in a scratch copy (webhook router mounted after express.json(), nothing else changed), 6 of the 11 failed: WEB-001, WEB-005, WEB-006, WEB-007, WEB-FIX-001 and WEB-FIX-004. The five rejection-only tests (WEB-002, WEB-003, WEB-004, WEB-FIX-002, WEB-FIX-003) still passed, which is why the defect hid for so long.

Test coverage matrix: Razorpay suite

Derived from the seven spec files in the repository: 37 test definitions, 34 CI-safe and 3 that need a live Razorpay Test Mode payment (tagged @razorpay-live and run separately). This lists what each test checks. It is not a coverage percentage.

Test counts by area
AreaCI-safeLive gatewayTotalSpec file
Health & configuration404health.spec.js (opens in a new tab)
Order creation & amount integrity303orders.spec.js (opens in a new tab)
Error handling & message sanitization505error-handling.spec.js (opens in a new tab)
Refund error handling202refund-error-handling.spec.js (opens in a new tab)
Payment signature verification527signature-verification.spec.js (opens in a new tab)
Webhook validation & idempotency11011webhook-validation.spec.js (opens in a new tab)
Checkout page (UI)415frontend.spec.js (opens in a new tab)
Total34337

Every test

All 37 Razorpay tests with what each checks
Test IDWhat it checksAreaRuns inDefect
HEALTH-001 (opens in a new tab)GET /health returns application healthHealth & configurationCI-safe
CONFIG-001 (opens in a new tab)GET /api/payments/config returns public configurationHealth & configurationCI-safe
CONFIG-002 (opens in a new tab)GET /api/payments/config does NOT expose secretsHealth & configurationCI-safe
CONFIG-003 (opens in a new tab)GET /api/payments/config properly reflects configuration statusHealth & configurationCI-safe
ORDER-002 (opens in a new tab)POST /api/payments/create-order handles both success and error gracefullyOrder creation & amount integrityCI-safe
ORDER-003 (opens in a new tab)client-supplied amount/currency cannot override the backend-controlled order amount In CI this cannot fail: its assertions run only when order creation returns 200, and with placeholder credentials it does not.Order creation & amount integrityCI-safe
ORDER-001 (opens in a new tab)POST /api/payments/create-order returns order structure (when configured)Order creation & amount integrityCI-safe
ERR-001 (opens in a new tab)Order creation error does NOT expose Razorpay API detailsError handling & message sanitizationCI-safe
ERR-002 (opens in a new tab)Payment verification error does NOT expose internal detailsError handling & message sanitizationCI-safe
ERR-003 (opens in a new tab)404 endpoint returns appropriate errorError handling & message sanitizationCI-safe
ERR-004 (opens in a new tab)Malformed JSON request returns 400 errorError handling & message sanitizationCI-safe
REF-001 (opens in a new tab)POST /api/refunds/create requires paymentIdError handling & message sanitizationCI-safe
REF-ERR-001 (opens in a new tab)a Razorpay-rejected refund returns a controlled error, not a crashRefund error handlingCI-safeDEF-001
REF-ERR-002 (opens in a new tab)the server remains available immediately after a refund errorRefund error handlingCI-safeDEF-001
SIG-001 (opens in a new tab)POST /api/payments/verify with valid signature accepts a real captured paymentPayment signature verificationLive gateway
REPLAY-001 (opens in a new tab)repeated verification of the same real payment stays idempotent (no duplicate/corrupted state)Payment signature verificationLive gateway
SIG-002 (opens in a new tab)POST /api/payments/verify with invalid signature rejects paymentPayment signature verificationCI-safe
SIG-003 (opens in a new tab)POST /api/payments/verify with tampered orderId rejects paymentPayment signature verificationCI-safe
SIG-004 (opens in a new tab)POST /api/payments/verify with missing signature field rejects requestPayment signature verificationCI-safe
SIG-005 (opens in a new tab)POST /api/payments/verify with missing orderId rejects requestPayment signature verificationCI-safe
SIG-006 (opens in a new tab)POST /api/payments/verify with missing paymentId rejects requestPayment signature verificationCI-safe
WEB-001 (opens in a new tab)POST /api/webhooks/razorpay with valid signature accepts webhookWebhook validation & idempotencyCI-safeDEF-002
WEB-002 (opens in a new tab)POST /api/webhooks/razorpay with invalid signature rejects webhookWebhook validation & idempotencyCI-safe
WEB-003 (opens in a new tab)POST /api/webhooks/razorpay with missing signature header rejects webhookWebhook validation & idempotencyCI-safe
WEB-004 (opens in a new tab)POST /api/webhooks/razorpay with missing event ID rejects webhookWebhook validation & idempotencyCI-safe
WEB-005 (opens in a new tab)Same webhook event ID sent twice is processed only once (idempotency)Webhook validation & idempotencyCI-safeDEF-002
WEB-006 (opens in a new tab)Different webhook event IDs are processed independentlyWebhook validation & idempotencyCI-safeDEF-002
WEB-007 (opens in a new tab)Webhook with tampered payload fails signature validationWebhook validation & idempotencyCI-safeDEF-002
WEB-FIX-001 (opens in a new tab)valid signature computed over the exact raw body is accepted and updates order stateWebhook validation & idempotencyCI-safeDEF-002
WEB-FIX-002 (opens in a new tab)a modified payload with the original signature is rejectedWebhook validation & idempotencyCI-safeDEF-002
WEB-FIX-003 (opens in a new tab)an invalid signature is rejectedWebhook validation & idempotencyCI-safeDEF-002
WEB-FIX-004 (opens in a new tab)a duplicate event ID is handled idempotentlyWebhook validation & idempotencyCI-safeDEF-002
UI-001 (opens in a new tab)Frontend page loads and displays payment formCheckout page (UI)CI-safe
UI-002 (opens in a new tab)Frontend amount displayed matches backend-controlled amountCheckout page (UI)Live gateway
UI-003 (opens in a new tab)Razorpay Checkout script loaded from official CDNCheckout page (UI)CI-safe
UI-004 (opens in a new tab)Page does NOT expose secrets in HTMLCheckout page (UI)CI-safe
UI-005 (opens in a new tab)Test Mode indicator is visible on pageCheckout page (UI)CI-safe

Not covered by any test: authentication and redirect flows, a real Razorpay-originated webhook, a successful refund, and the paid → refunded transition. See the scenario map on the case study.

Test runs

Latest GitHub Actions run of each public repository, read from GitHub on 15 Oct 2026. Run pages are public; job logs need a GitHub sign-in.

Latest GitHub Actions run per repository
ProjectWorkflowDateCommitResultLink
Razorpay Payment Gateway QA & AutomationPlaywright CI14 Sep 2026da9e7a2SuccessRun page (opens in a new tab)
RESTful Booker TypeScript APIPlaywright Tests10 Sep 2026a385240SuccessRun page (opens in a new tab)
ERPNext QA AutomationPlaywright Tests4 Sep 20268990014SuccessRun page (opens in a new tab)
Automation ExercisePlaywright Test Suite29 Aug 2026789d413SuccessRun page (opens in a new tab)

These are historical results. ERPNext's hosted test instance is offline, so a new run there would fail at the authentication step.

Playwright reports and traces (my own runs, 15 Oct 2026)

I cloned each public repository, installed it and ran the tests that need no secrets. The Razorpay run is the 34 CI-safe tests with the repository's own CI placeholder configuration; the 3 live-gateway tests were not run and no API keys were used or requested.

Playwright HTML report for the Razorpay suite showing 34 passed, 0 failed, 0 flaky and 0 skipped tests in Chromium
Playwright HTML report, CI-safe Razorpay suite: 34 passed, 0 failed (npm run test:ci, commit da9e7a2, 15 Oct 2026, 21.6 s). The 3 live-gateway tests were not run.
Playwright Trace Viewer for test WEB-FIX-001 showing the POST to /api/webhooks/razorpay, three assertions and a GET of the order, all passing
Trace Viewer, WEB-FIX-001 (a correctly signed webhook is accepted and moves the order to paid). A passing trace: POST /api/webhooks/razorpay with the X-Razorpay-Signature header, then the order read back.
Playwright report for WEB-FIX-001 failing with Expected 200 and Received 401 at line 277 of the webhook spec
The same test, WEB-FIX-001, against a scratch copy of the repository with the DEF-002 fix reverted (webhook router mounted after express.json()): expected 200, received 401. Six of the eleven webhook tests failed in that copy. Not the published code.
Playwright HTML report for the RESTful Booker suite showing 47 passed, 0 failed, 0 flaky and 0 skipped tests
Playwright HTML report, RESTful Booker TypeScript suite: 47 passed, 0 failed (npm test, commit a385240, 15 Oct 2026, 29.2 s, against the public restful-booker.herokuapp.com API).
Playwright report for the booking lifecycle test listing each request and assertion: POST auth, POST booking, GET, PUT, GET, DELETE and a final GET
One expanded test: the create, read, update, delete and verify workflow, showing each API call and assertion in order.

Not captured: a Trace Viewer for the ERPNext autocomplete test, because its hosted test instance is offline and the suite cannot run; and a report for Automation Exercise, whose setup step needs login credentials that I do not have in a clean clone.

Latest green GitHub Actions run, per repository

GitHub Actions run page for the Razorpay repository showing a successful Playwright CI run on commit da9e7a2
GitHub Actions, Razorpay repository: Playwright CI run #3 on commit da9e7a2, status Success, 46 s (14 Sep 2026). Job logs need a GitHub sign-in.
GitHub Actions run page for the RESTful Booker repository showing a successful Playwright Tests run on commit a385240
GitHub Actions, RESTful Booker repository: Playwright Tests run on commit a385240, status Success (10 Sep 2026).
GitHub Actions run page for the ERPNext repository showing a successful Playwright Tests run on commit 8990014
GitHub Actions, ERPNext repository: Playwright Tests run #6 on commit 8990014, status Success, 6 m 57 s (4 Sep 2026). This is the last run: the hosted test instance has since gone offline.
GitHub Actions run page for the Automation Exercise repository showing a successful Playwright Test Suite run on commit 789d413
GitHub Actions, Automation Exercise repository: Playwright Test Suite run on commit 789d413, status Success (29 Aug 2026). Its UI tests need login credentials, so I did not re-run this suite myself.

Want the full context?

The case study explains the strategy, both defects and what I could not prove.

Razorpay case study All proof of work