How to Test a Payment Gateway with Playwright: API, Webhooks, Refunds, and Failure Scenarios

A checkout that works once proves very little. The questions that matter in a payment integration are the ones a happy-path run never asks: what happens if the same event arrives twice, if the signature is wrong, if a failure notice arrives after a success notice, or if the gateway rejects a refund in a shape your error handler did not expect.

This article walks through how I approached those questions in an independent Razorpay Test Mode project, using Playwright Test for API-level checks. The code comes from the public repository (opens in a new tab). Where the repository does not cover something, I say so.

1. Why payment gateway testing is hard

A payment is not one request. It crosses the checkout UI, your application, the gateway and an asynchronous webhook, and it ends up as a record of state in your own database. Each of those can disagree with the others.

• The truth arrives twice. The browser reports success, and later the gateway reports it again by webhook. Both write to the same order. Either one can repeat, arrive late or arrive out of order.
• Duplicates are expected, not exceptional. Razorpay's webhook documentation says an endpoint might receive the same event multiple times and provides a per-event X-Razorpay-Event-Id header for deduplication. It also warns that events may not arrive in order.
• The cryptography is unforgiving. A webhook signature is an HMAC over the exact raw bytes that were sent. Anything that parses and re-serializes the body first breaks verification, often silently.
• You do not control the other side. The sandbox has its own limits, account restrictions and data retention. Tests that depend on one real historical payment cannot run in a clean environment.
• Mistakes cost money. A double-applied event or an accepted forged webhook is a financial and trust problem, not a cosmetic bug.

So the useful framing is state testing more than checkout testing. Most of the interesting failures live between the systems, not inside any single request.

2. Scope: scenarios worth testing, and what the project covers

Here is the scenario list I work from, mapped to what the repository actually implements. The statuses are deliberately blunt.

That table is the most useful artifact in the project. Writing it forced a distinction I now apply to every payment suite: implemented, automated and observed working end to end are three different claims.

3. Test architecture and environment

The system under test is deliberately small: an Express app that creates a Razorpay order, verifies the payment server-side, receives webhooks, supports refunds and tracks every order through an explicit state machine (created, authorized, paid, failed, refunded), persisted in SQLite. Small enough to own completely, real enough to behave like an integration.

Most tests use Playwright's request fixture to call the HTTP API directly. There is no browser involved, which makes them fast and deterministic. The committed UI tests use a real page in Chromium, but only for checks such as the Checkout script source and the absence of secrets in the HTML. Driving the hosted Checkout itself was done once, in a separate documented session.

// playwright.config.js (excerpt)
export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  use: {
    baseURL: process.env.APP_URL || 'http://localhost:3000',
    trace: 'on-first-retry',
  },
  webServer: {
    command: 'node src/app.js',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
});

Excerpt from the repository. Playwright starts the app itself, and CI uses one worker because tests share a single SQLite file.

Splitting CI-safe tests from live-gateway tests

The most important design decision was classifying tests by what they depend on. Of 37 tests, 34 are CI-safe: they need no real gateway account. The other 3 are tagged @razorpay-live and are excluded from CI:

"test:ci": "playwright test --project=chromium --grep-invert @razorpay-live",
"test:razorpay": "playwright test --project=chromium --grep @razorpay-live"

From package.json. Two of the live tests depend on one specific real payment; the third needs valid Test Mode credentials.

Mixing the live tests into CI would mean either committing real credentials or making the pipeline flaky against a service it does not control. Keeping them separate is what makes the 34 trustworthy as a regression signal.

One detail is easy to miss. The repository's CI documentation records that running the suite with no Razorpay environment variables at all produced 16 failures, because routes return 503 Razorpay not configured before reaching the logic under test. The workflow therefore sets three obviously fake placeholder values so the validation paths actually run:

env:
  NODE_ENV: test
  RAZORPAY_KEY_ID: rzp_test_ci_placeholder_not_real
  RAZORPAY_KEY_SECRET: ci_placeholder_secret_not_real
  RAZORPAY_WEBHOOK_SECRET: ci_placeholder_webhook_secret_not_real

From .github/workflows/playwright.yml. These are not secrets, and no call made with them can succeed against Razorpay.

4. Signatures and amount integrity

After Checkout, the browser hands the server an order ID, a payment ID and a signature. The server must recompute the HMAC and compare it, then confirm the payment with the gateway and check the amount against its own record. The client is never trusted for the amount.

// src/routes/payments.js (excerpt)
function verifyPaymentSignature(orderId, paymentId, signature) {
  const expectedSignature = crypto
    .createHmac('sha256', config.razorpay.keySecret)
    .update(`${orderId}|${paymentId}`)
    .digest('hex');

  return expectedSignature === signature;
}

The tests attack the two inputs that matter: a signature that is simply wrong, and a signature that is valid for a different order ID, which is what a tampering attempt looks like.

const generateSignature = (orderId, paymentId, secret) =>
  crypto.createHmac('sha256', secret).update(`${orderId}|${paymentId}`).digest('hex');

test('SIG-003: tampered orderId rejects payment', async ({ request }) => {
  const correctSignature = generateSignature(testOrderId, testPaymentId, APP_KEY_SECRET);

  const response = await request.post('/api/payments/verify', {
    data: {
      orderId: testOrderId + '_tampered',
      paymentId: testPaymentId,
      signature: correctSignature, // valid for the ORIGINAL orderId only
    },
  });

  expect(response.status()).toBe(401);
  expect((await response.json()).error).toBe('Unauthorized');
});

SIG-003, lightly condensed from tests/security/signature-verification.spec.js.

The test computes a genuinely valid signature and then changes the input it covers. That is a stronger check than sending random text, because it proves the server verifies the binding between the signature and the order.

5. Webhooks: raw bodies, tampering and idempotency

The webhook route is where this project found its most instructive defect. DEF-002: no correctly signed webhook could ever be accepted. The HMAC logic was right. The input was wrong. A global express.json() parser was mounted before the webhook route, so the body had already been parsed by the time the route's own express.raw() ran, and the signature was computed over the wrong bytes.

// src/app.js (excerpt)
// The webhook route needs the exact raw request body to verify the HMAC.
// It must be mounted BEFORE the global express.json() parser.
app.use('/api/webhooks', webhooksRouter);

app.use(express.json());

I found it by building valid signatures by hand with the right secret and watching them fail too. That ruled out the comparison logic and pointed at what the route actually received. Middleware order, in this case, is a security decision.

The regression tests sign a payload, deliver it, and then check the effect on the order, not just the HTTP status:

const sign = (body) =>
  crypto.createHmac('sha256', WEBHOOK_SECRET).update(body).digest('hex');

test('WEB-FIX-002: a modified payload with the original signature is rejected', async ({ request }) => {
  const orderId = 'order_webfix_' + Date.now();
  seedOrder(orderId);                                        // fresh order, status 'created'

  const payload = shapedPayload(orderId, paymentId);
  const signature = sign(payload);                           // signed over the ORIGINAL amount
  const modified = shapedPayload(orderId, paymentId, { amount: 1 });

  const response = await request.post('/api/webhooks/razorpay', {
    data: modified,
    headers: {
      'X-Razorpay-Signature': signature,
      'X-Razorpay-Event-Id': 'evt_webfix_tampered_' + Date.now(),
      'Content-Type': 'application/json',
    },
  });

  expect(response.status()).toBe(401);

  // The rejected webhook must not have touched the order.
  const order = await (await request.get(`/api/payments/orders/${orderId}`)).json();
  expect(order.status).toBe('created');
});

WEB-FIX-002, condensed from tests/security/webhook-validation.spec.js.

Two design choices in these tests are worth copying:

• Each test seeds its own order directly through the app's persistence layer. That keeps tests deterministic and portable to a clean CI checkout, instead of depending on a historical real transaction.
• Event IDs include Date.now(). Idempotency is stored durably in SQLite, so a hard-coded ID would pass the "first delivery" case only once, then report "already processed" on every later run.

Idempotency

The same event delivered twice must apply once. WEB-FIX-004 sends an identical signed event twice, expects the second response to say already processed, and then asserts the order is still paid with the same payment ID. That last assertion is the one that matters: the replay must not re-apply or duplicate the transition.

6. Refunds and gateway errors

The refund route is a good place to test a different class of failure: the gateway says no. DEF-001 was a server crash. When Razorpay rejected a refund, the catch block called error.message.includes(...), but the Razorpay SDK's error objects do not always set .message. Calling .includes() on undefined threw a second exception inside the error handler, and that took the process down.

test('REF-ERR-002: the server remains available immediately after a refund error', async ({ request }) => {
  // Trigger a Razorpay-rejected refund (an unknown payment ID).
  await request.post('/api/refunds/create', {
    data: { paymentId: 'pay_another_nonexistent_id_67890' },
  });

  // The defect crashed the whole process, so the real regression check is
  // that completely unrelated requests still work.
  const health = await request.get('/health');
  expect(health.status()).toBe(200);
});

REF-ERR-002, condensed. The liveness check is the regression assertion; the exact error status is secondary.

Notice what is being asserted: not the error message, but that the system survives. When a failure mode is "the process dies", the best regression test checks that something unrelated still responds.

7. Failure, cancellation and duplicate scenarios (recommended extension)

This is where the repository has gaps, and a payment suite without these is incomplete. The application already contains logic for them. For example, payment.failed maps to failed, and the state machine treats failed and refunded as terminal. But no committed test sends a failure event or attempts an illegal transition.

So I wrote extra checks against a scratch copy of the app (commit da9e7a2), using the same placeholder configuration CI uses. None of them are committed to the repository, and I have not changed it. I ran eight checks on 14 October 2026. Four confirmed intended behavior. Four exposed problems.

Confirmed intended behavior

• payment.failed moves a created order to failed.
• payment.authorized arriving after payment.captured does not regress a paid order.
• A full-amount refund.created webhook moves a paid order to refunded.
• Upper-case and truncated signatures are rejected.

Problems the checks exposed

The first one matters most. I wrote the test for the behavior I would expect, and it fails:

test('a successful retry after a failed attempt ends paid', async ({ request }) => {
  const orderId = seedOrder('order_retry');
  const paymentId = 'pay_retry_' + Date.now();

  await deliver(request, paymentEvent('payment.failed', orderId, paymentId, 'failed'),
    'evt_retry_a_' + Date.now());
  await deliver(request, paymentEvent('payment.captured', orderId, paymentId, 'captured'),
    'evt_retry_b_' + Date.now());

  expect(await orderStatus(request, orderId)).toBe('paid');
  // Observed against the current app: Received: "failed"
});

Scratch test, not in the repository. It fails against the repository as tested. deliver signs the body and posts it with the signature and event-ID headers; seedOrder and orderStatus are small helpers like those in the existing webhook tests.

1. A successful retry is ignored. The app answers 200 to the late payment.captured but leaves the order failed, so money is captured while the order says it failed. Razorpay's orders documentation (opens in a new tab) has no order-level failed status: an order stays attempted until a payment is captured, and its payment webhooks page (opens in a new tab) calls a payment.failed followed by payment.captured in the UPI retry case "expected behaviour". The state machine treats failed as terminal, which contradicts that. The client-side /verify route has a related weakness, which I read in the source but did not execute: when the state change is refused it logs a warning and still returns verified: true.
2. Partial refunds never add up. Two refund.created events of 25000 each on a 50000 order leave the order paid, because the code only moves to refunded when a single refund equals the order amount.
3. The idempotency ledger can be wiped without credentials. DELETE /api/webhooks/events clears every processed event ID, and GET on the same path lists them. After a wipe, a replayed event is processed again (Webhook received and validated instead of already processed). The repository's architecture notes list these as "debug only", but nothing stops them running outside debugging.
4. A replayed event ID with a bad signature gets 200. The handler looks up the event ID before verifying the signature, so a caller who knows an event ID can confirm it exists. No state changes, so it is low severity, but a signature check should come first.

I prototyped fixes for all four in the scratch copy, with regression tests. The tests fail without the fixes and pass with them, and the repository's own CI-safe suite still passes. I have not applied anything to the public repository.

These scenarios are still missing as committed tests, and I would add them next:

• Cancellation. Dismiss the Checkout modal and assert the order stays created, no payment is recorded and the user can retry cleanly. The Phase 3.3 report documents a manual run of this; a repeatable Playwright test against the hosted Checkout iframe is the extension.
• Duplicate submission. Double-click Pay and assert only one order is created. create-order currently builds a new order on every call, so this probably surfaces a real question for the application.
• Real webhook delivery. This needs a public URL, a configured dashboard webhook and test transactions. Razorpay's documentation notes that localhost cannot receive webhooks directly.

8. CI execution and debugging

The GitHub Actions workflow runs on push and pull request to main: Node 24, npm ci, Chromium only, then npm run test:ci. It uploads the Playwright HTML report on every run, and the test-results/ folder (traces are captured on the first retry) only when the job fails.

On 14 October 2026 I ran the same command on a clean copy of the repository, with the placeholder environment above, retries off:

npm run test:ci
...
  34 passed (42.1s)

My local run, not a CI log. The repository's last recorded CI run is documented on the project page; GitHub job logs require a sign-in, and report artifacts expire after 14 days.

Debugging habits that paid off

• Confirm the test count first. npx playwright test --list, with and without the --grep variants, shows exactly what a given command runs. That is how the 37 / 34 / 3 split was confirmed rather than assumed.
• Read the server log next to the test output. Playwright prints the app's console output. A failing webhook test is usually explained by a line such as Webhook rejected: Invalid signature.
• Vary one input at a time. DEF-002 was found by proving the HMAC logic was correct (a hand-built valid signature still failed), which moved suspicion to the request body.
• Treat CI retries with care. This config sets retries: 2 in CI. Retries tolerate a transient failure, but they can also hide a flaky test, so a green CI run is weaker evidence than a green run with retries off.

9. Evidence and limitations

Here is what the project does and does not prove.

• Demonstrated: signature verification and tamper rejection on payments and webhooks; webhook idempotency (processed event IDs are stored durably in SQLite); error sanitization; crash regression for refund errors; one real Test Mode payment completed manually and verified server-side. The CI-safe suite passes 34 of 34 locally.
• Not demonstrated: a webhook delivered by Razorpay itself; a successful refund; Checkout driven by a committed Playwright test (it was driven once, in a documented session); failure and cancellation flows in the repository's own tests.
• Sandbox, not production. Razorpay Test Mode is a controlled environment. It does not exercise real issuing banks, card networks, risk and fraud rules, live webhook configuration, settlement and reconciliation, rate limits or genuine traffic patterns. Validating a production payment integration is a separate piece of work, and it needs its own environment, access and agreed scope.
• Scale of the project. It is a single-product demo. Test counts show how much is checked, not how much of a real system is covered.

10. Lessons learned

1. Payment testing is state testing. The valuable assertions were about what the order looked like after an event, not about HTTP status codes.
2. Model the gateway's lifecycle, not your own guess. Marking an order failed on a failed attempt looks reasonable, and it contradicts how the gateway treats retries. A green suite cannot catch a wrong model; reading the gateway's documentation against the state machine can.
3. Middleware order is a security decision. The webhook defect was not in the cryptography. It was in what ran before it.
4. Do not assume SDK error shapes. The documented .message was not always there. Check what is actually thrown.
5. Conditional assertions can pass without checking anything. ORDER-003 asserts the amount only if (response.status() === 200). In CI the placeholder credentials make order creation fail (the server log shows Authentication failed), so that branch is not exercised. To check, I deliberately changed my scratch copy of the app to trust the client-supplied amount. The CI-safe suite still reported 34 of 34 passed. With real credentials the test does run (the Phase 3.3 report records it passing), but CI would not catch the regression. Marking the test as skipped when order creation fails makes CI report that honestly.
6. Order of checks matters. Verify the signature first, then deduplicate. Doing it the other way round, as the handler does today, leaks a small amount of information for no benefit.
7. A hardening idea, not a defect proven by a test. Both HMAC checks compare with ===. Node's crypto.timingSafeEqual is the usual choice for comparing secrets. I did not measure any timing difference, and a network timing attack on this app is unproven, but the change is small.