Admin Operations Playbook

20 operational procedures for diagnosing and resolving common issues.

This page is for internal operations staff only. It is not intended for merchants or partners.

1. Merchant can't log in

When to use: A merchant reports they cannot access their dashboard after entering their email.

Check the LoginLock table for the merchant's email. If a lock exists with locked_until in the future, the account is temporarily locked due to failed attempts.
If locked, either wait for the lock to expire (max 7 days) or delete the lock row manually in the database.
Check the Session table for expired sessions. If the merchant has no active session, they need to re-authenticate.
Verify the merchant's invite has been accepted. Check MerchantInvite for accepted_at. If null, the merchant never finished onboarding.
Verify Sequence authentication is working by checking the Sequence dashboard for the merchant's email. If the identity does not exist, the embedded wallet was never provisioned.
Check the ErrorLog table for recent auth errors related to the merchant's partner and email.

2. Payment stuck in pending

When to use: A payment has proof_status = attached but has not transitioned to verified for more than 10 minutes.

Steps

Look up the request by ID in the Request table. Confirm proof_status = attached and note the tx_hash.
Check the transaction on the appropriate block explorer (polygonscan.com, basescan.org, etc.). Verify the transaction exists and has been verified (not pending in the mempool).
Verify the transaction details match the payment request: correct recipient address, correct USDC contract, correct amount.
Check the Trails API status. Look at recent ErrorLog entries for rpc_error or rpc_unavailable codes. An RPC outage will delay verification.
If the transaction is valid but Trails has not picked it up, trigger a manual re-verification through the admin API: POST /api/admin/trails-reverify with the request ID.
If the transaction does not match (wrong amount, wrong recipient), the proof will eventually fail. Inform the merchant that the customer sent to the wrong address or with the wrong amount.

3. FX rates not refreshing

When to use: Merchants report that fiat amounts on payment pages look stale, or the admin dashboard shows the last FX refresh was hours ago.

Steps

Check the FxRate table for the partner. Look at updated_at to see when rates were last refreshed.
Try a manual refresh: POST /api/admin/fx-refresh with the partner ID. If it fails, check the response error.
Check the ErrorLog for fx_unavailable errors. This indicates the upstream FX rate provider is down.
If the FX provider is down, cached rates will be used as a fallback (5-minute stale window). Inform the partner that displayed amounts may be slightly off until the provider recovers.
Verify the partner's configured currency is in the list of 28 supported currencies. An unsupported currency will fail FX lookup.

4. Webhook delivery failing

When to use: A partner reports they are not receiving webhook notifications, or the WebhookDelivery table shows repeated failures.

Steps

Check the WebhookDelivery table for the partner. Filter by status = failed and look at the response_status and error_message columns.
If the response status is 4xx, the partner's endpoint is rejecting the delivery. Common causes: incorrect URL path, authentication required on their endpoint, or signature verification failing.
If the response status is 5xx, the partner's server is erroring. This is on their side.
If the error is a connection timeout or DNS failure, the partner's endpoint is unreachable. Verify the configured URL is correct and accessible from the internet.
Verify the webhook URL is HTTPS. HTTP URLs are rejected at configuration time but older records may exist.
Check that the partner's webhook secret matches what they are using for verification. Regenerating the secret invalidates the old one.
Manually trigger a test delivery from the admin portal to confirm the current configuration works.

5. Merchant requesting account disable

When to use: A merchant requests their account be deactivated.

Steps

Verify the request came from the merchant's registered email address.
Check for any pending payments (proof_status = attached) and wait for them to resolve before disabling.
Check for active recurring plans. All plans should be terminated before account deactivation.
Disable auto-forward if enabled, to prevent future forwarding attempts.
Set the merchant's disabled flag to true in the database. This prevents new logins and new payment requests.
Inform the partner that the merchant has been disabled. The partner may need to update their own records.
Do not delete payment history or wallet data — retain for reconciliation and audit purposes per the data retention policy.

6. Partner requesting new chain

When to use: A partner wants to enable a new blockchain network for their merchants.

Steps

Confirm the requested chain is in the list of supported chains: Polygon (137), Base (8453), Ethereum (1), Optimism (10), Arbitrum (42161).
If the chain is supported, update the partner's chain configuration in the admin portal.
Verify the USDC contract address for the chain is correctly configured in the platform's chain registry.
Run a test payment on the new chain to confirm Trails can verify transactions on it.
If the chain is not in the supported list, inform the partner that adding new chains requires engineering work (RPC integration, USDC contract verification, Trails support).

7. Reconciliation flagged discrepancy

When to use: The reconciliation cron job flags a mismatch between JoPay's records and on-chain balances.

Steps

Check the ReconciliationRun table for the latest run. Note the discrepancy_amount and merchant_id.
Compare the merchant's expected balance (sum of verified payments minus withdrawals) with the actual on-chain USDC balance at their wallet address.
Common causes of discrepancy: the merchant received USDC from a source outside JoPay (direct transfer), the merchant sent USDC outside JoPay (manual transfer), or an auto-forward partially failed.
Check ErrorLog for auto-forward failures for this merchant. A failed forward would leave the balance higher than expected.
If the discrepancy is from external transfers, note it in the reconciliation record as explained. No action needed.
If the discrepancy cannot be explained, escalate to engineering for a detailed transaction trace.

8. Recurring charges not firing

When to use: A merchant reports that scheduled recurring charges are not executing on time.

Steps

Check the RecurringPlan table for the plan ID. Verify status = active and that next_charge_at is in the past (meaning it should have fired).
Check the process-recurring cron job logs. Look for errors during the last execution window.
Verify the customer's embedded wallet has sufficient USDC balance. If the balance is insufficient, the charge will fail silently and retry on the next cycle.
Check that the customer has an embedded wallet (not an external wallet). Recurring charges require embedded wallets for server-initiated transactions.
Check the RecurringCharge table for recent charge attempts. Look at status and error_message for failed attempts.
If the cron job itself is not running, check the Vercel cron configuration in vercel.json to confirm the schedule is correct.

9. Auto-forward failing

When to use: A merchant has auto-forward enabled but funds are not being forwarded to their withdrawal address after verified payments.

Steps

Check the merchant's settings: auto_forward_enabled = true and withdraw_address is set and valid.
Verify the merchant has an embedded wallet. Auto-forward requires the server to sign transactions via Sequence.
Check the ErrorLog for auto-forward errors for this merchant. Common errors: insufficient gas (the wallet needs a small amount of native token for gas), RPC unavailable, or Sequence API errors.
Check the merchant's wallet balance for native gas token (MATIC on Polygon, ETH on Ethereum/Base/Optimism/Arbitrum). If the gas balance is zero, the forward transaction cannot be submitted.
If gas is the issue, a small amount of native token needs to be sent to the merchant's embedded wallet address.
Trigger a manual forward attempt through the admin API and monitor the response.

10. High error rate in ErrorLogs

When to use: Monitoring shows an elevated rate of errors in the ErrorLog table.

Steps

Query the ErrorLog table grouped by error_code for the last hour. Identify which error code is spiking.
If the spike is rpc_error or rpc_unavailable: the blockchain RPC provider is having issues. Check their status page. On-chain verification will be delayed but will resume automatically.
If the spike is rate_limited: a specific IP or merchant is hitting endpoints too aggressively. Check the rate limit tables for the top offenders.
If the spike is api_error (500s): check server logs for unhandled exceptions. This may indicate a code bug or infrastructure issue (database connection failures, memory pressure).
If the spike is fx_unavailable: the FX rate provider is down. Cached rates will be used as fallback. Monitor for recovery.
For sustained spikes, check the database connection pool. Pool exhaustion causes cascading 500 errors across all endpoints.

11. Purge cron failing

When to use: The data retention purge cron job reports errors or the data retention tables are growing beyond expected sizes.

Steps

Check the Vercel function logs for the purge cron endpoint. Look for timeout errors (the function may be hitting the execution time limit).
Check the batch sizes. The purge job deletes records in batches to avoid long-running transactions. If the table has grown very large, the batch size may need to be increased temporarily.
Verify the cron schedule in vercel.json. The purge should run daily.
Run a manual count of records past retention for each table: Events (90d), Sessions (14d), CSRF tokens (2d), Rate limit buckets (7d), Login locks (7d).
If the table is very large, consider running the purge manually with a larger batch size or multiple runs.
Check for database locks. A long-running query elsewhere could block the purge job's delete operations.

12. Rate limit table growing

When to use: The RateLimitBucket table row count is growing faster than expected or impacting database performance.

Steps

Check the purge cron is running and successfully deleting expired buckets (those older than 7 days).
Query the table for the top IP hashes by bucket count. A single abusive IP can create thousands of buckets.
If a single IP is dominating, consider blocking it at the infrastructure level (CDN/WAF) rather than relying on application-level rate limiting.
Check if a bot or crawler is aggressively hitting the pay page. The pay page rate limit is 60/min per IP, but a distributed bot could still create many buckets.
If the table is very large, run a manual cleanup: delete all rows where expires_at < NOW().

13. Partner webhook returning 4xx

When to use: Webhook deliveries to a partner are consistently failing with 4xx status codes.

Steps

Check the WebhookDelivery table for the specific HTTP status code. Common codes: 401 (auth required), 403 (forbidden), 404 (wrong path), 405 (method not allowed).
For 401/403: The partner's endpoint is requiring authentication that JoPay's webhook delivery does not provide. Advise the partner to allow unauthenticated POST requests to their webhook path and use HMAC signature verification instead.
For 404: The configured URL path is wrong. Verify the URL with the partner and update in the admin portal.
For 405: The partner's server is not accepting POST requests at that path. Advise them to configure their endpoint to accept POST.
Send a test delivery from the admin portal to verify after the partner makes changes.

14. Merchant wallet balance wrong

When to use: A merchant reports their dashboard balance does not match their expectations.

Steps

Check the on-chain USDC balance of the merchant's wallet address on the block explorer. This is the source of truth.
Compare to the cached balance in JoPay. The balance cache has a 30-second TTL with a 5-minute stale fallback. The dashboard may show a slightly stale value.
If the on-chain balance matches the dashboard: the merchant's expectation is incorrect. Help them understand their transaction history.
If the on-chain balance does not match the dashboard: the cache may be stale. Wait for the next refresh cycle (30 seconds) or trigger a manual balance check.
Check for recent auto-forward transactions. Auto-forwarded payments will reduce the balance immediately after verification.
Check if the merchant has sent funds outside JoPay (direct wallet transfer). These transactions are not tracked by JoPay and will cause a balance discrepancy in the dashboard.

15. Duplicate payment reports

When to use: A merchant or partner reports that a payment appears to have been counted twice.

Steps

Check the Request table for the reported amount and time range. Look for two requests with the same amount to the same merchant.
Check the tx_hash on both requests. If they have different transaction hashes, these are two separate legitimate payments. The merchant may have created two requests and the customer paid both.
If both requests reference the same tx_hash, this is a potential bug. A single transaction should only be matched to one request. Escalate to engineering.
Check the webhook deliveries. If the partner received duplicate webhook notifications for the same delivery_id, they should be deduplicating on their end. If they received different delivery_id values for the same payment, escalate.

16. CSRF validation failing

When to use: Merchants report that form submissions (settings updates, payment creation) fail with CSRF errors.

Steps

CSRF tokens expire after 2 days. If the merchant left a page open for a long time without refreshing, the token will have expired. Refreshing the page generates a new token.
Check if the merchant is using multiple browser tabs. Each tab gets its own CSRF token, and submitting from a stale tab will fail.
Verify the SameSite cookie attribute is set to Lax. If this was accidentally changed, cross-origin requests could strip the CSRF cookie.
Check if a browser extension or proxy is stripping cookies from requests.
If the issue is widespread (affecting many merchants), check the CSRF token generation code and the purge cron. If tokens are being purged too aggressively, valid tokens may be deleted.

17. Session expired prematurely

When to use: A merchant reports being logged out unexpectedly before their session should have expired.

Steps

Check the Session table for the merchant's most recent sessions. Compare created_at and expires_at to confirm the expected duration.
Sessions have a 14-day retention period. If the session was created more than 14 days ago, it was purged by the retention cron and the logout is expected.
Check if the merchant cleared their browser cookies. This removes the session cookie, forcing re-authentication.
Check if the purge cron accidentally deleted active sessions. The purge should only delete sessions past their expiration, not active ones. If active sessions are being purged, the retention window is misconfigured.
Verify the session cookie's max-age or expires attribute matches the intended session duration.

18. Admin audit log review

When to use: Periodic review of admin actions for compliance, or when investigating a specific incident.

Steps

Query the ErrorLog and admin action logs for the review period. Filter by admin user if investigating a specific actor.
Review all partner configuration changes: webhook URL updates, FX refreshes, chain changes, merchant invites.
Review all merchant disables/enables and the reason for each.
Check for unusual patterns: actions at odd hours, rapid successive changes, bulk operations.
Verify that all admin actions were performed by authorized personnel. Cross-reference with the admin access list.
Document findings and file the audit report per the compliance schedule.

19. Partner onboarding checklist verification

When to use: A new partner has been onboarded and you need to verify the setup is complete before going live.

Steps

Verify the partner record exists in the database with correct: name, slug, display currency, chain, asset (USDC or EURC), and branding configuration.
Verify API credentials have been generated and securely delivered to the partner.
Verify the webhook URL is configured and a test delivery succeeds with a 200 response.
Verify the partner can create a merchant invite and the invite email is delivered.
Complete a test payment end-to-end: create request, open payment page, send test USDC, verify proof, confirm webhook delivery.
Verify the partner's minimum fiat amount per request (min_fiat_per_request) is set correctly.
Verify FX rates are loading for the partner's configured currency.
Confirm rate limits are appropriate for the partner's expected volume.

20. Database connection pool exhaustion

When to use: Widespread 500 errors across all endpoints, with database connection timeout errors in the logs.

Steps

Check the database provider's dashboard (e.g., Supabase, PlanetScale) for active connection count. Compare to the configured pool size.
Check for long-running queries. A query that holds a connection for minutes can exhaust the pool. Look at the database's active query list.
Check if the purge cron is running a large delete operation without batching. Bulk deletes can hold connections for extended periods.
Check if there was a traffic spike. Sudden increases in request volume can exhaust the pool if the pool size is too small.
As an immediate mitigation, restart the application to release all connections. This will terminate in-flight requests but free the pool.
If the issue recurs, increase the connection pool size in the database configuration or optimize the slow queries identified in step 2.
Verify that Prisma is configured with appropriate pool timeout and connection limit values in the database URL.