[ใหม่] askmebet

Askmebet — Bug fixes to improve accuracy
This document outlines prioritized bug fixes, root cause analysis, test plans, and rollout strategies focused on improving accuracy across financial calculations, odds/price feeds, game outcomes, and reporting. The goal is to reduce reconciliation discrepancies, eliminate user visible errors (wrong payouts, incorrect balances), and restore trust through measurable quality improvements.
________________________________________
1. Scope & success criteria
• Reduce financial reconciliation variance to <0.1% of daily volume.
• Eliminate class A user complaints (wrong payout/balance) to <1 per 100k transactions.
• Achieve 99.9% parity between game engine outcomes and ledgered settlements.
• Automated tests covering fixed classes added to CI with no regressions.
________________________________________
2. High priority bug classes and fixes
1. Floating point rounding errors (balances, odds multipliers)
• Symptom: Small cent differences accumulate; payouts off by ±1–3 cents.
• Root cause: Use of IEEE 754 floats in business logic and presentation layers.
• Fix: Adopt integer arithmetic or fixed point libraries (store amounts in smallest currency unit, e.g., cents/units). Enforce decimal types in DB (DECIMAL/NUMERIC) for persistence. Add conversion utilities at I/O boundaries.
• Tests: Unit tests for arithmetic edge cases, cross language fixtures, and property based tests ensuring associativity and commutativity invariants.
2. Event ordering & eventual consistency (duplicate/late settlements)
• Symptom: Bets settled twice or out of order settlement after withdrawal.
• Root cause: Race conditions between matching engine, payment queue, and notification bus; lack of idempotency keys.
• Fix: Introduce global transaction idempotency_key for every financial intent; persist intent before external calls. Implement single writer for account balance updates (optimistic locking or serialized ledger writes). Use ordered, partitioned topics in the event bus keyed by account_id/game_id.
• Tests: Integration tests simulating concurrent plays and network partitions; reconciliation test harness verifying no duplicate settles.
3. Odds feed inconsistencies & stale caches
• Symptom: Users receive odds different from backend canonical values; slow market updates.
• Root cause: Multi layer caching (CDN, in memory gateways) with inconsistent invalidation; clocks skewed across servers.
• Fix: Move to authoritative odds service with versioned deltas; emit snapshot+delta model and sequence numbers. Implement cache invalidation using seq_no and short TTLs; use NTP/GPS time sync and reject out of order updates.
• Tests: End to end latency and correctness tests, simulated update bursts, and stale read detection.
4. Incorrect tax/fee calculation across jurisdictions
• Symptom: Wrong tax applied or edge cases unhandled for VIP/bonus flows.
• Root cause: Fragmented tax logic across services; hardcoded rates and missing exemptions.
• Fix: Centralize tax rules into a single tax service with a versioned rules engine and automated test suite. Expose API for other services. Add audit trail for every tax decision.
• Tests: Rule coverage matrix per jurisdiction, regression tests for promo combinations.
5. Timezone & daylight savings errors (settlement windows, tournament cutoffs)
• Symptom: Players credited for actions outside a tournament window or cutoffs wrong.
• Root cause: Mixed use of local time vs UTC and edge DST transitions.
• Fix: Normalize to UTC for all core events; store user timezone data separately for display only. Recompute windows using tz database where needed, with explicit DST rules.
• Tests: Boundary tests around DST rollovers and leap seconds; synthetic replay of historical events.
________________________________________
3. Testing & validation strategy
• Unit & integration tests for each fix; add to CI with coverage gates.
• Contract tests between services (tax, payment, odds) with schema and behavior verification.
• Synthetic replay harness: replay production like event streams to staging ledger and compare to golden outputs.
• Canary deployments: small percentage of traffic, shadow traffic runs for 24–72 hours with active reconciliation.
• Post deploy reconciliation: automated daily financial diff reports and anomaly detection (threshold alerts).
________________________________________
4. Instrumentation, monitoring & alerting
• Add message_id, correlation_id, and seq_no propagation to traces.
• Metrics: reconciliation_variance, duplicate_settlements_count, stale_odds_rate, tax_discrepancy_count. Create SLOs and p99 latency SLAs for tax/settlement APIs.
• Dashboards for real time reconciliation and pending DLQ items. Pager rules for critical thresholds (e.g., duplicate settlements > 1/10k).
________________________________________
5. Rollout & operational controls
• Branch → feature flag → internal QA → canary (1%) → ramp (10% → 50% → 100%) with automated health checks per phase.
• Pre signed rollback plan and DB migration rollback scripts. Migration steps must be idempotent and have backfill scripts (e.g., convert floats to integer cents).
• Post rollout audit window: manual review of DLQ and ledger reconciliation for first 7 days. Daily stakeholder summaries until metrics stable.
________________________________________
6. Long term prevention & engineering practices
• Adopt money safe libraries and code review checklist for all financial code.
• Enforce schema registry and consumer contract tests.
• Quarterly reconciliation drills and automated restore/replay tests.
• Post mortem culture: every discrepancy > threshold requires RCA, action items, and tracked closure.