Resilience and High Availability

CrowdHandler's infrastructure is designed for continuous availability and uptime. We employ the following resilience and fault-tolerance mechanisms:

AWS Availability Zones

We utilize multiple availability zones in each AWS region to provide redundancy at the data center level. This protects against failures that could impact an entire zone.

Database Multi-AZ Deployments

Our MySQL RDS databases are deployed in multi-AZ mode across 3 availability zones. This provides synchronous replication of data with automatic failover in the event of database instance impairments or AZ disruptions.

AWS Region Replication

We replicate all infrastructure and data from our primary US-East-1 region to US-East-2. This includes:

  • Database snapshots - We take hourly snapshots of production DBs which are then replicated to US-East-2. We can restore from a snapshot in the event of a region-level disaster.
  • S3 bucket replication - All data in S3 production buckets is asynchronously replicated to US-East-2 buckets.
  • AMI backups - Images of EC2 servers (where used) are backed up to US-East-2 in case we need to launch replacements.
  • Route53 health checks - We use Route53 health checks to monitor endpoints in both US-East-1 and US-East-2. Traffic can quickly be routed to the secondary region if needed.

Disaster Recovery Testing

We conduct annual disaster recovery testing to ensure we can redirect all traffic and cut over to US-East-2 in the event of a significant outage in US-East-1. This testing involves:

  1. Simulating an outage in US-East-1 that would require failover
  2. Promoting standby databases in US-East-2 to become primary
  3. Updating DNS or application origins to route all traffic to US-East-2
  4. Conducting tests to validate application availability and database access
  5. Switching back to US-East-1 once testing is complete

CrowdHandler's resilience measures provide safeguards against infrastructure failures, software disruptions and regional outages. Fault-tolerant systems, replication of data across regions and regular disaster recovery testing give us confidence in our ability to restore services with minimal downtime even after catastrophic events. Our availability architecture is designed for continuous uptime and a seamless customer experience.