Operations · Internal

Disaster Recovery Plan

Document Version: 1.0 · Last Updated: June 24, 2026 · Owner: IMFUTECH Engineering
This document describes how IMFUTECH detects, responds to, and recovers from infrastructure failures affecting the CARE RCM platform.

Recovery Time Objective (RTO)

≤ 4 hours

Target time to restore service after a full instance failure

Recovery Point Objective (RPO)

≤ 24 hours

Maximum acceptable data loss, bounded by daily snapshot frequency

1. Scope and Purpose

This plan covers the production CARE RCM application, hosted on Amazon Web Services (AWS), including the application server, database, file storage, and supporting infrastructure. It defines the procedures, responsible parties, and recovery targets for restoring service after a disruptive event — ranging from a single-server failure to data corruption or accidental deletion.

This plan does not cover third-party service outages (e.g., AWS regional outages affecting multiple customers, clearinghouse downtime, payment processor outages) beyond documenting how CARE RCM responds to and communicates about them — those events are outside IMFUTECH's direct control to remediate.

2. Current Architecture

As of this document's last update, CARE RCM runs on a single-instance architecture:

Component	Configuration
Compute	Single Amazon EC2 instance (t3.medium), Ubuntu 24.04 LTS, region us-east-2 (Ohio)
Web/App server	Nginx + PHP-FPM 8.2, serving the CARE RCM Laravel application
Database	MySQL 8.0, self-hosted on the same EC2 instance
File storage	Amazon S3 (bucket: carercm-prod-files), versioning enabled, encrypted at rest (SSE-S3)
DNS	Amazon Route 53, hosted zone for carercm.com
SSL/TLS	Let's Encrypt certificates via Certbot, auto-renewing
Backup	Amazon EBS snapshots via Data Lifecycle Manager (DLM), daily, 7-day rolling retention
Monitoring	Amazon CloudWatch (CPU, memory, disk, instance status checks) with SNS email alerting

Known limitation: This is a single-instance architecture with no automatic failover. A failure of the EC2 instance or its EBS volume requires manual recovery from the most recent snapshot, as described in Section 6. See Section 9 for planned improvements that would reduce recovery time and eliminate single points of failure.

3. Identified Risk Scenarios

Scenario	Likelihood	Impact
EC2 instance failure or unresponsiveness	Low	High — full outage until restored
EBS volume corruption or failure	Low	High — data loss back to last snapshot
Accidental data deletion (application-level)	Medium	Medium — recoverable via snapshot restore
Disk space exhaustion	Medium	Medium — detected via CloudWatch alarm before failure
SSL certificate expiry	Low	Medium — auto-renewal in place, manual fallback documented
AWS regional outage (us-east-2)	Very Low	High — outside IMFUTECH control; see Section 9
S3 bucket misconfiguration / accidental deletion	Low	Medium — versioning enabled, reduces permanent loss risk

4. Backup Strategy

4.1 Database and application server

The EBS root volume (containing the operating system, application code, and MySQL database files) is backed up daily via AWS Data Lifecycle Manager. Snapshots are incremental, encrypted, and retained on a 7-day rolling basis. This means at any point in time, IMFUTECH can restore the entire server state — application, configuration, and database — to any point within the last 7 days.

4.2 Patient documents and uploaded files

All patient-facing file uploads (medical records, claim batch files, EDI files) are stored in Amazon S3 rather than on the EC2 instance's local disk, with versioning enabled. This means these files are durable independently of the EC2 instance's health — an instance failure does not put uploaded documents at risk, and accidental overwrites/deletes can be reversed via S3's version history.

4.3 What is NOT currently covered

Point-in-time database recovery between snapshots (e.g., recovering to a specific minute rather than the daily snapshot time) is not currently available — see Section 9
Cross-region backup replication is not currently configured — all snapshots and the S3 bucket reside in us-east-2 only

5. Detection and Alerting

CloudWatch alarms continuously monitor the production instance and notify the on-call team via email (Amazon SNS) when thresholds are breached:

Alarm	Trigger Condition	Purpose
Instance status check failed	Status check fails for 1 evaluation period	Detects instance unresponsiveness or underlying AWS hardware issues
High CPU utilization	>80% for 3 consecutive 5-minute periods	Detects sustained load issues before they cause downtime
High disk usage	>80% utilization	Prevents MySQL write failures and audit log interruption from disk exhaustion
High memory usage	>85% utilization	Detects memory pressure before out-of-memory failures occur

These alarms have been tested and confirmed functional, including end-to-end email delivery, as of this document's last update.

6. Recovery Procedures

6.1 Instance failure or unresponsiveness

Confirm the failure via the AWS EC2 console and CloudWatch alarm history
Attempt a simple instance reboot first (resolves transient issues without data loss)
If reboot fails, launch a new EC2 instance from the most recent EBS snapshot
Reassociate the Elastic IP address with the new instance (no DNS change required, since DNS points to the Elastic IP, not the instance directly)
Verify Nginx, PHP-FPM, and MySQL services start correctly on the restored instance
Run application health checks (login, database connectivity, S3 connectivity) before declaring recovery complete

6.2 Data corruption or accidental deletion

Identify the approximate time the corruption/deletion occurred
Restore the affected EBS volume from the most recent snapshot taken before that time
For S3 files specifically, use S3 version history to restore the prior version of the affected object(s) without needing a full volume restore
Communicate the data loss window (if any) to affected practices in accordance with the Business Associate Agreement breach notification terms, if PHI integrity is affected

6.3 SSL certificate failure

Let's Encrypt certificates auto-renew via Certbot's scheduled renewal task. If auto-renewal fails, certificates can be manually renewed via sudo certbot renew, with a 30-day advance expiry warning built into Certbot's standard behavior.

7. Roles and Responsibilities

Role	Responsibility
Incident lead	[NAME/ROLE] — coordinates response, makes go/no-go recovery decisions
Infrastructure operator	[NAME/ROLE] — executes AWS console/CLI recovery steps
Customer communication	[NAME/ROLE] — notifies affected practices of outages or data issues

8. Testing and Review Schedule

Backup verification: confirm at least one snapshot restore is tested successfully within the first 90 days of production launch, and at minimum annually thereafter
Alarm testing: CloudWatch alarms were tested end-to-end (including email delivery) during initial setup; re-test after any significant infrastructure change
Plan review: this document should be reviewed and updated at minimum every 6 months, or immediately after any architecture change (e.g., moving to RDS, adding a load balancer, multi-AZ deployment)

9. Planned Resilience Improvements

The current single-instance architecture was chosen for initial production launch simplicity and cost. As CARE RCM scales, the following improvements are under consideration to reduce RTO/RPO and eliminate single points of failure:

Migrate MySQL to Amazon RDS with automated backups, point-in-time recovery (down to the second, rather than daily snapshots), and optional Multi-AZ failover
Cross-region snapshot replication to protect against a full regional AWS outage in us-east-2
Application Load Balancer + Auto Scaling Group across multiple Availability Zones, removing the single-EC2-instance dependency entirely
Automated failover testing as part of a regular game-day exercise schedule

These are documented here as a roadmap, not a current commitment with a fixed date — IMFUTECH will update this section as these improvements are implemented.