In a perfect world, my job would rarely require emergency support for critical failures. These events cause downtime and expose real weaknesses in your disaster recovery plans, and that is something I would never wish on anyone.
If you’ve been following our Managing GreenArrow series so far, you hopefully have regular management schedule dialed in and are feeling confident in the tools (or you at least have a good idea of what tools are in your toolbox, should you need them). In this blog post, we are going to look at three advanced tools that have foundations in the previous two blog posts, so if you haven’t already read them, I’d recommend reading the first and second in this series before continuing here.
Fair warning: I’m a “bad news first” and “rip the bandaid off fast” kind of guy.
Maybe it was an unexpected hardware failure or a weak ‘root’ user security policy that allowed a malicious 3rd party to compromise the integrity of your server beyond repair (or one of probably hundreds of other scenarios).
Whatever the reason, your server just went offline, and all of your data is gone or corrupt, and you don’t have any off-site backups.
All of your Mailing Lists and subscriber history, campaigns, and stats are lost.
Every click and open tracking link you’ve ever sent is now broken.
Any bounces, spam complaints, or unsubscribes that come back in will not function.
After you’ve caught your breath, it is time to reinstall GreenArrow and get back online. If you don’t have your license and repository key from when you first purchased GreenArrow, you’ll need to contact us to get those before you can proceed.
There is no way around it: this is truly a worst-case option. But if there is any silver lining, an event like this often marks the start of a more robust disaster recovery plan.
Restoring a Backup
Now think about how different the above scenario would be if you have been taking backups. Yes, there is still some downtime, but the total impact is far less when you can restore from a backup.
The actual amount of time it takes to restore will depend on the amount of data being restored and the performance of the server hardware you’re restoring to, but assuming everything goes smoothly, it typically takes about three hours. (Note: that is only the time it takes to restore GreenArrow; it does not include the time it takes to spin up a new server and copy the backup files to it.)
You’re also going to lose some data. If your server went offline at 8:00 PM and your most recent backup is from 3:00 AM, that is seventeen hours of data you’ve lost. The campaigns that you sent during those 17 hours between the backup and restore will also have broken links, and spam complaints and bounces that come in will not result in deactivation.
Now, this is a far better option than a clean install, but it is certainly not ideal.
Before Disaster Strikes: Migrating GreenArrow
If you’ve implemented the monitoring and regular health checks we discussed in the prior blog posts, then you’ve put yourself in a position where the best-case scenario may be possible, and you now have plenty of warning before you need to move GreenArrow to a new server.
Maybe you see early warning signs of hardware failure, or you’ve just grown enough that you’re hitting some performance bottlenecks. Regardless of “why” you need to move, actively monitoring your server will provide the flexibility to plan a migration to new hardware.
The primary benefit of migrating is that you are proactively managing the project outside of a critical malfunction. You have the time to put together a migration plan while everything is still running, and then execute that plan with minimal downtime.
Most migrations will require somewhere between 20 and 60 minutes of downtime, but I’ve done some with only 10 minutes of downtime.
Migrations also give you a replica of GreenArrow on your new system, with no data loss.
This is all-around the best option.
“An ounce of prevention…”
Given the worst-case scenario and the two alternatives, there is some serious incentive to keep up with regular monitoring and maintenance.
Even with the best planning and monitoring, you can never eliminate the possible need to restore from a backup, but every step you take toward this goal will move you farther away from a clean reinstall and closer to long-term success with GreenArrow.