Backing Up Production Machines

Photo courtesy of Jaymis Loveday

Photo courtesy of Jaymis Loveday

Last time we talked about my thoughts on upgrading (or not upgrading) your production machines. Today, I’m going to talk about creating a safety net for them. Again, these are lessons I’ve learned over 25+ years of managing mission critical systems, so learn from my mistakes. 

Maintain a Current Clone of Your Drives

Hard drives will fail. Usually at the worst time. Like Palm Sunday morning. Yeah, that happened. Even SSDs, which are proving to be pretty dang reliable, will fail at some point. The only way to get back up and running quickly is with a full image backup. For all my mission critical production machines, I had a small hard drive with a full, bootable clone backup on it. I used to have several to manage, but after a while, I just bought a 1 TB pocket drive and partitioned it to back up 2-3 machines on one drive.

On the Mac, you can use a program like Super Duper for my favorite, Carbon Copy Cloner to keep an exact copy of the drive. On Windows, you can use Ghost or a similar program. If the main drive fails, or an upgrade breaks stuff, you can boot from the backup, and effectively unwind time. Any time I made significant changes to the machine, I would update the clone, but only after I verified the changes worked properly. If an update went south, we broke out the backup and rolled it back. 

For those of use that find Macs make better PCs than PCs do and need to backup your Bootcamp partitions, my IT guy told me about a cool program called WinClone. To use it, boot into the Mac OS, and run WinClone. It will compress and clone your Windows “drive” and save it as a file to another drive. Should Windows go south, you can simply restore the Windows drive and it will be like nothing never happened.

This is great for the OS and applications, but you can easily lose files like ProPresenter songs, for example. But I have a solution for that, too.

Use Dropbox for Libraries.

I set up a Dropbox account for my important show files. I wrote a full guide to this in an article called Back It Up: Presentation, but the gist is that you store (or maintain a cloned copy) of your show files, songs, templates and maybe media in a Dropbox folder that is automatically updated to the cloud. If you have to blow the drive out and restore from a clone, Dropbox will put your library files back. I don’t think I would ever run a production machine without Dropbox.

Maintain Incremental Backups

I really like Time Machine, especially lately. It’s a lot lower overhead than it once was and can really save your bacon if you delete or mess up a file. But, don’t run Time Machine backups during the service. Time Machine can be processor and disk intensive, and it’s highly possible that it will mess up your media playback. I prefer to keep Time Machine on an external drive so I can simply leave the drive turned off during the service. If you simply must have it on an internal drive, use Time Machine Scheduler to avoid service times. But honestly, an external drive is easier. Time Machine won’t even try to back up if the drive isn’t there, so it’s foolproof.

Clone Before Updating

Before we upgraded our production machines to a new OS, I went through and cloned all the drives. I did this so we could go back if something didn’t work. It’s a whole lot easier to simply restore the clone than it is to downgrade the OS, then Time Machine everything back. When it comes to upgrading, clones are your best friend. And honestly, drives are cheap enough now, you can easily maintain several versions of the clone if you want. Do one right before you update, then one right after. Small pocket drives from WD and Seagate are perfect for this task. At under $100 each, you can afford to have several for each machine. 

Hopefully this has helpful for you. Now if you’ll excuse me, I have to update some of my backups…

Today's post is brought to you by Elite Core Audio. Elite Core Audio features a premium USA built 16 channel personal monitor mixing system built for the rigors of the road. For Personal Mixing Systems, Snakes, and Cases, visit Elite Core Audio.

Upgrading Production Machines

Not what you want to see on Sunday morning...

Not what you want to see on Sunday morning...

We got to talking about this topic on one of the recent CTW episodes, and I thought it would be a good post. When I was on staff as a TD, I had a pretty strict policy regarding our production machines. Now that I’m working as an integrator, I dread the days following a major Mac OS update. That’s because I know I will soon be getting calls that start with, “We just upgraded all our iMacs to latest, greatest OS X… and fill in the blank software doesn’t work right anymore…” At that point, all I can say is, “Yeah, I usually don’t upgrade right away. Or ever, really. But you have a full image backup from before the upgrade, right?” Silence…

So in the interest of preventing said calls and emails, let me give you a few pointers on how to manage production machines. These are lessons I learned—many of the them the hard way—over 25+ years of managing production computers. It’s important to note that production machines are different from office computers. If an office computer goes down, you may not be able to get to your email for a little bit (except through your phone), but otherwise, nothing bad really happens. 

If a production machine goes down on Sunday morning, bad, bad things happen. If you upgrade on Friday and break something, the next 36 hours will be stressful. You really don’t want to be beta testing new software on the weekend. Here’s my guide to keeping your sanity with your computers. 

Don’t Upgrade Unless You Have To

Most of the time, you don’t have to upgrade your production machines. When I was at Coast Hills, most of my machines were running the latest version of 10.6 until early 2014 when we upgraded to 10.7. Why? Because it worked. If everything works on the OS you have, don’t upgrade it. I really like computers that start up and go to work every time without any fanfare. Avoiding unnecessary updates helps this.

My triggers for updating the OS are twofold: First, if some production software updates and introduces new features that I really need, and it requires a newer version of the OS, then I’ll update. Second, if the OS update introduces new features I really need, I’ll update to that version after the next version comes out. I like to stay about 1 version back at least. 

Don’t Upgrade Right Away

Computer code has become so complex it’s almost impossible to catch all the bugs and problems in a program before release, let alone an operating system. Apple is pretty good, but there is no way they can know how a new OS will affect every user. And many churches are still using older hardware and peripherals like audio or video interfaces, and a new OS can break the drivers for a while or forever. This is perhaps my #1 rule of OS updates: DON’T UPGRADE RIGHT AWAY. Let others beta test it first. 

I stay behind by at least one version because that allows time to get drivers and software updated and working solidly. Remember, we prefer reliable performance to fancy new features. 

Turn Off Auto Updates

One of my biggest pet peeves for production machines is auto updates. Windows used to be the worst at this, but now Apple has joined the fun in the last two versions. Unless you configure it properly, both OS’s will happily install new software or system updates all on their own and that can easily break things. Until I figured out how to turn it off, we kept having Windows kick up a message saying it would reboot the machine in 10 minutes to install updates every Sunday morning! Google it to learn how to turn that off. 

This does mean you should stay on top of a manual update routine, especially for security updates. But do that on Monday or Tuesday, then test everything thoroughly during the week to make sure it works. If you leave your computers on all the time, you really need to be careful of this. The last thing you want is to come in on Sunday only to find your software updated and no longer works right.

Verify All Software Will Work—Including Drivers

I just upgraded my studio Mac Mini to Mavericks, mainly because I installed a second screen and wanted to take advantage of the updated Spaces functionality. I waited so long because I wanted to be sure all my audio interface software would be good. I use this machine every week for CTW, and it has to work.

If you use an external peripheral that relies on driver software, be sure it’s approved for the OS you want to use before upgrading. I’ve heard from several people that they decided to upgrade their OS and now some critical external piece doesn’t work anymore. Remember, unless you have to upgrade, don’t. 

Those are a few suggestions for the upgrade process. If you take anything from this, it’s don’t upgrade. At least not unless you absolutely have to. Next time, I’ll give you some suggestions for creating a safety net for your computers. In the meantime, my friend Joel Smith has written a great guide on keeping ProPresenter machines working reliably. You should go read it. 20 Steps To Maximizing ProPresenter For Mac

Roland

Today's post is brought to you by myMix. myMix is an intuitive, easy-to-use personal monitor mixing and multi-track recording system that puts each user in control of their own mix! myMix features two line-level balanced 1/4" TRS outputs and one 1/8" (3.5mm) headphone output, the ability to store up to 20 named profiles on each station, 4-band fully parametric stereo output EQ recording of up to 18 tracks plus stereo on an SD card. Learn more at myMixaudio.com