Backing Up Production Machines

Photo courtesy of Jaymis Loveday

Photo courtesy of Jaymis Loveday

Last time we talked about my thoughts on upgrading (or not upgrading) your production machines. Today, I’m going to talk about creating a safety net for them. Again, these are lessons I’ve learned over 25+ years of managing mission critical systems, so learn from my mistakes. 

Maintain a Current Clone of Your Drives

Hard drives will fail. Usually at the worst time. Like Palm Sunday morning. Yeah, that happened. Even SSDs, which are proving to be pretty dang reliable, will fail at some point. The only way to get back up and running quickly is with a full image backup. For all my mission critical production machines, I had a small hard drive with a full, bootable clone backup on it. I used to have several to manage, but after a while, I just bought a 1 TB pocket drive and partitioned it to back up 2-3 machines on one drive.

On the Mac, you can use a program like Super Duper for my favorite, Carbon Copy Cloner to keep an exact copy of the drive. On Windows, you can use Ghost or a similar program. If the main drive fails, or an upgrade breaks stuff, you can boot from the backup, and effectively unwind time. Any time I made significant changes to the machine, I would update the clone, but only after I verified the changes worked properly. If an update went south, we broke out the backup and rolled it back. 

For those of use that find Macs make better PCs than PCs do and need to backup your Bootcamp partitions, my IT guy told me about a cool program called WinClone. To use it, boot into the Mac OS, and run WinClone. It will compress and clone your Windows “drive” and save it as a file to another drive. Should Windows go south, you can simply restore the Windows drive and it will be like nothing never happened.

This is great for the OS and applications, but you can easily lose files like ProPresenter songs, for example. But I have a solution for that, too.

Use Dropbox for Libraries.

I set up a Dropbox account for my important show files. I wrote a full guide to this in an article called Back It Up: Presentation, but the gist is that you store (or maintain a cloned copy) of your show files, songs, templates and maybe media in a Dropbox folder that is automatically updated to the cloud. If you have to blow the drive out and restore from a clone, Dropbox will put your library files back. I don’t think I would ever run a production machine without Dropbox.

Maintain Incremental Backups

I really like Time Machine, especially lately. It’s a lot lower overhead than it once was and can really save your bacon if you delete or mess up a file. But, don’t run Time Machine backups during the service. Time Machine can be processor and disk intensive, and it’s highly possible that it will mess up your media playback. I prefer to keep Time Machine on an external drive so I can simply leave the drive turned off during the service. If you simply must have it on an internal drive, use Time Machine Scheduler to avoid service times. But honestly, an external drive is easier. Time Machine won’t even try to back up if the drive isn’t there, so it’s foolproof.

Clone Before Updating

Before we upgraded our production machines to a new OS, I went through and cloned all the drives. I did this so we could go back if something didn’t work. It’s a whole lot easier to simply restore the clone than it is to downgrade the OS, then Time Machine everything back. When it comes to upgrading, clones are your best friend. And honestly, drives are cheap enough now, you can easily maintain several versions of the clone if you want. Do one right before you update, then one right after. Small pocket drives from WD and Seagate are perfect for this task. At under $100 each, you can afford to have several for each machine. 

Hopefully this has helpful for you. Now if you’ll excuse me, I have to update some of my backups…

Today's post is brought to you by Elite Core Audio. Elite Core Audio features a premium USA built 16 channel personal monitor mixing system built for the rigors of the road. For Personal Mixing Systems, Snakes, and Cases, visit Elite Core Audio.

Upgrading Production Machines

Not what you want to see on Sunday morning...

Not what you want to see on Sunday morning...

We got to talking about this topic on one of the recent CTW episodes, and I thought it would be a good post. When I was on staff as a TD, I had a pretty strict policy regarding our production machines. Now that I’m working as an integrator, I dread the days following a major Mac OS update. That’s because I know I will soon be getting calls that start with, “We just upgraded all our iMacs to latest, greatest OS X… and fill in the blank software doesn’t work right anymore…” At that point, all I can say is, “Yeah, I usually don’t upgrade right away. Or ever, really. But you have a full image backup from before the upgrade, right?” Silence…

So in the interest of preventing said calls and emails, let me give you a few pointers on how to manage production machines. These are lessons I learned—many of the them the hard way—over 25+ years of managing production computers. It’s important to note that production machines are different from office computers. If an office computer goes down, you may not be able to get to your email for a little bit (except through your phone), but otherwise, nothing bad really happens. 

If a production machine goes down on Sunday morning, bad, bad things happen. If you upgrade on Friday and break something, the next 36 hours will be stressful. You really don’t want to be beta testing new software on the weekend. Here’s my guide to keeping your sanity with your computers. 

Don’t Upgrade Unless You Have To

Most of the time, you don’t have to upgrade your production machines. When I was at Coast Hills, most of my machines were running the latest version of 10.6 until early 2014 when we upgraded to 10.7. Why? Because it worked. If everything works on the OS you have, don’t upgrade it. I really like computers that start up and go to work every time without any fanfare. Avoiding unnecessary updates helps this.

My triggers for updating the OS are twofold: First, if some production software updates and introduces new features that I really need, and it requires a newer version of the OS, then I’ll update. Second, if the OS update introduces new features I really need, I’ll update to that version after the next version comes out. I like to stay about 1 version back at least. 

Don’t Upgrade Right Away

Computer code has become so complex it’s almost impossible to catch all the bugs and problems in a program before release, let alone an operating system. Apple is pretty good, but there is no way they can know how a new OS will affect every user. And many churches are still using older hardware and peripherals like audio or video interfaces, and a new OS can break the drivers for a while or forever. This is perhaps my #1 rule of OS updates: DON’T UPGRADE RIGHT AWAY. Let others beta test it first. 

I stay behind by at least one version because that allows time to get drivers and software updated and working solidly. Remember, we prefer reliable performance to fancy new features. 

Turn Off Auto Updates

One of my biggest pet peeves for production machines is auto updates. Windows used to be the worst at this, but now Apple has joined the fun in the last two versions. Unless you configure it properly, both OS’s will happily install new software or system updates all on their own and that can easily break things. Until I figured out how to turn it off, we kept having Windows kick up a message saying it would reboot the machine in 10 minutes to install updates every Sunday morning! Google it to learn how to turn that off. 

This does mean you should stay on top of a manual update routine, especially for security updates. But do that on Monday or Tuesday, then test everything thoroughly during the week to make sure it works. If you leave your computers on all the time, you really need to be careful of this. The last thing you want is to come in on Sunday only to find your software updated and no longer works right.

Verify All Software Will Work—Including Drivers

I just upgraded my studio Mac Mini to Mavericks, mainly because I installed a second screen and wanted to take advantage of the updated Spaces functionality. I waited so long because I wanted to be sure all my audio interface software would be good. I use this machine every week for CTW, and it has to work.

If you use an external peripheral that relies on driver software, be sure it’s approved for the OS you want to use before upgrading. I’ve heard from several people that they decided to upgrade their OS and now some critical external piece doesn’t work anymore. Remember, unless you have to upgrade, don’t. 

Those are a few suggestions for the upgrade process. If you take anything from this, it’s don’t upgrade. At least not unless you absolutely have to. Next time, I’ll give you some suggestions for creating a safety net for your computers. In the meantime, my friend Joel Smith has written a great guide on keeping ProPresenter machines working reliably. You should go read it. 20 Steps To Maximizing ProPresenter For Mac

Roland

Today's post is brought to you by myMix. myMix is an intuitive, easy-to-use personal monitor mixing and multi-track recording system that puts each user in control of their own mix! myMix features two line-level balanced 1/4" TRS outputs and one 1/8" (3.5mm) headphone output, the ability to store up to 20 named profiles on each station, 4-band fully parametric stereo output EQ recording of up to 18 tracks plus stereo on an SD card. Learn more at myMixaudio.com

Why Your Online Video Has To Be Good

Image courtesy of Steve Bowbrick

Image courtesy of Steve Bowbrick

Last time, we touched on the issue of live streaming or not. I am hearing more and more churches that want to live stream their services and I always ask why? I’m not going to rehash that here; go back and read the last post. Today, I want to focus on those that have made the decision to have an online video presence. And I want to tell you why it has to be good. 

The Competition is Fierce

North Point, Life Church, Church on the Move, Willow Creek, Saddleback and dozens more giga-churches stream their services every weekend. And they do a great job. So if someone wants to go online and watch a well-produced online church service, it’s not hard to find one. Now, you don’t necessarily have to go to that extreme; but you had best not simply throw a consumer grade camera up at the back of the room and post the resulting video online. Not only will no one watch it, there could be more harm than just a low view count. 

Who Watches Anyway?

There are several classes of people who watch church services online. The first class is your own congregation; they couldn’t make it that weekend for whatever reason, and wanted to see what the sermon was about. They are probably the most tolerant of poor video quality. But even then, if the shots are grainy, out of focus, or poorly cut together, or if the audio is poor, they won’t last long. It’s a lot harder for most people to just get up and leave a service in the middle if it’s not meeting their expectations; but closing a window online is easy. You have to do a good job to keep people engaged. 

Another class is the church shopper. We’re finding that more and more people check out a church’s website before visiting the first time. And this may seem like a great reason to post videos of your service. And it is. But only if those videos are good. If the video quality—technically or artistically—is subpar, you have probably lost the chance make a personal impression. Poor video tells people you don’t care enough about church to do this well. It tells them that your church is not worthy of their time. For this group, no video is better than bad video. With no video, they have to attend your church at least once to see if they like it. That gives you at least one shot at making it a great experience for them. 

Don’t Do What You Can’t Do Well

A lot of churches will justify poor online video by saying, “We’re just getting started, we don’t have to have it all dialed in at the beginning.” I would suggest another approach. Start small, but start well. A smaller church probably can’t afford to jump right into a 5 camera shoot with a jib and a full broadcast mix. That’s OK. But start off with a single, high-quality, manned camera and just do the sermon. Get that nailed. Make sure your lighting is great, the image quality is excellent and the audio is top-notch. This isn’t that hard, though it’s also not necessarily cheap. 

Later on, you can add additional cameras and a switcher for more visual interest. You can even start adding graphics. Only after that’s fully dialed in should you attempt the music set as that is easily the hardest. Your lighting will need to change, and you’ve got to figure out audio. It’s not impossible, but it is difficult to do well. If you’re a volunteer-run church, plan on spending some money to have professionals come in and help you get that set up. That doesn’t guarantee every weekend will sound amazing, but it’s a good first step. 

It’s a classic walk before you run situation. Start small, but start well. Don’t make the mistake of putting poor video online and thinking that just because it’s there people will watch it. Pastors (hopefully) don’t phone in their sermons because it’s not that big a deal (it is). We shouldn’t be doing a poor job on video simply because it’s online. If anything, being online should mean it’s more important because anyone can see it. Put your best foot forward and do a great job in everything you do. Hey, that sounds scriptural!

  Whatever you do, work heartily, as for the Lord and not for men, knowing that from the Lord you will receive the inheritance as your reward. You are serving the Lord Christ. Colossians 3:23-24

Today's post is brought to you by Elite Core Audio. Elite Core Audio features a premium USA built 16 channel personal monitor mixing system built for the rigors of the road. For Personal Mixing Systems, Snakes, and Cases, visit Elite Core Audio.