When I started doing live production in my church 20+ years ago, every piece of equipment we used was hardware-based. When we turned it on each weekend, it worked pretty much every time; unless we had a hardware failure. A hardware failure meant you were out of business until it was physically fixed (unless you had a backup). The good news of that system was that generally speaking things were pretty much the same week to week. You wouldn’t come in and find that a software update broke something. Today, it’s a different story.
If you have a smartphone, chances are you can’t use it for 7 minutes before an update badge appears in your app store. It is a rare day when I can turn on my computer and don’t have some app telling me there is a new version available for download. Most of the time when I’m on my laptop, I’ll just hit “Update” and let it do it's thing. On a production machine, it’s a different story.
If you’ve been doing this for any length of time, you’ve been burned by an update that breaks something. Chances are, it’s happened just before a service and you were left scrambling for a fix. We all know not to update our audio consoles on show day (we do know that, right?), but sometimes seemingly innocuous updates on presentation, video and lighting computers can put us out of business.
To be fair, it’s hard for developers to imagine every possible software/hardware interaction scenario. Which is why it’s incumbent on us to make sure we don’t put ourselves at risk for problems. Here are some things I’ve learned (and am still learning) that help avoid taking a critical system down over the weekend.
Turn Off Automatic Updates
Almost every application has an option to check for updates at startup. Turn this off. It’s way too tempting to just click through the update dialog box on Sunday morning, only to find out something is broken. I turn off all application auto-update and update check features as well as the OS-level updates. Once I get a system stable, I don’t want an app upsetting the apple cart and breaking something, especially on the weekend.
Most times, updates are a good thing, but I like to do them in a controlled manner during the week when I have time to test everything just to make sure nothing broke. I don’t always live by this credo, and when I don’t, I am often sorry for it. To keep things running smoothly, I will check for updates manually during the week, evaluate if the update is necessary or needed, then update with caution.
Keep Good Backups
I really like the Mac App store. It makes it so easy to manage all my app purchases, licenses and updates. However, it makes it really difficult to roll back to a previous version if there is a problem. And if you’re not using the App Store, chances are, your applications are doing in-place updates (downloading and installing the updates for you). Both of those update processes delete the old versions of the software (sometimes they just move old versions to the trash, so check there first).
Having an up-to-date backup of your system will help ensure you can roll back if necessary. Time Machine is actually quite good at this; you can enter Time Machine and restore a previous version of the app (before the upgrade) as well as any pref files or application support files that may have changed (usually in user/Library/Application Support/app name or developer and user/Library/Preferences).
Don’t Update on Show Day
This seems very obvious, but again, it’s so easy to click “OK” on that dialog that tells you there is a new version available that has just a few minor improvements. Sometimes updates don’t actually break anything, but may still effect your workflow. Occasionally keyboard shortcuts change, or a dialog updates which could throw off a volunteer or break a script.
Make sure you update during the week, giving yourself ample time to test everything if the machine is mission critical. And by during the week, I don’t mean at 4:55 PM on Friday. If it’s not a mission critical system, the rules are a little different.
I run LAMA on a Mac at FOH for RTA and SPL monitoring. But I don’t have to use it, so if something breaks, the show goes on. However, we have another Mac at FOH that runs Mixx (for walk in/walk out), Wireless Workbench and the Roland RCS software. Those have to work every weekend so I don’t update those applications (or that OS) lightly.
We’re still running older versions of Mac OSX on many of our FOH machines because they are working just fine right now. At some point we will have to update them, but I won’t do it until I have to.
Updates can be a great thing. The fact that we’re working on computers means that new features can be enabled by downloading software. However, things can also go horribly wrong if we’re not careful. Be safe, update with caution and give yourself plenty of time to fix things if they go wrong.