“This will take a while.” Great…

“This will take a while.” Great…

It’s that time of year again. The time of year when a young tech directors thoughts turn to…software upgrades. With the recent release of ProPresenter 7, the internets are all abuzz with discussions about upgrading software. Now, this article isn’t about ProPresenter per se. I have no experience with it and no particular thoughts about it one way or another. It’s simply a catalyst for me thinking about upgrades in general.

On the bookFace groups I follow, I often see questions about upgrading software stuff. Having done my fair share of it, I thought I would share some observations and ideas on what I consider best practices.

Not on the Weekend
The first rule for me of upgrading is to never, no not ever upgrade on the weekend. I turn off all automatic updates on all my production machines and make sure they are not even checking on Saturday or Sunday. In fact, it’s a good argument for production machines being on their own network, air-gapped to the outside worlds. Windows 10 can’t install its next group of productivity enhancing features providing unparalleled levels of productivity if it can’t talk to the servers.

You never know when an upgrade won’t work, or will break something else. The last thing you want to be doing during what was supposed to be soundcheck is tracking down new drivers and updates so the teleprompter…er…confidence monitor will work.

I personally put Wednesday as my hard stop for updates. That was the last day I would attempt software or firmware updates. If I ran out of time, I did not do it on Thursday, nor Friday. Why? Well, one time I installed firmware and it bricked the hardware. I had to have new hardware overnighted to get it working. Well, it would have been overnight except I found myself with bricked hardware on Friday afternoon. We had missed the shipping cutoff for the day and it wasn’t going out until Monday. That was an uncomfortable weekend.

If you break something on Wednesday, you have all day Thursday to get something new moving your way and there’s a good chance you can have something delivered Friday or maybe even Saturday. Now, to be fair, that rarely happens. But it only takes one time.

If it ain’t Broke…
I generally take the approach to software updates that if what I’m using isn’t broken, I don’t try to “fix” it. Over the years, I’ve had updates or “upgrades” cause more problems than they solved—especially when there weren’t problems in the first place. Sometimes it’s not even the software in question that causes the problem.

You might, for example, find that to upgrade to the latest and greatest version of software X that you need to update the OS. No problem, you think, it’s time to do that anyway. So, you update the OS and X turns out to be super snazzy. You play around with new features of X and it’s great. Then on Sunday you show up and find out that Software Y now needs to be updated, and you need it for the weekend. Worse, once you update Y, you discover the drivers for hardware Z no longer work. Three hours later, you’re finally back in business; right about the time the pastor is starting his message…

It’s even worse when you upgrade and then discover your hardware no longer works with the new software. I’ve seen this with video and audio interfaces. Or maybe the drivers simply haven’t caught up yet. This is a great way to take an entire system off line if you’re not careful.

If You Don’t Need It

I always read over the feature upgrades or bug fixes in software updates on my production machines. If there isn’t a really compelling reason to update the software or firmware, I don’t do it. Sometimes an update will fix a bug I’ve never experienced. And since I’ve never seen an update that doesn’t fix bugs introduced in the last “bug fix”, it seems each update introduces other bugs that will then be fixed in the next release. If I’m not dealing with those bugs now, I don’t want potential new ones unless there’s a feature I can really use.

For example, I’ve noticed that newer versions of Mac OS X have really jacked up wired networking. I had nothing but trouble with it using my (former) new MacBook Pro running Mojave. It’s (former) because I just sold it and went to a Lenovo laptop that cost 25% as much and actually works with Ethernet. Since much of my work revolves around connecting to hardware via Ethernet, that cool new MBP wasn’t much of an upgrade. In fact, my 2013 running High Sierra is much more reliable with Ethernet. Why High Sierra? See point #2.

You really have to ask if those new features outweigh the potential risks of breaking something that’s working. Back when I was a TD, our ProPresenter machine was 5 years old, running 4 versions back of the Mac OS and I think two version back on ProPresenter. Why? Because it worked every weekend and the new versions of all that software offered nothing compelling enough to risk changing it. In fact, I remember updating ProPresenter one week and finding a bug that really caused some significant issues. They fixed the following week, but that weekend wasn’t fun. I’m not sure I ever upgraded any of that stuff again.

Again, this is not a dig on ProPresenter in particular. There are simply too many combinations of hardware and software in the world for any developer to ensure their software is going to work without issue with all of them. Renewed Vision is generally really good at getting their software up and running again, but you have to give them a little time. And you may not have that time.

Even Hardware Upgrades are Scary

As I’m writing this out, I’m remembering a time with a client of mine that found some very interesting issues with upgrading. They were in an area that had very unstable power delivery. I believe their incoming power varied from 57 to 64 Hz or something like that. Most equipment can handle that, but it was causing problems with lip sync between the ProVideo Player/Blackmagic Ultrastudio and the switcher. As there sermon went on, the lip sync would drift, but it drifted both forward and back as the power drifted.

So, we decided to genlock the Ultrastudio. That fixed the lip sync problem. However, it introduced a really weird AM modulated digital noise into the audio signal. My client spent an inordinate (or normal, really) amount of time on the phone with Blackmagic and it was discovered that the genlock input leaks voltage to the audio output stage. This caused the amplitude modulated distortion.

The ultimate solution was to use a Radial USB audio DI to send audio into the mixer and only use the Ultrastudio for video. Now, that’s a border case that you don’t come across every day—which is one reason it took so long to solve. But it illustrates the point that upgrading one thing can have a cascading effect on other things that you hadn’t considered; and that goes for hardware as well as software.

Which is why we should approach all upgrades with care. If at all possible, have a test bed for any new upgrades and put them through their paces off line first before going live on a weekend. This is often easier with software than hardware, which is why I have the Wednesday rule.

Proceed With Caution
At the end of the day, my advice is to approach all upgrades and updates with extreme caution. With our own personal machines, it’s fun to upgrade and try new things. If we’re practicing best practices, we have backups that we can restore to if things go horribly wrong. But with mission critical production systems, at least for me, there has to be a really good reason to upgrade. Some things to think about.