This week Van interviews Brent Allan. They discuss the importance of knowing your systems. This is especially true for when you come into a new position. Get to know how things work in your room.
This week Mike talks tech booths. He goes through best practices on size, design elements, how to lay them out and how to keep coffee cups from accumulating on the front wall.
It’s that time of year again. The time of year when a young tech directors thoughts turn to…software upgrades. With the recent release of ProPresenter 7, the internets are all abuzz with discussions about upgrading software. Now, this article isn’t about ProPresenter per se. I have no experience with it and no particular thoughts about it one way or another. It’s simply a catalyst for me thinking about upgrades in general.
On the bookFace groups I follow, I often see questions about upgrading software stuff. Having done my fair share of it, I thought I would share some observations and ideas on what I consider best practices.
Not on the Weekend
The first rule for me of upgrading is to never, no not ever upgrade on the weekend. I turn off all automatic updates on all my production machines and make sure they are not even checking on Saturday or Sunday. In fact, it’s a good argument for production machines being on their own network, air-gapped to the outside worlds. Windows 10 can’t install its next group of productivity enhancing features providing unparalleled levels of productivity if it can’t talk to the servers.
You never know when an upgrade won’t work, or will break something else. The last thing you want to be doing during what was supposed to be soundcheck is tracking down new drivers and updates so the teleprompter…er…confidence monitor will work.
I personally put Wednesday as my hard stop for updates. That was the last day I would attempt software or firmware updates. If I ran out of time, I did not do it on Thursday, nor Friday. Why? Well, one time I installed firmware and it bricked the hardware. I had to have new hardware overnighted to get it working. Well, it would have been overnight except I found myself with bricked hardware on Friday afternoon. We had missed the shipping cutoff for the day and it wasn’t going out until Monday. That was an uncomfortable weekend.
If you break something on Wednesday, you have all day Thursday to get something new moving your way and there’s a good chance you can have something delivered Friday or maybe even Saturday. Now, to be fair, that rarely happens. But it only takes one time.
If it ain’t Broke…
I generally take the approach to software updates that if what I’m using isn’t broken, I don’t try to “fix” it. Over the years, I’ve had updates or “upgrades” cause more problems than they solved—especially when there weren’t problems in the first place. Sometimes it’s not even the software in question that causes the problem.
You might, for example, find that to upgrade to the latest and greatest version of software X that you need to update the OS. No problem, you think, it’s time to do that anyway. So, you update the OS and X turns out to be super snazzy. You play around with new features of X and it’s great. Then on Sunday you show up and find out that Software Y now needs to be updated, and you need it for the weekend. Worse, once you update Y, you discover the drivers for hardware Z no longer work. Three hours later, you’re finally back in business; right about the time the pastor is starting his message…
It’s even worse when you upgrade and then discover your hardware no longer works with the new software. I’ve seen this with video and audio interfaces. Or maybe the drivers simply haven’t caught up yet. This is a great way to take an entire system off line if you’re not careful.
If You Don’t Need It
I always read over the feature upgrades or bug fixes in software updates on my production machines. If there isn’t a really compelling reason to update the software or firmware, I don’t do it. Sometimes an update will fix a bug I’ve never experienced. And since I’ve never seen an update that doesn’t fix bugs introduced in the last “bug fix”, it seems each update introduces other bugs that will then be fixed in the next release. If I’m not dealing with those bugs now, I don’t want potential new ones unless there’s a feature I can really use.
For example, I’ve noticed that newer versions of Mac OS X have really jacked up wired networking. I had nothing but trouble with it using my (former) new MacBook Pro running Mojave. It’s (former) because I just sold it and went to a Lenovo laptop that cost 25% as much and actually works with Ethernet. Since much of my work revolves around connecting to hardware via Ethernet, that cool new MBP wasn’t much of an upgrade. In fact, my 2013 running High Sierra is much more reliable with Ethernet. Why High Sierra? See point #2.
You really have to ask if those new features outweigh the potential risks of breaking something that’s working. Back when I was a TD, our ProPresenter machine was 5 years old, running 4 versions back of the Mac OS and I think two version back on ProPresenter. Why? Because it worked every weekend and the new versions of all that software offered nothing compelling enough to risk changing it. In fact, I remember updating ProPresenter one week and finding a bug that really caused some significant issues. They fixed the following week, but that weekend wasn’t fun. I’m not sure I ever upgraded any of that stuff again.
Again, this is not a dig on ProPresenter in particular. There are simply too many combinations of hardware and software in the world for any developer to ensure their software is going to work without issue with all of them. Renewed Vision is generally really good at getting their software up and running again, but you have to give them a little time. And you may not have that time.
Even Hardware Upgrades are Scary
As I’m writing this out, I’m remembering a time with a client of mine that found some very interesting issues with upgrading. They were in an area that had very unstable power delivery. I believe their incoming power varied from 57 to 64 Hz or something like that. Most equipment can handle that, but it was causing problems with lip sync between the ProVideo Player/Blackmagic Ultrastudio and the switcher. As there sermon went on, the lip sync would drift, but it drifted both forward and back as the power drifted.
So, we decided to genlock the Ultrastudio. That fixed the lip sync problem. However, it introduced a really weird AM modulated digital noise into the audio signal. My client spent an inordinate (or normal, really) amount of time on the phone with Blackmagic and it was discovered that the genlock input leaks voltage to the audio output stage. This caused the amplitude modulated distortion.
The ultimate solution was to use a Radial USB audio DI to send audio into the mixer and only use the Ultrastudio for video. Now, that’s a border case that you don’t come across every day—which is one reason it took so long to solve. But it illustrates the point that upgrading one thing can have a cascading effect on other things that you hadn’t considered; and that goes for hardware as well as software.
Which is why we should approach all upgrades with care. If at all possible, have a test bed for any new upgrades and put them through their paces off line first before going live on a weekend. This is often easier with software than hardware, which is why I have the Wednesday rule.
Proceed With Caution
At the end of the day, my advice is to approach all upgrades and updates with extreme caution. With our own personal machines, it’s fun to upgrade and try new things. If we’re practicing best practices, we have backups that we can restore to if things go horribly wrong. But with mission critical production systems, at least for me, there has to be a really good reason to upgrade. Some things to think about.
Van’s back in the truck and he’s talking about how to spark passion in your volunteer team. Some team members seem content to do average work every week. Van has some thoughts on how to help move them to the next level.
This week Mike shares some tips on how to prepare for big services—Christmas, Easter, VBS—without going crazy. The key is in thinking ahead—way ahead. Plan early and your life is better.
This week we talk about contracts! Many don’t fully understand the types of contracts they will be entering into when they embark on a AVL project (or building project). This week, we’ll delve into the main types you’ll encounter and how to make sure you get what you want.
This week Mike tells you why it’s more important to develop skills than buy gear. If you take the time to build you skills, the gear won’t matter. But if you think buying better gear will make you better, you should listen to this show.
Hey y’all. As we wind down summer, a little tidbit came across the desk of one of our engineers, Steve Lund. He was pursuing some info put out by James Stoffo of Radio Active Designs. James does a lot of work coordinating wireless gear for things like the Super Bowl, the CMAs, and pretty much every major award show. I actually met him at the CMAs a few years back. Great dude.
Anyway, shared this map that came from Professional Wireless Systems. This shows the counties that T-Mobile is planning on lighting up starting November 1. Of this year. 2017. So if you happen to live in one of the little green squares, and you have a wireless mic or IEM running above 617 MHz, you’re going to need to shut it down after Nov. 1.
Remember, it’s not a matter of interference. By law, once the new owners of the spectrum start testing, every other operator needs to cease using it. Could you get away with it? Maybe. Should you? No.
There’s a little early Christmas gift for you. Time to order up those new wireless units, or better yet, wire everything you can. This isn’t going to get easier going forward…
UPDATE: I’ve been asked about a larger map. Can’t find a larger map, but I did find this spreadsheet of counties that presumably generated the map in question. So, there you go.
This week, we discuss when it’s appropriate to bring an AVL integrator into the discussion about a new building or renovation project. In our experience, too many churches wait too long. Once plans are nearly finalized–or worse, concrete is poured–it becomes increasingly difficult to make simple, cost-effective changes to the building, add infrastructure or just make it more production friendly. Don’t waste money; start sooner!
This week, it’s old guys vs. young guys! Well, not really. But Van & Mike talk with two younger tech leaders about the challenges and rewards of leading technical ministries as a 20-30-something. Warning, these two guys are really sharp–after listening to them, you’re going to need to up your game!