Old Production Takes From an Old Guy

Everything’s shiny, Cap’n. Not to fret.

 

The crew of the Serenity. Photo courtesy of 20th Century Fox.

Ah Firefly, how we miss you…

I thought of that quote last week when our Paradigm system completely crapped out. We had been noticing some minor issues, things not quite acting right, so we did some investigation. It seems the IP address changed somehow, so we re-booted. When it came up again, it failed to load the Arch Config File and we were out of business. About a half-dozen phone calls later, it was determined that the Paradigm controller was hosed and needed to go back to ETC for service. Only one problem; it was 5 PM on Friday afternoon. West Coast time. Super. 

Thankfully, the Net3 system was still working fine, and we have control of the lights from the Hog. We can still pull off the weekend, it’s just a little less convenient without Paradigm. ETC will be shipping us a replacement brain this week and we should be back up and running in no time. At least in theory. And I’ll be working on my days off making sure that actually happens. Living the dream as a TD…

But I digress. The reason for this post is to ask the question, “What do you do when you have a catastrophic system failure?” It’s a bit of a rhetorical question as it all depends on what system has failed. But I want to ask the question because most of the time, we expect our systems to just work. And most of the time they do. But when they don’t we should have a plan. In fact, we should have a plan before they don’t work. 

One plan would be to have backup equipment in place. For example, we always have an extra wireless handheld on hand for every service, just in case we lose the pastor’s mic. Plans like that are easy. But how about a main system processor, or a Paradigm brain? Chances are, it’s not practical (or budget-friendly) to have backups of those on hand). And how would you decide what to have on hand anyway? 

But it’s a good exercise to think about what you would do in the event of a major system failure. Let’s say your mixing desk doesn’t power up next weekend. What do you do? It’s Saturday afternoon or Sunday morning, so you can’t get a rental in. There will still be a few hundred (or thousand) people showing up for church in a few hours, what’s your plan? Or let’s say your main projector won’t light up. Do you have another one you can throw up there to get through a service in a pinch? 

Now, I’m not going to even attempt to detail contingency plans here; there are far too many possibilities to even consider. But I will say that the most important thing in an emergency like that is to not panic. Chances are, everyone around you will go into meltdown mode, and it’s important that you remain calm and solutions-focused. Of course, having spent some time actually considering options ahead of time will make it easier for you to develop with a solution quickly and calmly. 

Really, this post is just an attempt to get you thinking. I don’t have a whole more to say at this point, and I really have to get back to solving my lighting issue. Do you have a catastrophic system failure story you’d like to share? Leave a comment…

This post is brought to you by CCI Solutions. With a reputation for excellence, technical expertise and competitive pricing, CCI Solutions has served churches across the US in their media, equipment, design and installation needs for over 35 years.

15 Comments

  1. chris@behindthemixer.com

    We've got a spare mixer handy. And I've had to use it. The question to ask is "what must I absolutely have to present a good sound (lighting/video)?" From there, you get a good idea of what you should keep as a backup – where financially possible. Darn those dollars!

  2. chris@behindthemixer.com

    We've got a spare mixer handy. And I've had to use it. The question to ask is "what must I absolutely have to present a good sound (lighting/video)?" From there, you get a good idea of what you should keep as a backup – where financially possible. Darn those dollars!

  3. Anthony

    We just had our console at our portable campus meltdown and was in the shop for two weeks. Luckily we had replaced another console a year ago and still had a spare just lying around.

  4. phil@philrowley.net

    We have dual SC-48 consoles, one at FOH and one doing monitor duties. We have enough tie lines between FOH and stage that if one died, it could easily be swapped out and things could be patched in enough to make the weekend "go". I usually leave some of those patches plugged in "just in case"!

  5. phil@philrowley.net

    We have dual SC-48 consoles, one at FOH and one doing monitor duties. We have enough tie lines between FOH and stage that if one died, it could easily be swapped out and things could be patched in enough to make the weekend "go". I usually leave some of those patches plugged in "just in case"!

  6. marcus@burge.com

    This sunday morning we had a momentary loss of power. Just brief enough that over 1/2 the audience members didn't even notice the lights blink. As the house power amps relayed back on I had many people turn to look at FOH, while we just smiled as the amps and projector came back to life and the band continued on. At the end of the service I had many people ask what happened to the audio system and projector, and we had to tell them there was actually a power loss, they had no clue.

    Just a friendly reminder to stay calm and collected. Assess the situation and act on a solution. And if you have nothing else to do, just smile and wave when you're caught in the headlights.

  7. marcus@burge.com

    This sunday morning we had a momentary loss of power. Just brief enough that over 1/2 the audience members didn't even notice the lights blink. As the house power amps relayed back on I had many people turn to look at FOH, while we just smiled as the amps and projector came back to life and the band continued on. At the end of the service I had many people ask what happened to the audio system and projector, and we had to tell them there was actually a power loss, they had no clue.

    Just a friendly reminder to stay calm and collected. Assess the situation and act on a solution. And if you have nothing else to do, just smile and wave when you're caught in the headlights.

  8. jonlillie@compasschurch.org

    I've found that panicing does nothing more than make the fixes come slower and the situation worse. As for failures? Had every aux on a desk go dead during the 15 mins between sound check and service…. luckily we had an aviom system hooked up and just put everyone that was on a wedge on ears. They lived, one die hard wedge guy even liked it more.

    Other than that one time I have had nothing major happen. [I was off the weekend the power went out in the business complex where the church is located, πŸ™‚ ] I've been very blessed over the last 15 years of working tech.

  9. jonlillie@compasschurch.org

    I've found that panicing does nothing more than make the fixes come slower and the situation worse. As for failures? Had every aux on a desk go dead during the 15 mins between sound check and service…. luckily we had an aviom system hooked up and just put everyone that was on a wedge on ears. They lived, one die hard wedge guy even liked it more.

    Other than that one time I have had nothing major happen. [I was off the weekend the power went out in the business complex where the church is located, πŸ™‚ ] I've been very blessed over the last 15 years of working tech.

  10. madmoe@gmail.com

    Several months ago, at the beginning of our 2nd service, we lost all sound from the wireless mics. I glanced at the rack where the receivers are, and noticed that the entire rack was black, no readouts except a single light on the UPS. I opened the back of the rack, figured out where the Mic Receivers and S-1608 (where all the audio components in the rack were plugged for our Roland M400 Mixer). In the meantime the singers were learning how to project their voices. Ran an extension cord to the rack plugged the essentials into that, bringing up the vocals back. I was later told by one of the others in the booth that they were surprised that I didn't look frantic or panic. I said I was just focused on find the problem and fixing it, panic would take too long.

    Later we determined that the UPS had failed and the company replaced it. On a second note we had 2 of the UPS units, one in the remote rack where the amps were, and that one had failed about 2 months prior.

  11. madmoe@gmail.com

    Several months ago, at the beginning of our 2nd service, we lost all sound from the wireless mics. I glanced at the rack where the receivers are, and noticed that the entire rack was black, no readouts except a single light on the UPS. I opened the back of the rack, figured out where the Mic Receivers and S-1608 (where all the audio components in the rack were plugged for our Roland M400 Mixer). In the meantime the singers were learning how to project their voices. Ran an extension cord to the rack plugged the essentials into that, bringing up the vocals back. I was later told by one of the others in the booth that they were surprised that I didn't look frantic or panic. I said I was just focused on find the problem and fixing it, panic would take too long.

    Later we determined that the UPS had failed and the company replaced it. On a second note we had 2 of the UPS units, one in the remote rack where the amps were, and that one had failed about 2 months prior.

  12. crissniemann@gmail.com

    Although our audio system functions on several digital audio transports over various connectivity methods, driven from a digital console and processing, we did not get rid of our old copper lines. We can have a complete system malfunction and be up and running analogue style in about 20 min with our portable mixer and head end (which happens to be the old FOH system.) The old copper comes in handy from time to time too because its easily patchable in and out of our system.
    Our lighting system is very simple and if our console goes down, our hand full of simple scenes can be recalled via computer.
    In the end, I think you can only be as redundant as your budget and reality allows. I have an extra bulb for the projector, but not an extra switcher. For our situation, lighting can fail (as long as we have house lights) and video/projection can fail but the word must be heard. For that reason our redundancy is built around audio.

  13. crissniemann@gmail.com

    Although our audio system functions on several digital audio transports over various connectivity methods, driven from a digital console and processing, we did not get rid of our old copper lines. We can have a complete system malfunction and be up and running analogue style in about 20 min with our portable mixer and head end (which happens to be the old FOH system.) The old copper comes in handy from time to time too because its easily patchable in and out of our system.
    Our lighting system is very simple and if our console goes down, our hand full of simple scenes can be recalled via computer.
    In the end, I think you can only be as redundant as your budget and reality allows. I have an extra bulb for the projector, but not an extra switcher. For our situation, lighting can fail (as long as we have house lights) and video/projection can fail but the word must be heard. For that reason our redundancy is built around audio.

  14. Mslaird1976@gmail.com

    Three weeks ago, we experienced a sound system shut-down. In the middle of the first song, in our first service, on Sunday morning. Long story short, the mics and keys were useless, but I still had our pastor's countryman and the sermon dvd (video-venue multisite). Speakers worked, but we went "unplugged" for the remainder of the service. As audio tech, I was calmly pulling out my hair trying to troubleshoot. Here's the great thing–other than leadership, NO ONE knew what was going on. The worship pastor received more compliments than ever. What went out and what worked couldn't be explained by the tripped circuit breaker, so I'm chalking it to a lesson in humility and surrender.

  15. Mslaird1976@gmail.com

    Three weeks ago, we experienced a sound system shut-down. In the middle of the first song, in our first service, on Sunday morning. Long story short, the mics and keys were useless, but I still had our pastor's countryman and the sermon dvd (video-venue multisite). Speakers worked, but we went "unplugged" for the remainder of the service. As audio tech, I was calmly pulling out my hair trying to troubleshoot. Here's the great thing–other than leadership, NO ONE knew what was going on. The worship pastor received more compliments than ever. What went out and what worked couldn't be explained by the tripped circuit breaker, so I'm chalking it to a lesson in humility and surrender.

© 2021 ChurchTechArts

Theme by Anders NorenUp ↑