Old Production Takes From an Old Guy

Month: May 2013 (Page 1 of 3)

Backing Up Lighting


A few weeks ago, I wrote a post called, “When Things Go Horribly Wrong. Again.” In that post, I talked about our lighting console crashing (turns out we had a bad SSD), and how we recovered from it. That crash brought to light some major gaps in our backup process; ones I now hope we’ve closed.

In the weeks leading up to the backup, I thought everything was fine. We had an external backup drive attached and backup software that was supposed to be backing up every Saturday night. Turns out “supposed to” is the operative phrase. Apparently, it wasn’t backing up, and hadn’t for quite some time. I don’t know why.

Normally, I’m a belt and suspenders kind of guy. I don’t consider any files safe unless they’re on at least two preferably three) drives or media, with one in the cloud. I like the 3-2-1 backup strategy (the link takes you to a PDF report of that strategy) that was developed in conjunction with the National Archives. All files should exist in 3 different places, on 2 different media with at least 1 off-site. I do that with our audio console show files, but we never set it up for lighting. Now we have. Here’s what we did.

Backup, Backup, Backup

It turns out that in the new version of Hog PC (v. 4), they included a handy-dandy backup button. It’s a one-stop-shop for making a backup of your show file. Somehow, it makes the backup files very small, but it’s still the whole show file. How can we not take advantage of that? Turns out, we weren’t. Now we do.

The standard shut down procedure on Saturday night is to first hit “Backup” to make a backup show file. That way, if things go south on Sunday, we can get right back to where we were. But that only protects us from a show file crash; if the hard drive goes again, we’re hosed. So, we back up the backup.

Dropbox for the Win!

Dropbox is perhaps the easiest way to back things up. Now, we could go into the Hog’s backup folder and manually copy the latest backup over to the Dropbox folder and let it do it’s thing. But that takes time and is a pain. So I automate. 

Using a free utility from Microsoft called Sync Toy, I built a little sync script that will, when fired, sync the backups folder with the Lighting Backups folder in Dropbox. And again, we could launch Sync Toy manually, run the sync, wait for Dropbox to upload, then shut down. Again, a pain. And boring. So, we automate once more.

I stole a little batch file that Isaiah Franco made for me back when he worked here. It’s super simple, and basically does three things. First, it asks if you really want to go through with this. Second, it launches Sync Toy and syncs the files. Third it waits one minute for Dropbox to upload and then shuts down the computer. 

We leave a shortcut on the desktop labeled “Shutdown PC,” and have the guys fire that to shut down. For those that are interested, here’s the .bat file. Feel free to mod it as needed if you like.

ECHO Are you sure you want to sync the SD-8 files and shutdown the PC

set /P c=[Y/N]
if /I "%c%" EQU "Y" goto :shutdown
if /I "%c%" EQU "N" goto :abort

goto :choice

"C:\Program Files\SyncToy 2.1\SyncToyCmd.exe" -R
ECHO Commencing shutdown in 60 seconds.
shutdown /s /t 60

:abort@ECHO OFF

When this runs, it simply launches Sync Toy and runs whatever sync folders we have set up. Within a few seconds, I can see Dropbox notifications on other computers that the files are updating. Cool.

Backup Hardware

One thing we noticed during that crash was that if the Mac Mini (running Win 7) goes down, we’re kind of dead in the water. Although, since it’s Hog PC, we just need another PC. So, I configured my laptop, which already had Win7 installed in VMWare, with the Hog software, and linked it to the Dropbox folder. So now, if the Mac dies, we grab my laptop, fire up VMWare and in a few minutes, we’re back in action. Maybe we’re limited in our ability to build big shows due to the monitor limitations, but we can run the weekend. But what if we loose our DMX widget?

Backup, Backup Hardware

We’d actually have to loose both widgets because we have two, but what if we did? We still have the ETC Paradigm system in-house, with a really cool touch screen in the booth. So one of our projects this summer will be to update the control programming on that system to include all the moving lights and LEDs, as well as a simple 10-cue page that would get us through a service in case everything else failed. 

So that’s our system right now. In the last few weeks, I’ve been checking to see that the backups are actually working, and being pushed up to Dropbox. I now feel like I have one fewer thing to keep me awake on Saturday night.

Today’s post is brought to you by GearTechs. Technology for Worship is what they do. Audio, video and lighting; if it’s part of your worship service, and it has to do with technology, GearTechs can probably help. Great products, great advice, GearTechs.

Why Hire an Integrator?

Photo courtesy of   NCinDC

Photo courtesy of NCinDC

I was recently asked by a reader for some suggestions on helping his church’s leadership understand the need to hire an integrator for new building project. If you’ve read this blog at all, or listened to ChurchTechWeekly, you know it’s something we talk about a lot. Sadly, most churches don’t do this well. In the case of our reader, his leadership thinks that they can do the job in-house, and things will be just fine. As someone who has been working in churches a long time (20+ years…), I can tell you that in-house jobs rarely end up in the same zip code as fine.

Now, to be sure, there are some churches that can tackle an in-house install. Typically those churches are fairly large and have multiple technical personnel on staff. That technical staff has collectively many years of experience doing design, and installing equipment. They also have a great relationship with an equipment vendor who they can ask advice on things they don’t know about. 

Those churches are rare, however. Most churches are smaller with less experienced (if any) technical staff. Now make no mistake; I think the small-church technical leader or volunteers are real heroes. They get stuff done in the face of non-existent budgets, time and equipment. But there is a vast difference between making things happen on a weekend and designing and installing a full-blown A/V/L system in a new (or renovated) building. 

Designing a full system takes a lot of know-how, and requires an encyclopedic knowledge of equipment. Given the wildly varying standards we’re dealing with right now, it’s far too easy to end up with systems that won’t work together. Designing speaker systems is far more complex than hanging a few boxes in the air (which also needs to be done safely), and typically DIY projects are less than desirable.

As someone who for whatever reason seems to be the Mike Holmes of church tech, I can tell you’ve I’ve pulled out a lot of gear that was installed by well-meaning but terribly uninformed people. It can take years and hundreds of thousands of dollars to fix mistakes that were done in the name of saving money. Don’t do it! 

Here are the top five reasons you should not do it yourself. Feel free to send this link to your pastor…

You Don’t Know Enough

I don’t mean you’re stupid; I mean designing and installing an A/V/L system requires a specialized skill set and unless you’ve been doing it for a few years, you don’t have it. I have been at this game for 25 years, working with professional AV equipment and systems both inside and outside the church. When it came time to design a PA for my church, I looked to experts at my integrator (and speaker manufacturer). Why? Because I don’t design speaker systems for a living. I have ideas and opinions, but they have the real expertise. When it comes time to do the install, they will install it. Why? Because I don’t routinely hang 1800 pounds of gear over people’s heads. They do. 

Chances are, you simply don’t know enough gear options to make good decisions. I go into so many churches who simply bought was was for sale at Guitar Center, never realizing that for 10% more, they could have bought a far, far better solution. Sometimes, it even costs less. Integrators spend their lives designing and installing systems while constantly keeping up on new equipment and technology. You don’t. It’s far better for you to do what you do and let them do what they do. 

You Don’t Have the Time

Designing and installing a complete system—even a fairly simple one—takes an incredible amount of time. If your church tech team is made up of volunteers, chances are they don’t have a spare 40 hours a week they can give to the project for a month. At least not if you want to keep doing services. 

Last year, we did part of the install in our student wing, working along side our integrator. Even that load just about killed us. It took me 2 full months to recover, and every Sunday during the install and recovery, I debated whether or not I wanted to come in to work or not. It was brutal. And we know what we’re doing and have a staff. Don’t kill your people just to save a few bucks. It’s not worth it.

You’ll Have to Live With the Building a Long Time

I can’t tell you how many times I’ve had to say no to our leadership on doing something they want to do because we don’t have the infrastructure in our building. Opened in 1993, we’ve been fighting it since day one. Why? Because they cut the AV budget and didn’t use an integrator. There are no conduit runs between the tech booth and the stage, or the amp room. How did that get missed? How impossible is it to fix now? (Answer: Very!)

No building is perfect, even those build with an integrator on board early. However, the worst buildings, the ones that are the hardest to deal with are ones that were DIY projects, especially with architects who haven’t done churches before. Or the churches they have done are very traditional and don’t require any production equipment. If I see another church with a single 3/4” conduit between the booth and the stage, I’m going to be sick. Please, get someone involved who actually knows what they’re doing.

You Will Waste Money On Your Own

Everyone thinks going solo will save money. I have bad news for you: It rarely ever does. Oh sure, maybe in the short run, you might save a few thousand dollars—maybe even $20-30K. But here’s the reality—if what you buy and install turns out to be the wrong stuff installed improperly, how much will you spend tearing out and doing it again? As someone who has been part of multiple churches that are on their second or third PA system, I can tell you it’s a lot. 

A good integrator will make sure you get what you need, and will ensure that it’s installed properly. Moreover, they will consider future needs and be thinking of additional cable and power runs that you’ll miss. Remember, once the building is built, it’s there. Do you want to fight with it every day of the rest of your life, or actually enjoy working there? 

You Are Setting Your Team Up To Fail

Pastors, this one is for you. You may not realize it, but when you ask your volunteer (or one-man) tech team to do the whole design and install themselves to save money, you are setting them up to fail. Why? For all the reasons above. They don’t have the knowledge base, experience or time to do it right. And here’s the kicker. Once it’s installed, and it doesn’t really work they way you wanted it to, you will blame them. 

Instead of equipping them to succeed, you’ve set them up to fail. For the next several years, you will be frustrated with the way the equipment  works (or doesn’t) and you will become bitter toward the team that installed it. Eventually, they will get tired of feeling your disappointment (or wrath) and they will leave. Some will leave the Church for good. You will then go out and hire an integrator to “fix what those knuckleheads screwed up.” Don’t do it!

Think you’re above all that? Trust me, you’re not. I’ve been in their shoes and have experienced it first-hand. I also talk with tech leaders all over the country and hear the same story over and over again. 

Do yourself, your church and your tech team a favor and hire the right people for the job. Don’t cut your tech team out of the loop, either. The best church buildings are built by a partnership between leadership, the builder, the tech team and the integrator. Work hard to foster that team approach. Everyone will be happier in the end, I promise you.

Today’s post is brought to you by Bose Professional Systems Division, committed to developing best-in-class products, tools, and services to create original audio experi
ences. The chief advantage products like RoomMatch® array module loudspeakers and our line of PowerMatch® amplifiers offer for worship are clear natural sound that makes voices and music seem more real.

And by Ultimate Ears. Housed within a custom shell designed to fit your ears, high quality multiple armature speaker systems provide an unparalleled sound environment, as well as 26 dB of passive noise cancellation.

Church Tech Weekly Episode 149: Temporal Fusion


It’s all about reverbs and delays this week! Our panel of experts talk about how they use reverb & delay, what settings they start with and how to get the most out of the gear you have.


Today’s post is brought to you by myMix. myMix is an intuitive, easy-to-use personal monitor mixing and multi-track recording system that puts each user in control of their own mix! myMix features two line-level balanced 1/4″ TRS outputs and one 1/8″ (3.5mm) headphone output, the ability to store up to 20 named profiles on each station, 4-band fully parametric stereo output EQ recording of up to 18 tracks plus stereo on an SD card. Learn more at myMixaudio.com

CTW Coverage: Gurus 2013—Curtis Templeton

Curtis talks about the pastoral side of technical direction, and what it means like to “smell like your sheep.” A great reminder that TDs are pastors, too.

Today’s post is brought to you by DPA Microphones. DPA’s range of microphones have earned their reputation  for exceptional clarity,  high resolution, above all, pure, uncolored accurate sound. Whether recording or sound reinforcement, theatrical or broadcast, DPA’s miking solutions have become the choice of professionals with uncompromising demands for sonic excellence.

Church Tech Weekly Episode 148: Live at Gurus


This past week was Gurus 2013 and CTW was there! In this special episode, we interview 40 people who were there, and we did it live at the show! There’s lots of gold to mine in this episode, trust me.


Today’s post is brought to you by BargeHeights. Bargeheights offers cost effective lighting and LED video gear for churches. Coupled with unique visual design, Bargeheights transforms worship venues of all sizes.

CTW Coverage: Gurus 2013—Andrew Stone & Whitney George

Andrew and Whitney shared how they developed trust with one another and why it’s so important in the church production setting.

Today’s post is brought to you by Heil Sound. Established in 1966, Heil Sound Ltd. has developed many professional audio innovations over the years, and is currently a world leader in the design and manufacture of large diaphragm dynamic, professional grade microphones for live sound, broadcast and recording.

CTW Coverage: Gurus 2013—Steve Carter

After challenging the room full of technical artists to pay attention to the spiritual side of what we do, we caught up with Steve for a few additional thoughts.

Today’s post is brought to you by the Roland R-1000. The R-1000 is a multi-channel recorder/player ideal for the V-Mixing System or any MADI equipped console or environment. Ideal for virtual sound checks, multi-channel recording, and playback.

When Things Go Horribly Wrong. Again.

Photo courtesy: Amy Dominello  /   News & Record

Photo courtesy: Amy Dominello / News & Record

I don’t know why it always happens around Easter week. Perhaps it’s a 12-month duty cycle, or perhaps it’s spiritual warfare leading up to the week, but for the last 3 years we’ve had something significant break on Palm Sunday or the weekend before. This year, I thought we had escaped. Until Palm Sunday morning.

I was mixing and my lighting guy caught my attention and said, “Uh, Mike, the Hog keeps freezing.” I told him to re-start the computer. He replied that he had already done that twice. Uh oh. I moved over there to take a look. He reported that the console would work for about 5 minutes then freeze up. It became completely unresponsive and we couldn’t do anything. By now, it was about 8:30 and we had service in 30 minutes. Ugh.

Over the next few weeks, I’m going to write a series of posts that will detail some of our backup processes. Before I do that, I will tell you that we’re still in progress and our system is far from perfect. But we spend a lot of time on it and are moving toward a place where I feel like we can handle about anything. I will also tell you that our system was not working on Palm Sunday this year. That’s something else we learned. But before we get into all of that, I want to share something else I’ve learned over the years, and it’s probably more important than any of this. 

The way in which you respond to a system failure is more important than getting the system back online. That is to say, if you start freaking out, yelling and running around like a madman, while you might get the system back online, you will have a lot more damage to clean up afterwards. If you stay calm, cool and focused on the big picture, not only will you likely recover from the disaster, you will have also gained a lot of respect from your team and leadership.

Now I’ll be the first to admit that I don’t always do this well. Sometimes I can be completely cool and collected, working through a problem with no real issue. Other times, I tend to grumble under my breath, complaining about this “crappy technology that never works…” I’ve been doing this long enough that I rarely go into full-on meltdown mode, but I can become a grumbler while I’m fixing something. It’s something I know about and am working on. Here are a few things that I think are key to recovering from a system failure.

Stay solutions focused

The moments after the lighting console craps out (again) are not when we should be complaining that they should not have cut the new one out of the budget (again), researching new options or blaming the previous guy for buying that one. What we need to do is fix the problem; in that case, get the lights back on. 

We first thought we had a corrupt show file, so we tried opening a backup. That’s when we discovered our backup system hadn’t been working. We eventually found another backup show file, but the problem persisted. With time and options running out, we managed to program 3 quick looks (walk in/out, music, teaching) into the Paradigm controller before the console froze again. That got us through the weekend (albeit with the simplest lighting anyone’s ever seen there…), and we had time to really asses the situation. 

Put the team first

That weekend, my lighting operator was a high school student who does great work, but has limited experience with the console. So rather than throw him into the middle of troubleshooting, I worked with him to try to come up with a solution. While the situation was a bit tense, I tried to make light of it and remind him that it was not the end of the world if we only had 3 lighting looks that weekend. 

We joked that I would give him the cue when it was time to change from “music” to “teaching” so he wouldn’t miss it. Instead of getting upset, we just moved forward and tried to make it fun. 

Find something that works and build from there

We discovered that we could get the console up for a few minutes before it froze up. That gave us enough time to program those three scenes into the Paradigm (ETC’s architectural control system), which got us through the weekend. As we had less than 30 minutes to figure it out before the service started, I didn’t have time to go into full-blown troubleshooting mode on the console. We just needed something to work. 

A year or so ago, I had a power supply go bad in my stage rack, and somehow it took an input card with it. What was happening didn’t make any sense, I just lost 8 input channels. I spent a few minutes figuring out what happened, then simply re-patched those inputs to a card that seemed to be OK, and updated the patch on the console. We made it through the weekend, which then gave me time to figure out what was happening in the relative calm of a Sunday afternoon.

I think it’s important to not be surprised by calamitous events. If we’re not running systems that have grown to be complex and highly inter-connected, we’re running systems that are old and frail. It’s no surprise that things fail; all equipment, no matter how good or expensive has a finite lifespan. We simply need to prepare for it and remain calm when it all goes wrong. In upcoming posts, I’ll detail some of our process for surviving system failures.

Today’s post is brought to you by DiGiCo. DiGiCo audio mixing consoles deliver solutions that provide extreme flexibility, are easy to use and have an expandable infrastructure, while still providing the best possible audio quality. Visit their website to learn more.

« Older posts

© 2021 ChurchTechArts

Theme by Anders NorenUp ↑