Lessons Learned at a Concert

Last week while in Atlanta for the Catalyst conference, I got to hang out with my friend Nick Rivero and his new wife, Sarah. Not only did we have some great BBQ, but we had a great time talking about production, touring, and building stuff. As we talked, I was reminded of a night a few years ago when Thomas and I got to hang out with him at the Staples Center in LA. Nick was touring with Lady Antebellum as a video system tech. We got a tour of the entire production; backstage, the stage itself and FOH. And then there was the show, which was also pretty impressive. 

I won’t go into a lot of details about the evening itself as that is not the point of this article. I will say the show was great; and everyone we met was very gracious and friendly. We felt very welcome there and stayed all the way to the end of tear down. I tend to really pay close attention whenever I get to do something like this as there are always things I can learn to make my processes better. What follows are some of my key takeaways.

Predictability and Consistency is Important

On multiple occasions I asked Nick how much things changed up for the evening’s show over the course of the tour. And on each occasion he said, “Not much.” At one point we were talking about how they wire their system up each day and he said (and I’m quoting from memory, so it may not be word for word, but it’s close), “When you’re doing this every day, you want things the same as possible.” Another time I asked him about the walk-in playlist, wondering if it changed up each night. Again, the answer was, “Nope. It’s kind of nice to have that predicability. After a while, you know when you get to this song, you have a few minutes to go.”

As I am a huge fan of consistency, this was really more of a reinforcement for me than a lesson. But still, it’s always good to challenge your thinking, then come to the conclusion that you’re headed in the right direction. Doing things the exact same way has saved me on many, many occasions. Once I find a process that works, I repeat it over and over again until I come up with a better process. 

Many Hands Don’t Make Light Work; Many Skilled Hands Do

We’ve heard that verse repeated over and over again in church and in many respects it’s true. However, I loved watching the union hands come out to strike the show after the concert was over. The various team leaders on the tour would give a bunch of guys some brief instructions and off they would go. Near the beginning, I watched Nick say, “OK, green shirt guys, my name is Nick. I need you to take all these cables, coil them up to the end and bring them back here. Thank you.” Within a few minutes all the projector and camera looms were neatly coiled and returned to video world. Even more fun to watch were the riggers. Without hardly a word, trusses starting coming down, fixtures were removed and returned to road cases and lines coiled. The whole show was loaded out in a few hours.

Contrast that to the typical church “y’all come help teardown” party. It’s chaos! One or two technical people will try to herd the army of cats on stage as cables get coiled improperly, equipment gets put away in the wrong spot (or worse, lost), things get taken down that should be left up. Sometimes, people even get hurt.

I prefer fewer people with more skill. Give me a few guys that know how to wrap cable and ask good questions over a dozen completely unskilled people every time. During my last few years as a TD, I adopted this approach for our big events. Instead of recruiting huge teams, I hand-picked a few guys who I had worked with and knew would do things the right way. It went much better. 

Being Together Still Matters

With the rise of “online church,” some of us are wondering what the future holds for the weekend service experience. At the show, I was again reminded of the power of being in a large group, experiencing something together. I am not typically a fan of large crowds, but I will say there was an energy and excitement in the room that doesn’t exist when I listen to their CD or watch a video in my office. One of the big things we still do well as the Church is bring people together for a common experience. When God is doing something in our midst, it is palpable, and being part of that with a few hundred or a few thousand others is powerful. We mustn’t forget that.

What do you learn when you go to a concert?

Setting Up a Redundant Dante Network

Last time, I shared with you the problem you can have if you set up a Dante network improperly. Without redundancy, a single break anywhere in the system can cause major problems. With redundancy, if you cross the streams, the whole network refuses to work properly. Based on my experiences a few weeks back and some conversations I had with people who know a lot more about this than I do, I have revised my set up process to make sure things work properly. 

Configure First

My biggest mistake was wiring everything together then powering it all up. I freaked the switches out without even knowing it, then nothing I did after that worked properly. So the new tactic is to configure first, then wire. In some cases (like Yamaha Rio boxes), this will mean flipping dip switches for redundant mode. In others, it will mean powering it up and selecting redundant mode in the setup menu. No Cat5 cables will be connected until each piece of gear is verified to be in redundant mode.

But it’s not time to connect anything yet. The next step is to set up the switches. It’s important to get the QoS settings correct, build VLANs, configure settings for wireless use and disable energy efficient modes. This is all best done without anything connected to the switch. In fact, the next time I do it, I’ll be programming switches in the office before heading out to the field. 

Wire Primary First, Test

Once everything is configured properly, and we’ve triple-checked to be sure everything is in redundant mode, we will wire up the primary network side and make sure that all works. The system will function just fine on just the primary network, and this is the time to mount all stage racks to consoles, make sure signal is flowing between devices and the system is functioning as expected. 

If everything checks out with just the primary network connected, then it’s time to connect the secondary network. If everything was configured properly, nothing should happen. If the system freaks out when you plug in a device’s secondary port, you have the streams crossed somewhere. At that point, disconnect the secondary, power cycle everything and check your settings again. 

Once you get everything working with both networks, you can test the failover by pulling the primary from one device. Audio should keep on flowing and stay working when you plug primary back in.

Label Well

The take away for us on this install was to make sure everything was labeled well once we got it all working. We further hosed ourselves during trouble shooting by accidentally plugging a primary port into a secondary switch. I didn’t do it for that install, but in figure ones, I will even use different colors for the Cat5e cables and patch cords to make sure the two networks stay separate. And I’ll standardize on those colors so as we build networks all over the country, we will always know what is primary and secondary. 

It’s Not That Hard, Just Different

I know a lot of people are afraid of the digital network revolution in sound systems. It’s true that when we used big copper snakes, it was a little easier to troubleshoot things like bad cables and improper patches. However, we still had to make sure the system was wired correctly with regard to polarity. We still had to pay attention to power and grounding. The transition between balanced and unbalanced connections still had to be handled properly. There were plenty of places for things to go horribly wrong. The biggest difference was we could typically physically see the problem. 

In the networked world, we can have problems that we can’t physically see. The problems can exist inside a switch and it takes a different set of troubleshooting skills to figure it out. But it’s not really all that hard once you do it a few times and get some basic knowledge of the system. Yamaha actually has some great information on setting up network systems and switches, and I highly recommend you check it out before you set up your first Dante network. This post is not meant to be an exhaustive guide, just an overview. 

It’s a brave new world out there, folks. I’ve been saying for a few years that our job as technical artists will involve a lot more network skills. Now is the time to beef up that skill set!

Roland

Today's post is brought to you by Pacific Coast Entertainment. Pacific Coast Entertainment is the premier event production company servicing Southern California and the western states. PCE offers a complete line of Lighting, Audio, Video, and Staging equipment for rentals, sales and installs. Where old fashion customer service meets high tech solutions. PCE, your one stop tech resource.

Important Safety Tip: With Dante, Don't Cross the Streams

One of my favorite scenes in Ghostbusters is when Egon gives the guys a warning about crossing the streams. As we know, it’s not good. Not good at all.

Dr. Egon Spengler: There's something very important I forgot to tell you.

Dr. Peter Venkman: What?

Dr. Egon Spengler: Don't cross the streams.

Dr. Peter Venkman: Why?

Dr. Egon Spengler: It would be bad.

Dr. Peter Venkman: I'm fuzzy on the whole good/bad thing. What do you mean, "bad"?

Dr. Egon Spengler: Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light.

Dr Ray Stantz: Total protonic reversal.

Dr. Peter Venkman: Right. That's bad. Okay. All right. Important safety tip. Thanks, Egon.

 

That scene played out in real life for me a few weeks back. Well, sort of. I didn’t exactly experience total protonic reversal, but I did discover that it’s a bad, bad thing to cross the streams in a redundant Dante network. But let’s back up a step. 

Two Types of Dante Networks

There are a few different ways to lay out a Dante network. Ignoring topologies for a moment, you can either set up a redundant network or a switched network. All Dante devices have two Dante network ports on them, and they will be labeled Primary and Secondary. By default, most devices have those two ports connected together in switched mode. That means you can plug one device into the primary and another device into the secondary and data will flow freely between all three devices. 

The benefit of this method is that you don’t need any switches. Just keep daisy chaining all the devices together and the network works quite well. The downside is that if any link in the chain breaks, audio stops flowing. If you had a console and computer at FOH, two stage racks, a processor and a few amps on stage and someone breaks the cable between the two stage racks, you will lose audio in the house as the data is no longer flowing from the console all the way through to the system processor and amps. 

Now, you could just take all the primary ports and connect them to an external switch. That way, if you lose one link, only that device is affected; the rest of the system still functions. This is clearly a step up in reliability from the daisy chain, but you can still lose a device if a cable fails. 

That’s why the better option is a redundant network. In a redundant network, you connect all the primary ports to a switch (or series of switches, depending on the layout) and all the secondary ports to another switch or set of switches. I highlight “another” because it’s very important that the networks remain separate. 

Don’t Cross the Streams

I learned this lesson the hard way a few weeks ago. It turns out that if you wire the system up properly, with all the primary ports going to one switch and the secondary ports to another, but have one device set up as a switched device, you essentially build a bridge between the two networks. And that’s bad. While you don’t have total protonic reversal, you do get what is called a broadcast storm, and the switches freak out. The only way to fix it is to separate the networks and power cycle everything on the network. 

Dante devices will stop working properly, probably won’t pass audio and devices will not mount properly. Basically, it’s bad. I chased this problem for a while before my friend Jake Cody helped me figure it out. I learned an important lesson that day, and it changed the way I will set up a Dante network next time. 

Why Redundant?

You might be asking, why bother with redundancy if it’s so fraught with peril to set up? Well, it’s really a safety net. When we used analog copper snakes, one wire could go bad, and we’d still have signal in the others. In say, a 56-channel snake, if you lost one channel, you still had 55. On a 16-channel drive snake, you could patch around a channel that went bad. Even if you lost 2-3 of them, most of the system would still function. And if you got creative, you could make it all work. 

But when the entire system depends on four 24 gauge solid copper conductors, it doesn’t take much to wipe the whole system out. If one of those wires breaks, you don’t lose 75% of the system; you lose it all. Or if a single switch freaks out, you can lose the whole network. And the only way to fix it is to stop audio, and reboot or reconfigure. 

Building a redundant network will let you fix a problem in one leg while the other still passes audio. For any system that is mission critical—and that pretty much means all of them—you should go redundant. 

Next time, I’ll share with you my revised process for setting up a network that will work the first time.

“Gear

Today's post is brought to you by Heil Sound. Established in 1966, Heil Sound Ltd. has developed many professional audio innovations over the years, and is currently a world leader in the design and manufacture of large diaphragm dynamic, professional grade microphones for live sound, broadcast and recording.