Back It Up For Easter

Image courtesy of Jon Ross

Image courtesy of Jon Ross

This is one of those quick-hit posts that may save you a lot of headaches. A few years ago, we spent the entire week before Palm Sunday setting the stage, hanging fixtures and programming lights. It was a long week, but we got a lot done and we were ready to go for the Easter week rehearsals. 

On Palm Sunday morning, my lighting tech started having problems with the console. Some cue stacks wouldn’t fire, and some lights weren’t responding. He rebooted and it stopped working altogether. Several more power cycles and re-boots and we were able to set a few quick looks and save them in to our ETC Paradigm for playback from the touch screen (really glad I had 6 recordable scene buttons set up before that day!). That got us through service. It wasn’t great, but there was light. 

After services I grabbed my IT guy and we determined fairly quickly that the hard drive on the console was dead. That would be the hard drive with all the programming for Easter on it. Yeah, that hard drive. 

We ordered up a new computer (the existing one was coming up for replacement anyway) with an SSD and a backup drive and IT got it configured by Monday afternoon. Thankfully, Thomas, my LD had backed up most of his work to his hard drive before leaving the week before. So he didn’t have nearly as much to do in order to be ready for rehearsals. Still, it was a nerve-wracking few days. 

Moral of the story: back it up! If you don’t have at least 3 copies of every media and show file in at least 2 different media, you are at high risk of losing the show. Because if a drive or computer is going to fail, it’s going to fail right before a big weekend like Easter. Don’t fall prey to that. Some simple backups strategies will save your sanity.

Dropbox

There’s really no reason why you shouldn’t have all your show and media files in Dropbox. It’s free for 2 gigs, which is more than enough for most everything important. I personally pay for the pro version every year and have a Terabyte of storage at my disposal. After each programming session, copy all your files to Dropbox and you’ll have several copies of them (especially if you have that Dropbox folder set up to sync to your laptop or desktop computer). 

We had Dropbox on all our tech booth computers, which meant every file from every system was on every machine. Local sync makes that fast and easy. And, with all those local backups, restoring is super-fast, too. 

Clone Drives

Another lesson we learned after the Great Lighting Console Crash of ’11 was that we needed to have fully bootable backups of the system drives of each mission critical computer. For Macs, I use Carbon Copy Cloner. We had another program we used to copy the entire Windows partition of Bootcamped Macs, but I don’t recall what it was. You can probably find it on Google. Norton Ghost is something you can use to clone PC hard drives. 

The main reason I clone drives is that it makes it faster to get up and running. Sure, you could start with a new computer, re-install all your software and configure all your settings. But that takes time, and you might not have it. That may be something you want to do, but if you need to get up and running fast, plug in the clone and get back to work.

Of course, for clones to be useful, they need to be current. We updated our clone drives every month. That way, we were never too far out of sync. When prepping for big weekends, we’d update the clones as soon as programming was done. 

Console Show Files

A show file will get corrupted at the worst times. I always save copies of my show file to a thumb drive when I’m done mixing. Even when I’m volunteering on a weekend at my church, I save the show file to my thumb drive Saturday night. At Coast Hills, we always ran an external computer sync’d to the console, and that show file was auto-sync’d to Dropbox on shutdown, which meant we had about 10 copies of the show file in just a minute after we shut the console down. Never hurts to have a spare. 

Hopefully this will give you the incentive you need to back everything up this week before you roll into Easter week. My hope is that this will save at least one person a lot of grief next week.

Elite Core

DMX Over Cat5, Pt. 2

Last time around, we talked about using Cat5 cable to distribute DMX signals. In that implementation, it is really cable replacement. Instead of pulling DMX cable (not mic cable—there is a difference), we pull Cat5 for our backbone distribution runs. Fixture to fixture cables are normal DMX cables. Today, I want to talk a little bit about using Ethernet to distribute DMX. This will be an overview article as there is way too much information to contain in a single post. Also, some of the standards are still evolving, and it’s not always simple, especially when mixing multiple manufacturers. Come to think of it, we need to do a podcast on this…

Here is a basic DMX network diagram. This is courtesy of Pathway Connectivity. 

Here is a basic DMX network diagram. This is courtesy of Pathway Connectivity

The Original Ethernet—DMX Protocols

In the beginning, we had things like: 

  • Strand Shownet
  • ETC Net1
  • ETC Net2
  • ArtNet
  • Pathport

Each of those protocols use Ethernet wiring and switchgear to distribute multiple universes of DMX throughout a facility. All of them require some time of break in and break out adapter, as well as at least one Ethernet switch to get all the nodes talking to each other. In and of themselves, they were fine. The problem was, none of them talked to each other. Some devices could speak multiple languages, but the languages themselves were not compatible. If having an all ETC Net2 system was what you needed for example, it worked well. But introducing another standard into the mix was problematic. 

Still, those protocols worked well. They offered up to 128 universes, unlimited outputs, signal management (splitting, routing, prioritizing), and because it was all based on Ethernet standards, it was inexpensive to install and manage. So far so good. But you were using Ethernet, and RJ45 connectors aren’t the most robust. And Category cable is fragile compared to a regular DMX cable. 

The New Hotness—ACN

As often happens, when engineers see protocol soup like we have above, they look for a way to create a new one that will do everything the old ones would do, and more, and do it easier. That’s the promise of ACN. ACN stands for Architecture for Control Networks and defines a series of nested Protocol Data Units—a whole series of TLAs (Three Letter Acronyms) defining how data gets moved around.

What’s cool about ACN is that it is media agnostic; you can use whatever cable you want. It’s designed to be interoperable, so multiple manufacturers equipment can be used together. It’s supposed to be plug and play, which simplifies setup. It’s also two-way, meaning the end devices can report their capabilities to the controller, and the controller will know what to do with it. In theory, this means we can get rid of fixture libraries someday. 

ACN uses an Ethernet backbone, so configuration and system architecture is familiar. I’ve been telling you that as a technical leader, you’re going to need to know more about networking. We know that’s true of audio, and it’s becoming more and more true of lighting and media servers. 

What’s Available Now?

Like many new standards, it will take time to implement. While there are some media servers and the like on the market that use ACN, there are few fixtures that do. Hopefully that changes in the next few years. Right now, we have ETC’s variant of ACN known as Net3. Pathway Connectivity uses sACN (Streaming ACN) in their Pathport products. And believe it or not, these two can talk to each other!

The good news is that we can install ACN backbone systems now, and simply break in and out to DMX as needed. Someday when ACN becomes commonplace on fixtures as well as controllers, we remove the adapters and everything talks ACN. And this is happening; many of the Jands consoles for example, already speak sACN and will simply output their DMX universes straight to the network. 

This is an exciting time to be in this industry. I was with a friend the other day and he showed me an installation that required hundreds of universes of DMX to manage. There’s no way anything like that would even be conceivable using regular old DMX. But with ACN, it’s easily possible. 

If you want to learn more about this, check out Pathway Connectivities Resource page. They have some articles and a Power Point presentation with good info (it’s where I got some of this content—thanks for that, guys!). Now is a great time to begin learning more about ACN, as it will be the standard going forward. Hopefully, we don’t have to wait 10 years before we start seeing native ACN fixtures…

Roland

Today's post is brought to you by DPA Microphones. DPA's range of microphones have earned their reputation  for exceptional clarity,  high resolution, above all, pure, uncolored accurate sound. Whether recording or sound reinforcement, theatrical or broadcast, DPA's miking solutions have become the choice of professionals with uncompromising demands for sonic excellence.

Backing Up Production Machines

Photo courtesy of Jaymis Loveday

Photo courtesy of Jaymis Loveday

Last time we talked about my thoughts on upgrading (or not upgrading) your production machines. Today, I’m going to talk about creating a safety net for them. Again, these are lessons I’ve learned over 25+ years of managing mission critical systems, so learn from my mistakes. 

Maintain a Current Clone of Your Drives

Hard drives will fail. Usually at the worst time. Like Palm Sunday morning. Yeah, that happened. Even SSDs, which are proving to be pretty dang reliable, will fail at some point. The only way to get back up and running quickly is with a full image backup. For all my mission critical production machines, I had a small hard drive with a full, bootable clone backup on it. I used to have several to manage, but after a while, I just bought a 1 TB pocket drive and partitioned it to back up 2-3 machines on one drive.

On the Mac, you can use a program like Super Duper for my favorite, Carbon Copy Cloner to keep an exact copy of the drive. On Windows, you can use Ghost or a similar program. If the main drive fails, or an upgrade breaks stuff, you can boot from the backup, and effectively unwind time. Any time I made significant changes to the machine, I would update the clone, but only after I verified the changes worked properly. If an update went south, we broke out the backup and rolled it back. 

For those of use that find Macs make better PCs than PCs do and need to backup your Bootcamp partitions, my IT guy told me about a cool program called WinClone. To use it, boot into the Mac OS, and run WinClone. It will compress and clone your Windows “drive” and save it as a file to another drive. Should Windows go south, you can simply restore the Windows drive and it will be like nothing never happened.

This is great for the OS and applications, but you can easily lose files like ProPresenter songs, for example. But I have a solution for that, too.

Use Dropbox for Libraries.

I set up a Dropbox account for my important show files. I wrote a full guide to this in an article called Back It Up: Presentation, but the gist is that you store (or maintain a cloned copy) of your show files, songs, templates and maybe media in a Dropbox folder that is automatically updated to the cloud. If you have to blow the drive out and restore from a clone, Dropbox will put your library files back. I don’t think I would ever run a production machine without Dropbox.

Maintain Incremental Backups

I really like Time Machine, especially lately. It’s a lot lower overhead than it once was and can really save your bacon if you delete or mess up a file. But, don’t run Time Machine backups during the service. Time Machine can be processor and disk intensive, and it’s highly possible that it will mess up your media playback. I prefer to keep Time Machine on an external drive so I can simply leave the drive turned off during the service. If you simply must have it on an internal drive, use Time Machine Scheduler to avoid service times. But honestly, an external drive is easier. Time Machine won’t even try to back up if the drive isn’t there, so it’s foolproof.

Clone Before Updating

Before we upgraded our production machines to a new OS, I went through and cloned all the drives. I did this so we could go back if something didn’t work. It’s a whole lot easier to simply restore the clone than it is to downgrade the OS, then Time Machine everything back. When it comes to upgrading, clones are your best friend. And honestly, drives are cheap enough now, you can easily maintain several versions of the clone if you want. Do one right before you update, then one right after. Small pocket drives from WD and Seagate are perfect for this task. At under $100 each, you can afford to have several for each machine. 

Hopefully this has helpful for you. Now if you’ll excuse me, I have to update some of my backups…

Today's post is brought to you by Elite Core Audio. Elite Core Audio features a premium USA built 16 channel personal monitor mixing system built for the rigors of the road. For Personal Mixing Systems, Snakes, and Cases, visit Elite Core Audio.

Upgrading Production Machines

Not what you want to see on Sunday morning...

Not what you want to see on Sunday morning...

We got to talking about this topic on one of the recent CTW episodes, and I thought it would be a good post. When I was on staff as a TD, I had a pretty strict policy regarding our production machines. Now that I’m working as an integrator, I dread the days following a major Mac OS update. That’s because I know I will soon be getting calls that start with, “We just upgraded all our iMacs to latest, greatest OS X… and fill in the blank software doesn’t work right anymore…” At that point, all I can say is, “Yeah, I usually don’t upgrade right away. Or ever, really. But you have a full image backup from before the upgrade, right?” Silence…

So in the interest of preventing said calls and emails, let me give you a few pointers on how to manage production machines. These are lessons I learned—many of the them the hard way—over 25+ years of managing production computers. It’s important to note that production machines are different from office computers. If an office computer goes down, you may not be able to get to your email for a little bit (except through your phone), but otherwise, nothing bad really happens. 

If a production machine goes down on Sunday morning, bad, bad things happen. If you upgrade on Friday and break something, the next 36 hours will be stressful. You really don’t want to be beta testing new software on the weekend. Here’s my guide to keeping your sanity with your computers. 

Don’t Upgrade Unless You Have To

Most of the time, you don’t have to upgrade your production machines. When I was at Coast Hills, most of my machines were running the latest version of 10.6 until early 2014 when we upgraded to 10.7. Why? Because it worked. If everything works on the OS you have, don’t upgrade it. I really like computers that start up and go to work every time without any fanfare. Avoiding unnecessary updates helps this.

My triggers for updating the OS are twofold: First, if some production software updates and introduces new features that I really need, and it requires a newer version of the OS, then I’ll update. Second, if the OS update introduces new features I really need, I’ll update to that version after the next version comes out. I like to stay about 1 version back at least. 

Don’t Upgrade Right Away

Computer code has become so complex it’s almost impossible to catch all the bugs and problems in a program before release, let alone an operating system. Apple is pretty good, but there is no way they can know how a new OS will affect every user. And many churches are still using older hardware and peripherals like audio or video interfaces, and a new OS can break the drivers for a while or forever. This is perhaps my #1 rule of OS updates: DON’T UPGRADE RIGHT AWAY. Let others beta test it first. 

I stay behind by at least one version because that allows time to get drivers and software updated and working solidly. Remember, we prefer reliable performance to fancy new features. 

Turn Off Auto Updates

One of my biggest pet peeves for production machines is auto updates. Windows used to be the worst at this, but now Apple has joined the fun in the last two versions. Unless you configure it properly, both OS’s will happily install new software or system updates all on their own and that can easily break things. Until I figured out how to turn it off, we kept having Windows kick up a message saying it would reboot the machine in 10 minutes to install updates every Sunday morning! Google it to learn how to turn that off. 

This does mean you should stay on top of a manual update routine, especially for security updates. But do that on Monday or Tuesday, then test everything thoroughly during the week to make sure it works. If you leave your computers on all the time, you really need to be careful of this. The last thing you want is to come in on Sunday only to find your software updated and no longer works right.

Verify All Software Will Work—Including Drivers

I just upgraded my studio Mac Mini to Mavericks, mainly because I installed a second screen and wanted to take advantage of the updated Spaces functionality. I waited so long because I wanted to be sure all my audio interface software would be good. I use this machine every week for CTW, and it has to work.

If you use an external peripheral that relies on driver software, be sure it’s approved for the OS you want to use before upgrading. I’ve heard from several people that they decided to upgrade their OS and now some critical external piece doesn’t work anymore. Remember, unless you have to upgrade, don’t. 

Those are a few suggestions for the upgrade process. If you take anything from this, it’s don’t upgrade. At least not unless you absolutely have to. Next time, I’ll give you some suggestions for creating a safety net for your computers. In the meantime, my friend Joel Smith has written a great guide on keeping ProPresenter machines working reliably. You should go read it. 20 Steps To Maximizing ProPresenter For Mac

Roland

Today's post is brought to you by myMix. myMix is an intuitive, easy-to-use personal monitor mixing and multi-track recording system that puts each user in control of their own mix! myMix features two line-level balanced 1/4" TRS outputs and one 1/8" (3.5mm) headphone output, the ability to store up to 20 named profiles on each station, 4-band fully parametric stereo output EQ recording of up to 18 tracks plus stereo on an SD card. Learn more at myMixaudio.com