Old Production Takes From an Old Guy

A Few System Design Principles

This might seem like an IT specific post, but stick with me. I think there are principles that can apply to any type of system you’re designing and be helpful. First, a little background. Upper Room started as a sub-minisry of Christ Presbyterian Church. As a sub-ministry, we’ve been utilizing the IT resources of CPC since our inception. And for the most part, that’s been fine. However, with the planting of Upper Room as an independent church, we had to start looking into our own IT system. As the resident geek (and with no budget to hire a real IT guy), the task fell to me. And I’m not completely unqualified, I’ve been doing IT related stuff on and off since 1991 when I convinced my then employer to fire the company we bought our new computer system from and promote me to IT Director. And that worked out OK, so…

Anyway, I digress. The task at hand was to build a new IT system. Our needs are really pretty modest. We need to access the internet quickly and without at lot of barriers. We need to share and store some files on a central drive accessible by all. We need mail and some basic calendar sharing. And we need a way to back up our laptop and desktop machines. As with a lot of things, there are many different ways we could accomplish that. And therein lies the first principle.

Decide what the real priorities are.

I spent several hours with the church leadership trying to figure out exactly what the real needs where. Not the stated needs, but the real ones. The biggies turned out to be network backup, e-mail and calender sharing (actually, that last one was for one person only, but he’s the boss so…). The trick to this step is that sometimes people will say one thing, but mean another. For example, I know of one church that wanted to ban all cords from the stage. They wanted a clean look, and wanted to go all wireless…for everything. That was the stated need. Then they were presented with the cost. The real need turned out to be a neat stage. After the sound guys learned this, they started dressing cables neater, and keeping the stage tidy. As far as I know there hasn’t been a huge desire to go all wireless anymore.

Next, determine the timeline.

If I were looking at a short-term solution, I could accomplish the above goals pretty simply. Finding a solid internet connection isn’t that hard, and a garden variety firewall would keep out most of the bad guys. We had already switched to Google apps for our e-mail, and it’s working well enough, so that’s solved. I could easily throw a simple NAS (network attached storage) drive on the network and call that shared storage. And I could have picked up some inexpensive USB drives and enabled Time Machine.

However, we’re in a very unique situation right now. We are not really a church plant, because we have a staff of 8-10 people. And we’ve become very used to things working at an enterprise-class of service. And since we’re “planting” with probably 600-700 people on day one, we can expect some significant growth over the next year or two. So this means I can’t lowball the system. Even if I only spent a few thousand dollars on some hardware, it would be largely wasted when we upgraded to a “real” network system in a year or two. We’re also in a situation where we are being graciously supported by CPC, which means we have a few more dollars at our disposal than we might in a few years. So I had to start thinking long-term. What type of system would work for us in 3-5 years, if not longer?

Immediately, I thought XServe. It’s way more hardware than we need right now, but it will take a while for us to outgrow it. I also started thinking of going with a hardware RAID 5 for network storage and Time Machine backup. Storage is pretty cheap right now, so I budgeted for 4.5 TB. It’s probably 3-4 times what we need right now, but I’m thinking about what will work in 3 years. I also started looking at a hardware-based firewall that is designed to protect small enterprises. I put together the budget and turned it in. Amazingly, it was approved! Which brings me to my next principle.

Well almost. I’ll bring you that one (and a few more) tomorrow.

14 Comments

  1. jdboyd@jdboyd.net

    I found that ZFS is a lot nicer than hardware raid. While software raid has a reputation for being slow, ZFS shows that it doesn’t have to be slow any more. Also, ZFS is extremely easy to operate. Probably the only RAID system that is simpler is the Drobo.

    You might consider either stalling until OSX Server 10.6 comes out, or try your hand at a Solaris server, and then use the money that would have been spent on the hardware raid on more disk space or more servers.

    Solaris is free to use, and the support contract is pretty affordable. Solaris support NFS, CIFS, and iSCSI out of box. It is pretty easy to set up mDNS on it (that is what Apple is using to power Bonjour). iSCSI means that you can concentrate all of your server storage in one big box and have additional servers just boot from the network, internal flash, or internal mirrored disks. Solaris no longer requires Sun hardware (although I very much like Sun hardware). It also is an orderable option on some Dell PowerEdges and some IBMs, as well as being installable after the fact on a wider range of hardware.

  2. jdboyd@jdboyd.net

    I found that ZFS is a lot nicer than hardware raid. While software raid has a reputation for being slow, ZFS shows that it doesn’t have to be slow any more. Also, ZFS is extremely easy to operate. Probably the only RAID system that is simpler is the Drobo.

    You might consider either stalling until OSX Server 10.6 comes out, or try your hand at a Solaris server, and then use the money that would have been spent on the hardware raid on more disk space or more servers.

    Solaris is free to use, and the support contract is pretty affordable. Solaris support NFS, CIFS, and iSCSI out of box. It is pretty easy to set up mDNS on it (that is what Apple is using to power Bonjour). iSCSI means that you can concentrate all of your server storage in one big box and have additional servers just boot from the network, internal flash, or internal mirrored disks. Solaris no longer requires Sun hardware (although I very much like Sun hardware). It also is an orderable option on some Dell PowerEdges and some IBMs, as well as being installable after the fact on a wider range of hardware.

  3. jrygel@hotmail.com

    I have had the privelidge of working with churches for the last several years that are forward thinking like this, or at least, they allow me (and those who have gone before me) the budget to be forward thinking. I am very definitely NOT an IT guy (getting an LS9 connected to a computer tested the limit of my networking abilities), but the same can be said of sound, lights, and video. It’s always difficult to balance between the demands of the present (budget) and the future (expanded capability).

    I am curious, do you have a permanent space for your new ‘plant’ or are you going to be a porta-church for a while? This usually makes the disparity between current needs and future needs that much greater, especially for sound, video, and sometimes lights.

  4. jrygel@hotmail.com

    I have had the privelidge of working with churches for the last several years that are forward thinking like this, or at least, they allow me (and those who have gone before me) the budget to be forward thinking. I am very definitely NOT an IT guy (getting an LS9 connected to a computer tested the limit of my networking abilities), but the same can be said of sound, lights, and video. It’s always difficult to balance between the demands of the present (budget) and the future (expanded capability).

    I am curious, do you have a permanent space for your new ‘plant’ or are you going to be a porta-church for a while? This usually makes the disparity between current needs and future needs that much greater, especially for sound, video, and sometimes lights.

  5. ryan@nerdhook.com

    Have you found Time Machine to actually be a reliable backup?

    I’ve had one too many problems with the sparse image bundle file becoming corrupt to have much faith in it. I still use it, since it’s dead simple, and as you said, storage is cheap.

    But for actual critical backups, I use Chronosync (+ Drobo).

    Just curious if my Time Machine experiences are the exception or the rule…

  6. ryan@nerdhook.com

    Have you found Time Machine to actually be a reliable backup?

    I’ve had one too many problems with the sparse image bundle file becoming corrupt to have much faith in it. I still use it, since it’s dead simple, and as you said, storage is cheap.

    But for actual critical backups, I use Chronosync (+ Drobo).

    Just curious if my Time Machine experiences are the exception or the rule…

  7. gccjason@gmail.com

    Another critical piece to consider is support. What if God calls you to do ministry elsewhere? Is the system you build able to easily be handled by an outsourced company or volunteers?

    RE: backups – what if the building has a fire or natural disaster? Or someone steals the drobo? (I know several churches that were broken into and all their computers (and thus data) were lost). I’d recommend checking out an offsite solution like mozypro.com and get at least the mission critical data offsite.

    If you need to bounce ideas off someone or see what other churches are doing check out our huge list of church IT resources http://www.jasonpowell.net/jason_powell_church_it/2008/10/church-it-resou.html … especially the live IRC chat.

    Jason

  8. gccjason@gmail.com

    Another critical piece to consider is support. What if God calls you to do ministry elsewhere? Is the system you build able to easily be handled by an outsourced company or volunteers?

    RE: backups – what if the building has a fire or natural disaster? Or someone steals the drobo? (I know several churches that were broken into and all their computers (and thus data) were lost). I’d recommend checking out an offsite solution like mozypro.com and get at least the mission critical data offsite.

    If you need to bounce ideas off someone or see what other churches are doing check out our huge list of church IT resources http://www.jasonpowell.net/jason_powell_church_it/2008/10/church-it-resou.html … especially the live IRC chat.

    Jason

  9. dennyd@triple-d.us

    http://aws.amazon.com“ rel=”nofollow”>Amazon S3 is a nice solution for back-up storage. Very affordable, and it can scale as your needs change over time. There’s no need to spend a lot of time/money to acquire/maintain/repair/upgrade/replace backup hardware. http://jungledisk.com“ rel=”nofollow”>Jungle Disk (backup client) is working on a Time Machine restore interface for an upcoming release as well. Might not suit your specific needs, but it may be worth checking out.

  10. dennyd@triple-d.us

    http://aws.amazon.com“ rel=”nofollow”>Amazon S3 is a nice solution for back-up storage. Very affordable, and it can scale as your needs change over time. There’s no need to spend a lot of time/money to acquire/maintain/repair/upgrade/replace backup hardware. http://jungledisk.com“ rel=”nofollow”>Jungle Disk (backup client) is working on a Time Machine restore interface for an upcoming release as well. Might not suit your specific needs, but it may be worth checking out.

  11. Church Tech Arts » A Few

    […] we’re back to looking at some principles of system design. If you missed yesterday’s post, here is the Reader’s Digest version: First find the real needs, next determine your […]

  12. mike@churchtecharts.org

    Great thoughts, all! Thanks for participating in the discussion. I wanted to address the questions that have arisen…

    jdboyd–I’ve been watching ZFS for a bit now. Looks like 10.6 include it. Sadly, due to our timeline and the inevitable delay in the actual release date of Snow Leopard server, I don’t think we can wait. I’m going with a RocketRaid 3522 for the hardware RAID, which has a pretty easy to use interface. And it’s wicked fast with a Mini-SAS interface.

    Justin–I’m not sure if we’ll be truly portable or not at this point. We’re looking at renting space in another church, and would hope to “permanently” install some of our stuff. If we go really portable, you’re right, that’s a game changer. Then I design it like I would a touring rig. As it stands right now, it looks like we’ll be doing more of a permanent install, that may have to be undone when we leave.

    Ryan–So far so good for me with Time Machine. I’ve used it several times to recover files I lost or deleted. Like you, I believe in a belt and suspenders approach, so my critical files will all be backed up (several times) with various approaches to ensure redundancy.

    Jason–You’re right, support is critical. One of my very first design principles (that I didn’t mention here) is that it has to be volunteer friendly. I never design a system or process that I don’t think could be easily handed off to a volunteer. I’m a big proponent of working myself out of a job. I’ve inherited enough badly designed and implemented systems were the previous person assumed they would be the last person to ever work on it to do that to anyone else.

    Offsite–right there with you. I’ll be backing up all the critical shared storage files on external, removable drives weekly (at least) and taking those off-site. Everything will be set up to run on scripts automatically, so it could easily be handed off to a volunteer.

    Denny–Good call on the online stuff. I’m not sure we’re ready for S3 backup just yet, but it’s good to keep an eye on.

  13. mike@churchtecharts.org

    Great thoughts, all! Thanks for participating in the discussion. I wanted to address the questions that have arisen…

    jdboyd–I’ve been watching ZFS for a bit now. Looks like 10.6 include it. Sadly, due to our timeline and the inevitable delay in the actual release date of Snow Leopard server, I don’t think we can wait. I’m going with a RocketRaid 3522 for the hardware RAID, which has a pretty easy to use interface. And it’s wicked fast with a Mini-SAS interface.

    Justin–I’m not sure if we’ll be truly portable or not at this point. We’re looking at renting space in another church, and would hope to “permanently” install some of our stuff. If we go really portable, you’re right, that’s a game changer. Then I design it like I would a touring rig. As it stands right now, it looks like we’ll be doing more of a permanent install, that may have to be undone when we leave.

    Ryan–So far so good for me with Time Machine. I’ve used it several times to recover files I lost or deleted. Like you, I believe in a belt and suspenders approach, so my critical files will all be backed up (several times) with various approaches to ensure redundancy.

    Jason–You’re right, support is critical. One of my very first design principles (that I didn’t mention here) is that it has to be volunteer friendly. I never design a system or process that I don’t think could be easily handed off to a volunteer. I’m a big proponent of working myself out of a job. I’ve inherited enough badly designed and implemented systems were the previous person assumed they would be the last person to ever work on it to do that to anyone else.

    Offsite–right there with you. I’ll be backing up all the critical shared storage files on external, removable drives weekly (at least) and taking those off-site. Everything will be set up to run on scripts automatically, so it could easily be handed off to a volunteer.

    Denny–Good call on the online stuff. I’m not sure we’re ready for S3 backup just yet, but it’s good to keep an eye on.

  14. Church Tech Arts » Cutti

    […] I discussed in a few previous posts on design principles (here and here), the first thing we went back to is this: What are our values? This is still a work in […]

© 2021 ChurchTechArts

Theme by Anders NorenUp ↑