I’ve just read an amusing series of blog posts about bad wiring [1]. I’ve seen my share of wiring horror in the past. There are some easy ways of minimising wiring problems which seem to never get implemented.
The first thing to do is to have switches near computers. Having 48 port switches in a server room and wires going across the building causes mess and is difficult to manage. A desktop machine doesn’t need a dedicated Gig-E (or even 100baseT) connection to the network backbone. Cheap desktop switches installed on desks allow one cable to go to each group of desks (or two cables if you have separate networks for VOIP and data). If you have a large office area then a fast switch in the corner of the room connecting to desktop switches on the desks is a good way to reduce the cabling requirements. The only potential down-side is that some switches are noisy, the switches with big fans can be easily eliminated by a casual examination, but the ones that make whistling sounds from the PSU need to be tested first. The staff at your local electronics store should be very happy to open one item for inspection and plug it in if you are about to purchase a moderate number (they will usually do so even if you are buying a single item).
A common objection to this is the perceived lack of reliability of desktop switches. One mitigating factor is that if a spare switch is available the people who work in the area can replace a broken switch. Another is that my observation is that misconfiguration on big expensive switches causes significantly more down-time than hardware failures on cheap switches ever could. A cheap switch that needs to be power-cycled once a month will cause little interruption to work, while a big expensive switch (which can only be configured by the “network experts” – not regular sysadmins such as me) can easily cause an hour of down-time for most of an office during peak hours. Finally the reliability of the cables themselves is also an issue, having two cables running to the local switch in every office can allow an easy replacement to fix a problem – it can be done without involving the IT department (who just make sure that both cables are connected to the switch in the server room). If there is exactly one cable running to each PC from the server room and one of the cables fails then someone’s PC will be offline for a while.
In server rooms the typical size of a rack is 42RU (42 Rack Units). If using 1RU servers that means 42 Ethernet cables. A single switch can handle 48 Ethernet ports in a 1RU mount (for the more dense switches), others have 24 ports or less. So a single rack can handle 41 small servers and a switch with 48 ports (two ports to go to the upstream switch and five spare ports). If using 2RU servers a single rack could handle 20 servers and a 24port switch that has two connections to the upstream switch and two spare ports. Also it’s generally desirable to have at least two Ethernet connections to each server (public addresses and private addresses for connecting to databases and management). For 1RU servers you could have two 48 port switches and 40 servers in a rack. For 2RU servers you could have 20 servers and either two 24port switches or one 48port switch that supports VLANs (I prefer two switches – it’s more difficult to mess things up when there are two switches, if one switch fails you can login via the other switch to probe it, and it’s also cheaper). If the majority of Ethernet cables are terminated in the same rack it’s much harder for things to get messed up. Also it’s very important to leave some spare switch ports available as it’s a common occurrence for people to bring laptops into a server room to diagnose problems and you really don’t want them to unplug server A to diagnose a problem with server B…
Switches should go in the middle of the rack. While it may look nicer to have the switch at the top or the bottom, that means that the server which is above or below it will have the cables for all the other switches going past it. Ideally the cables would go in neat cable runs at the side of the rack but in my experience they usually end up just dangling in front. If the patch cables are reasonably short and they only dangle across half the servers things won’t get too ugly (this is harm minimisation in server room design).
The low end of network requirements is usually the home office. My approach to network design for my home office is quite different, I have no switches! I bought a bunch of dual-port Ethernet cards and now every machine that I own has at least two Ethernet ports (and some have as many as four). My main router and gateway has four ports which allows connections from all parts of my house. Then every desktop machine has at least two ports so that I can connect a laptop in any part of the house. This avoids the energy use of switches (I previously used a 24 port switch that drew 45W [2]), switches of course also make some noise and are an extra point of failure. While switches are more reliable than PCs, as I have to fix any PC that breaks anyway my overall network reliability is increased by not using switches.
For connecting the machines in my home I mostly use bridging (only the Internet gateway acts as a router), I have STP enabled on all machines that have any risk of having their ports cross connected but disable it on some desktop machines with two ports (so that I can plug my EeePC in and quickly start work for small tasks).
I think you overestimate the average end-user’s willingness to do any work that is not part of their job description. In an ideal world, switches for each group of desks would work, but in the real world there are problems:
1) In any company over about 20 employees, the vast majority of employees will have zero vested interest in the company, and will therefore either be reluctant or utterly unwilling to do anything outside of their job description, even if it just involves power-cycling a switch.
2) End users are downright terrified of technology. They’ll treat the switch on their desk as though it’s an angry cobra and will refuse to get near it.
3) If the switch needs to be reset every month, the IT department in the company is not doing its job.
Wiring is a dirty fact of modern technology. Some of it is awful, and some of it just LOOKS awful. (Many of the pictures in the blog you referenced were of REALLY good wiring jobs, they just look bad because there are so many wires.) The fact of the matter is that wiring is an “expense” of technology, and it is up to us, as technology professionals, to deal with that wiring so that our end users can focus on their jobs.
The basic business responsibilities perspective notwithstanding, however, there are still a few purely technical issues:
1) Once an organization grows beyond a 24-port switch, it’s time to invest in good switches that have individually configurable 8-12 port modules. A mis-configuration will now only knock out a few users, and those modules can be swapped out quickly if necessary.
2) Having individual switches on desks encourages quick and dirty solutions like stringing switches together. Put any more than 3 switches in a line and you start getting nasty network issues.
3) With a proper naming scheme for the wires and a few simple wire testing tools it’s quite easy to diagnose wiring issues.
Just my observations. Most of the ideas you present are very good overall, but in my experience, they just don’t work nearly as well in PRACTICE as it seems that they should. :)
Alex: 1) In my experience users are more than willing to power-cycle devices. My experience (being involved in supporting Internet gateways for a number of companies) is that when things stop working everything in the area ends up power cycled, the DSL modem, the router, and the PC on the desk (in that order). Sometimes it’s just a remote web site that has gone down or the company has been black-listed for having a virus-ridden Windows box that sent out spam, but everything gets rebooted anyway.
2) Some users might, but there are always a few computer hobbyists in every office.
3) I agree that it would be bad to seek out switches that need to be power cycled every month. But you seem to end up with some anyway. If the switch in question cost $5000 then you are stuck with it and just need to automate the reboot. If it cost $100 then you have a choice of having the nearest person power-cycle it (which they probably won’t even tell you about) or just spending another $100. Both options are viable.
I agree with your point about some of the wiring being good. But while it may be technically good to have an ordered and labeled set of 100 cables, having an architecture that allows an ordered and labeled set of 5 cables is better.
Let’s not forget that organisation structures change over time. While you might have a team of people who can do a really good wiring job now, next year you might have people who are less skilled – and then things go horribly wrong. Designing things so that they can be maintained by someone with less skill than you is a good idea for many reasons.
In regard to your second points:
1) So you have several switches in one box? I’d rather just have several switches so that one failure CAN’T cause other failures.
2) Well, it’s not as if that doesn’t happen already…
3) Sure, and someone doesn’t work while waiting for the person with the wire testing tools to arrive and they disturb others while waiting.
I prefer the rack installations which use a patch panel at the top (or bottom) of each rack, leading to whatever (usually large) switch(es) as appropriate.
This keeps the network tech within the rack as simple as possible (it’s just a wire), keeps the labelling simple (“Server gandalf, NIC #2 Switch 4, Port 43”), and allows for centralised management of the network infrastructure (allowing you to use things like tagged VLANs usefully too). Of course, that assumes a centralised and competent network admin team; without that, you may as well try to keep everything as flat and simple as possible, still try to avoid Alex’s “Issue #2” above.
With the above example, you maintain docs saying that in Cab 3, Gandalf’s NIC2 goes to local Patch Panel port 23, which is connected to Cab 1’s Patch Panel port 3.23, which goes to Switch 4 Port 43.
Steve: You assume a centralised and competent network admin team that actually controls everything. All it takes is one guy to go in there to fix a server in an emergency and change a few cables without documenting it and then it becomes a total mess. The patch panel and big central switch idea means that it’s difficult and complicated to do things right, easy to do things wrong, and time consuming to fix problems once they have been created.
I think that there should be a design goal to make it easy to do things right and easy to fix problems once they occur.