I've seen a handful of datacenters, run by different companies, claiming "2N power", where two "power rooms" or "UPS rooms" labeled "A" and "B", power the servers; and where each of the two UPS rooms contained a cluster of UPSes, instead of just one big UPS per room. The interesting thing is that, within a single "power room", each and every UPS in the cluster needs to be on-line to meet "design load" (power the servers), IF there were only one working power room. (if both A and B rooms are working, then the servers can get by with one UPS off-line in each UPS room).
For example, a server room could be powered from an "A" and a "B" UPS room. Each UPS room might have two UPSes paralleled together. If the whole "A" UPS room were down, then both UPSes in the B-room would need to be on-line in order to power the server load.
In the datacenters I saw, the operator claimed that this arrangement allowed them to do maintenance on one UPS in the "B-side" UPS room, without putting the whole "B-side" UPS cluster into bypass mode. That's nice: both A and B UPS rooms would produce clean power (not on bypass). But you can't parallel an on-line UPS with a UPS in bypass, so I guess that they must shut-down the UPS they are working on(?), and let the remaining one in the room carry the load. Or if they have 3 UPSes in the UPS room, then they shut down one and let the other 2 carry tho load.
I am wondering about this arrangement. If there were a failure in the UPS room which isn't having maintenance work (for example, transformer failure), then all the servers would take power from the remaining UPS room... but that room can't give enough power because one UPS is off.
Wouldn't it be better to transfer load for the "B" room to generator before working on one UPS, so that if the "A" UPS room has a failure, critical load is supported? If you really want to keep the "B" room on-line during maintenance, it seems like you should have N+1 UPSes parallled there, and not just N.
I agree. The ideal situation would be 2(N+1); otherwise, when room A is down a UPS and can't carry the full load, room B has no backup in case something fails. Another option might be 2N+1, where an extra UPS is switchable between banks, but I've never actually seen this in practice.
In an 2N or N+2 you should have 4 power paths- A&B fully redundant power path on both separate power path , SO you should still be running in what you described but may have a blip on what that individual UPS is supporting.
Thanks, Ken. The safest practice may be to bypass the whole UPS cluster (on generator, of course), when maintaining either UPS. That said, I have executed this bypass in facilities where A and B-side PDUs are single-feed, and some operators have serious concerns about using the unconditioned generator power on B-side. It doesn't matter if the generators are multi-meg, with TVSS conditioned output, and you show the operators the power quality measurements.
My thought is that if the equipment is expensive enough (e.g. MRI scanner), then it needs 2N+1 to avoid this problem. If the load is consumer electronics, then the reduction in power supply MTTF from using generator power for a few hours per year is probably less than the variation caused by other factors. For example, these same careful DC operators rarely make their techs wear grounding straps when touching servers, and airflow blanking is rarely perfect.
Personally, I'd rather wraparound the whole cluster during maintenance, rather than keep the cluster on-line, but at half-capacity. But I can see the pressure that would drive someone to take chances, to try to avoid wraparound.
Some companies have taken redundancy to extremes with 2(N+1) where the entire A and B planes are redundant all the way to the Utility feeds. Even to separate buildings for each power source with N+1 in each (UPS/Gen/Transformer) and yet human error has triumphed over engineering excellence.
In my experience, no matter how much redundancy and complexity we build into our DC’s, humans without and malicious intent will always find a way to kill it.
I now realize that humans are simple creatures and require simple easy to understand systems and work practices. I would now recommend a simple A and B plane with the minimum number of components in the chain. If UPS maintenance is required, the entire plane (A or B) is bypassed. There are a few things you will need to consider with these designs:
You need to allow more than the normal transformer capacity. The full site load may be running on the same transformer that is also supplying the UPS load test power.
The site must also have a very well designed Surge Diverter system to ensure that nothing harmful gets to the load.
The main disadvantage with this topology is that unconditioned power is provided to one plane of the load. Unless you live in a third work country (or Enron is your utility) the power quality will be acceptable.
Always keep in the back of your mind that humans will do the wrong thing for all the right reasons. As the streaker said to the Judge “It seemed like a good idea at the time”.
2N power, with clustered-N?
