KVM/IP makes it possible to work remotely 99.9% of the time. The only time you need to be in the datacenter is when physical interaction is needed (swap hard drive, fix servers, etc).
I personally prefer to be on there. I have only ever worked in one Data Center before. They were not the best data center and for the most part i was the only tech on duty. I always prefer being there as I feel like I have more control. It is a lot more hectic but... well you cant beat the control you have with actually being there
Yes, there is no doubt that most things can be done remotly. That is why there is a large market for remote system administration. You can even really do remote reboots now. The only thing you can not do is hardware related jobs...
THis is the sort of argument of working at home isnt it. We had it on WHC. If your office is the Data Center then I would spend more time there. I am comforted by all the expensive machinery around me