Posted by Mark Harris on Tue, Jul 06, 2010 @ 11:24 AM
For those of you following my writings, you'll know that I've "been around the block" a few times in the IT world. For 30 years I've seen alot of technologies come and go. Various technologies always seem to sound great at first (to some people), but how those technologies play out over time is a measure of the capabilities, costs, and timing BUT more importantly a bit of 'chance'. Sometimes it just doesn't add up or make any sense, and yet certain solutions thrive while others fail. Data Center Management has always been considered an "ART" rather than a science. Emotions, previous experiences and personal focal points drive investments within the data center. The ART of data center design varies widely from company to company.
That background is a good point of reference when considering the task at hand for today: Explaining just WHY it has been so hard to deploy data center monitoring, and include both IT gear AND facilities equipment in the standard IT planning and performance processes. As it turns out, IT gear vendors have done a fairly good job of standardizing management protocols and access mechanisms, but there have been simply too many incompatible facilities gear management systems over the years. Many are still very proprietary and/or undocumented or poorly documented. Additionally, the equipment manufacturers have been in NO hurry to make their devices communicate any better with anything in the outside world. "Their equipment, their tools" has been the way of life for facilities gear vendors. (I call it "Vendor-Lock").
Ironically, these "facilities" sub-systems (like power and cooling) would likely be considered today as THE most mission critical part of running IT cost-effectively. We need to find any answer....
So, we have two factors to consider:
1. Data Center Design is considered by many to be an ART rather than a science. Depending on the leadership, various differing levels of emphasis is paid in different technology areas.
2. Data Center Monitoring has been historically viewed as difficult to deploy across the field of devices and the resulting limited reports and views to be insignificant and non-impactful to the bigger picture.
Well, times have changed. Senior leadership across corporations are asking the IT organization to behave more like a science. The days of 'art' are drawing to a close. Accountability, effectiveness, ROI, efficiency are all part of the new daily mantra within IT. Management needs repeatable, defendable investments that can withstand any challenge, and yet allow for any change.
Additionally, with the price of energy over 3 years surpassing the initial acquisition price of that same gear, the most innovative Data Center Managers are taking a fresh new look at deploying active, strategic Data Center Monitoring as part of their baseline efforts. How else would a data center manager know where to make new investments in energy efficiency technologies without some means to establish baselines and continuous measurements towards results? How would they know if they succeeded?
Data Center Monitoring can be easily deployed today, accounting for all of any company's geographically distributed sites, leveraging all of their existing instrumentation (shipping in newly purchased IT gear for the past few years), and topping it off with a whole slew of amazing wireless sensor systems to augment it all.
Today, integrated data center monitoring across IT gear and facilities equipment is not only possible, but quite EASY to deploy for any company that chooses to do so.
You just Gotta-Wanna!
Posted by Mark Harris on Tue, Jun 22, 2010 @ 05:15 PM
Over the course of meeting with perhaps 100 customers over the last 6 months, it has become painfully clear to me that there is widescale and growing confusion about Real-Time Data Center Monitoring.
I would suggest that Real-Time monitoring which answers MOST customers' needs MUST have a number of specific capabilities which the vast majority of what's available today do NOT:
1. Scale. Most shipping Data Center Management and Monitoring solutions fail to realize that SCALE is a big deal. Monitoring 100 devices on a trade show floor demo is entirely different that deploying true monitoring across 20 sites, each with thousands of devices. You simply can't use the same ARCHITECTURE, and all the marketing fluff in the world won't solve this fundamental structure issue. The ONLY way to scale this is using a DISTRIBUTED architecture.
2. Device Coverage. These same vendors will tell you that they speak SNMP and that everything you need to monitor speaks SNMP. Nonsense! Firstly, there are many protocols including Mod-Bus, SNMP, BACnet, WMI, Serial, etc, etc. Secondly, just supporting the protocol doesn't get you much closer to the device knowledge. Each device has to be specifically understood to read the required values. In most vendor's proposals, this shows up as "Professional Services" which means 'We'll figure it out on the job, on your dime'.
3. Real-Time Monitoring MUST store observed metrics and KPIs over long periods of time. I would suggest that while there are many reasons why most customers want to see real-time monitoring, the vast majority of these reasons are TIME-BASED. The monitored values or metrics need to be collected, time-stamped, stored, and available openly to run analysis upon. While customers may want to know that the data center is consuming 350kW this instant, what they REALLY want to know is that the data center WAS consuming 275kW 3 months ago, 310kW last month, 350kW today, and then PROJECT the future date of the wall that they will hit of the 500kW feed from the power utility.
The road ahead will continue to be littered with failed deployments of real-time management solutions which do NOT realize the dream of Data Center Monitoring. Customers should challenge their vendors to answer ALL of the tough questions. Consider the old-school 'Get it in Writing' approach, and then be very specific about your expectations, needs, and acceptance criterior...
Let's ALL win this GREEN game!
Posted by Mark Harris on Fri, Jun 04, 2010 @ 01:34 PM
I was flipping through the 2007 report to congress issued by Jonathan Koomey ("Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431") and on Page 10 came across a very easy to read, but impactful diagram which provides some great insight into the future of the IT industry, and can be discussed in terms of end-users as well.
I suspect that this chart could be applied more or less to ANY individual company in their quest for energy efficiency. If there is some level of 'greening' at play in a corporation, then this chart can be a crystal ball into your 5 possible futures.
You can see from the diagram varying impacts on energy consumption, (starting at the top) going from taking NO NEW ACTION, all the way through DOING EVERYTHING POSSIBLE. I would suggest today that most companies are somewhere approaching the "Improved Operations Scenerio". If you look above, you'll see this green curve essentially takes the overhead out of operations, but does very little to have any significant long term effect on the SLOPE of the curve.
In the chart, the "State of the Art Scenerio" is a good depiction of what is POSSIBLE (expected) if all business processes are tuned and all equipment is refreshed with the latest. This would create a real-time infrastructure ("RTI" as defined by Gartner) that self-tunes itself based upon demand. Most importantly... It would also lower the most basic cost per transaction. A CPU cycle would actually cost less!
These are very exciting times ahead...
Posted by Mark Harris on Fri, May 28, 2010 @ 04:54 PM
While I've seen my share of some pristine new data centers over the past few years, as well as a huge number of large scale retro-fit projects where old centers are being turned into new usable data center space, I have also seen an alarming number of older 'house of cards' data centers that are up in modern production and appear to be 'hands-off'.
These data centers are typically chock full of older devices and interconnects that were passed down from generation to generation of IT managers, only to realize that what they inherited was unmanageable. While it is true that these data centers will ultimately find their way into extinction in a world focused on operational efficiency and pro-active management and best practices, we can all feel the pain involved when we encounter something like this.
Above is one of the most interesting centers I've seen, and would appear to have conflicting priorities as to what is required to move forward. While I don't have a comprehensive sequence of steps required to migrate to a highly supportable, efficient and monitored data center, let me suggest one step that will help tremendously... Find the YELLOW patch cord and disconnect it.
Seriously, when I saw this photo I had to laugh and take a second look. Was it some new thermal blanketing technology? Or a way to eliminate blanking panels? The reason I make light here is that there are countless data centers that are in similiar out-of-spec designs and would benefit from adopting new data center technologies, new power distribution, cooling and monitoring solutions, but are challenged by WHERE TO BEGIN and the magnitude of the task at hand.
In the monitoring world for instance where Modius delivers value, we regularly find data centers with NO VISIBILITY to their energy usage and easily can identify hundreds or thousands of points of monitorable data that would help get energy usage under control. We are ready willing and able to take on chaos and make sense of it.
Posted by Mark Harris on Tue, May 25, 2010 @ 11:23 AM
Think back to the last time your purchased a new car. I would bet that within the first 30 minutes of actually looking at the brochures or sitting in the car, the attention turned to the Leather seats, body color, Stereo system and electronics package.
By inference, the consumer (you) had already assumed and agreed that the car foundation itself was as stated in the data sheet and their design engineers had done their job building a functional car. It had a chassis, it had an engine of a certain size, and it was as speedy and efficient as the TV commercial showed. No need to be concerned that the physical layer had any issues. Somehow the car would perform.
Instead, your attention was to the 'soft' details. There you are, buying a $30,000 car, and most of the sales configuration and cost discussion was about the $3000-$4000 worth of options. Most people don't even know how big the gas tank is when they drive home in the car!
The Data Center is much the same. The underpinnings for most data centers have for the most part been specified by the building design engineers of record, built per spec, and typically installed far away from view. The mechanical and electrical structures were designed and installed based upon equipment resource requirements and assumptions at the time, and at the end of the day, the IT organization ultimately 'inherited' what was installed. How many watts per square foot were really possible? What is the redundant Cooling capacity? None of these critical resource available capacities or real-time usage is actually well understood or even visible to the IT organization over time. (And UNTIL LATELY, not even much concern about it). This situation is compounded by the fact that all of the major IT vendors are now selling boxes that consume 2-4 times the amount of power in the same space as the units shipped just two years ago. It can be seen that the data center is a VERY dynamic system, and the most valueable on-going data center analysis and KPIs must be based upon it's real-time aspects.
While IT as a whole has focused for years on their own 'Fine Corinthian Leather", (like virtualization/operating systems, storage and networks), the real challenge at hand today is to better understand the real-time performance of the chassis. The amount of fuel in the gas tank and it's current efficiency, the engine performance, the available redundancy systems, etc.
Don't get me wrong, I am a huge fan of Fine Corinthian Leather, but I think it's prudent to understand the bigger picture before claiming victory...
Posted by Mark Harris on Wed, May 05, 2010 @ 09:00 AM
Having walked through my share of data centers, it is always interesting to see such a heterogeneous amalgamation of IT gear that has accumulated since the data center itself was commissioned. While every data center designer and manager starts out with wild fanciful ideals about the pristine architecture of the data center, the actual complexion of the data center changes dramatically over time and we are left with rows and rows of assorted gear, all happily consuming power, blinking LEDs, and perhaps 20%-30% of these devices no longer in use... Zombies abound!
Perhaps Zombies is a harsh word, but the concept is the same. A non-trivial portion of the devices in the data center are powered, generating heat, consuming precious IP addresses, and yet performing NO actual work. Why? Their intended application changed over time, the project was never completed, their original workload was shifted elsewhere, a test bed that was never dismantled, and a dozen other reasons exist for large quantities of machines entering the Zombie realm, but there we have it, machine after machine that is in the living dead state, and WORSE THAN THAT, we do not have enough information about these devices to TURN THEM OFF. So they sit, consuming resources in the safety of the data center, avoiding decommissioning... And here's the myth/rub: A server just idling along just running the operating system consumes 60%-70% of its total power before any workload is applied! A server doing NO work is wasting almost two-thirds of its maximum rated power! Note to self: this is a real issue and not something we can choose to overlook any longer. With the price of power at record highs, and power increasing by 7% per year as far as we can see in the future, WE HAVE to find these Zombies and kill them.
How can we reclaim the resources being consumed by these Zombies? We have to build designs that intelligently monitor power consumption and pro-actively and continually test to see if those resources are efficiently doing work. We need to observe power consumption either directly using embedded sensors (such as the Energy-Star compliance servers) or with intelligent power distribution devices (ideally with per-outlet metrics). Here is the secret: Zombies all have a similar trait... they stay fairly constant in their power consumption. A server will likely consume almost two-thirds of its maximum power before any loads and work is applied. A Zombie server therefore will continue to consume the same two-thirds of its rated values every time you look at it.
Creating new IT best practices which identify the need for per-device power monitoring is the first step. And the second step is deploying an intelligent monitoring tool which has the ability to look over longer periods of time at the energy being consumed on a device level basis. Some simple standard deviation math will result in servers that can no longer hide their 'walking dead' status. Pro-Active monitoring will identify Zombies and allow you to reclaim power, space and cooling quite easily!
Posted by Mark Harris on Thu, Apr 29, 2010 @ 07:00 AM
It is funny how many times I have recently visited larger data centers considering 'Green IT' or other Efficiency initiatives and find high priority funded projects for wireless temperature sensors. True wireless environment technology is some of the coolest 'tangible' stuff I have seen in a long time. It is a high-tech version of the kind of technologies that we all grew up with, things we all just inherently 'get'. Temperature and Humidity Sensors. What could be simplier?
Here it is in 2010, and is important to realize that finally there are a number of great choices for wireless sensor solutions out there either using Active RFID or 802.15.4 (zigbee) technologies. A customer today really can deploy a fairly granular 'mesh' of sensors in data centers and related facilities areas without much difficulty. The sensors are simple, small, have long battery lives (> 3 years each) and low-cost. All of the solutions have easy to install packaging with double-sided tape or velcro. How easy is that?
Well, I would argue that the REAL VALUE for wireless temperature and humidity environmental sensors are NOT the sensors themselves, nor the data derived from each individual sensor but the aggregation of all of the data from all of the devices, rolled together with the metric data from the co-located IT gear and the facilities deployed HVAC gear, all normalized and easily accessible using ordinary tools. EXCEL anyone? (Or for the web-bies in the crowd, "Xcelsius Anyone?"). Imagine being able to plot the PUE of your data center as a function of outside temperature, or the total power consumption as a function of actual CPU processing (IT load). Remember, sensors can be found everywhere in your data center as discrete wired and wireless boxes, as well as embedded in every IT device purchased in the past 3 years, such as your servers, routers, firewalls and storage directors as well as in every PDU or iPDU (power strip). Sensors are everywhere just waiting to be queried for their metrics!
Customers should think BIGGER. Push the envelope and think PAST the wireless sensors (which ARE very cool), think PAST the pretty pictures that any one of the wireless vendors can draw, and focus on how to transform ALL of the data that you can get your hands on into actionable, cost saving information that can be directly applied in the BIGGER picture of running the IT structure at the lowest cost possible, supporting SLAs, etc.