The Cisco production data center in Richardson, Texas, is a model of the service-oriented data center, supporting the business with agility and resiliency.

The new Cisco data center in Richardson, Texas, which opened its doors, or servers, in June of this year, is the company’s first full-fledged service-oriented data center (SODC). The data center is designed to get applications up and running quickly and support them reliably and resiliently. It is also prepared to support any new business strategies, trends, and models.

Located just outside Dallas, Richardson will become one half of the primary North American production data center pair for Cisco, with 30,000 square feet of raised floor space. Cisco plans to build another data center of the same capacity nearby so the two can be used as active-active centers; each carrying half the load and ready to take over if one fails.

The Richardson data center employs the Cisco VFrame Data Center (VFrame DC) solution for rapid advanced policy-based provisioning of virtual resources within the SODC. Cisco VFrame DC enables end-to-end orchestration of data center resources to support Cisco's business model in real time.

At the beginning of planning some four years ago, says Andy Broer, IT manager of data center infrastructure at Cisco, “When I asked our then-CIO what was most important in this new data center, he gave me one word: reliability.”

The data center’s electrical systems are designed to Tier 4 standards while mechanical systems are designed to Tier 3 standards: “system-plus-system,” Broer says.  For example, the data center is served by two geographically separated electrical substations, each delivering 10 megawatts of power. It is also served by two separate OC-48 (2.5 Gbps) fiber-optic cables from different service providers.  Two separate 14-inch mains convey chilled water  to and hot water from the 86 computer room air-handling units.

Cisco chose the site after an extensive search that narrowed 420 prescreened locations down to one, in large part to avoid natural disasters such as earthquakes, hurricanes, and tornadoes. “When hurricanes go inland, they lose strength without a warm water source,” says Broer. “While there is an elevated tornado risk where we are, it isn’t until you get further north into Oklahoma and beyond where they start to become very frequent.”

To resist weather-related events, the building has been reinforced to withstand winds of 125 to 145 mph. The interior has been designed to further maximize reliability. Eighty-six 30-ton air handlers are installed in walled-off, separated maintenance galleries around the raised floor area, offering another solid barrier, and a reinforced cement wall sits between the galleries and the outside of the building for additional protection.

Power and Other Essentials

Other important criteria were reliable sources of electricity, a good pool of IT talent, and ready accessibility. The Dallas-Forth Worth airport is just 25 miles down the road.

Electricity supply drove some decisions. Working backward from the 5 megawatts available for the data center itself out of the 20 available to the building, Broer says that the design group calculated it could afford approximately 1200 equipment cabinets, each drawing an average of 4.2 kilowatts. Cabinets filled with blade servers would, of course, draw more and need to be offset for cooling reasons by “white space” in others.

“We calculated a lower power draw per cabinet than some other companies are using,” says Todd Glenn, IT manager for the data center project, “but sometimes having a bit more space is useful in the future.”   

The network within the building was designed for full service as well. Broer says that each location where a cabinet could be sited is provided with 10-Gbps service and multiple fiber and copper ports.

Architecture Designed for Service

The basic building block of the Richardson facility is the “pod.” Each pod contains a given amount of server, storage, and network capacity so it can support nearly any application.  Actually, Broer explains, it can support many, because its components are virtualized; that is, combined into pools of server and storage space. One server or storage unit can support many applications, giving each of them its own virtual machine; servers can run virtual machines that use different operating systems. 

Virtualization saves money and energy by using resources most efficiently. Cisco has already virtualized a large percentage of the servers in other production data centers, avoiding the expenditure of many millions of dollars. Virtualization also makes the support of services and applications agile. If an application needs more capacity, it receives it, without its user having to purchase another server or storage device. “There are efficiencies in implementing a consistent architecture,” says Glenn.

The goal is to virtualize up to 80 percent of the servers in this data center, Glenn says.  Users are encouraged to opt for this type of service by making it the default and asking them to verify that their applications cannot run on a virtual machine.

Cisco uses VFrame DC to orchestrate and provision virtual and physical machines within the SODC Pods.  VFrame, says Glenn, reduces provisioning time from weeks to a standard three days, or in some cases, a few hours.

Environmental Impact Considerations

Within the data center, provisions have been made to use energy efficiently and reduce overall consumption. For example, Broer says, “We switched to lower-voltage lighting. We chose white cabinets, rather than gray, so more light would be reflected from them.”

Servers and storage units are located to eliminate hot spots that can disrupt efficient cooling. The cabinets are designed so air comes in the front and is collected at the back, where it is channeled to the coolers, which are located external to the data center itself. “That way,” Broer says, “they can draw in hotter air so they are loaded as fully as possible because they work more efficiently that way.”

Virtualization within the data center provides environmental benefits by doing more with less. Increasing storage and server utilization, replacing appliances with a service-module architecture, and managing power consumption through network-based management is expected to slow power consumption and enable the use of previously unidentified power capacity.

In addition, on the grounds of the data center, existing trees onsite were dug up, bagged, and cared for during work on the building and then replanted in a new landscape design. The center was created by transforming an empty building that Cisco had already constructed earlier in the decade.

Lessons Learned

One of the most important lessons, Broer says, is designing a production data center within the context of a global data center plan. Given that framework, you can match the design to what you actually need. “If we had been assured that there would be another data center as an active-active pair, we might have decided, for example, that meeting Tier 2 reliability standards for each was enough,” says Broer.

Other lessons include:

• Try to get a high-level executive champion who is willing to sign on for the long haul, so you have consistency in goals from year to year.
• Testing of absolutely everything is key. This should include all the mechanical as well as IT equipment. "We didn’t have a good idea of how our IT equipment would interact with the automated failover systems we installed." Glenn says.
• If the data center is being planned by two or more groups, make sure to build close cooperation from the start, so everyone understands everyone else’s documented needs and business commitments.
• Designs need to be under change management control to avoid "scope creep" and added costs.

Broer believes the innovative design of the new Richardson data center will support the company’s business innovations successfully.

For More Information

Cisco IT Data Center Solutions
VFrame Data Center First Customer
Design for a Productive Data Center
Server Virtualization on Fast Track
Storage Virtualization a Work in Progress