40GBASE-T in Data Center Networks

01. November 2012

10 Gigabit Ethernet is de-facto standard in today’s data center with growing adoption of 40G. While 40G Ethernet standards already exist for SM fiber and MPO based MM fiber cables, standards bodies are currently developing 40GBASE-T Ethernet over twisted pair copper cabling systems. Such high networking speed imposes strict performance requirements for cable components and cabling systems. This article examines challenges in ensuring adequate performance of installed cabling, with specific attention to certification testing in field.

40G Ethernet Physical Layer Alternatives


Singlemode Fiber

Due to its long reach and superior transmission performance, singlemode fiber is specified for carrying 40Gbps data up to a distance as long as 10km (40GBASE-LR4). The physical layer electronics and optics consist of four channels, each carrying 10Gbps data with different wavelengths. Singlemode fiber is preferred option where budget is not a constraint, or when the link distances are long.

Multimode Fiber

Multimode fiber with parallel optics MPO interface is the most popular medium for 40G Ethernet today (40GBASE-SR4). The networking hardware is cheaper compared to singlemode, and it supports all typical link lengths (up to 100m for OM3 cable and 125m for OM4) in a data center network.

Copper Twinax
For short reach channels up to a length of 7m, 40GBASE-CR4 standard specifies use of twinax copper cable assembly. Typical use is interconnecting networked devices which are physically located adjacent to each other.

Copper Twisted Pair
Recent developments suggest that copper structured cabling systems are here to stay and will make an important alternative to fiber links for 40G Ethernet. Twisted pair copper cabling is likely to retain its cost advantage over fiber at least for next several years. Copper cables are perceived to be easier to install and maintain. Importantly, BASE-T networking standards over twisted pair cable are backwards compatible with auto-negotiation capability. This enables organizations to upgrade to higher speeds incrementally, enabling better control over capital expenses. 
 
IEEE initiated a formal project for defining 40GBASE-T standard using twisted pair cabling in 2012. Cabling standardization bodies are not behind. TIA is developing specifications for “Category 8” cabling systems suitable for 40GBASE-T.  ISO/IEC has a similar project that aims to define two variants of cabling systems that will support 40GBASE-T. These new cabling systems are being called Class 1 (using CAT6A like components with higher capacity), and Class 2 (using CAT7A like components with higher capacity).  Additionally, ISO/IEC is defining recommendations about using existing cabling systems such as Class FA for 40GBASE-T application.

Selecting Transmission Bandwidth for Twisted Pair Cabling for 40GBASE-T

One of the key tasks in defining Ethernet standards is to determine appropriate RF bandwidth for communication. For example, 10GBASE-T uses 400MHz bandwidth, which roughly means that every Hz of RF spectrum carries 25 bits of binary data, or in other words, channel capacity utilization is 25 bits/Hz.  Higher order and more complex modulation schemes can increase capacity utilization. There is a maximum limit to channel capacity, known as Shannon Capacity. This limit is decided by electromagnetic noise experienced by the channel. Noise comes from external and internal sources. Examples of internal noise sources are cross-talk and return-loss. Ethernet Physical layer devices use sophisticated signal processing techniques to predict and cancel the effects of internal noise sources, thereby increasing capacity of the channel. However, there is a price to pay: higher power consumption leading to heat generation.

Learning from the 10GBASE-T Experience
High power consumption was the single largest reason why 10GBASE-T adoption lagged all predictions made when the standard was released in 2006. This problem has been largely overcome today through innovative designs and semiconductor technology advancements. With this 10GBASE-T experience fresh in mind, the experts developing 40GBASE-T standard are not in a mood to increase the target for capacity utilization significantly.  

But 40G is four times as much data as 10G. One way of squeezing more data, without significantly changing modulation density (capacity utilization), is to increase the bandwidth. In this case, it would mean a four-fold increase from 400MHz to 1,600MHz. This is what the 40GBASE-T standard seems to be driving towards.

There is one issue with increase of bandwidth. On twisted pair cables, signal attenuates rapidly with increasing frequency. This means, received signal at 1,600MHz is significantly smaller than received signal at 100MHz. This phenomenon imposes restriction on the length of cable. With a 100m cable, received signal at high frequencies would be buried in the noise.  A compromise therefore has to be made on the maximum supportable link distance.  

The net effect of all these considerations is:
- 40GBASE-T will use bandwidth spectrum from 1MHz- approximately 1600MHz
- The maximum length of the cable will be limited to around 30m
- The cabling channel may be specified for fewer than four connectors

The good news is, a large percentage of data center link lengths are well within this 30m constraint.  Studies have indicated that more than 80% of data center links are 30m or shorter, and therefore eligible to benefit from 40GBASE-T.

Field Testing of 40G Copper Cabling
While cabling technologies and semiconductor technologies can ensure feasibility of supporting 40Gbps Ethernet over twisted pair copper cables, wide spread market adoption calls for additional considerations.  One of the key elements is availability of field test instruments to characterize and certify installed cabling for suitability for 40GbE.   

Measurement Bandwidth
Just like 300mL soft-drink needs a bottle with 400mL capacity for easy filling, cabling systems for supporting 1,600MHz transmission will be specified to 2,000MHz, and field testers will typically support even higher measurement bandwidths. Among many aspects of 40 GBASE-T that are still under early stages of development, field testing is a noteworthy exception. Psiber Data’s WireXpert is an example of field tester that provides sufficient measurement bandwidth to qualify 40GBASE-T cabling during draft standardization stages and after standardization. A number of IEEE studies on RF performance of cabling systems have been done using this instrument.

Conclusion

Despite the growth in wireless and fiber infrastructure, copper cabling will still be the dominant media for enterprise networks in a foreseeable future. When designing infrastructure for use over next 15 to 20 years, one must consider the fact that there is a high likelihood that 40GBASE-T systems will be defined, and become common-place in 5-10 years. There are technical challenges in supporting such high data rates, one of the main challenges being complexity of physical layer devices.  In order to create a complete eco system for adoption of technologies like 40GBASE-T, the industry will need cabling systems, networking devices, standardization, and also field test instruments suitable for that technology. Field testing over wider bandwidth has been constrained in the past due to several factors, but now at least one commercially available field tester features capability of certifying cabling to bandwidths higher than 2,000 MHz, which is expected to meet field testing needs for future 40GBASE-T systems.  

DOWNLOAD WHITEPAPER>>