Need for Speed: 40 GBASE-T in Data Center Networks

Data center network infrastructure is witnessing a transformation, driven by growing bandwidth and network performance demand. 10 Gigabit Ethernet is de-facto standard in today’s data center with growing adoption of 40 G. While 40 G Ethernet standards already exist for SM fiber and MPO based MM fiber cables, standards bodies have developed 40 GBASE-T Ethernet over twisted pair copper cabling systems. Such high networking speed imposes strict performance requirements for cable components and cabling systems. This article examines challenges in ensuring adequate performance of installed cabling, with specific attention to certification testing in field.

40 G Ethernet Physical Layer Alternatives Singlemode Fiber

Due to its long reach and superior transmission performance, singlemode fiber is specified for carrying 40 Gbps data up to a distance as long as 10 km
(40 GBASE-LR4). The physical layer electronics and optics consist of four channels, each carrying 10 Gbps data with different wavelengths. Singlemode fiber is the preferred option where budget is not a constraint, or when the link distances are long.

Multimode Fiber

Multimode fiber with parallel optics MPO interface is the most popular medium for 40 G Ethernet today (40 GBASE-SR4). The networking hardware is cheaper compared to singlemode, and it supports all typical link lengths (up to 100 m for OM3 cable and 125 m for OM4) in a data center network.

Copper Twinax

For short reach channels up to a length of 7 m (40 GBASE-CR4) standard specifies use of twinax copper cable assemblies. Typical use is interconnecting networked devices which are physically located adjacent to each other.

Copper Twisted Pair

Recent developments suggest that copper structured cabling systems are here to stay and will be an important alternative to fiber links for 40 G Ethernet. Twisted pair copper cabling is likely to retain its cost advantage over fiber for the next several years. Copper cables are perceived to be easier to install and maintain. Importantly, BASE-T networking standards over twisted pair cable are backwards compatible with auto-negotiation capability. This enables organizations to upgrade to higher speeds incrementally, with better control over capital expenses.

IEEE initiated a formal project for defining 40 GBASE-T standard using twisted pair cabling in 2012. Cabling standardization bodies are also updating their specifications. TIA has developed specifications for “Category 8” cabling systems suitable for 40 GBASE-T.

ISO/IEC has a similar project that aims to define two variants of cabling systems that will support 40 GBASE-T. These new cabling systems are being called Class I (using CAT 6A like components with higher capacity), and Class II (using CAT 7A like components with higher capacity). Additionally, ISO/IEC is defining recommendations about using existing cabling systems such as Class FA for 40 GBASE-T
application.

One of the key tasks in defining Ethernet standards is to determine appropriate RF bandwidth for communication. For example, 10 GBASE-T uses 400 MHz bandwidth, which roughly means that every Hz of RF spectrum carries 25 bits of binary data, in other words, channel capacity utilization is 25 bits/Hz. Higher order and more complex modulation schemes can increase capacity utilization. There is a maximum limit to channel capacity, known as Shannon Capacity.

This limit is due to the electromagnetic noise experienced by the channel. Noise comes from external and internal sources. Examples of internal noise sources are cross-talk and return-loss. Ethernet physical layer devices use sophisticated signal processing techniques to predict and cancel the effects of internal noise sources, thereby increasing capacity of the channel. However, the trade-off is higher power consumption and the resulting heat generation.

Learning from the 10GBASE-T Experience

High power consumption was the single largest reason why 10 GBASE-T adoption lagged all predictions made when the standard was released in 2006. This problem has been largely overcome today through innovative designs and semiconductor technology advancements.

With this 10 GBASE-T experience fresh in mind, the experts developing 40 GBASE-T standard are reluctant to increase the target for capacity utilization significantly.

But 40 G is four times as much data as 10 G. One way of squeezing more data, without significantly changing modulation density (capacity utilization), is to increase the bandwidth. In this case, it would mean a four-fold increase from 400 MHz to 1,600 MHz. This is what the 40GBASE-T standard seems to be driving towards.

There is one issue with increase of bandwidth. On twisted pair cables, signal attenuates rapidly with increasing frequency. This means, received signal at
2,000 MHz is significantly smaller than received signal at 100 MHz. This phenomenon imposes restrictions on the length of cable.

With a 100 m cable, received signal at high frequencies would be buried in the noise. A compromise therefore hasto be made on the maximum supportable link distance.

The Net Effect of all these Considerations is:

  • 40 GBASE-T uses bandwidth spectrum from 1 MHz- 2,000 MHz
  • The maximum length of the cable is limited to 30 m
  • The cabling channel is specified for two connectors

The good news is, a large percentage of data center link lengths are
well within this 30 m constraint. Studies have indicated that more than 80% of data center links are 30 m or shorter, and therefore eligible to benefit from
40 GBASE-T.

Field Testing of 40 G Copper Cabling

While cabling technologies and semiconductor technologies can support 40 Gbps Ethernet over twisted pair copper cables, wide spread market adoption calls for additional considerations. One of the key elements is availability of field test instruments to characterize and certify installed cabling for suitability for 40 GbE.

Conclusion

Despite the growth in wireless and fiber infrastructure, copper cabling will still be the dominant media for enterprise networks in a foreseeable future. When designing infrastructure for use over next 15 to 20 years, one must consider the fact that there is a high likelihood that 40 GBASE-T systems will be defined, and become common place in 5-10 years. There are technical challenges in supporting such high data rates, one of the main challenges being complexity of physical layer devices. In order to create a complete eco system for adoption of technologies like 40 GBASE-T, the industry will need cabling systems, networking devices, standardization, and also field test instruments suitable for that technology.

Autor: Thomas Hüsch, Technical Support & Training

Richard-Reitzner-Allee 6
85540 Haar
München, Deutschland
Tel: +49 (0) 89/45656-612
Fax: +49 (0) 89/45656-656
E-Mail: thomas.huesch@softing.com