High-performance computing requires high-performance I/O. The industry has been working hard to improve high-bandwidth remote solutions for some time. Last year Intel and Xilinx both launched FPGAs with 56G I/O. The long-reach 112G SerDes PHY has been announced, and with the upcoming 5nm node, the state may be even better. Looking further ahead, the industry roadmap beyond 112G is full of uncertainty. For distances spanning tens of meters or more, optical communication will be used instead of electrical signal communication. Unfortunately, these products tend to be inversely correlated. The farther away from the physical die, the higher the cost of transmission. Also, higher data transfer rates often sacrifice density, so monolithic silicon photonic chips are seen as a better solution for information processing.
Terabit PHY, or TeraPHY for short, is the first product from Ayar Labs. This is a prototype photonics chip designed to be integrated in a system in a package with a CPU, GPU or FPGA. RE is just a small problem - optics and electronics don't quite communicate with each other. The secret sauce? Ayar's design leverages GlobalFoundries' 45nm RF SOI (radio frequency silicon-on-insulator) process, which allows them to develop monolithic integrated designs that integrate optics and complex circuits around the optics. In short: this allows them to provide electrical I/O interfaces on one side and optical interfaces on the other.

Intel has developed an extensive chip architecture around its Layer 10 FPGA family. But all of these chips are developed in-house. The good news is that the architecture uses the AIB interface to communicate between the main FPGA chip and the various chips. As part of the DARPA ERI project, the interface was also made available as an open standard, so it is no longer an Intel or EMIB patent. The Stratix 10 multi-chip architecture lends itself to what TeraPHY provides—replacing an electronic transceiver module with a TeraPHY chiplet, as long as the interface is compatible. This is the route chosen by Ayar Labs.

For TeraPHY, Ayar integrates 24 lanes of AIB interface. In fact, the maximum number of channels allowed per column is 24 channels (and AUX blocks). Each channel represents a group of signals. At the current bump pitch of 55 microns, this means twenty transmit data signals and twenty receive data signals. It operates at speeds up to 2GT/s. Ayar said that for their TeraPHY chiplets, the total interface bandwidth is 960Gbps, which indicates that they are using the 1GT/s AIB base specification rather than the 2GT/s AIB Plus specification.

FPGA integrates photonic chip for the first time with bandwidth up to 5.12 Tbps

Since the bump pitch used by the AIB interface is small, it can be used on silicon. In the StraTIx 10 case, that means using Intel's EMIB technology. In the unfinished package below, there are two TeraPHY chiplets to the right of the large StraTIx 10 FPGA chip. The location of the EMIB is clearly visible at the edges of all dies. Note that there may be other chiplets on the other side of the FPGA.

Located between the AIB interface and the optical interface is configurable cross glue logic that maps AIB channels to optical channels. Crossbars allow one-to-many connections. A single electrical signal channel can be sent over multiple optical interfaces and vice versa. The TeraPHY chiplet integrates ten photonic macro pairs, one for transmit and one for receive.

Light within the chip travels through the waveguide. Due to the properties of light, multiple wavelengths of light can travel along the same waveguide without interfering with each other. Wavelength division multiplexing (WDM) technology is used to introduce multiple such wavelengths into a waveguide to increase the amount of data that can be transmitted over the same fiber link. To achieve this, Ayar used multiple micro-ring resonators on the same waveguide, using different wavelengths from the waveguide to convert data to light or electricity. Individual low-power silicon photonic ring resonators are locked to the specific wavelengths at which they operate. These ring resonators are driven by CMOS drivers that interface with the reset of the digital logic on the die.

Each macro pair contains a set of PLLs, TRXSlices, and other logic needed to make them all work. If you look at the GDSII screenshot, you can derive eight TRX slices, one for each wavelength. The various PLLs in each macro are designed so that the data rate can be configured in increments of up to 2x. Current TeraPHY chiplets allow data rates of 16 Gbps, 25.6 Gbps and up to 32 Gbps. Since there are eight wavelengths per waveguide, you're looking at a configurable aggregate bandwidth of 128 Gbps to 256 Gbps per macro.

Current TeraPHY chiplets contain 10 macro pairs. This means it is capable of delivering up to 2.56 Tb/s of aggregated bandwidth across all optical macros. That's a lot more than all the features on the AIB link. It is not clear why they are so unbalanced, but since a single AIB channel can be routed to multiple optical channels, there may be topologies when doing this type of communication. For example, one SoC routes traffic to two other SoCs. It is worth adding that it uses the NRZ modulation format on the optical channel since no error correction is required at the receiving end.

For the partnership with Intel, two TeraPHYs are integrated into the StraTIx 10 FPGA. This means a total optical bandwidth of 5.12 Tbps per FPGA. Two small chips less than 50mm square are impressive!

So how do you physically remove it from the chip? The input or output waveguide terminates in a grating coupler, which is a device with ridges and grooves on its top surface that allow light to scatter out of the waveguide at a certain angle. Here, the fiber is pulled close enough that it can collect the scattered light. For Intel StraTIx 10 FPGAs, the light emanates from the top. In other words, TeraPHY assembly involves aligning and bonding optical fibers through the backside of the chip. The fiber optic connector runs directly through an opening from the top of the cover to the TeraPHY chiplet.

"We're seeing a surge in data center workloads that have an unmet need for bandwidth and the need to connect devices over rack-level distances," said Vince Hu, vice president of product strategy and innovation for Intel's FPGAs. The best way to do this is to use optical interconnects, and with Ayar Labs chiplets, we can achieve very high bandwidth with low latency and low power consumption."

Since the AIB has an extremely low latency of only about 3ns, the round-trip communication through the AIB to TeraPHY and through the AIB is less than 10ns, and the delay per meter is about 5ns. Optical fiber (depending on the configuration of the entire system), up to 2 km. The total energy efficiency of TeraPHY is just under 5 pJ/bit. This figure contains the AIB interface, crossbar switch and optical macroblocks.