Effective Bandwidth

Ethernet supports a payload size from 64 to 1500 bytes (up to over 9000 with jumbo packets) and its efficiency is best with a maximum payload, although this comes at the cost of increased latency jitter. RapidIO technology transports 1 to 256 bytes, balancing large payload jitter against small payload inefficiency.

For control plane applications that cannot tolerate packet loss, an Ethernet fabric must be significantly over-provisioned to avoid packet loss and limit associated latency and jitter. Given 25- 35% usage for many applications, this translates to a sustainable effective throughput for layer 2 traffic of ≈250 Mbps for 1 GE and 2.5 Gbps for 10 GE, depending on average packet size. Note that performance is defined not by PHY symbol rate, but rather the effective rate in which a protocol reliably transports data. Additionally, even with over-provisioning, end-to-end latency can still run in milliseconds since traffic must traverse multiple software stacks.

By implementing protocol processing in hardware, RapidIO technology greatly reduces effective latency in comparison to Ethernet and can deliver substantially higher fabric utilization in complex topologies. For control plane applications, link-level error correction minimizes latency jitter caused by soft errors, potentially reducing end-to-end latency below 500 ns.

Throughput is also affected by overhead for operations such as reading, writing, and messaging. Ethernet RDMA provides read and write operations, but as a layer 4 protocol, its high overhead is not well-suited for small control-oriented load/store operations. TCP/IP services resemble RapidIO messaging, but where RapidIO messaging supports convenient 4Kbyte messages and is often fully implemented in hardware, TCP/IP supports 64Kbyte messages that are much more dependent upon software for processing. Additionally, the RapidIO standard defines a protocol for keeping caches coherent across the interconnect, a feat ineffective to implement in Ethernet due to low header efficiency, high latency, and inconsistent levels of hardware support.

Extensive packet handling by Ethernet switches also increases overall complexity and processing load. When IP packets are routed, numerous fields must be updated and the FCS recalculated. RapidIO switches typically only update the AckID field, which is not covered by the CRC and so does not force a recalculation.