AMD 64 manual A.2.3 What Role Do Buffers Play in the Throughput Observed?, Appendix A

40555 Rev. 3.00 June 2006	Performance Guidelines for AMD Athlon™ 64 and AMD Opteron™
	ccNUMA Multiprocessor Systems

4.4GB/s necessary. The two coherent HyperTransport links are loaded at 3.5 GB/s each. Thus the utilization of each of the two coherent HyperTransport links that connect node 0 and node 1 equals 87% (3.5÷4).

A.2.3 What Role Do Buffers Play in the Throughput Observed?

Node 0 queues up packets in HyperTransport buffers and sends them on the outgoing link only if node 1 can accommodate them. Likewise node 1 queues up packets in HyperTransport buffers and sends them on the outgoing link only if node 0 can accept them.

When the HyperTransport buffers are saturated, they can prevent the coherent HyperTransport links from reaching their full throughput capacity of 4GB/s and, thus, full 100% utilization.

Also, saturating the HyperTransport buffers in the XBar has a domino effect on the other buffers in the system. Remember, the SRI is connected to the XBar, which is connected to the coherent HyperTransport links.

When packets are stalled in the XBar buffer queue to be sent over the coherent HyperTransport links, a chain effect can cause packets stall in the SRI buffer queue to be sent to the XBar.

AMD makes several event profiling tools available under NDA to monitor the HyperTransport bandwidth and buffer queue usage patterns.

The buffer lengths are BIOS configurable within some hardware-specific limits that are specified in the appropriate BIOS Kernel and Developers Guide for the processor under consideration. Following AMD recommendations, the BIOS allocates these buffers on a link-by-link basis to optimize for the most common workloads.

A.2.4 What Resources Are Used When Write-Only Threads Do Not Fire at Each Other (No Crossfire) on an Idle System?

Now consider the case in which the writer threads do not fire at each other: i.e., the first thread runs on node 0 and writes to memory on node 1 and second thread runs on node 1 and writes to memory on node 3.

In this case, the bidirectional link from node 0 to node 1 is in under substantial use (60% utilization in each direction). In addition, the bidirectional link from node 1 to node 3 is also under substantial use (54% utilization in each direction).

As the load is now spread over two bidirectional links instead of 1, the performance is better than in the crossfire case.

Appendix A 41