Manuals / AMD / Computer Equipment / Computer Hardware

AMD 64 manual A.5 Why Is 0 Hop-1 Hop Case Slower Than

Models: 64

1 48

Download 48 pages 55.63 Kb

Page 43

A.5 Why Is 0 Hop-1 Hop Case Slower Than

40555 Rev. 3.00 June 2006	Performance Guidelines for AMD Athlon™ 64 and AMD Opteron™
	ccNUMA Multiprocessor Systems

A.5 Why Is 0 Hop-1 Hop Case Slower Than

0 Hop-0 Hop Case on a System under High Background Load (High Subscription) for Write- Only Threads?

When a 0 hop-0 hop scenario is subjected to a very high background load, the system sees the following traffic pattern, where each node gets memory requests from the threads as described:

•Node 0: 2 foreground threads.

•Node 1: 1 background thread.

•Node 3: 1 background thread.

•Node 2: 1 background thread.

In the 0 hop-1 hop case, the system sees the following traffic pattern:

•Node 0: 1 foreground thread

•Node 1: 1 foreground and 1 background threads.

•Node 3: 1 background thread.

•Node 2: 1 background thread.

The 0 hop-1 hop case suffers from a greater load imbalance than the 0 hop-0 hop case, with node 1 suffering the worst effect of this imbalance.

Each of the background threads, as before, asks for data at a rate of 4GB/s and each of the foreground threads asks for data at a rate of 2.98 GB/s.

Data shows that there is a total memory access rate of 4.78 GB/s on node 1 and several buffer queues on node 1 are saturated and cannot absorb the data provided by the memory controller any faster.

A.6 Support for a ccNUMA-Aware Scheduler for AMD64 ccNUMA Multiprocessor Systems

Developers should ensure that the OS is properly configured to support ccNUMA. All versions of Microsoft® Windows® XP for AMD64 and Windows Server for AMD64 support ccNUMA without any configuration changes. The 32-bit versions of Windows Server 2003, Enterprise Edition and Windows Server 2003, Datacenter Edition require the /PAE boot parameter to support ccNUMA. For 64-bit Linux®, there may be separate kernels supporting ccNUMA that should be selected. The 2.6.x Linux kernels feature NUMA awareness in the scheduler[11]. Most SuSE and Red Hat Enterprise distributions of 64-bit Linux have the ccNUMA aware kernel. Solaris 10 and subsequent versions of Solaris for AMD64 support ccNUMA without any changes.

Appendix A

43

Image 43

AMD 64 manual A.5 Why Is 0 Hop-1 Hop Case Slower Than

Contents

Application Note Issue Date JunePerformance Guidelines for AMD Athlon 64 and AMD Opteron ccNUMA Multiprocessor SystemsTrademarks 2006 Advanced Micro Devices, Inc. All rights reservedChapter 3 Analysis and Recommendations ContentsChapter 2 Experimental Setup ContentsA.2.1 What Resources Are Used When a Single Read-Only or List of Figures List of Figures40555 Rev. 3.00 June List of FiguresPerformance Guidelines for AMD Athlon 64 and AMD Opteron ccNUMA Multiprocessor SystemsPerformance Guidelines for AMD Athlon 64 and AMD Opteron Revision HistoryRevision History ccNUMA Multiprocessor Systems40555 Rev. 3.00 June Revision HistoryPerformance Guidelines for AMD Athlon 64 and AMD Opteron ccNUMA Multiprocessor SystemsChapter Chapter 1 IntroductionIntroduction 10 http//msdn2.microsoft.com/en-us/library/ms186255SQL.90.aspx 1.1 Related DocumentsIntroduction Chapter 14 http//msdn2.microsoft.com/en-us/library/tt15eb9t.aspxIntroduction Chapter IntroductionPerformance Guidelines for AMD Athlon 64 and AMD Opteron ccNUMA Multiprocessor SystemsChapter Chapter 2 Experimental SetupExperimental Setup 2.1 System UsedExperimental Setup Figure 1. Quartet TopologyChapter 2.2 Synthetic TestFigure 2. Internal Resources Associated with a Quartet Node Experimental SetupExperimental Setup Data Access Rate QualifiersChapter 2.3 Reading and Interpreting Test Graphs2.3.1 X-Axis Display Experimental Setup2.3.3 Y-Axis Display 2.3.2 Labels UsedExperimental Setup Analysis and Recommendations Chapter 3 Analysis and Recommendations3.1 Scheduling Threads 3.1.1 Multiple Threads-Independent DataAnalysis and Recommendations 3.2 Data Locality Considerations3.1.2 Multiple Threads-Shared Data 3.1.3 Scheduling on a Non-Idle System100% ChapterAnalysis and Recommendations 2 HopAnalysis and Recommendations 3.2.1 Keeping Data Local by Virtue of first TouchAnalysis and Recommendations ChapterAnalysis and Recommendations afterwords no longer needs the data structure and if only one of the worker threads needs the data structure. In other words, the data structure is not truly shared between the worker threads3.4.1 Myth All Equal Hop Cases Take Equal Time Threads access local data3.3 Avoid Cache Line Sharing 3.4 Common Hop Myths DebunkedAnalysis and Recommendations Threads firing at each other crossfireChapter Node 0 Core Node 1 Core Node 2 Core Node 3 CoreAnalysis and Recommendations Analysis and Recommendations Next, we increase the number of background threads to six, running on216% 3.4.2 Myth Greater Hop Distance Always Means Slower TimeChapter Analysis and RecommendationsAnalysis and Recommendations Both threads access memory on nodeChapter 0 hop-0 hop case for the write-only threadsAnalysis and Recommendations Analysis and Recommendations In addition, three background threads are running on nodes 1, 2 andHigh Total Time for both threads write-write ChapterAnalysis and Recommendations Medium Total Time for both threads write-write158% 158% 3.5 LocksAnalysis and Recommendations 147%Analysis and Recommendations ChapterChapter Analysis and RecommendationsPerformance Guidelines for AMD Athlon 64 and AMD Opteron ccNUMA Multiprocessor SystemsConclusions Chapter 4 ConclusionsChapter Conclusions Data placement tools can also come in handy when a thread needs more data than the amount of physical memory available on a node. Certain OSs also allow data migration with these tools or API. Using this feature, data can be migrated from the node where it was first touched to the node where it is subsequently accessed. There is a cost associated with this migration and it is not advised to use it frequently. For additional details on the tools and APIs offered by various OS for thread and memory placement refer to Section A.7 on pageAppendix A Appendix AA.1 Description of the Buffer Queues Figure 16. Internal Resources Associated with a Quartet NodePage Appendix A A.2.3 What Role Do Buffers Play in the Throughput Observed?0 Hop-1 Hop Case on an Idle System for Write- Only Threads? A.4 Why Is 0 Hop-0 Hop Case Slower Than theA.5 Why Is 0 Hop-1 Hop Case Slower Than Controlling Process and Thread Affinity A.7.1 Support Under LinuxAppendix A A.7.2 Support under SolarisA.7.3 Support under Microsoft Windows Controlling Memory AffinityA.8.2 Support under Solaris A.8.1 Support under LinuxA.8.3 Support under Microsoft Windows Appendix A A.8.4 Node Interleaving Configuration in the BIOS40555 Rev. 3.00 June Performance Guidelines for AMD Athlon 64 and AMD OpteronccNUMA Multiprocessor Systems Appendix A