Advanced configuration example

This example server has four NUMA nodes with eight hyper-threaded cores per node (16 logical processors per node, a total of 64 logical processors in the system). This system also uses the expansion configuration and has 11 PCIe expansion slots. During system boot, the system BIOS assigns PCIe slots 1-6 to NUMA node 2 and PCIe slots 7-11 to NUMA node 0. NUMA nodes 1 and 3 have no assigned PCIe slots. This configuration creates a load balancing problem in the system when IO Accelerator devices are under heavy traffic. During these periods of high use, half of the processors in the system sit idle while the other half of the processors are 100% utilized, thus limiting the throughput of the IO Accelerator devices.

To avoid this situation, you must manually configure the affinity of the IO Accelerator devices using the FIO_AFFINITY configuration parameter to distribute the work load across all NUMA nodes. This parameter overrides the default behavior of the IO Accelerator driver. For more information about the FIO_AFFINITY configuration parameter, refer to the syntax explanation below.

Syntax:

The following is an example of how to configure 10 HP IO Accelerator ioDrive Duo devices (each with two IO Accelerator devices) in a HP DL580 G7 system manually as described in the preceding paragraphs. Slot 1 is a Generation 1 PCIe slot, so it is not compatible with an ioDrive Duo device. Therefore you can fill slots 2-11 with ioDrive Duo devices. Because each ioDrive Duo device has two IO Accelerator devices, each ioDrive Duo devices has two device numbers (one for each IO Accelerator device). Each slot has two device numbers.

The following tables list the default BIOS NUMA node assignments.

BIOS assigned	PCIe slots	FCT device numbers	Processor Affinity
NUMA node

0	7-11	8,9,13,14,18,19,23,24,28,29	All processors in the node

1	None	None	None
2	2-6	135,136,140,141,145,146,150,151,	All processors in the node
		155,156
3	None	None	None

Assigned	PCIe slots	FCT device numbers	Processor Affinity
NUMA node

0	7-9	8,9,13,14,18,19	All processors in the node (no hex mask)
1	10-11	23,24,28,29	All processors in the node (no hex mask)
2	2-3	135,136,140,141	All processors in the node (no hex mask)
3	4-6	145,146,150,151,155,156	All processors in the node (no hex mask)

In this example, the BIOS creates a load imbalance by assigning the cards to only two NUMA nodes in the system. To balance the work load, enter the following settings:

To configure the IO Accelerator driver with these override settings, run fio-configwith the following string:

fio-config -p FIO_AFFINITY 8,n0;9,n0;13,n0;14,n0;18,n0;19,n0;23,n1;24,n1;28,n1;29,n1;135,n2;136,n2;140, n2;141,n2;145,n3;146,n3;150,n3;151,n3;155,n3;156,n3

No <hex mask> was stipulated for any of the devices in this example (making each <affinity specification> a couplet rather than a triplet). Therefore, each device is shared among all of the

NUMA configuration 68