NUMA configuration
Introduction to NUMA architecture
Servers with NUMA
On servers with NUMA architecture, during system boot, the BIOS on some systems will not distribute PCIe slots evenly among the NUMA nodes. Each NUMA node contains multiple CPUs. This imbalanced distribution means that, during high workloads, half or more of the CPUs will remain idle while the rest are 100% utilized. To prevent this imbalance, you must manually assign IO Accelerator devices equally among the available NUMA nodes.
Configuring the IO Accelerator devices for servers FIO_AFFINTIY parameter with the
with NUMA architecture requires the use of the utility.
FIO_AFFINITY parameter
The FIO_AFFINITY parameter is a list of <affinity specification> triplets that specify the affinity settings of all adapters in the system. Each item in the triplet is separated by a comma, and each triplet set is separated by a semicolon.
Syntax
Where each <affinity specification> has the following syntax:
<fct number>,[gn]<group or node number>[,<hex mask>]
If there is no g or n character before the group or node number, then the number is assumed to be a group number.
The hex mask is optional. If it is not present, the mask is assumed to be 0xffffffffffffffff. Also, the 0x prefix is optional.
If the hex mask is a node mask, then the mask is relative to the node, not the group to which the node belongs.
Simple example
fio-config -p FIO_AFFINITY 4,n1,0xf;5,n0;7,g1;9,g2,0xff0
This command creates the following:
Device | Node/Group | Processor Affinity |
|
|
|
fct4 | node 1 | Processors 0 to 3 in the node (mask 0xf) |
fct5 | node 0 | All processors in the node (no hex mask) |
fct7 | group 1 | All processors in the group (no hex mask) |
fct9 | group 2 | Processors 4 to 11 in the group (mask 0xff0) |
NUMA configuration 67