The 16 echelons in the memory subsystem can be subdivided into four independent memory quadrants accessed by four independent MID buses. Each quadrant contains two independent SDRAM buses. Four echelons can be installed on each SDRAM bus. The CC contains four MBATs, one for each memory quadrant. Each MBAT contains eight sets of routing CSRs (one per rank). Each routing CSR specifies the bits of the address that are masked or compared to select the corresponding rank, referred to as interleave bits. The routing CSR also specifies how the remaining address bits are routed to bank, row, and column address bits.
To optimize bandwidth, consecutive memory accesses target echelons that are as far from each other as possible. For this reason, the interleaving algorithm programs the MBATs so that consecutive addresses target echelons in an order that skips first across quadrants, then across SDRAM buses, then across echelons per SDRAM bus, then across banks per rank.
Cell Map
Cell mapping creates a scheme that is easy to implement in hardware. It enables easy calculation of the interleaving parameters for software. In order to do this part of the physical address to perform a lookup into a table, it gives the actual physical cell and ways of interleaving into memory at this address. Be aware of the following:
•A portion of memory that is being interleaved across must start at an offset that is a multiple of the memory chunk for that entry. For example, to interleave across 16 GB of memory with one entry, the starting address for this chunk must be 0 GB, 16 GB, 32 GB, 48 GB, or 64 GB. If using three 2 GB entries to interleave across three cells, then the multiple must be 2 GB, not 6 GB.
•Interleaving is performed across the actual cells within the system. Interleaving can be done across a minimum of 0.5 GB on a cell, and a maximum interleave across 256 GB per cell.
•Each cell in an interleave group must have the same amount of memory interleaved. That is, you cannot interleave 2 GB in one cell and 4 GB in another cell.
The cell map remains the same size as in previous HP Integrity CECs.
Link Interleaving
The link interleaving functionality did not exist in sx1000. This logic is new for the sx2000 CC. The sx2000 enables cells to be connected through multiple paths. In particular, each CC chip has three crossbar links. When one CC sends a packet to another CC, it must specify which link to use.
The CC is the sx2000 chipset cell controller. It interfaces to processors, main memory, the crossbar fabric, an I/O subsystem, and
Memory Error Protection
All of the CC cache lines are protected in memory by an error correction code (ECC). The sx2000 memory ECC scheme is significantly different from the sx1000 memory ECC scheme. An ECC code word is 288 bits long: 264 bits of payload (data and tag) and 24 bits of redundancy. An ECC code word is contained in each pair of
CPUs and Memories 35