IBM 10 SP1 EAL4 Figure 5-64: NUMA Design

5.5.3.1 Support for NUMA servers

NUMA is an architecture wherein the memory access time for different regions of memory from a given

processor varies according to the nodal distance of the memory region from the processor. Each region of

memory, to which access times are the same from any CPU, is called a node. The NUMA architecture

surpasses the scalability limits of SMP (Symmetric Multi-Processing) architecture.

With SMP, all memory accesses are posted to the same shared memory bus. This works fine for a relatively

small number of CPUs, but there is a problem with hundreds of CPUs competing for access to this shared

memory bus. NUMA alleviates these bottlenecks by limiting the number of CPUs on any one memory bus

and connecting the various nodes by means of a high-speed interconnect. NUMA allows SLES Server to

scale more efficiently for systems with dozens or hundreds of CPUs, because CPUs can access a dedicated

memory bus for local memory. It also supports multiple interconnected memory nodes, each supporting a

smaller number of CPUs.

Figure 5-64 shows a sample NUMA design. Each node in the system is simply a 4 processor SMP system.

Each CPU in the node contains a L1 and L2 cache. The node contains an L3 cache, which is shared by all

processors in the node. Nodes are connected together via an Interface/Node, which contains the SCI

interface.

The SLES kernel includes various data structures and macros for determining the layout of the memory and

processors on the system. These enable the VM subsystem to make decisions on the optimal placement of

memory for processes. For more information about NUMA, see http://lse.sourceforge.net/numa/.

5.5.3.2 Reverse map Virtual Memory

Reverse map Virtual Memory (Rmap VM) improves the operation of the kernel memory management

subsystem, resulting in improved performance for memory constrained systems, NUMA systems, and

systems with large aggregate virtual address spaces. Previously, the Linux memory management system

could efficiently perform virtual-to-physical address translation using page tables. However, translating a

physical address to its corresponding virtual addresses was inefficient and time consuming, because it

required the memory management software to traverse every page table in the system. In today's large

124

Figure 5-64: NUMA Design

Contents

Main Page Table of Contents Page Page Page Page Page Page Page Page Page 1 Introduction 1.1 Purpose of this document 1.2 Document overview 1.3 Conventions used in this document 1.4 Terminology 2 System Overview Figure 2-1: Series of TOE systems connected by a physically protected LAN 2.1 Product history 2.1.1 SUSE Linux Enterprise Server 2.1.2 eServer systems 2.2 High-level product overview 2.2.1 eServer host computer structure Figure 2-2: Overall structure of the TOE Page 2.2.2 eServer system structure 2.2.3 TOE services 2.2.4 Security policy Figure 2-3: Local and network services provided by SLES 2.2.5 Operation and administration 2.2.6 TSF interfaces 2.3 Approach to TSF identification Page Page 3 Hardware architecture 3.1 System x 3.1.1 System x hardware overview 3.1.2 System x hardware architecture 3.2 System p 3.2.1 System p hardware overview 3.2.2 System p hardware architecture 3.3 System z 3.3.1 System z hardware overview 3.3.2 System z hardware architecture 3.4 eServer 326 Figure 3-1: z/VM as hypervisor 3.4.1 eServer 326 hardware overview 3.4.2 eServer 326 hardware architecture Figure 3-2: AMD x86-64 architecture in compatibility mode Page 4 Software architecture 4.1 Hardware and software privilege 4.1.1 Hardware privilege 4.1.1.1 Privilege level Figure 4-1: Levels of Privilege 4.1.2 Software privilege 4.1.2.1 DAC 4.1.2.1.1 Subjects and objects 4.1.2.1.2 Attributes 4.1.2.1.3 Access control rules 4.1.2.1.4 Software privilege 4.2 TOE Security Functions software structure Figure 4-2: TSF and non-TSF software 4.2.1 Kernel TSF software 4.2.1.1 Logical components Figure 4-3: Logical kernel subsystems and their interactions 4.2.1.2 Execution components Figure 4-4: Kernel execution components 4.2.1.2.1 Base kernel 4.2.1.2.2 Kernel threads 4.2.1.2.3 Kernel modules and device drivers 4.2.2 Non-kernel TSF software Page 4.3 TSF databases 4.4 Definition of subsystems for the CC evaluation 4.4.1 Hardware 4.4.2 Firmware 4.4.3 Kernel subsystems 4.4.4 Trusted process subsystems 4.4.5 User-level audit subsystem Page 5 Functional descriptions 5.1 File and I/O management Figure 5-1: File and I/O subsystem and its interaction with other subsystems 5.1.1 Virtual File System Figure 5-2: ext3 and CD-ROM file systems before mounting Figure 5-3: ext3 and CD-ROM file systems after mounting Figure 5-4: Virtual file system 5.1.1.1 Pathname translation Page 5.1.1.2 open() Figure 5-6: VFS data structures and their relationships with each other 5.1.1.3 write() 5.1.1.4 mount() 5.1.1.5 Shared subtrees 5.1.2 Disk-based file systems 5.1.2.1 Ext3 file system 5.1.2.1.1 Extended Attributes 5.1.2.1.2 Data structures Figure 5-7: Security attributes, extended security attributes, and data blocks for the ext3 inode Page Page Figure 5-9: Access control on ext3 file system 5.1.2.2 ISO 9660 file system for CD-ROM 5.1.2.2.1 Data structures and algorithms 5.1.3 Pseudo file systems Figure 5-10: File lookup on CD-ROM file system 5.1.3.1 procfs 5.1.3.2 tmpfs 5.1.3.3 sysfs 5.1.3.4 devpts 5.1.3.5 rootfs 5.1.3.6 binfmt_misc 5.1.3.7 securityfs 5.1.4 inotify 5.1.5 Discretionary Access Control (DAC) 5.1.5.1 Permission bits 5.1.5.2 Access Control Lists 5.1.5.2.1 Types of ACL tags 5.1.5.2.2 ACL qualifier 5.1.5.2.3 ACL permissions 5.1.5.2.4 Relationship to file permission bits 5.1.5.2.5 ACL_MASK 5.1.5.2.6 Default ACLs and ACL inheritance 5.1.5.2.7 ACL representations and interfaces 5.1.5.2.8 ACL enforcement 5.1.6 Asynchronous I/O 5.1.7 I/O scheduler 5.1.7.1 Deadline I/O scheduler 5.1.7.2 Anticipatory I/O scheduler 5.1.7.3 Completely Fair Queuing scheduler 5.1.7.4 Noop I/O scheduler 5.1.8 I/O interrupts 5.1.8.1 Top halves 5.1.8.2 Bottom halves 5.1.8.3 Softirqs 5.1.8.4 Tasklets 5.1.9 Processor interrupts 5.1.10 Machine check 5.2 Process control and management Figure 5-11: Process subsystem and its interaction with other subsystems 5.2.1 Data structures Page Figure 5-12: The task structure 5.2.2 Process creation and destruction 5.2.2.1 Control of child processes 5.2.2.2 DAC controls 5.2.2.2.1 setuid()and setgid() 5.2.2.2.2 seteuid()and setegid() 5.2.2.2.3 setreuid()and setregid() 5.2.3 Process switch 5.2.4 Kernel threads 5.2.5 Scheduling Figure 5-13: O(1) scheduling 5.2.6 Kernel preemption Figure 5-14: Hyperthreaded scheduling 5.3 Inter-process communication 5.3.1 Pipes Figure 5-15: Pipes Implementation 5.3.1.1 Data structures and algorithms 5.3.2 First-In First-Out Named pipes 5.3.2.1 FIFO creation 5.3.2.2 FIFO open 5.3.3 System V IPC 5.3.3.1 Common data structures 5.3.3.2 Common functions 5.3.3.2.1 ipc_alloc() 5.3.3.2.2 ipcperms() 5.3.3.3 Message queues 5.3.3.3.1 msg_queue Page Page 5.3.4 Signals 5.3.4.1 Data structures 5.3.4.2 Algorithms 5.3.5 Sockets 5.4 Network subsystem Figure 5-16: Object reuse handling in socket allocation 5.4.1 Overview of the network protocol stack Figure 5-17: Network subsystem and its interaction with other subsystems Figure 5-18: How data travels through the Network protocol stack 5.4.2 Transport layer protocols 5.4.2.1 TCP 5.4.2.2 UDP 5.4.3 Network layer protocols 5.4.3.1 Internet Protocol Version 4 (IPv4) 5.4.3.2.1 Addressing 5.4.3.2.2 IPv6 Header 5.4.3.2.3 Flow Labels 5.4.3.2.4 Security 5.4.3.3 Transition between IPv4 and IPv6 5.4.3.4 IP Security (IPsec) 5.4.3.4.1 Functional Description of IPsec Page Page 5.4.4 Internet Control Message Protocol (ICMP) 5.4.4.1 Link layer protocols 5.4.4.1.1 Address Resolution Protocol (ARP) 5.4.5 Network services interface Figure 5-19: Server and client operations using socket interface 5.4.5.1 socket() Figure 5-20: bind() function for internet domain TCP socket 5.4.5.2 bind() Figure 5-21: bind() function for UNIX domain TCP socket 5.4.5.3 listen() 5.4.5.4 accept() 5.4.5.5 connect() 5.4.5.6 Generic calls 5.5 Memory management Figure 5-22: Mapping read, write and close calls for sockets Figure 5-23: Memory subsystem and its interaction with other subsystems 5.5.1 Four-Level Page Tables Figure 5-24: Previous three-level page-tables architecture 5.5.2 Memory addressing 5.5.2.1 System x Figure 5-25: New page-table implementation: the four-level page-table architecture Figure 5-26: System x virtual addressing space Figure 5-27: Logical Address Translation 5.5.2.1.1 Segmentation Figure 5-28: Access control through segmentation 5.5.2.1.2 Paging Figure 5-29: Contiguous linear addresses map to contiguous physical addresses Page Figure 5-32: Access control through paging Page Figure 5-33: Paging data structures 5.5.2.2 System p Figure 5-34: Logical partitions Figure 5-35: Machine state register Page Page 5.5.2.2.1 Address Translation on LPARs 5.5.2.2.2 Hypervisor 5.5.2.2.3 Real mode addressing 5.5.2.2.4 Virtual mode addressing 5.5.2.2.5 Access to I/O address space 5.5.2.2.6 Direct Memory Access addressing Figure 5-38: DMA addressing 5.5.2.2.7 Run-Time Abstraction Services 5.5.2.2.8 Preventing denial of service 5.5.2.3 System p native mode Figure 5-39: Effective address Figure 5-40: Virtual address 5.5.2.3.1 Machine State Register Figure 5-41: Block address Figure 5-42: Machine state register 5.5.2.3.2 Page descriptor 5.5.2.3.3 Segment descriptor 5.5.2.3.4 Block descriptor Figure 5-43: Page table entry Figure 5-44: Segment Table Entry 5.5.2.3.5 Address translation mechanisms Figure 5-46: Address translation method selection Figure 5-45: Block Address Translation entry Figure 5-47: Block Address Translation access control 5.5.2.3.6 Page Address Translation and access control Page Page 5.5.2.4 System z 5.5.2.4.1 Native hardware mode 5.5.2.4.2 LPAR mode 5.5.2.4.3 z/VM Guest mode 5.5.2.4.4 Address types 5.5.2.4.5 Address sizes 5.5.2.4.6 Address spaces 5.5.2.4.7 Address translations Page Figure 5-51: Address translation modes Page 5.5.2.4.8 Memory protection mechanisms Figure 5-53: Low-address protection on effective address Page Page Page Figure 5-56: Key match logic for key-controlled protection Figure 5-57: Fetch protection override for key-controlled 5.5.2.5 eServer 326 5.5.2.5.1 Logical address 5.5.2.5.2 Effective address Figure 5-58: eServer 326 address types and their conversion units 5.5.2.5.3 Linear address 5.5.2.5.4 Physical address 5.5.2.5.5 Segmentation Figure 5-59: Data access privilege checks 5.5.2.5.6 Paging Page Page Page 5.5.2.5.7 Translation Lookaside Buffers 5.5.3 Kernel memory management 5.5.3.1 Support for NUMA servers Figure 5-64: NUMA Design 5.5.3.2 Reverse map Virtual Memory Figure 5-65: Rmap VM 5.5.3.3 Huge Translation Lookaside Buffers Figure 5-66: TLB Operation 5.5.3.4 Remap_file_pages Figure 5-67: Remap_ file_ pages for database applications 5.5.3.5 Page frame management 5.5.3.6 Memory area management 5.5.3.7 Noncontiguous memory area management 5.5.4 Process address space Figure 5-68: Object reuse handling while allocating new linear address 5.5.5 Symmetric multiprocessing and synchronization 5.5.5.1 Atomic operations 5.5.5.2 Memory barriers 5.5.5.3 Spin locks 5.5.5.4 Kernel semaphores 5.6 Audit subsystem 5.6.1 Audit components Figure 5-69: Audit framework components 5.6.1.1 Audit kernel components 5.6.1.1.1 Kernel-userspace interface 5.6.1.1.2 Syscall auditing Figure 5-70: Audit Kernel Components 5.6.1.1.3 Filesystem watches 5.6.1.1.4 Task structure Figure 5-71: Task Structure 5.6.1.1.5 Audit context fields 5.6.1.2 File system audit components 5.6.1.3 User space audit components 5.6.2 Audit operation and configuration options Figure 5-72: Audit User Space Components 5.6.2.1 Configuration Page 5.6.2.2 Operation 5.6.3 Audit records 5.6.3.1 Audit record generation 5.6.3.1.1 Kernel record generation Figure 5-73: Audit Record Generation 5.6.3.1.2 Syscall audit record generation Figure 5-74: Extension to system calls interface 5.6.3.1.3 File system audit record generation 5.6.3.1.4 Socket call and IPC audit record generation Figure 5-75: User Space Record Generation 5.6.3.1.5 Record generation by trusted programs 5.6.3.2 Audit record format Page Page 5.6.4 Audit tools 5.6.4.1 auditctl 5.6.4.2 ausearch 5.6.5 Login uid association 5.7 Kernel modules 5.7.1 Linux Security Module framework Page 5.7.2 LSM capabilities module Figure 5-76: LSM hook architecture 5.7.3 LSM AppArmor module 5.8 AppArmor 5.8.1 AppArmor administrative utilities 5.8.2 AppArmor access control functions 5.8.3 securityfs 5.9 Device drivers 5.9.1 I/O virtualization on System z 5.9.1.1 Interpretive-execution facility 5.9.1.2 State description 5.9.1.3 Hardware virtualization and simulation 5.9.2 Character device driver 5.9.3 Block device driver Figure 5-77: Setup of f_op for character device specific file operations 5.10 System initialization Figure 5-78: Setup of f_op for block device specific file operations 5.10.1 init 5.10.2 System x 5.10.2.1 Boot methods 5.10.2.2 Boot loader 5.10.2.3 Boot process Page Page 5.10.3 System p 5.10.3.1 Boot methods 5.10.3.2 Boot loader 5.10.3.3 Boot process Page Page 5.10.4.1 Boot process Page 5.10.5 System z 5.10.5.1 Boot methods 5.10.5.2 Control program 5.10.5.3 Boot process Page 5.10.6 eServer 326 Figure 5-82: System z SLES boot sequence 5.10.6.1 Boot methods 5.10.6.2 Boot loader 5.10.6.3 Boot process Page 5.11 Identification and authentication Figure 5-83: eServer 326 SLES boot sequence 5.11.1 Pluggable Authentication Module 5.11.1.1 Overview 5.11.1.2 Configuration terminology 5.11.1.3 Modules Page 5.11.2 Protected databases 5.11.2.1 Access control rules 5.11.2.1.1 DAC 5.11.2.1.2 Software privilege 5.11.3 Trusted commands and trusted processes 5.11.3.1 agetty 5.11.3.2 gpasswd 5.11.3.3 login 5.11.3.4 mingetty 5.11.3.5 newgrp 5.11.3.6 passwd 5.11.3.7 su 5.11.4 Interaction with audit 5.12 Network applications 5.12.1 OpenSSL Secure socket-layer interface Figure 5-84: SSL location in the network stack 5.12.1.1 Concepts 5.12.1.1.1 Encryption Figure 5-86: Decryption Figure 5-85: Encryption Figure 5-87: Encryption Algorithm and Key Page 5.12.1.1.2 Message digest 5.12.1.1.3 Message Authentication Code (MAC) 5.12.1.1.4 Digital certificates and certificate authority 5.12.1.2 SSL architecture Figure 5-90: SSL Protocol 5.12.1.2.1 SSL handshake protocol Page Figure 5-92: SSL protocol action 5.12.1.3 OpenSSL algorithms 5.12.1.4 Symmetric ciphers 5.12.1.4.1 Asymmetric ciphers 5.12.1.4.2 Certificates 5.12.1.4.3 Hash functions 5.12.2 Secure Shell 5.12.2.1 SSH client 5.12.2.2 SSH server daemon 5.12.3 Very Secure File Transfer Protocol daemon 5.12.4 CUPS 5.12.4.1 cupsd 5.12.4.2 ping 5.12.4.3 ping6 5.12.4.4 openssl 5.12.4.5 stunnel 5.12.4.6 xinetd 5.13 System management 5.13.1 Account Management 5.13.1.1 chage 5.13.1.2 chfn 5.13.1.3 chsh 5.13.2 User management 5.13.2.1 useradd 5.13.2.2 usermod 5.13.2.3 userdel 5.13.3 Group management 5.13.3.1 groupadd 5.13.3.2 groupmod 5.13.3.3 groupdel Page 5.13.4 System Time management 5.13.4.1 date 5.13.4.2 hwclock 5.13.5 Other System Management 5.13.5.1 AMTU 5.13.5.1.1 Memory 5.13.5.1.2 Memory separation 5.13.5.1.3 I/O controller and network 5.13.5.1.4 I/O controller and disk 5.13.5.1.5 Supervisor mode instructions Page 5.13.5.1.6 AMTU output 5.13.5.2 star Page 5.13.6 I&A support 5.13.6.1 pam_tally 5.13.6.2 unix_chkpwd 5.14 Batch processing 5.14.1 Batch processing user commands 5.14.1.2 at 5.14.2 Batch processing daemons 5.14.2.1 cron 5.14.2.2 atd 5.15 User-level audit subsystem 5.15.1 Audit daemon 5.15.2 Audit utilities 5.15.2.1 aureport 5.15.2.2 ausearch 5.15.3 Audit configuration files 5.15.4 Audit logs 5.16 Supporting functions 5.16.1 TSF libraries Library Description 5.16.2 Library linking mechanism 5.16.3 System call linking mechanism 5.16.3.1 System x 5.16.3.2 System p 5.16.3.3 System z 5.16.4 System call argument verification Page 6 Mapping the TOE summary specification to the High-Level Design Page Page 6.7.4 Trusted processes (TP.4) 6.7.5 TSF Databases (TP.5) 6.7.6 Internal TOE protection mechanisms (TP.6) 6.7.7 Testing the TOE protection mechanisms (TP.7) 6.8 Security enforcing interfaces between subsystems 6.8.1 Summary of kernel subsystem interfaces 6.8.1.1 Kernel subsystem file and I/O 6.8.1.1.1 External Interfaces 6.8.1.1.2 Internal Interfaces 6.8.1.1.3 6.8.1.1.4 Data Structures 6.8.1.2 Kernel subsystem process control and management 6.8.1.2.1 External interfaces (system calls) 6.8.1.2.2 Internal Interfaces 6.8.1.2.3 Data Structures 6.8.1.3 Kernel subsystem inter-process communication 6.8.1.3.1 External interfaces (system calls) 6.8.1.3.2 Internal Interfaces 6.8.1.3.3 Data Structures 6.8.1.4 Kernel subsystem networking 6.8.1.4.1 External interfaces (system calls) 6.8.1.4.2 Internal interfaces 6.8.1.4.3 Data Structures 6.8.1.5 Kernel subsystem memory management 6.8.1.5.2 Internal interfaces 6.8.1.5.3 Data Structures 6.8.1.6 Kernel subsystem audit 6.8.1.6.1 External interfaces 6.8.1.6.2 Internal interfaces 6.8.1.6.3 Data structures 6.8.1.7 Kernel subsystem device drivers 6.8.1.7.1 External interfaces (system calls) 6.8.1.7.2 Internal interfaces Page 6.8.2 Summary of trusted processes interfaces 7 References