[20191202]关于hugepages相关问题.txt

[20191202]关于hugepages相关问题.txt --//上午在测试hugepages相关问题时遇到一个问题,自己以前疏忽了,就是在use_large_pages=true的情况下. --//如果在/etc/sysctl.conf参数文件中vm.nr_hugepages定义不够大,即使定义vm.nr_overcommit_hugepages足够. --//也不会向vm.nr_overcommit_hugepages借用内存,导致进入混合模式. --//通过例子说明: 1.环境: SCOTT@book> @ ver1 PORT_STRING                    VERSION        BANNER ------------------------------ -------------- -------------------------------------------------------------------------------- x86_64/Linux 2.4.xx            11.2.0.4.0     Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production SCOTT@book> alter system set use_large_pages=true scope=spfile; System altered. # grep "^vm.nr" /etc/sysctl.conf vm.nr_hugepages = 104 vm.nr_overcommit_hugepages = 1000 # sysctl -p 2.测试: --//重启数据库看看,检查alert文件发现: Adjusting the default value of parameter parallel_max_servers from 480 to 170 due to the value of parameter processes (200) Starting ORACLE instance (normal) ************************ Large Pages Information ******************* Per process system memlock (soft) limit = 51 GB Total Shared Global Region in Large Pages = 208 MB (33%) Large Pages used by this instance: 104 (208 MB) Large Pages unused system wide = 0 (0 KB) Large Pages configured system wide = 104 (208 MB) Large Page size = 2048 KB RECOMMENDATION:   Total System Global Area size is 618 MB. For optimal performance,   prior to the next instance restart:   1. Increase the number of unused large pages by  at least 205 (page size 2048 KB, total size 410 MB) system wide to   get 100% of the System Global Area allocated with large pages ******************************************************************** --//可以发现仅仅使用定义vm.nr_hugepages = 104. $ grep -i page /proc/meminfo AnonPages:        165700 kB PageTables:        18904 kB AnonHugePages:         0 kB HugePages_Total:     104 HugePages_Free:       49 HugePages_Rsvd:       49 HugePages_Surp:        0 Hugepagesize:       2048 kB $ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps 60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1 60c00000 default file=/SYSV00000000\040(deleted) huge 6a400000 default file=/SYSV00000000\040(deleted) huge 6c400000 default file=/SYSV00000000\040(deleted) huge 6cc00000 default file=/SYSV00000000\040(deleted) huge --//仅仅5个共享内存段使用hugepages. 6d000000 interleave:0-1 file=/SYSV00000000\040(deleted) dirty=785 mapmax=25 active=245 N0=400 N1=385 86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) dirty=1 mapmax=25 N1=1 $ sysresv IPC Resources for ORACLE_SID "book" : Shared Memory: ID              KEY 413040641       0x00000000 413073410       0x00000000 413106179       0x00000000 413138949       0x00000000 413171718       0x00000000 413204487       0x00000000 413237256       0xe8a8ec10 Semaphores: ID              KEY 335151104       0x6aa88594 Oracle Instance alive for sid "book" --//也就是在use_large_pages=true的情况下不会向vm.nr_overcommit_hugepages参数借用内存. 3.测试: SYS@book> alter system set use_large_pages=only scope=spfile; System altered. # grep "^vm.nr" /etc/sysctl.conf vm.nr_hugepages = 104 vm.nr_overcommit_hugepages = 204 # sysctl -p .. SYS@book> startup ORA-27137: unable to allocate large pages to create a shared memory segment Linux-x86_64 Error: 12: Cannot allocate memory Additional information: 2097152 --//104+204 = 308,还不够大. # grep "^vm.nr" /etc/sysctl.conf vm.nr_hugepages = 104 vm.nr_overcommit_hugepages = 1000 # sysctl -p .. SYS@book> startup ORACLE instance started. Total System Global Area  643084288 bytes Fixed Size                  2255872 bytes Variable Size             205521920 bytes Database Buffers          427819008 bytes Redo Buffers                7487488 bytes Database mounted. Database opened. --//alert输出如下: Adjusting the default value of parameter parallel_max_servers from 480 to 170 due to the value of parameter processes (200) Starting ORACLE instance (normal) ************************ Large Pages Information ******************* Parameter use_large_pages = ONLY Per process system memlock (soft) limit = 51 GB Large Pages unused system wide = 308 (616 MB) Large Pages configured system wide = 308 (616 MB) Large Page size = 2048 KB ERROR:   Failed to allocate shared global region with large pages, unix errno = 12.   Aborting Instance startup.   ORA-27137: unable to allocate Large Pages to create a shared memory segment ACTION:   Total System Global Area size is 624 MB. Prior to next instance restart:  1. Increase the number of unused large pages (page size 2048 KB)  to at least 312 (624 MB) to allocate 100% System Global Area  with large pages. ******************************************************************** --//以上是报ORA-27137的情况. Mon Dec 02 15:59:56 2019 Adjusting the default value of parameter parallel_max_servers from 480 to 170 due to the value of parameter processes (200) Starting ORACLE instance (normal) ************************ Large Pages Information ******************* Parameter use_large_pages = ONLY Per process system memlock (soft) limit = 51 GB Total Shared Global Region in Large Pages = 618 MB (100%) Large Pages used by this instance: 309 (618 MB) Large Pages unused system wide = 0 (0 KB) Large Pages configured system wide = 309 (618 MB) Large Page size = 2048 KB ******************************************************************** $ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps 60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1 60c00000 default file=/SYSV00000000\040(deleted) huge dirty=37 mapmax=25 N0=30 N1=7 86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) huge dirty=1 mapmax=25 N0=1 $ sysresv IPC Resources for ORACLE_SID "book" : Shared Memory: ID              KEY 413401089       0x00000000 413433858       0x00000000 413466627       0xe8a8ec10 Semaphores: ID              KEY 335314944       0x6aa88594 Oracle Instance alive for sid "book" 3.总结: --//设置内核参数vm.nr_overcommit_hugepages ,仅仅在use_large_pages=only的情况下才会有效.