Topic
1 reply Latest Post - ‏2012-12-18T09:56:40Z by iroman
SystemAdmin
SystemAdmin
2404 Posts
ACCEPTED ANSWER

Pinned topic Performance Issue

‏2012-12-18T09:10:49Z |
Hi,

I once was an oracle dba and now deal with software engineering. Tracking performance of our oracle servers, I found two opinions within my enterprise: one colleague says "Don't worry, be happy", the other one discoverd some performance leaks. So who is right:

-topas_nmon--h=Help-------------Host=ORA11---Refresh=2 secs---11:01.01---------------------------------------------------------------------------------
¦ Memory ------------------------------------------------------------------------------------------------------------------------------------------------------¦
¦ Physical PageSpace | pages/sec In Out | FileSystemCache ¦
¦% Used 99.3% 5.1% | to Paging Space 0.0 0.0 | (numperm) 8.8% ¦
¦% Free 0.7% 94.9% | to File System 8.7 252.9 | Process 74.5% ¦
¦MB Used 48830.5MB 627.3MB | Page Scans 0.0 | System 16.0% ¦
¦MB Free 321.5MB 11660.7MB | Page Cycles 0.0 | Free 0.7% ¦
¦Total(MB) 49152.0MB 12288.0MB | Page Steals 0.0 |
¦
¦ | Page Faults 2400.2 | Total 100.0% ¦
¦------------------------------------------------------------ | numclient 8.8% ¦
¦Min/Maxperm 1430MB( 3%) 42898MB( 87%) <--% of RAM | maxclient 87.3% ¦
¦Min/Maxfree 960 1088 Total Virtual 60.0GB | User 80.5% ¦
¦Min/Maxpgahead 2 8 Accessed Virtual 42.0GB 70.0%| Pinned 17.3% ¦
¦--------------------------------------------------------------------------------------------------------------------------------------------------------------¦
...
+-topas_nmon--+=slower-----------Host=ORA11---Refresh=2 secs---11:16.25----------------------------------------------------------------------------------+
¦ Memory -------------------------------------------------------------------------------------------------------------------------------------------------------¦
¦ Physical PageSpace | pages/sec In Out | FileSystemCache ¦
¦% Used 99.9% 5.1% | to Paging Space 0.0 0.0 | (numperm) 9.0% ¦
¦% Free 0.1% 94.9% | to File System 12948.9 73.0| Process 74.9% ¦
¦MB Used 49092.8MB 628.7MB | Page Scans 49041.3 | System 16.0% ¦
¦MB Free 59.2MB 11659.3MB | Page Cycles 0.0 | Free 0.1% ¦
¦Total(MB) 49152.0MB 12288.0MB | Page Steals 13061.4 |
¦
¦ | Page Faults 142.5 | Total 100.0% ¦
¦------------------------------------------------------------ | numclient 9.0% ¦
¦Min/Maxperm 1430MB( 3%) 42898MB( 87%) <--% of RAM | maxclient 87.3% ¦
¦Min/Maxfree 960 1088 Total Virtual 60.0GB | User 81.0% ¦
¦Min/Maxpgahead 2 8 Accessed Virtual 42.2GB 70.3%| Pinned 17.3% ¦
¦---------------------------------------------------------------------------------------------------------------------------------------------------------------¦

  1. vmstat -I 30

System configuration: lcpu=12 mem=49152MB ent=2.50

kthr memory page faults cpu

-----------
------------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
2 0 0 11028640 49945 3 102 0 0 0 0 235 15058 1746 27 7 65 0 0.92 36.6
4 0 0 11025745 52650 0 67 0 0 0 0 238 33100 1991 26 13 60 0 1.05 42.1
2 1 0 11024425 53956 0 54 0 0 0 0 176 7866 1378 12 5 83 0 0.46 18.5
3 1 0 11033620 44706 0 51 0 0 0 0 174 49817 2075 30 11 59 0 1.07 42.6
4 2 0 11038535 15474 122 759 0 0 0 0 362 18913 1757 27 8 64 1 0.93 37.0
3 1 0 11056621 15141 375 1892 0 0 2750 3764 423 7825 1771 16 8 74 2 0.63 25.2
3 2 0 11044570 21265 3975 465 0 0 4065 9238 1012 37284 4015 38 16 42 4 1.43 57.2
3 1 0 11041423 17917 210 174 0 0 87 156 680 19260 2375 28 10 60 2 1.01 40.4
3 1 0 11041302 24787 150 255 0 0 521 1051 740 41315 2844 35 11 53 2 1.20 48.2
3 1 0 11047396 20417 255 195 0 0 450 2252 666 16090 2274 25 7 66 2 0.86 34.2
3 1 0 11048629 16378 424 203 0 0 442 1098 826 38731 3014 35 11 51 3 1.23 49.3
3 1 0 11046328 15566 3442 450 0 0 3588 15423 992 20375 3117 32 13 53 2 1.19 47.8
5 2 0 11048362 15327 8003 912 0 0 8560 18528 1713 29916 4354 30 19 47 5 1.30 52.0
3 1 0 11042275 25349 3950 133 0 0 4125 7246 1237 34536 3593 44 19 34 3 1.67 67.0
3 1 0 11042131 16820 303 315 0 0 176 377 765 18449 2645 55 6 37 1 1.61 64.4
3 1 0 11042499 16566 823 566 0 0 1306 2773 1000 44225 4448 37 15 45 2 1.40 56.0
3 1 0 11042431 15017 226 830 0 0 910 3293 825 21529 3609 34 8 56 2 1.14 45.4
3 1 0 11041707 21116 428 145 0 0 658 1789 756 41008 2991 36 12 49 2 1.29 51.6
4 1 0 11035412 20764 2395 192 0 0 2276 6263 800 16832 2630 52 8 38 2 1.58 63.3
3 1 0 11057935 19570 10314 140 0 0 11030 26850 844 41278 3738 76 20 3 1 2.49 99.6
kthr memory page faults cpu

-----------
------------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
3 1 0 11043955 16228 642 170 0 0 158 1121 742 16821 2808 39 8 49 3 1.27 50.7
3 1 0 11035900 22827 751 457 0 0 869 2244 1141 17685 3713 50 10 35 4 1.60 63.9
3 1 0 11032091 24156 1007 206 0 0 988 1532 965 38823 3223 34 15 47 4 1.30 51.9
4 1 0 11031926 20134 614 273 0 0 585 647 853 35167 3090 38 12 48 3 1.31 52.6
3 1 0 11034236 33130 702 166 0 0 1222 2798 796 36626 3020 44 15 39 2 1.55 62.0
3 1 0 11031295 26830 255 121 0 0 0 0 619 18611 2363 34 9 55 2 1.14 45.6
4 1 0 11033034 20590 251 184 0 0 158 700 772 43835 3254 46 13 39 1 1.56 62.5
3 2 0 11033248 20460 190 104 0 0 215 354 388 11939 1704 16 6 77 1 0.60 23.9
3 1 0 11033243 17865 64 231 0 0 0 0 596 20596 2402 11 6 81 1 0.48 19.2
3 1 0 11033231 15867 205 166 0 0 145 387 464 15817 2234 14 10 73 2 0.66 26.5
2 1 0 11033282 15894 211 96 0 0 220 1216 278 5910 1561 14 5 79 3 0.50 20.2
3 1 0 11034119 20021 129 104 0 0 302 1599 291 32062 1975 21 14 64 1 0.93 37.2
2 1 0 11033386 20644 2 77 0 0 4 4 212 7721 1352 9 5 85 0 0.39 15.7
3 1 0 11035046 16193 53 210 0 0 45 58 362 30045 2018 13 11 76 0 0.63 25.3
2 1 0 11035090 16287 14 146 0 0 68 178 215 5402 1344 9 4 87 0 0.35 14.2
3 2 0 11035459 16036 2 103 0 0 33 81 232 29415 1850 19 11 70 0 0.79 31.7
3 1 0 11039503 17481 2 147 0 0 229 1895 235 14149 1627 18 8 74 0 0.69 27.7
3 1 0 11053651 18186 469 157 0 0 984 5958 430 52104 2091 21 12 66 1 0.88 35.1
3 0 0 11062284 17280 79 130 0 0 340 1115 381 66324 2146 25 14 61 0 1.03 41.1
2 1 0 11069156 16098 273 124 0 0 467 2704 529 37421 1974 24 8 67 1 0.85 34.0

best regards

Matthias
Updated on 2012-12-18T09:56:40Z at 2012-12-18T09:56:40Z by iroman
  • iroman
    iroman
    13 Posts
    ACCEPTED ANSWER

    Re: Performance Issue

    ‏2012-12-18T09:56:40Z  in response to SystemAdmin
    You have a performance issue because there are blocked processes in the queue. Blocked processes are due to the scans to free up memory pages. You have to tune the memory settings. For the current workload (as in vmstat) you need 43 GB of memory,so the memory is not overcomitted yet (there is no paging activity as well so far). Check the filesystem cache and try to adjust the limits.Check as well if you are short of 64K pages (vmstat -P ALL ).
    Conclusion is that too many scans for releasing memory pages slow down application.