관련 로그 첨부드립니다. /var/log/messages 의 최근내역들 첨부해드렸고 grep이나 서치로 보시면, 중간에 Killed process 5744 (ken_cypress)와 그 위에 Out of memory 보실 수 있습니다.
메모리 부족이 정말 원인이 맞는 것 같네요… 흑흑
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: CPU: 8 PID: 17196 Comm: mysqld Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org 04/01/2014
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Call Trace:
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d61dc1>] dump_stack+0x19/0x1b
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d5c7ea>] dump_header+0x90/0x229
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2700f72>] ? ktime_get_ts64+0x52/0xf0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2757f7f>] ? delayacct_end+0x8f/0xb0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27ba274>] oom_kill_process+0x254/0x3d0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b9d1d>] ? oom_unkillable_task+0xcd/0x120
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b9dc6>] ? find_lock_task_mm+0x56/0xc0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27baab6>] out_of_memory+0x4b6/0x4f0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d5d2ee>] __alloc_pages_slowpath+0x5d6/0x724
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27c0e95>] __alloc_pages_nodemask+0x405/0x420
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb280dcf8>] alloc_pages_current+0x98/0x110
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b60d7>] __page_cache_alloc+0x97/0xb0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b8d38>] filemap_fault+0x298/0x490
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffc058a186>] ext4_filemap_fault+0x36/0x50 [ext4]
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e41da>] __do_fault.isra.59+0x8a/0x100
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e478c>] do_read_fault.isra.61+0x4c/0x1b0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e9134>] handle_pte_fault+0x2f4/0xd10
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb26d0880>] ? finish_task_switch+0x50/0x1c0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27ebc6d>] handle_mm_fault+0x39d/0x9b0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6f5e3>] __do_page_fault+0x203/0x500
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6f9c6>] trace_do_page_fault+0x56/0x150
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6ef42>] do_async_page_fault+0x22/0xf0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6b788>] async_page_fault+0x28/0x30
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Mem-Info:
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: active_anon:15027577 inactive_anon:815546 isolated_anon:0#012 active_file:591 inactive_file:366 isolated_file:66#012 unevictable:0 dirty:9 writeback:0 unstable:0#012 slab_reclaimable:21922 slab_unreclaimable:366443#012 mapped:2584 shmem:694159 pagetables:32891 bounce:0#012 free:83293 free_pcp:251 free_cma:0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA free:15908kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 1803 64245 64245
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA32 free:251584kB min:1896kB low:2368kB high:2844kB active_anon:1036220kB inactive_anon:524000kB active_file:124kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080624kB managed:1847120kB mlocked:0kB dirty:20kB writeback:0kB mapped:84kB shmem:29972kB slab_reclaimable:9316kB slab_unreclaimable:20360kB kernel_stack:768kB pagetables:2728kB unstable:0kB bounce:0kB free_pcp:128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:629 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 0 62442 62442
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 Normal free:65268kB min:65668kB low:82084kB high:98500kB active_anon:59074088kB inactive_anon:2738184kB active_file:2240kB inactive_file:1496kB unevictable:0kB isolated(anon):0kB isolated(file):264kB present:65011712kB managed:63943664kB mlocked:0kB dirty:16kB writeback:0kB mapped:10252kB shmem:2746664kB slab_reclaimable:78372kB slab_unreclaimable:1445412kB kernel_stack:5264kB pagetables:128836kB unstable:0kB bounce:0kB free_pcp:1092kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6578 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 0 0 0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA32: 551*4kB (UEM) 378*8kB (UEM) 632*16kB (UEM) 568*32kB (UEM) 496*64kB (UEM) 306*128kB (UE) 208*256kB (UEM) 90*512kB (UE) 41*1024kB (UEM) 3*2048kB (UEM) 0*4096kB = 251884kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 Normal: 9365*4kB (UEM) 3208*8kB (UEM) 295*16kB (UEM) 4*32kB (E) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 67972kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 695171 total pagecache pages
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 44 pages in swap cache
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Swap cache stats: add 23791418, delete 23791376, find 657061/670559
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Free swap = 0kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Total swap = 524284kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 16777082 pages RAM
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 0 pages HighMem/MovableOnly
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 325409 pages reserved
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2663] 0 2663 9866 1467 26 0 0 systemd-journal
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2691] 0 2691 11146 150 24 17 -1000 systemd-udevd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 3215] 0 3215 15512 155 29 1 -1000 auditd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 3875] 81 3875 14561 158 33 8 -900 dbus-daemon
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4347] 0 4347 6704 200 18 0 0 systemd-logind
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4389] 0 4389 118413 517 85 5 0 NetworkManager
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4707] 999 4707 179150 2368 66 0 0 polkitd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4758] 0 4758 5403 76 16 0 0 irqbalance
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5392] 0 5392 28192 255 58 2 -1000 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5396] 0 5396 186779 11176 273 9 0 rsyslogd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5397] 0 5397 143457 2680 98 109 0 tuned
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5410] 0 5410 27524 34 10 0 0 agetty
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5412] 0 5412 27524 34 13 0 0 agetty
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15515] 1000 15515 32086 277 17 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15516] 1000 15516 29434 683 14 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [10377] 1000 10377 18079 194 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [16997] 1000 16997 18079 194 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [13982] 1000 13982 28295 47 13 0 0 sh
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [13989] 1000 13989 204580 5625 93 0 0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [16004] 1000 16004 18079 194 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [17192] 27 17192 548619 95924 280 3295 0 mysqld
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 9154] 1001 9154 18079 194 38 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [24112] 1001 24112 18079 193 36 1 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [18954] 1001 18954 18079 193 37 1 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [24747] 1001 24747 18079 194 38 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6399] 1002 6399 18079 194 38 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 8959] 1002 8959 18079 195 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11254] 1001 11254 18079 194 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11393] 1001 11393 31976 153 18 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11394] 1001 11394 28895 124 14 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15920] 1001 15920 18079 195 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [21299] 1001 21299 18079 87 37 108 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [25266] 1001 25266 18079 195 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5141] 1001 5141 18079 194 38 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6997] 1001 6997 18079 194 39 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [20455] 1001 20455 18079 194 39 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22680] 1001 22680 18079 194 36 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2649] 1001 2649 18079 194 35 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [32102] 1001 32102 18079 194 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2563] 1001 2563 18079 195 37 0 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [32148] 1001 32148 18079 183 37 12 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15742] 1001 15742 18079 193 39 1 0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14096] 1000 14096 31976 154 19 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14097] 1000 14097 29434 683 15 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14369] 1000 14369 215616 2024 91 0 0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14381] 1000 14381 191879 6568 175 0 0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2049] 0 2049 31580 170 19 0 0 crond
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22052] 1000 22052 32075 242 18 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22053] 1000 22053 29472 684 14 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22821] 1000 22821 27000 25 11 0 0 tail
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 1173] 1000 1173 31943 132 17 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 1175] 1000 1175 215964 12714 189 0 0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4538] 0 4538 39291 339 81 0 0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4540] 1000 4540 39291 351 77 0 0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4541] 1000 4541 29438 678 14 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5375] 1000 5375 31910 65 21 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5430] 0 5430 39291 339 78 0 0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5432] 1000 5432 39291 340 76 0 0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5433] 1000 5433 29438 677 15 0 0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5744] 1000 5744 15630547 14980153 29687 0 0 ken_cypress
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6328] 1000 6328 31943 118 18 0 0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6329] 1000 6329 248226 18532 183 0 0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6393] 1000 6393 40508 202 35 0 0 top
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Out of memory: Kill process 5744 (ken_cypress) score 905 or sacrifice child
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Killed process 5744 (ken_cypress) total-vm:62522188kB, anon-rss:59920612kB, file-rss:0kB, shmem-rss:0kB
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Created slice User Slice of root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868163 of user root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868162 of user root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868161 of user root.