Klaytn Endpoint Node Dead

관련 로그 첨부드립니다. /var/log/messages 의 최근내역들 첨부해드렸고 grep이나 서치로 보시면, 중간에 Killed process 5744 (ken_cypress)와 그 위에 Out of memory 보실 수 있습니다.

메모리 부족이 정말 원인이 맞는 것 같네요… 흑흑

Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: CPU: 8 PID: 17196 Comm: mysqld Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org 04/01/2014
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Call Trace:
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d61dc1>] dump_stack+0x19/0x1b
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d5c7ea>] dump_header+0x90/0x229
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2700f72>] ? ktime_get_ts64+0x52/0xf0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2757f7f>] ? delayacct_end+0x8f/0xb0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27ba274>] oom_kill_process+0x254/0x3d0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b9d1d>] ? oom_unkillable_task+0xcd/0x120
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b9dc6>] ? find_lock_task_mm+0x56/0xc0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27baab6>] out_of_memory+0x4b6/0x4f0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d5d2ee>] __alloc_pages_slowpath+0x5d6/0x724
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27c0e95>] __alloc_pages_nodemask+0x405/0x420
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb280dcf8>] alloc_pages_current+0x98/0x110
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b60d7>] __page_cache_alloc+0x97/0xb0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27b8d38>] filemap_fault+0x298/0x490
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffc058a186>] ext4_filemap_fault+0x36/0x50 [ext4]
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e41da>] __do_fault.isra.59+0x8a/0x100
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e478c>] do_read_fault.isra.61+0x4c/0x1b0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27e9134>] handle_pte_fault+0x2f4/0xd10
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb26d0880>] ? finish_task_switch+0x50/0x1c0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb27ebc6d>] handle_mm_fault+0x39d/0x9b0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6f5e3>] __do_page_fault+0x203/0x500
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6f9c6>] trace_do_page_fault+0x56/0x150
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6ef42>] do_async_page_fault+0x22/0xf0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [<ffffffffb2d6b788>] async_page_fault+0x28/0x30
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Mem-Info:
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: active_anon:15027577 inactive_anon:815546 isolated_anon:0#012 active_file:591 inactive_file:366 isolated_file:66#012 unevictable:0 dirty:9 writeback:0 unstable:0#012 slab_reclaimable:21922 slab_unreclaimable:366443#012 mapped:2584 shmem:694159 pagetables:32891 bounce:0#012 free:83293 free_pcp:251 free_cma:0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA free:15908kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 1803 64245 64245
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA32 free:251584kB min:1896kB low:2368kB high:2844kB active_anon:1036220kB inactive_anon:524000kB active_file:124kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080624kB managed:1847120kB mlocked:0kB dirty:20kB writeback:0kB mapped:84kB shmem:29972kB slab_reclaimable:9316kB slab_unreclaimable:20360kB kernel_stack:768kB pagetables:2728kB unstable:0kB bounce:0kB free_pcp:128kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:629 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 0 62442 62442
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 Normal free:65268kB min:65668kB low:82084kB high:98500kB active_anon:59074088kB inactive_anon:2738184kB active_file:2240kB inactive_file:1496kB unevictable:0kB isolated(anon):0kB isolated(file):264kB present:65011712kB managed:63943664kB mlocked:0kB dirty:16kB writeback:0kB mapped:10252kB shmem:2746664kB slab_reclaimable:78372kB slab_unreclaimable:1445412kB kernel_stack:5264kB pagetables:128836kB unstable:0kB bounce:0kB free_pcp:1092kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6578 all_unreclaimable? yes
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: lowmem_reserve[]: 0 0 0 0
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 DMA32: 551*4kB (UEM) 378*8kB (UEM) 632*16kB (UEM) 568*32kB (UEM) 496*64kB (UEM) 306*128kB (UE) 208*256kB (UEM) 90*512kB (UE) 41*1024kB (UEM) 3*2048kB (UEM) 0*4096kB = 251884kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 Normal: 9365*4kB (UEM) 3208*8kB (UEM) 295*16kB (UEM) 4*32kB (E) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 67972kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 695171 total pagecache pages
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 44 pages in swap cache
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Swap cache stats: add 23791418, delete 23791376, find 657061/670559
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Free swap  = 0kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Total swap = 524284kB
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 16777082 pages RAM
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 0 pages HighMem/MovableOnly
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: 325409 pages reserved
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2663]     0  2663     9866     1467      26        0             0 systemd-journal
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2691]     0  2691    11146      150      24       17         -1000 systemd-udevd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 3215]     0  3215    15512      155      29        1         -1000 auditd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 3875]    81  3875    14561      158      33        8          -900 dbus-daemon
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4347]     0  4347     6704      200      18        0             0 systemd-logind
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4389]     0  4389   118413      517      85        5             0 NetworkManager
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4707]   999  4707   179150     2368      66        0             0 polkitd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4758]     0  4758     5403       76      16        0             0 irqbalance
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5392]     0  5392    28192      255      58        2         -1000 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5396]     0  5396   186779    11176     273        9             0 rsyslogd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5397]     0  5397   143457     2680      98      109             0 tuned
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5410]     0  5410    27524       34      10        0             0 agetty
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5412]     0  5412    27524       34      13        0             0 agetty
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15515]  1000 15515    32086      277      17        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15516]  1000 15516    29434      683      14        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [10377]  1000 10377    18079      194      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [16997]  1000 16997    18079      194      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [13982]  1000 13982    28295       47      13        0             0 sh
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [13989]  1000 13989   204580     5625      93        0             0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [16004]  1000 16004    18079      194      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [17192]    27 17192   548619    95924     280     3295             0 mysqld
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 9154]  1001  9154    18079      194      38        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [24112]  1001 24112    18079      193      36        1             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [18954]  1001 18954    18079      193      37        1             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [24747]  1001 24747    18079      194      38        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6399]  1002  6399    18079      194      38        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 8959]  1002  8959    18079      195      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11254]  1001 11254    18079      194      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11393]  1001 11393    31976      153      18        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [11394]  1001 11394    28895      124      14        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15920]  1001 15920    18079      195      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [21299]  1001 21299    18079       87      37      108             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [25266]  1001 25266    18079      195      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5141]  1001  5141    18079      194      38        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6997]  1001  6997    18079      194      39        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [20455]  1001 20455    18079      194      39        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22680]  1001 22680    18079      194      36        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2649]  1001  2649    18079      194      35        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [32102]  1001 32102    18079      194      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2563]  1001  2563    18079      195      37        0             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [32148]  1001 32148    18079      183      37       12             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [15742]  1001 15742    18079      193      39        1             0 ssh-agent
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14096]  1000 14096    31976      154      19        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14097]  1000 14097    29434      683      15        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14369]  1000 14369   215616     2024      91        0             0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [14381]  1000 14381   191879     6568     175        0             0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 2049]     0  2049    31580      170      19        0             0 crond
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22052]  1000 22052    32075      242      18        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22053]  1000 22053    29472      684      14        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [22821]  1000 22821    27000       25      11        0             0 tail
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 1173]  1000  1173    31943      132      17        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 1175]  1000  1175   215964    12714     189        0             0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4538]     0  4538    39291      339      81        0             0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4540]  1000  4540    39291      351      77        0             0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 4541]  1000  4541    29438      678      14        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5375]  1000  5375    31910       65      21        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5430]     0  5430    39291      339      78        0             0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5432]  1000  5432    39291      340      76        0             0 sshd
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5433]  1000  5433    29438      677      15        0             0 bash
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 5744]  1000  5744 15630547 14980153   29687        0             0 ken_cypress
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6328]  1000  6328    31943      118      18        0             0 screen
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6329]  1000  6329   248226    18532     183        0             0 node
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: [ 6393]  1000  6393    40508      202      35        0             0 top
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Out of memory: Kill process 5744 (ken_cypress) score 905 or sacrifice child
Jul 27 06:19:38 Klaytn-Endpoint-Node-Tokyo kernel: Killed process 5744 (ken_cypress) total-vm:62522188kB, anon-rss:59920612kB, file-rss:0kB, shmem-rss:0kB
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Created slice User Slice of root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868163 of user root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868162 of user root.
Jul 27 06:20:01 Klaytn-Endpoint-Node-Tokyo systemd: Started Session 868161 of user root.
2개의 좋아요