En을 ami를 통해 생성했을 때 싱크 속도 문제 발생

Amazon Linux, m5.2xlarge로 Endpoint node를 운영하고 있습니다.
새로운 Endpoint node를 띄우기 위해서 최신 블록까지 싱크가 맞추어져 있는 노드의 Instance Snapshot을 찍어서 새로운 인스턴스를 생성했는데 생성한 인스턴스에서 블록 싱크에 시간이 너무 오래 걸려서 문제 해결 방법을 찾고 있습니다. 혹시 노드가 생성될 때 unique idnetifier가 local에 생성되는게 있나요?

새로운 인스턴스 생성
=> Github release에서 Ken 다운받아서 설치
=> Snapshot archive에서 최신 블록 데이터 받아서 Fast Sync
=> kend 실행, 최신 블록까지 싱크 완료
=> kend stop
=> instance snapshot

=> 생성된 AMI로 새로운 instance 생성
=> 새로운 instance에서 kend start
=> 싱크 속도 문제…

이 순서로 진행했습니다. 혹시 실수가 있었나 해서 여러번 동일한 방법으로 시도해봤지만 계속 같은 문제가 발생했고 어디서 문제가 생기는 건지 전혀 감이 오질 않네요. 한 블록 가져오는데 체감상 2~3초씩은 걸리는 것 같고 블록 싱크 속도가 블록 생성 속도보다 느리다 보니… 마냥 기다릴 수도 없어서 질문 남깁니다.

kend.out 로그입니다.

INFO[08/15,03:14:36 Z] [31] Enabling metrics collection              
INFO[08/15,03:14:36 Z] [31] Enabling Prometheus Exporter             
INFO[08/15,03:14:36 Z] [18] Setting connection type                   nodetype=en conntype=1
INFO[08/15,03:14:36 Z] [18] Cypress bootnodes are set                
INFO[08/15,03:14:36 Z] [18] Bootnode - Add Seed                       Node="kni://18b36118cce093673499fc6e9aa196f047fe17a0de35b6f2a76a4557802f6abf9f89aa5e7330e93c9014b714b9df6378393611efe39aec9d3d831d6aa9d617ae@52.78.22.32:32323?ntype=bn"
INFO[08/15,03:14:36 Z] [18] Bootnode - Add Seed                       Node="kni://63f1c96874da85140ecca3ce24875cb5ef28fa228bc3572e16f690db4a48fc8067502d2f6e8f0c66fb558276a5ada1e4906852c7ae42b0003e9f9f25d1e123b1@52.194.200.217:32323?ntype=bn"
INFO[08/15,03:14:36 Z] [18] Bootnode - Add Seed                       Node="kni://94cc15e2014b86584908707de55800c0a2ea8a24dc5550dcb507043e4cf18ff04f21dc86ed17757dc63b1fa85bb418b901e5e24e4197ad4bbb0d96cd9389ed98@18.136.251.28:32323?ntype=bn"
INFO[08/15,03:14:36 Z] [18] Setting MaxPhysicalConnections            MaxPhysicalConnections=10
INFO[08/15,03:14:36 Z] [18] Cypress network ID is set                 networkid=8217
INFO[08/15,03:14:36 Z] [18] Raised fd limit to process's maximum value  fd=65535
INFO[08/15,03:14:36 Z] [18] Cypress network ID is set                 networkid=8217
INFO[08/15,03:14:36 Z] [18] Archiving mode of this node               isArchiveMode=false
INFO[08/15,03:14:36 Z] [40] Starting peer-to-peer node                instance=Klaytn/v1.6.3+78c0483cd2/linux-amd64/go1.15.7
INFO[08/15,03:14:36 Z] [46] Non-single database is used for persistent storage  DBType=LevelDB
INFO[08/15,03:14:36 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/misc levelDBCacheSize=16 openFilesLimit=655 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:37 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/header levelDBCacheSize=38 openFilesLimit=1638 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:37 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/body   levelDBCacheSize=38 openFilesLimit=1638 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:37 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/receipts levelDBCacheSize=38 openFilesLimit=1638 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:37 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/statetrie_migrated_62853677/0 levelDBCacheSize=76 openFilesLimit=3276 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:37 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/statetrie_migrated_62853677/1 levelDBCacheSize=76 openFilesLimit=3276 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:38 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/statetrie_migrated_62853677/2 levelDBCacheSize=76 openFilesLimit=3276 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:38 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/statetrie_migrated_62853677/3 levelDBCacheSize=76 openFilesLimit=3276 useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:38 Z] [46] Created a sharded database                dbType=statetrie numShards=4
INFO[08/15,03:14:38 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/txlookup                      levelDBCacheSize=16 openFilesLimit=655  useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:38 Z] [46] LevelDB configurations                    path=/home/ec2-user/kend_home/klay/chaindata/bridgeservice                 levelDBCacheSize=16 openFilesLimit=327  useBufferPool=true usePerfCheck=true compressionType=none compactionTableSize(MB)=2 compactionTableSizeMultiplier=1.000
INFO[08/15,03:14:38 Z] [41] Initialised chain configuration           config="{ChainID: 8217 IstanbulCompatibleBlock: <nil> SubGroupSize: 22 UnitPrice: 25000000000 DeriveShaImpl: 2 Engine: istanbul}"
INFO[08/15,03:14:38 Z] [30] Successfully loaded governance state from database  blockNumber=67273728
INFO[08/15,03:14:38 Z] [41] Initialising Klaytn protocol              versions=[64] network=8217
INFO[08/15,03:14:38 Z] [5] Using DeriveShaConcat!                   
INFO[08/15,03:14:38 Z] [47] Initializing local trie node cache (fastCache)  MaxMiB=9510 FilePath=/home/ec2-user/kend_home/fastcache
INFO[08/15,03:14:38 Z] [47] Initialized local trie node cache (fastCache)  LoadedMiB=0 LoadedEntries=0 elapsed=1.928155ms
INFO[08/15,03:14:38 Z] [5] Loaded most recent local header           number=67273019 hash=8bc2f0…2b89a4 td=67273020 age=27m17s565ms
INFO[08/15,03:14:38 Z] [5] Loaded most recent local full block       number=67273019 hash=8bc2f0…2b89a4 td=67273020 age=27m17s565ms
INFO[08/15,03:14:38 Z] [5] Loaded most recent local fast block       number=67273019 hash=8bc2f0…2b89a4 td=67273020 age=27m17s565ms
INFO[08/15,03:14:38 Z] [5] prefetchTxWorkers are started             num=32
INFO[08/15,03:14:38 Z] [5] Loaded local transaction journal          transactions=0 dropped=0
INFO[08/15,03:14:38 Z] [5] Regenerated local transaction journal     transactions=0 accounts=0
INFO[08/15,03:14:38 Z] [40] Starting P2P networking                  
INFO[08/15,03:14:40 Z] [34] UDP listener up                           self="kni://a96179143046427113cacb24e96b21980da852ec626365cf1d33a82633d6641a8d6ccf0e2a045792a8fd33215e9c89f5919efa192222c63cf4c655937613cd98@[::]:32323?ntype=en"
INFO[08/15,03:14:40 Z] [40] Started P2P server                        id=a961791430464271 multichannel=true
INFO[08/15,03:14:40 Z] [40] RLPx listener up                          self="kni://a96179143046427113cacb24e96b21980da852ec626365cf1d33a82633d6641a8d6ccf0e2a045792a8fd33215e9c89f5919efa192222c63cf4c655937613cd98@[::]:32323?ntype=en"
INFO[08/15,03:14:40 Z] [40] RLPx listener up                          self="kni://a96179143046427113cacb24e96b21980da852ec626365cf1d33a82633d6641a8d6ccf0e2a045792a8fd33215e9c89f5919efa192222c63cf4c655937613cd98@[::]:32323?ntype=en"
INFO[08/15,03:14:40 Z] [44] Start listening chain head event to update stakingInfoCache. 
INFO[08/15,03:14:40 Z] [40] IPC endpoint opened                       url=/home/ec2-user/kend_home/klay.ipc
INFO[08/15,03:14:40 Z] [40] FastHTTP endpoint opened                  url=http://0.0.0.0:8551               cors=* vhosts=*
INFO[08/15,03:14:40 Z] [40] FastWebSocket endpoint opened             url=ws://0.0.0.0:8552
INFO[08/15,03:14:44 Z] [33] [Dial] Add dial candidate from static nodes  id=d8e7917913f8056b NodeType=2 ip=54.95.22.119 mainPort=32323 port=[32323]
INFO[08/15,03:14:44 Z] [33] [Dial] Add dial candidate from static nodes  id=301c8d7d8f3b0352 NodeType=2 ip=49.50.164.186 mainPort=32323 port="[32323 32324]"
INFO[08/15,03:14:44 Z] [33] Added a multichannel P2P Peer             id=301c8d7d8f3b0352 conn=trusted-staticdial peerID=301c8d7d8f3b0352
INFO[08/15,03:14:44 Z] [33] Added a multichannel P2P Peer             id=d8e7917913f8056b conn=trusted-staticdial peerID=d8e7917913f8056b
INFO[08/15,03:14:47 Z] [28] Block synchronisation started            
ERROR[08/15,03:14:47 Z] [25] Drop an empty message from timeout channel 
INFO[08/15,03:14:47 Z] [49] Mining aborted due to sync               
INFO[08/15,03:14:47 Z] [28] Downloader queue stats                    receiptTasks=0 blockTasks=53 itemSize=806.18B throttle=8192
INFO[08/15,03:14:51 Z] [5] Inserted a new block                      number=67273020 hash=7a3131…a7995c txs=4 gas=4876673 elapsed=3.892s     processTxs=3.882s finalize=3.556ms validateState=528.623µs totalWrite=3.033273ms trieWrite=2.821371ms
INFO[08/15,03:14:56 Z] [5] Inserted a new block                      number=67273021 hash=f334c1…7d7945 txs=4 gas=379925  elapsed=5.366s     processTxs=5.364s finalize=647.415µs validateState=32.528µs  totalWrite=1.091659ms trieWrite=908.21µs
INFO[08/15,03:14:56 Z] [5] Imported new chain segment                number=67273021 hash=f334c1…7d7945 blocks=2 txs=8 elapsed=9.258s     trieDBSize=263.24kB mgas=5.257 mgasps=0.568 age=27m33s670ms
INFO[08/15,03:14:56 Z] [5] Inserted a new block                      number=67273022 hash=5b519f…43280c txs=1 gas=86416   elapsed=26.023ms   processTxs=688.711µs finalize=748.876µs validateState=9.064µs   totalWrite=1.247227ms trieWrite=1.017197ms
INFO[08/15,03:14:58 Z] [5] Inserted a new block                      number=67273023 hash=1d7d3f…e555aa txs=4 gas=4817735 elapsed=1.869s     processTxs=1.819s    finalize=46.398ms  validateState=948.932µs totalWrite=2.563919ms trieWrite=2.367489ms
INFO[08/15,03:15:03 Z] [5] Inserted a new block                      number=67273024 hash=680d56…f57bf9 txs=3 gas=602741  elapsed=5.084s     processTxs=5.038s    finalize=44.625ms  validateState=77.347µs  totalWrite=1.435957ms trieWrite=1.22279ms
INFO[08/15,03:15:03 Z] [5] Inserted a new block                      number=67273025 hash=6a2e8f…830251 txs=0 gas=0       elapsed=1.536ms    processTxs=200ns     finalize=562.644µs validateState=7.199µs   totalWrite=753.841µs  trieWrite=601.96µs

안녕하세요. Klaytn 개발팀의 Aidan입니다.

공유주신 로그에서 processTxs가 유독 오래걸리는 것을 보아 데이터 읽기에 시간이 많이 소요되는 것으로 추정됩니다. Snapshot을 통해 새로운 Instance를 생성한 후, 처음 노드를 실행할 때 DB compaction 등이 일어난 것으로 보이는데 이 부분은 추가 확인하여 개선하도록 하겠습니다.

@11156 님의 상황처럼 노드를 처음 실행하였을 때 블록처리 속도가 유난히 낮으면, 아래 API를 이용하여 문제를 해결 할 수 있습니다. 이 명령어를 사용하면 State Cache WarmUp이 실행하시면 수 분 ~ 수 시간 이내에 블록처리 속도가 많이 개선되는 것을 확인하실 수 있을것입니다.

debug.startWarmUp()

WarmUp에 소요되는 시간은 Disk IOPS나 Cache 사이즈에 따라 다르며 Cache가 충분히 WarmUp이 된 경우 자동으로 정지됩니다. 또한, debug.stopWarmUp() 명령어를 사용하시면 WarmUp을 즉시 정지시킬 수 있습니다.

궁금하신 점 있으시면 추가적으로 문의 남겨주세요.
감사합니다.

안녕하세요.
같은 문제점을 겪고 있는데, 혹시 해결하신 사항이 있으신지 궁금하여 글 남깁니다.