geth 1.8.15 fatal error runtime out of memory

Posted on Edited on

Get a fatal error: runtime: out of memory in geth. Env is as below:

  • geth version: 1.8.15-stable
  • go version: 1.11 linux/amd64
  • Linux version 4.14.47-64.38.amzn2.x86_64
  • (mockbuild@ip-10-0-1-219)
  • (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC))
  • aws型号 c5.large | vCPU：2 | 内存 (GiB)：4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
INFO [11-22|01:36:30.725] Submitted transaction                    fullhash=0x583c48c36a165daed190212facf12813b2a5331b05e43ede7a1e8426936d83d6 recipient=0xe2E55079E31cf95589B5433cF06beb448624d0A7
INFO [11-22|01:36:31.286] Committing block                         stateHash=cc2f6d…6fbe5b blockHash=3b92a3…2a3b13
INFO [11-22|01:36:31.287] Imported new chain segment               blocks=1 txs=1 mgas=0.021 elapsed=428.986µs mgasps=48.953 number=3 hash=607b71…154d5e cache=964.00B
INFO [11-22|01:36:31.287] Commiting                                module=oval        blockHash=3b92a30f0a4b332a95b5bf304d133b5ebe76210f6c329b14148069a77c2a3b13 cbr=cc2f6d729a06198723f407523b4ac61796a0198aa9e9e3d5a1c17e37e56fbe5b
INFO [11-22|01:42:00.336] Committing block                         stateHash=de1af6…3b83f8 blockHash=71c97f…2c1bdc
INFO [11-22|01:42:00.337] Imported new chain segment               blocks=1 txs=1 mgas=0.075 elapsed=456.862µs mgasps=163.472 number=4 hash=295456…55f08d cache=1.27kB
INFO [11-22|01:42:00.337] Commiting                                module=oval        blockHash=71c97f9a55e28a03d9c114a9749d6a8f50dccbc2a3538f4ce6818fb2c52c1bdc cbr=de1af60245d823ff58d280ced344046ef4348957b83900d55cfd8e108d3b83f8
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x10bd3c9, 0x16)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xc0cc000000, 0x10000000, 0x217fdf8)
	/usr/local/go/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0x2165e60, 0x10000000, 0x313d68, 0x7fcb9139acd8)
	/usr/local/go/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0x2165e60, 0x8000, 0x0)
	/usr/local/go/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0x2165e60, 0x8000, 0x217fe08, 0x448fab)
	/usr/local/go/src/runtime/mheap.go:848 +0x337
runtime.(*mheap).alloc_m(0x2165e60, 0x8000, 0x7fcb9cf00101, 0x7fcb9cf025a0)
	/usr/local/go/src/runtime/mheap.go:692 +0x119
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:759 +0x4c
runtime.(*mheap).alloc(0x2165e60, 0x8000, 0x7fcb91010101, 0x42baf5)
	/usr/local/go/src/runtime/mheap.go:758 +0x8a
runtime.largeAlloc(0x10000000, 0x460101, 0x7fcb9d0896c0)
	/usr/local/go/src/runtime/malloc.go:1019 +0x97
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:914 +0x46
runtime.systemstack(0xc000308480)
	/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1229
......

【原因分析】
geth莫名其妙自动关闭，日志未呈现异常。此问题是因为服务器内存不足触发Linux的OOM killer操作，被杀掉了。
【解决方案】

  1. 设置 –cache，并不能解决问题。
  2. 除了升级内存，没有太好的办法。只能依靠监控程序报警，发现问题重启即可。
  3. 需要在节点间设置 rpc 通信的白名单，防止黑客暴力破解 keystore 的密码。
  4. 其中折中的办法是设置swap。具体命令如下：
1
2
3
4
5
6
7
8
free -m
dd if=/dev/zero of=/swap bs=10M count=800
chmod 600 /swap
ll /swap
mkswap /swap
swapon /swap
swapon -s
free -m

相关链接：