What does “mvcc: database space exceeded” mean and how do I fix it?
The multi-version concurrency control data model in etcd keeps an exact history of the keyspace. Without periodically compacting this history (e.g., by setting --auto-compaction), etcd will eventually exhaust its storage space. If etcd runs low on storage space, it raises a space quota alarm to protect the cluster from further writes. So long as the alarm is raised, etcd responds to write requests with the error mvcc: database space exceeded.
可以确认下使用空间
- ETCDCTL_API=3 etcdctl --write-out=table endpoint status
解决办法：
1. 手动清理历史数据
-
# get current revision
-
$ rev=$(ETCDCTL_API=3 etcdctl --endpoints=:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
-
# compact away all old revisions
-
$ ETCDCTL_API=3 etcdctl compact $rev
-
compacted revision 1516
-
# defragment away excessive space
-
$ ETCDCTL_API=3 etcdctl defrag
-
Finished defragmenting etcd member[127.0.0.1:2379]
-
# disarm alarm
-
$ ETCDCTL_API=3 etcdctl alarm disarm
-
memberID:13803658152347727308 alarm:NOSPACE
-
# test puts are allowed again
-
$ ETCDCTL_API=3 etcdctl put newkey 123
- OK
Auto Compaction
etcd can be set to automatically compact the keyspace with the --auto-compaction-* option with a period of hours:
# keep one hour of history $ etcd --auto-compaction-retention=1