Fork me on GitHub

安装部署3节点Etcd高可用集群(3节点同一套证书)

发表于 |
字数统计: 2,580

环境

IP 主机名
10.1.80.91 s1
10.1.80.92 s2
10.1.80.93 s3

安装cfssl

cfssl版本 可执行文件存放目录 证书存放目录
1.6.2 /usr/local/bin /tmp/certs

下载地址
https://github.com/cloudflare/cfssl/releases

1
2
3
4
5
6
7
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssl_1.6.2_linux_amd64 -o /tmp/cfssl
chmod +x /tmp/cfssl
mv /tmp/cfssl /usr/bin/cfssl

curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.2/cfssljson_1.6.2_linux_amd64 -o /tmp/cfssljson
chmod +x /tmp/cfssljson
mv /tmp/cfssljson /usr/bin/cfssljson

生成自签名CA证书

可通过http://play.etcd.io/install生成配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
mkdir -p /tmp/certs

cat > /tmp/certs/root-ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "Meeleet",
      "OU": "Meeleet Security",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "C": "CN"
    }
  ],
  "CN": "root-ca"
}
EOF
cfssl gencert --initca=true /tmp/certs/root-ca-csr.json | cfssljson --bare /tmp/certs/root-ca

# verify
openssl x509 -in /tmp/certs/root-ca.pem -text -noout


# cert-generation configuration
cat > /tmp/certs/gencert.json <<EOF
{
  "signing": {
    "default": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "876000h"
    }
  }
}
EOF

结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# CSR configuration
/tmp/certs/root-ca-csr.json

# CSR
/tmp/certs/root-ca.csr

# self-signed root CA public key
/tmp/certs/root-ca.pem

# self-signed root CA private key
/tmp/certs/root-ca-key.pem

# cert-generation configuration for other TLS assets
/tmp/certs/gencert.json

生成本地颁发的带有私钥的证书

证书相关操作

由于3个节点用同一套证书，所以只需生成一套即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
mkdir -p /tmp/certs

cat > /tmp/certs/etcd-ca-csr.json <<EOF
{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "Meeleet",
      "OU": "Meeleet Security",
      "L": "Hangzhou",
      "ST": "Zhejiang",
      "C": "CN"
    }
  ],
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "localhost",
    "10.1.80.91",
    "10.1.80.92",
    "10.1.80.93"
  ]
}
EOF
cfssl gencert \
  --ca /tmp/certs/root-ca.pem \
  --ca-key /tmp/certs/root-ca-key.pem \
  --config /tmp/certs/gencert.json \
  /tmp/certs/etcd-ca-csr.json | cfssljson --bare /tmp/certs/etcd

# verify
openssl x509 -in /tmp/certs/etcd.pem -text -noout

结果

1
2
3
4
5
6
7
8
9
-rw-r--r--  1 root root  323 Sep 30 05:28 etcd-ca-csr.json
-rw-r--r--  1 root root 1098 Sep 30 05:28 etcd.csr
-rw-------  1 root root 1679 Sep 30 05:28 etcd-key.pem
-rw-r--r--  1 root root 1493 Sep 30 05:28 etcd.pem
-rw-r--r--  1 root root  205 Sep 30 05:25 gencert.json
-rw-r--r--  1 root root 1017 Sep 30 05:25 root-ca.csr
-rw-r--r--  1 root root  221 Sep 30 05:25 root-ca-csr.json
-rw-------  1 root root 1679 Sep 30 05:25 root-ca-key.pem
-rw-r--r--  1 root root 1346 Sep 30 05:25 root-ca.pem

将证书传到s91, s92, s93,方便起见，就把所有文件都传了

1
2
3
scp -r /tmp/certs/ root@10.1.80.91:/root/
scp -r /tmp/certs/ root@10.1.80.92:/root/
scp -r /tmp/certs/ root@10.1.80.93:/root/

这里请更改成您自己所用的系统用户名

安装etcd

s91,s92,s93上安装etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
ETCD_VER=v3.4.21
ETCD_INSTALL_PATH=/opt/etcd3

# choose either URL
# GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}

mkdir -p ${ETCD_INSTALL_PATH}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o ${ETCD_INSTALL_PATH}/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf ${ETCD_INSTALL_PATH}/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /opt/etcd3 --strip-components=1
rm -f ${ETCD_INSTALL_PATH}/etcd-${ETCD_VER}-linux-amd64.tar.gz

${ETCD_INSTALL_PATH}/etcd --version
${ETCD_INSTALL_PATH}/etcdctl version

frontend 运行etcd(不推荐，只用于测试功能用)

分别在s91,s92,s93上运行etcd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# make sure etcd process has write access to this directory
# remove this directory if the cluster is new; keep if restarting etcd
# rm -rf /tmp/etcd


/opt/etcd3/etcd --name s1 \
  --data-dir /tmp/etcd/s1 \
  --listen-client-urls https://10.1.80.91:2379 \
  --advertise-client-urls https://10.1.80.91:2379 \
  --listen-peer-urls https://10.1.80.91:2380 \
  --initial-advertise-peer-urls https://10.1.80.91:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/opt/etcd3/etcd --name s2 \
  --data-dir /tmp/etcd/s2 \
  --listen-client-urls https://10.1.80.92:2379 \
  --advertise-client-urls https://10.1.80.92:2379 \
  --listen-peer-urls https://10.1.80.92:2380 \
  --initial-advertise-peer-urls https://10.1.80.92:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/opt/etcd3/etcd --name s3 \
  --data-dir /tmp/etcd/s3 \
  --listen-client-urls https://10.1.80.93:2379 \
  --advertise-client-urls https://10.1.80.93:2379 \
  --listen-peer-urls https://10.1.80.93:2380 \
  --initial-advertise-peer-urls https://10.1.80.93:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem

–initial-cluster-token tkn “tkn”可以替换成你自己需要的值

检查etcd集群状态

1
2
3
4
5
6
ETCDCTL_API=3 /opt/etcd3/etcdctl \
  --endpoints 10.1.80.91:2379,10.1.80.92:2379,10.1.80.93:2379 \
  --cacert ${HOME}/certs/root-ca.pem \
  --cert ${HOME}/certs/etcd.pem \
  --key ${HOME}/certs/etcd-key.pem \
  endpoint health

systemd运行方式(推荐)

s1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# after transferring certs to remote machines
mkdir -p ${HOME}/certs
cp /tmp/certs/* ${HOME}/certs


# make sure etcd process has write access to this directory
# remove this directory if the cluster is new; keep if restarting etcd
# rm -rf /tmp/etcd/s1



# to write service file for etcd
cat > /tmp/s1.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/opt/etcd3/etcd --name s1 \
  --data-dir /tmp/etcd/s1 \
  --listen-client-urls https://10.1.80.91:2379 \
  --advertise-client-urls https://10.1.80.91:2379 \
  --listen-peer-urls https://10.1.80.91:2380 \
  --initial-advertise-peer-urls https://10.1.80.91:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/s1.service /etc/systemd/system/s1.service



# to start service
sudo systemctl daemon-reload
sudo systemctl enable s1.service
sudo systemctl start s1.service

# cat s1.service
sudo systemctl cat s1.service

# to get logs from service
sudo systemctl status s1.service -l --no-pager
sudo journalctl -u s1.service -l --no-pager|less
sudo journalctl -f -u s1.service

# to stop service
sudo systemctl stop s1.service
sudo systemctl disable s1.service
1
# to start service

s2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# after transferring certs to remote machines
mkdir -p ${HOME}/certs
cp /tmp/certs/* ${HOME}/certs


# make sure etcd process has write access to this directory
# remove this directory if the cluster is new; keep if restarting etcd
# rm -rf /tmp/etcd/s2



# to write service file for etcd
cat > /tmp/s2.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/opt/etcd3/etcd --name s2 \
  --data-dir /tmp/etcd/s2 \
  --listen-client-urls https://10.1.80.92:2379 \
  --advertise-client-urls https://10.1.80.92:2379 \
  --listen-peer-urls https://10.1.80.92:2380 \
  --initial-advertise-peer-urls https://10.1.80.92:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/s2.service /etc/systemd/system/s2.service



# to start service
sudo systemctl daemon-reload
sudo systemctl enable s2.service
sudo systemctl start s2.service

# cat s2.service
sudo systemctl cat s2.service

# to get logs from service
sudo systemctl status s2.service -l --no-pager
sudo journalctl -u s2.service -l --no-pager|less
sudo journalctl -f -u s2.service

# to stop service
sudo systemctl stop s2.service
sudo systemctl disable s2.service

s3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# after transferring certs to remote machines
mkdir -p ${HOME}/certs
cp /tmp/certs/* ${HOME}/certs


# make sure etcd process has write access to this directory
# remove this directory if the cluster is new; keep if restarting etcd
# rm -rf /tmp/etcd/s3



# to write service file for etcd
cat > /tmp/s3.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/opt/etcd3/etcd --name s3 \
  --data-dir /tmp/etcd/s3 \
  --listen-client-urls https://10.1.80.93:2379 \
  --advertise-client-urls https://10.1.80.93:2379 \
  --listen-peer-urls https://10.1.80.93:2380 \
  --initial-advertise-peer-urls https://10.1.80.93:2380 \
  --initial-cluster s1=https://10.1.80.91:2380,s2=https://10.1.80.92:2380,s3=https://10.1.80.93:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file ${HOME}/certs/root-ca.pem \
  --cert-file ${HOME}/certs/etcd.pem \
  --key-file ${HOME}/certs/etcd-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file ${HOME}/certs/root-ca.pem \
  --peer-cert-file ${HOME}/certs/etcd.pem \
  --peer-key-file ${HOME}/certs/etcd-key.pem

[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/s3.service /etc/systemd/system/s3.service



# to start service
sudo systemctl daemon-reload
sudo systemctl enable s3.service
sudo systemctl start s3.service

# cat s3.service
sudo systemctl cat s3.service

# to get logs from service
sudo systemctl status s3.service -l --no-pager
sudo journalctl -u s3.service -l --no-pager|less
sudo journalctl -f -u s3.service

# to stop service
sudo systemctl stop s3.service
sudo systemctl disable s3.service

Check status:

1
2
3
4
5
6
ETCDCTL_API=3 /opt/etcd3/etcdctl \
  --endpoints 10.1.80.91:2379,10.1.80.92:2379,10.1.80.93:2379 \
  --cacert ${HOME}/certs/root-ca.pem \
  --cert ${HOME}/certs/etcd.pem \
  --key ${HOME}/certs/etcd-key.pem \
  endpoint health

注意切换到指定用户

若结果如下，表示集群状态正常

1
2
3
10.1.80.92:2379 is healthy: successfully committed proposal: took = 9.184858ms
10.1.80.93:2379 is healthy: successfully committed proposal: took = 8.681243ms
10.1.80.91:2379 is healthy: successfully committed proposal: took = 11.855811ms

参考

etcd Labs

cfssl ca证书有效期修改

分享到： 微信 新浪微博 Twitter Facebook 复制网址 邮件 收藏夹 一键分享 更多