集群服务原理图

整体原理图

主节点(Primary)搭建在物理主机

副本集原理图

集群规划

一个Primary节点，两个Secondary节点

ip port description
192.168.1.167 27017 primary, 搭建在物理主机
172.17.0.5 27017 secondary, mongod镜像
172.17.0.6 27017 secondary, mongod镜像

配置Primary节点服务

MongoDB Offical Doc——Internal Authentication_Enforce Keyfile Access Control in a Replica Set
MongoDB Offical Doc——Internal Authentication_Deploy Replica Set With Keyfile Access Control

mongodkey 文件

用openssl生成验证文件keyFile

1
2

openssl rand -base64 756 > mongodkey
chmod 400 mongodkey

Note:

  • 集群成员间通过keyfile作安全验证
  • 客户端和集群服务器间通过账号作安全验证
  • 生产环境应该采用：x.509 Internal Authentication
  • keyFile文件格式：
    • mongodkey可成功开启服务
    • mongod.key开启服务失败(具体原因还未验证,猜测文件格式有影响)
  • MongoDB角色权限问题

mongod.conf文件

mongod服务启动参数

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#
# Program: mongod.conf 
# History: zhezhang	2017/05/25
#

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /home/mongo/log/mongod.log

# where and how to store data.
storage:
  dbPath: /home/mongo/data/db
  journal:
    enabled: true

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile

# network interfaces
net:
  port: 27017
  bindIp: 192.168.1.169  # Listen to local interface only, comment to listen on all interfaces.

# authorization
security:
  #authorization: enabled
  #authorization: disabled
  keyFile: /home/mongo/mongodkey

# replica set
replication:
  replSetName: rs

Note

  • 在Replica Set集群未初始化前先不开启MongoDB安全验证，即 security.authorization=disabled
  • 在Replica Set集群搭建，集群成员间需要通过 keyFile 进行验证-Reference (接下来的从节点配置也需要keyFile)

mongod.service文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=root
Group=root
Environment="OPTIONS=--quiet -f /home/mongo/mongod.conf"
ExecStart=/usr/bin/mongod $OPTIONS run
ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb
ExecStartPre=/usr/bin/chown root:root /var/run/mongodb
ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb
PermissionsStartOnly=true
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target

运行·停止·状态

1
2
3

systemctl start mongod.service
systemctl stop mongod.service
systemctl status mongod.service

配置Secondary节点服务

Secondary 节点以Docker Container方式运行

Dockerfile文件

编写构建MongoDB容器的Dockerfile文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

#
# Program: Dockerfile for MongoDB ReplicaSet
# 
# History: zhezhang 2017/06/21
#

# Build Images:
# docker build -t mgo:rs .
#
# Create Container:
# docker run \ 
# --name container_name \
# -v /home/docker/mongo/conf:/home/mongo/conf \ 
# -v /home/docker/mongo/log:/home/mongo/log \
# -d mgo:rs
# 
# Start:
# docker start container_name ... 
#
# Stop:
# docker stop container_name ...
# 
# Delete:
# docker rm -v container_name ...
# 
# Get Container Ip:
# docker-ip container_name ...
# 
# Connect bash
# docker exec -it container_name /bin/bash
#

FROM centos:centos7

MAINTAINER <boyzhz@163.com>  

ADD ./mongodb-org-3.4.repo /etc/yum.repos.d/

RUN yum -y update && \
    yum -y install mongodb-org-server && \
    yum clean all && \
    mkdir -p /home/mongo/data/db && \
    mkdir -p /home/mongo/log/ && \
    mkdir -p /home/mongo/conf && \
    touch /home/mongo/log/mongod.log && chmod 666 /home/mongo/log/mongod.log && \
    mkdir -p /home/mongo/pid/ 

COPY mongod.conf mongodkey run.sh /home/mongo/conf/

EXPOSE 27017
                
CMD ["/home/mongo/conf/run.sh"]

# ENTRYPOINT ["/usr/bin/mongod"]
# CMD ["-f", "/home/mongo/mongod.conf"]

MongoDB镜像源：

1
2
3
4
5
6

[mongodb-org-3.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc

Note：

mongod.conf文件

mongod服务启动参数

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#
# Program: mongod.conf 
# History: zhezhang	2017/05/25
#

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /home/mongo/log/mongod.log

# Where and how to store data.
storage:
  dbPath: /home/mongo/data/db
  journal:
    enabled: true

# how the process runs
#processManagement:
#  fork: true  # fork and run in background
#  pidFilePath: /home/mongo/pid/mongod.pid  # location of pidfile

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.

# authorization
security:
  #authorization: enabled
  #keyFile: /home/mongo/conf/mongodkey
  authorization: disabled

# replica set
replication:
  replSetName: rs

注：

  • keyFile 用于ReplicaSet集群内部节点之间的验证，所有节点服务都要存储一份该文件
  • keyFile 选项默认开启了 authorization=enabled, 详细说明在接下来的安全验证部分

run.sh 文件

run.sh 用于容器启动时开启mongod服务

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#!/bin/sh
#
# Program: start api service
#
# History: zhezhang	2017/06/07
# 

echo -e "mongod service is starting..."
# mongod config

WorkDir=/home/mongo
Config=${WorkDir}/conf/mongod.conf
Options="--quiet --config ${Config}"
CMD=/usr/bin/mongod

${CMD} ${Options}

构建MongoDB镜像

  • 构建mongo基镜像：mgors:base

    docker build -t mgors:base .

Note: Secondary节点以mgors:base基础进行构建

运行·停止·删除

  • 启动容器

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    docker run \ 
  --name rs1 \
  --network docker1 \
  --ip 172.18.0.167 \
  --restart always \
  -v /home/docker/mongo/conf:/home/mongo/conf \ 
  -v /home/docker/mongo/log:/home/mongo/log \
  -d mgors:base

docker run \ 
  --name rs2 \
  --network docker1 \
  --ip 172.18.0.168 \
  --restart always \
  -v /home/docker/mongo/conf:/home/mongo/conf \ 
  -v /home/docker/mongo/log:/home/mongo/log \
  -d mgors:base

    Note: -v 将本地主机目录(文件)挂载到容器中，修改总是时时更新的

  • 启动|停止|删除

    1
2
3

    docker start rs1 rs2 
docker stop rs1 rs2 
docker rm -v rs1 rs2

    -v 指定在删除容器的同时删除数据卷

配置Replica Set副本集

连接到Primary节点服务

Replica Set 未配置完成，直接登陆即可

1

mongo 192.168.1.169：27017

配置初始化参数

首先切换到 admin 数据库：use admin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

cfg = {
  "_id": "rs",
  "members": [
    {
      "_id": 0,
      "host": "192.168.1.169:27017"
    },
    {
      "_id": 1,
      "host": "172.17.0.5:27017"
    },
    {
      "_id": 2,
      "host": "172.17.0.6:27017"
    }
  ]
}

执行初始化

1
2
3

rs.initiate(cfg)

# return { "ok" : 1 }, represent success.

查看初始化状态及配置

1
2

rs.status()
rs.config()

创建数据库账号

首先切换到 primary 节点，然后切换到admin数据库，再创建账号

创建角色为 root 的超级管理员账号

1
2
3
4
5
6
7
8
9

use admin

db.createUser({
    user: "root",
    pwd: "root09",
    customData:{description:"超级管理员"},
    roles: [{ role: "root", db: "admin" }]
})
db.auth("root", "root09")

创建角色为 userAdminAnyDatabase 的管理员账号

1
2
3
4
5
6
7
8
9

use admin

db.createUser({
    user: "admin",
    pwd: "admin09",
    customData:{description:"管理员"},
    roles: [{ role: "userAdminAnyDatabase", db: "admin" }]
})
db.auth("admin", "admin09")

创建应用数据库账号

1
2
3
4
5
6
7
8

use cloud-v0

db.createUser({
    user: "hzqx",
    pwd: "hzqx09",
    roles: [ { role: "readWrite", db: "cloud-v0" } ]
})
db.auth("hzqx", "hzqx09")

连接集群节点&查看状态

连接到primary节点(账号+密码)

1
2
3

mongo 192.168.1.169:27017 -u root -p root09

rs.status() // 执行该命令即可查看集群状态

连接到secondary节点(账号+密码)

1
2
3
4

mongo 172.17.0.5:27017 -u root -p root09
mongo 172.17.0.6:27017 -u root -p root09

rs.status() // 执行该命令即可查看集群状态

Over here，Deploy a MongoDB Replica Set with Docker and Keyfile Access Control Finished.

FAQ

  • Dockefile配置错误

    • RUN 指令下的多条子命令要用 && 连接，但是最后一条指令一定不能添加 && 连接符，否则会导致COPY作为RUN的子命令，进而导致错误：/bin/sh: COPY: command not found

  • 基于镜像启动mongod服务时：mongod.conf中ip写死导致只有一个mongod服务启动，

    • docker run –name mgors1 mongod:rs ——> Exited (48) 17 seconds ago

  • MongodDB角色权限问题

    • root： 登陆验证后执行rs.status()…等等命令成功

    • admin：登陆验证后执行rs.status()…等等相关命令失败

      1
2
3
4
5
6
7

      rs:OTHER> rs.initiate(cfg)
{
    "ok" : 0,
    "errmsg" : "not authorized on admin to execute command { replSetInitiate: { _id: \"rs\", members: [ { _id: 0.0, host: \"172.17.0.5:27017\" }, { _id: 1.0, host: \"172.17.0.6:27017\" }, { _id: 2.0, host: \"172.17.0.7:27017\" } ] } }",
    "code" : 13,
    "codeName" : "Unauthorized"
}

  • 创建账号必须在primary节点，其他节点报错

    • Error: couldn’t add user: not master :

将来优化方向

See Also

Thanks to the authors 🙂