【ocfs2】集群环境部署

一 安装虚拟机

采用vmware最新版workstation

安装3台基于centos8.1的操作系统

虚拟机配置

磁盘  30G

CPU  2个

内存  2G

网络   采用桥接方式   自动分配ip

主机            ip

Ocfs2-1     192.168.199.131/24

Ocfs2-2     192.168.199.243/24

Ocfs2-3     192.168.199.176/24

 

二 安装ocfs2驱动

采用升级带有ocfs2驱动内核的方法，安装驱动；

具体  在文章https://blog.csdn.net/weixin_40747106/article/details/104873093 中 准备好的内核rpm包

rpm -Uvh kernel-5.4.0-1.x86_64.rpm

设置grub 默认启动的内核

grub2-set-default 0

重启操作系统

 

三 安装用户态工具及配置相关文件

 

1  准备集群配置文件

空节点配置文件 cluster.conf  拷贝至/etc/ocfs2/cluster.conf 

cluster.conf 

cluster:
	heartbeat_mode = local
	node_count = 0
	name = defaultpool

2 安装用户态工具包ocfs2-tool

安装文章https://blog.csdn.net/weixin_40747106/article/details/104880783中准备好的rpm包

rpm -ivh rpm -ivh ocfs2-tools-1.8.6-1.x86_64.rpm

过程中需要有依赖包 libaio-devel 安装下就可以。

 

3 准备iscsi存储

需要准备一个iscsi服务端 提供iscsi存储；

采用模拟的方法

创建一个虚拟机iscsi-work   安装centos7 操作系统；

在iscsi-work 上安装scsi-target-utils

因为centos7. 8 没有现成的scsi-target-utils rpm包需要编译；

以centos7 为例  下载源码包

http://rpm.pbone.net/index.php3/stat/26/dist/93/size/319390/name/scsi-target-utils-1.0.55-4.el7.src.rpm

下载后编译得到scsi-target-utils rpm

编译过程中需要一个perl包  perl-Config-General-2.51-2.el7.noarch.rpm  网上找下，能找到；

编译src包能得到

scsi-target-utils-1.0.55-4.el7.x86_64.rpm            scsi-target-utils-gluster-1.0.55-4.el7.x86_64.rpm
scsi-target-utils-debuginfo-1.0.55-4.el7.x86_64.rpm

安装第一个即可

iscsi 配置参考

https://www.linuxidc.com/Linux/2017-05/143877.htm

我的配置

[root@dev-159 x86_64]# cat /etc/tgt/targets.conf 
# This is a sample config file for tgt-admin.
#
# The "#" symbol disables the processing of a line.

# Set the driver. If not specified, defaults to "iscsi".
default-driver iscsi

# Set iSNS parameters, if needed
#iSNSServerIP 192.168.111.222
#iSNSServerPort 3205
#iSNSAccessControl On
#iSNS On

# Continue if tgtadm exits with non-zero code (equivalent of
# --ignore-errors command line option)
#ignore-errors yes


<target iqn.2020-2.com.gs.wwwww:iscsi_test>
	backing-store /dev/sdb
	initiator-address ALL
</target>
[root@dev-159 x86_64]# 

配置所有ip可以访问

如果有类似Smatx 这种存储直接配置iscsi服务端也可以。

 

4 配置节点集群

前提条件：

3台主机 升级完内核，安装好ocfs2-tools工具

关闭3台主机 防火墙  systemctl disable firewalld.service

设置selinux  为disabled

启动o2cb, ocfs2 服务 

A  配置cluster.conf 文件

每台主机 添加节点

o2cb add-node --ip 192.168.199.176 --port 7777 --number 1 defaultpool dev-176

o2cb add-node --ip 192.168.199.243 --port 7777 --number 2 defaultpool dev-243

o2cb add-node --ip 192.168.199.131 --port 7777 --number 3 defaultpool dev-131

执行完生成的cluster.conf 文件

[root@dev-131 ~]# cat /etc/ocfs2/cluster.conf 
cluster:
	heartbeat_mode = local
	node_count = 3
	name = defaultpool

node:
	number = 1
	cluster = defaultpool
	ip_port = 7777
	ip_address = 192.168.199.176
	name = dev-176

node:
	number = 2
	cluster = defaultpool
	ip_port = 7777
	ip_address = 192.168.199.243
	name = dev-243

node:
	number = 3
	cluster = defaultpool
	ip_port = 7777
	ip_address = 192.168.199.131
	name = dev-131

[root@dev-131 ~]# 

也可以将生成的cluster.conf 文件拷贝到另外2台主机

B  配置o2cb

o2cb 服务有自己的配置文件  o2cb,  在ocfs2-tools rpm包中有自带的o2cb 配置文件，拷贝到路径/etc/sysconfig/o2cb路径下

参考o2cb 配置如下

[root@dev-131 ~]# cat o2cb 
#
# This is a configuration file for automatic startup of the O2CB
# driver.  It is generated by running /etc/init.d/o2cb configure.
# On Debian based systems the preferred method is running
# 'dpkg-reconfigure ocfs2-tools'.
#

# O2CB_ENABLED: 'true' means to load the driver on boot.
O2CB_ENABLED=true

# O2CB_STACK: The name of the cluster stack backing O2CB.
O2CB_STACK=o2cb

# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=defaultpool

# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
O2CB_HEARTBEAT_THRESHOLD=61

# O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is considered dead.
O2CB_IDLE_TIMEOUT_MS=90000

# O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive packet is sent
O2CB_KEEPALIVE_DELAY_MS=10000

# O2CB_RECONNECT_DELAY_MS: Min time in ms between connection attempts
O2CB_RECONNECT_DELAY_MS=2000

[root@dev-131 ~]# 

相关参数说明  参考  man   o2cb.sysconfig

/etc/sysconfig/o2cb(5)                     OCFS2 Manual Pages                    /etc/sysconfig/o2cb(5)

NAME
       /etc/sysconfig/o2cb - Cluster configuration file for the o2cb cluster stack.

SYNOPSIS
       The  configuration  file  /etc/sysconfig/o2cb  stores the active cluster stack, its name and the
       various cluster timeouts for the o2cb cluster stack.

DESCRIPTION
       This file can be populated using the o2cb init script. An example of the same is illustrated  in
       the examples section.

       The list of configurable parameters in this file are:

       O2CB_STACK
              Name  of  the  cluster  stack.  The possible values are o2cb, pcmk and cman.  o2cb is the
              default cluster stack of the OCFS2 file system. pcmk (Pacemaker) and cman (rgmanager) are
              the two other cluster stacks that are supported by the same file system.

       O2CB_BOOTCLUSTER
              Name of the active cluster. While /etc/ocfs2/cluster.conf can hold descriptions of multi‐
              ple clusters, only one can be active at any one time. The name of that active cluster  is
              specified  here. The name itself can be up to 16 alphanumeric characters [0-9A-Za-z] with
              no special characters.

       The remaining configurable parameters (cluster timeouts) are only relevant for the o2cb  cluster
       stack.  These cluster timeouts are used by the o2cb cluster stack to determine whether a node is
       dead or alive. The default timeouts are just a guide and may need to be tweaked depending on the
       hardware the software is running on.

       The various cluster timeouts for the o2cb cluster stack are:

       O2CB_HEARTBEAT_THRESHOLD
              The  disk  heartbeat timeout is th

man 手册里面解释的很清楚；

 5 格式化磁盘

将iscsi共享过来的磁盘格式化成ocfs2 文件系统， 这里由于实验，没有配置多路径。

用iscsiadm  相关命令发现， 登录设备；

iscsiadm  -m session  

[root@dev-131 ~]# iscsiadm  -m session 
tcp: [1] 192.168.199.159:3260,1 iqn.2020-2.com.gs.wwwww:iscsi_test (non-flash)

发现131 已经有iscsi session

blkid 查看所有文件系统

[root@dev-131 ~]# blkid 
/dev/sr0: UUID="2020-01-03-21-42-40-00" LABEL="CentOS-8-1-1911-x86_64-dvd" TYPE="iso9660" PTUUID="087a73cc" PTTYPE="dos"
/dev/sda1: UUID="0ddeeea6-97f0-4958-a1db-e64e4e7cc8ff" TYPE="ext4" PARTUUID="505dc1d6-01"
/dev/sda2: UUID="b1f20fdf-cb23-4bef-a1e4-634c92a78e19" TYPE="swap" PARTUUID="505dc1d6-02"
/dev/sda3: UUID="b789a557-678b-4efb-a7c3-f50b535d9829" TYPE="xfs" PARTUUID="505dc1d6-03"
/dev/sdb: LABEL="test" UUID="ae3bd57a-78d5-4212-acb6-b8b7334c5e31" TYPE="ocfs2"

/dev/sdb  已经是ocfs2文件系统；

我们可以不格；

格式化命令 

mkfs.ocfs2 -b 4k -C 1M -L test  -N 3 -T  vmstore -J block64 -F -x /dev/sdb

相关参数  据磁盘大小 和节点个数   -N 选择3 -C 选择1 M  -b 最好是4k    -T  vmstore（用于存放磁盘文件

命令详解介绍    man  mkfs.ocfs2

6 挂载

每个节点使用命令mount.ocfs2   

mount.ocfs2 -o coherency=buffered  /dev/sdb  /mnt/ocfs2

 

2 使用

在节点131 上 使用ocfs2， 在另外2个节点能看到内容变化

131 

[root@dev-131 ocfs2]# cat t.txt 
asdflkajdfka
[root@dev-131 ocfs2]# 

243

[root@dev-243 ocfs2]# cat t.txt 
asdflkajdfka
[root@dev-243 ocfs2]# 

176 

[root@dev-176 ocfs2]# cat t.txt 
asdflkajdfka
[root@dev-176 ocfs2]# 

 

 3个节点上的数据一致。