部署swift

黄俊智

2023-12-01

1.概念

对象存储包含以下组件

a. 对象服务：管理实际存储的数据

b. 容器服务：管理容器

c. 账号服务：管理用户账号信息

c. 代理服务：对外提供访问接口，可以有多个。

d. 其它服务：必要的后台处理程序，例如副本维护，状态更新等。

部署方式

一般分两种服务器：

a. 存储服务器：运行前两种服务，同时还运行sqlite和rsync

b.代理服务器：运行代理服务

存储网络分为三种

a. 对外网络：即代理服务器的对外接口网络，外部主机可以通过该接口访问swift存储服务。

b. 存储网络：用于集群内部通讯。

c. 复制网络：专用于数据复制的通讯。

复制网络不是必须的。

如果有复制网络，则对象服务和容器/账号服务除监听STORAGE_LOCAL_NET之外，还得监听STORAGE_REPLICATION_NET，但是rsync服务只需监听STORAGE_REPLICATION_NET即可

zone

一个zone包含一个或多个存储设备，其宕机不会影响集群运行。

基本安装方式中可以把一台主机作为一个zone，建议一个集群至少包含5个zone。

ring

实现对象存储数据到实际存储设备之间的映射。

3.安装MySQL

3.1在控制节点安装MySQL服务器

yum install mysql mysql-server MySQL-python

vim /etc/my.cnf

[mysqld]
bind-address=xxxx
default-storage-engine = innodb
innodb_file_per_table
collation-server = utf8_general_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8

# service mysqld start

# chkconfig mysqld on

# mysql_install_db
# mysql_secure_installation

3.1在集群其它节点安装MySQL客户端

yum install MySQL-python

4.在集群所有节点安装openstack的安装源

yum install yum-plugin-priorities

yum install -y http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-3.noarch.rpm

yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum install -y openstack-utils openstack-selinux

yum upgrade

reboot

4.在控制节点安装消息队列

yum install qpid-cpp-server

vim /etc/qpidd.conf

</pre><p>[mysqld]</p><p><pre name="code" class="plain">auth=no

service qpidd start

chk config qpidd on

5.部署keystone

由于认证需要，部署swift之前必须先部署keystone。

5.1安装keystone

5.1.1在控制节点安装keystone service和client

yum install openstack-keystone python-keystoneclient

5.2配置keystone服务使用的数据库

# openstack-config --set /etc/keystone/keystone.conf database connection mysql://keystone:KEYSTONE_DBPASS@controller/keystone

KEYSTONE_DBPASS修改为想要的密码，controller修改为数据库所在主机的地址。

5.3在MySQL数据库中创建keystone数据库

用户名设置为keystone，设置keystone用户对其有完全访问权限。密码设置为和前面的KEYSTONE_DBPASS一样。

$ mysql -u root -p
mysql> CREATE DATABASE keystone;
mysql> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \
IDENTIFIED BY 'KEYSTONE_DBPASS';
mysql> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \
IDENTIFIED BY 'KEYSTONE_DBPASS';
mysql> exit

5.4创建数据库表

su -s /bin/sh -c "keystone-manage db_sync" keystone

5.5定义一个认证token，所有openstack的其他服务和keystone服务之间都使用这个token

# ADMIN_TOKEN=$(openssl rand -hex 10)
# echo $ADMIN_TOKEN
# openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $ADMIN_TOKEN

5.6设置PKI token

# keystone-manage pki_setup --keystone-user keystone --keystone-group
keystone
# chown -R keystone:keystone /etc/keystone/ssl
# chmod -R o-rwx /etc/keystone/ssl

5.7启动keystone服务

# service openstack-keystone start
# chkconfig openstack-keystone on

5.8定期将过期的token删除

# (crontab -l -u keystone 2>&1 | grep -q token_flush) || echo '@hourly /usr/bin/keystone-manage token_flush >/var/log/keystone/keystone-tokenflush.log 2>&1' >> /var/spool/cron/keystone

5.2定义用户，租户和角色

定义用户，租户和角色的目的是为了使服务和端点的访问变得有效。

5.2.1设置环境变量

$ export OS_SERVICE_TOKEN=ADMIN_TOKEN # ADMIN_TOKEN见上文

$ export OS_SERVICE_ENDPOINT=http://controller:35357/v2.0# controller即控制节点的ip

5.2.2创建管理员

创建管理员用户：keystone user-create --name=admin --pass=ADMIN_PASS --email=ADMIN_EMAIL

创建管理员角色：keystone role-create --name=admin

创建管理员租户：keystone tenant-create --name=admin --description="Admin Tenant"

把以上三者联系起来：keystone user-role-add --user=admin --tenant=admin --role=admin

把admin和_member_角色联系起来：keystone user-role-add --user=admin --role=_member_ --tenant=admin

5.2.3为日常维护创建用户

keystone user-create --name=demo --pass=DEMO_PASS --email=DEMO_EMAIL

keystone tenant-create --name=demo --description="Demo Tenant"

keystone user-role-add --user=demo --role=_member_ --tenant=demo

5.2.4为所有的服务建立service租户

keystone tenant-create --name=service --description="Service Tenant"

5.3定义服务和API端点

为了跟踪那些服务和API端点已经安装，首先需要注册这些服务。

5.3.1 注册keystone服务：

keystone service-create --name=keystone --type=identity --description="OpenStack Identity"

keystone endpoint-create \
--service-id=$(keystone service-list | awk '/ identity / {print $2}') \
--publicurl=http://controller:5000/v2.0 \
--internalurl=http://controller:5000/v2.0 \
--adminurl=http://controller:35357/v2.0

5.4验证keystone安装

5.4.1取消前面的环境变量设置

unset OS_SERVICE_TOKEN OS_SERVICE_ENDPOINT

5.4.2执行以下命令，观察输出是否正常

$ keystone --os-username=admin --os-password=ADMIN_PASS \
--os-tenant-name=admin --os-auth-url=http://controller:35357/v2.0 \
token-get

5.5在集群所有主机上安装keystone客户端

5.5.1安装客户端

yum install python-keystoneclient

5.5.2设置环境变量

vim 创建admin-openrc.sh

export OS_USERNAME=admin
export OS_PASSWORD=ADMIN_PASS
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://controller:35357/v2.0

cp admin-openrc.sh /etc/profile.d/

source admin-openrc.sh

5.5.3测试

keystone token-get

keystone user-list

6.安装swift

6.1基本步骤

6.1.1.创建swift用户，该用户属于service租户，admin组

$ keystone user-create --name=swift --pass=SWIFT_PASS --email=swift@example.com
$ keystone user-role-add --user=swift --tenant=service --role=admin

6.1.2创建swift服务

$ keystone service-create --name=swift --type=object-store --description="OpenStack Object Storage"

6.1.3创建swift的访问端点

keystone endpoint-create \
--service-id=$(keystone service-list | awk '/ object-store / {print $2}') \
--publicurl='http://controller:8080/v1/AUTH_%(tenant_id)s' \
--internalurl='http://controller:8080/v1/AUTH_%(tenant_id)s' \
--adminurl=http://controller:8080

6.1.4在集群的所有主机上创建swift用户，配置目录和文件

# adduser swift

# mkdir -p /etc/swift

# vim /etc/swift/swift.conf

[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_prefix = xrfuniounenqjnw
swift_hash_path_suffix = fLIbertYgibbitZ

6.2安装和配置存储节点

安装swift的相关服务

yum install openstack-swift-account openstack-swift-container openstack-swift-object xfsprogs xinetd

准备一个独立的存储块设备（设为/dev/sdb），将其格式化为xfs文件系统。

# yum install -y xfsprogs

#################################################

#如果是独立存储设备

# fdisk /dev/sdb

# mkfs.xfs /dev/sdb1
# echo "/dev/sdb1 /srv/node/sdb1 xfs noatime,nodiratime,nobarrier,logbufs= 8 0 0" >> /etc/fstab
# mkdir -p /srv/node/sdb1
# mount /srv/node/sdb1
# chown -R swift:swift /srv/node

#################################################

#如果是模拟文件系统

fallocate -l 200G /data/swift-data/xfs.disk
losetup -d /dev/loop0
losetup /dev/loop0 /data/swift-data/xfs.disk
mkfs.xfs /dev/loop0
mkdir -p /srv/node/loop0
mount /dev/loop0 /srv/node/loop0
chown -R swift:swift /srv/node

vim /etc/rsyncd.conf

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = STORAGE_LOCAL_NET_IP<span style="white-space:pre">	</span><span style="color:#ff0000;">#如果有独立的副本同步网络，则替换为STORAGE_REPLICATION_NET_IP</span>
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
<h3>lock file = /var/lock/container.lock</h3>[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

vim /etc/xinetd.d/rsync

disable = no

# service xinetd start
# mkdir -p /var/swift/recon
# chown -R swift:swift /var/swift/recon

6.3安装和配置代理节点

6.3.1安装软件

yum install openstack-swift-proxy memcached python-swiftclient pythonkeystone-auth-token

6.3.2配置memcached

vim /etc/sysconfig/memcached

OPTIONS="-l PROXY_LOCAL_NET_IP"<span style="white-space:pre">	</span>#本机的私网地址

6.3.3配置proxy

vim /etc/swift/proxy-server.conf

[DEFAULT]
bind_port = 8080
user = swift
[pipeline:main]
pipeline = healthcheck cache authtoken keystoneauth proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = Member,admin,swiftoperator
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers ('.r:*').
delay_auth_decision = true
# auth_* settings refer to the Keystone server
auth_protocol = http
auth_host = controller
auth_port = 35357
# the service tenant and swift username and password created in Keystone
admin_tenant_name = service
admin_user = swift
admin_password = SWIFT_PASS
[filter:cache]
use = egg:swift#memcache
[filter:catch_errors]
use = egg:swift#catch_errors
[filter:healthcheck]
use = egg:swift#healthcheck

6.3.4创建账号，容器和对象的ring

# cd /etc/swift
# swift-ring-builder account.builder create 18 3 1
# swift-ring-builder container.builder create 18 3 1
# swift-ring-builder object.builder create 18 3 1

6.3.5为存储节点上的每个存储设备加入ring的入口点

# swift-ring-builder account.builder add zZONE-STORAGE_LOCAL_NET_IP:6002[RSTORAGE_REPLICATION_NET_IP:6005]/DEVICE 100
# swift-ring-builder container.builder add zZONE-STORAGE_LOCAL_NET_IP_1:6001[RSTORAGE_REPLICATION_NET_IP:6004]/DEVICE 100
# swift-ring-builder object.builder add zZONE-STORAGE_LOCAL_NET_IP_1:6000[RSTORAGE_REPLICATION_NET_IP:6003]/DEVICE 100

如果没有专用的复制网络，那么无需填写 STORAGE_REPLICATION_NET_IP:6003。

假设一个存储节点的地址是10.0.0.1，存储网络地址是10.0.1.1，存储设备挂载到/srv/node/sdb1，/etc/rsyncd.conf文件中配置的path是/srv/node/，那么设备就是sdb1，命令如下：

# swift-ring-builder account.builder add z1-10.0.0.1:6002R10.0.1.1:6005 sdb1 100
# swift-ring-builder container.builder add z1-10.0.0.1:6001R10.0.1.1:6004 sdb1 100
# swift-ring-builder object.builder add z1-10.0.0.1:6000R10.0.1.1:6003 sdb1 100

假设5台主机分为5个zone，那么zone依次加1

6.3.6验证每个ring的内容

# swift-ring-builder account.builder
# swift-ring-builder container.builder
# swift-ring-builder object.builder

6.3.7重新平衡ring

# swift-ring-builder account.builder rebalance

# swift-ring-builder container.builder rebalance
# swift-ring-builder object.builder rebalance

6.3.8复制以下文件到每一个proxy和存储节点

/etc/swift/account.ring.gz

/etc/swift/container.ring.gz

/etc/swift/object.ring.gz

6.3.9确保swift用户对这些文件的所有权

chown -R swift:swift /etc/swift

6.3.10启动代理服务

# service openstack-swift-proxy start
# chkconfig openstack-swift-proxy on

6.4在所有的存储节点上启动以下服务

# for service in \
openstack-swift-object openstack-swift-object-replicator openstack-swiftobject-updater openstack-swift-object-auditor \
openstack-swift-container openstack-swift-container-replicator openstackswift-container-updater openstack-swift-container-auditor \
openstack-swift-account openstack-swift-account-replicator openstack-swiftaccount-reaper openstack-swift-account-auditor; do \
service $service start; chkconfig $service on; done

或者

# swift-init all start

6.5.验证安装

从代理服务器运行以下命令

$ source admin-openrc.sh

$ swift stat

上传文件：

$ swift upload myfiles test.txt # myfiles是容器
$ swift upload myfiles test2.txt

下载myfiles容器内的所有文件

swift download myfiles

7.加入新的proxy

1.以相同的方式加入代理服务器

2.为了使访问统一，需要增加一台反向代理服务器

3.更新memcached服务器列表：