Openstack swift常见问题
a) 如何查看.data文件的xattr属性
root@KVM-21:/mnt/sdb1/1/node/sdb1/objects/857/483/d666fd8b3ff3177e8aaa3790323ad483# ll
total 8
drwxr-xr-x 2 root root 34 11月 14 08:55 ./
drwxr-xr-x 3 root root 45 11月 14 08:55 ../
-rw------- 1 root root 108 11月 14 08:55 1415926512.64242.data
root@KVM-21:/mnt/sdb1/1/node/sdb1/objects/857/483/d666fd8b3ff3177e8aaa3790323ad483# python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xattr
>>> xattr.listxattr("1415926512.64242.data")
(u'user.swift.metadata',)
>>> xattr.getxattr("1415926512.64242.data","user.swift.metadata")
'\x80\x02}q\x01(U\x0eContent-Lengthq\x02U\x03108U\x04nameq\x03U$/AUTH_test/mycontainers/getAuthTokenq\ x04U\x13X-Object-Meta-MtimeU\x111415861238.176129q\x05U\x04ETagq\x06U bae6b14c0b1e486868f3c40c3b6bf35b q\x07U\x0bX-Timestampq\x08U\x101415926512.64242U\x0cContent-Typeq\tU\x18application/octet-streamq\nu.'
>>> metadata = xattr.getxattr("1415926512.64242.data","user.swift.metadata")
>>> import pickle
{'Content-Length': '108', 'name': '/AUTH_test/mycontainers/getAuthToken', 'Content-Type': 'application/octet-stream', 'ETag': 'bae6b14c0b1e486868f3c40c3b6bf35b', 'X-Timestamp': '1415926512.64242', 'X-Object-Meta-Mtime': '1415861238.176129'}
>>>
b) 如何查看hashes.pkl文件内容
root@sws50:/srv/node/sdc/objects/100000# ls 8bd hashes.pkl
In [1]: with open('hashes.pkl', 'rb') as fp:
...: import pickle
...: hashes = pickle.load(fp)
In [2]: hashes
Out[2]: {'8bd': '9e99c8eedaa3197a63f685dd92a5b4b8'}
hashes.pkl是存放在每个partition中的一个2进制pickle化文件，‘8bd’是suffix_dir，而9e99c8eedaa3197a63f685dd92a5b4b8则是该partition下所有文件名字的md5哈希值。
验证方法如下：
root@VirtualBox:/mnt/sdb1/1/node/sdb1/objects/596# ll
total 4
drwxr-xr-x 3 root root 45 Jan 20 11:37 ./
drwxr-xr-x 8 root root 67 Jan 20 10:33 ../
drwxr-xr-x 3 root root 45 Jan 20 10:33 5e6/
-rw------- 1 root root 48 Jan 20 11:37 hashes.pkl
-rwxr-xr-x 1 root root 0 Jan 20 10:33 .lock*
Python 2.7.3 (default, Feb 27 2014, 20:00:17)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from hashlib import md5
>>> etag = md5()
>>> etag.update("1421721221.89468.data")
>>> etag.hexdigest()
'1effa0b1557c0a5020f4fdcdbfc5d067'
>>> import pickle
>>> with open("/mnt/sdb1/1/node/sdb1/objects/596/hashes.pkl") as fp:
... a = pickle.load(fp)
...
>>> a
{'5e6': '1effa0b1557c0a5020f4fdcdbfc5d067'}
c) Ring文件的操作
root@osd-20:/etc/swift# swift-ring-builder account.builder
account.builder, build version 4
1024 partitions, 3.000000 replicas, 1 regions, 4 zones, 4 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
0 1 1 127.0.0.1 6012 127.0.0.1 6012 sdb1 1.00 768 0.00
1 1 2 127.0.0.1 6022 127.0.0.1 6022 sdb2 1.00 768 0.00
2 1 3 127.0.0.1 6032 127.0.0.1 6032 sdb3 1.00 768 0.00
3 1 4 127.0.0.1 6042 127.0.0.1 6042 sdb4 1.00 768 0.00
root@osd-20:/etc/swift# swift-ring-builder account.builder rebalance
No partitions could be reassigned.
Either none need to be or none can be due to min_part_hours [1].
root@osd-20:/etc/swift# swift-ring-builder account.builder
account.builder, build version 4
1024 partitions, 3.000000 replicas, 1 regions, 4 zones, 4 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
0 1 1 127.0.0.1 6012 127.0.0.1 6012 sdb1 1.00 768 0.00
1 1 2 127.0.0.1 6022 127.0.0.1 6022 sdb2 1.00 768 0.00
2 1 3 127.0.0.1 6032 127.0.0.1 6032 sdb3 1.00 768 0.00
3 1 4 127.0.0.1 6042 127.0.0.1 6042 sdb4 1.00 768 0.00
root@osd-20:/etc/swift# swift-ring-builder account.builder validate
root@osd-20:/etc/swift# swift-ring-builder account.builder search z1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
0 1 1 127.0.0.1 6012 127.0.0.1 6012 sdb1 1.00 768 0.00
root@osd-20:/etc/swift# swift-ring-builder account.builder search 6022
No matching devices found
root@osd-20:/etc/swift# swift-ring-builder account.builder search :6022
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 2 127.0.0.1 6022 127.0.0.1 6022 sdb2 1.00 768 0.00
root@osd-20:/etc/swift# swift-ring-builder account.builder set_weight :6022 100
d1r1z2-127.0.0.1:6022R127.0.0.1:6022/sdb2_"" weight set to 100.0
root@osd-20:/etc/swift# swift-ring-builder account.builder search :6022
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 2 127.0.0.1 6022 127.0.0.1 6022 sdb2 100.00 768 -74.25
root@osd-20:/etc/swift# swift-ring-builder account.builder set_info :6022 10.20.1.216:6022/sdb2_"account"
Device d1r1z2-127.0.0.1:6022R127.0.0.1:6022/sdb2_"" is now d1r1z2-10.20.1.216:6022R127.0.0.1:6022/sdb2_"account"
root@osd-20:/etc/swift# swift-ring-builder account.builder set_info :6022 10.20.1.216:6022/sdb2_"account2"
Device d1r1z2-10.20.1.216:6022R127.0.0.1:6022/sdb2_"account" is now d1r1z2-10.20.1.216:6022R127.0.0.1:6022/sdb2_"account2"
root@osd-20:/etc/swift# swift-ring-builder account.builder search :6022
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 2 10.20.1.216 6022 127.0.0.1 6022 sdb2 100.00 768 -74.25 account2
root@osd-20:/etc/swift# swift-ring-builder account.builder rebalance
Reassigned 1024 (100.00%) partitions. Balance is now 1368.55.
-------------------------------------------------------------------------------
NOTE: Balance of 1368.55 indicates you should push this
ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------
root@osd-20:/etc/swift# swift-ring-builder account.builder search :6022
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
1 1 2 10.20.1.216 6022 127.0.0.1 6022 sdb2 100.00 1792 -39.92 account2
root@osd-20:/etc/swift# swift-ring-builder account.builder
account.builder, build version 6
1024 partitions, 3.000000 replicas, 1 regions, 4 zones, 4 devices, 1368.55 balance
The minimum number of hours before a partition can be reassigned is 1
Devices: id region zone ip address port replication ip replication port name weight partitions balance meta
0 1 1 127.0.0.1 6012 127.0.0.1 6012 sdb1 1.00 418 1301.50
1 1 2 10.20.1.216 6022 127.0.0.1 6022 sdb2 100.00 1792 -39.92 account2
2 1 3 127.0.0.1 6032 127.0.0.1 6032 sdb3 1.00 438 1368.55
3 1 4 127.0.0.1 6042 127.0.0.1 6042 sdb4 1.00 424 1321.61
d) Swift中如何使用curl
curl -k -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://localhost:8080/auth/v1.0
curl -k -v -H 'X-Auth-Token: AUTH_tk1036384315d64640ab97b3294ec6fc1d' http://localhost:8080/v1/AUTH_test
e) Swift中如何使用swift client
swift -U test:tester -K testing -A http://localhost:8080/auth/v1.0 list
swift -U test:tester -K testing -A http://localhost:8080/auth/v1.0 list foobar
swift -U test:tester -K testing -A http://localhost:8080/auth/v1.0 upload foobar test3
swift -U test:tester -K testing -A http://localhost:8080/auth/v1.0 post foobar -r ".r:*,.rlistings"
f) 查看日志
tail -f /var/log/syslog kinetic swift日志放在这个地方，swift的日志在SAIO环境搭建中有说明
g) Hash路径生成原理
Paths={acc, con, obj}
md5(HASH_PATH_PREFIX + '/' + '/'.join(paths)
+ HASH_PATH_SUFFIX).hexdigest()
h) Db文件存储路径
G:\kinetic-swift-master\swift\swift\container\backend.py
DATADIR = 'containers'
def storage_directory(datadir, partition, name_hash):
return os.path.join(datadir, str(partition), name_hash[-3:], name_hash)
drive: drive that holds the container like sda/sdb…
self.root = conf.get('devices', '/srv/node')
hsh = hash_path(account, container)
db_dir = storage_directory(DATADIR, part, hsh)
db_path = os.path.join(self.root, drive, db_dir, hsh + '.db')
i) Async_pending文件存储路径
文件路径的组成如下所示，其中数据名称是由hash_path后面紧跟’-‘，后面是以发送container request的header中包含的时间戳所产生
j) Rsync 备份工具
rsync的命令格式可以为以下六种：
rsync [OPTION]... SRC DEST
rsync [OPTION]... SRC [USER@]HOST:DEST
rsync [OPTION]... [USER@]HOST:SRC DEST
rsync [OPTION]... [USER@]HOST::SRC DEST
rsync [OPTION]... SRC [USER@]HOST::DEST
rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
对应于以上六种命令格式，rsync有六种不同的工作模式：
1)拷贝本地文件。当SRC和DES路径信息都不包含有单个冒号":"分隔符时就启动这种工作模式。
2)使用一个远程shell程序(如rsh、ssh)来实现将本地机器的内容拷贝到远程机器。当DST路径地址包含单个冒号":"分隔符时启动该模式。
3)使用一个远程shell程序(如rsh、ssh)来实现将远程机器的内容拷贝到本地机器。当SRC地址路径包含单个冒号":"分隔符时启动该模式。
4)从远程rsync服务器中拷贝文件到本地机。当SRC路径信息包含"::"分隔符时启动该模式。
5)从本地机器拷贝文件到远程rsync服务器中。当DST路径信息包含"::"分隔符时启动该模式。
6)列远程机的文件列表。这类似于rsync传输，不过只要在命令中省略掉本地机信息即可。
Object path生成过程
object的存储路径由object server进程内部称为DiskFile类初始化时产生，过程如下：
1.由文件所属的account、container和object名称产生'/account/container/object'格式的字符串，和HASH_PATH_SUFFIX组成新的字符串，调用hash_path函数，生成md5 hash值name_hash。其中HASH_PATH_SUFFIX作为salt来增加安全性，HASH_PATH_SUFFIX值存放在/etc/swift/swift.conf中。
2. 调用storage_directory函数，传入DATADIR, partition, hash_path参数生成DATADIR/partition/name_path[-3:]/name_path格式字符串
3. 连结path/devcie/storage_directory(DATADIR, partition,name_ hash)生成数据存储路径datadir
4. 调用normalize_timestamp函数生成“16位.5位”的时间戳+扩展名的格式生成对象名称
例如，某object的存储路径为：/srv/node/sdc/objects/19892/ab1/136d0ab88371e25e16663fbd2ef42ab1/1320050752.09979.data