应用挂载beegfs指定目录_BeeGFS元数据组织方式和路径寻址分析
元数据根节点的确定和获取
管理节点处理元数据节点的心跳信息时，如果发现目前没有Root节点，则会在已经注册的节点中选择ID最小的那个注册为元数据Root节点(这些信息最后都会保存在磁盘上)：
// fhgfs_mgmtd\source\components\HeartbeatManager.cpp
/**
* @param rootIDHint empty string to auto-define root or a nodeID that is assumed to be the root
* @return true if a new root node has been defined
*/
bool HeartbeatManager::initRootNode(NumNodeID rootIDHint, bool rootIsBuddyMirrored)
{
// be careful: this method is also called from other threads
// note: after this method, the root node might still be undefined (this is normal)
bool setRootRes = false;
if( (rootIDHint != 0) || (metaNodes->getSize()) )
{ // check whether root has already been set
if(rootIDHint == 0)
rootIDHint = metaNodes->getLowestNodeID();
// set root to lowest ID (if no other root was set yet)
setRootRes = metaNodes->setRootNodeNumID(rootIDHint, false, rootIsBuddyMirrored);
if(setRootRes)
{ // new root set
log.log(Log_CRITICAL, "New root directory metadata node: " +
Program::getApp()->getMetaNodes()->getNodeIDWithTypeStr(rootIDHint) );
notifyAsyncAddedNode("", rootIDHint, NODETYPE_Meta); /* (real string ID will
be retrieved by notifier before sending the heartbeat) */
}
}
return setRootRes;
}
客户端在处理与管理节点的心跳信息时，会从中获取并设置元数据Root节点信息：
// fhgfs_client_module\source\common\net\message\nodes\HeartbeatMsgEx.c
/**
* Handles the contained root information.
*/
void __HeartbeatMsgEx_processIncomingRoot(HeartbeatMsgEx* this, App* app)
{
Logger* log = App_getLogger(app);
const char* logContext = "Heartbeat incoming (root)";
NodeStoreEx* metaNodes;
bool setRootRes;
NodeOrGroup rootOwner = this->rootIsBuddyMirrored
? NodeOrGroup_fromGroup(this->rootNumID.value)
: NodeOrGroup_fromNode(this->rootNumID);
NumNodeID rootNumID = HeartbeatMsgEx_getRootNumID(this);
// check whether root info is defined
if( (HeartbeatMsgEx_getNodeType(this) != NODETYPE_Meta) || (NumNodeID_isZero(&rootNumID)))
return;
// try to apply the contained root info
metaNodes = App_getMetaNodes(app);
setRootRes = NodeStoreEx_setRootOwner(metaNodes, rootOwner, false);
if(setRootRes)
{ // found the very first root
Logger_logFormatted(log, Log_CRITICAL, logContext, "Root (by Heartbeat): %hu",
HeartbeatMsgEx_getRootNumID(this) );
}
}
元数据根目录的确定和获取
客户端在挂载文件系统，初始化超级块时，会向元数据Root节点获取根目录的DirEntry信息：
// fhgfs_client_module\source\filesystem\FhgfsOpsSuper.c
/**
* Fill the file system superblock (vfs object)
*/
int FhgfsOps_fillSuper(struct super_block* sb, void* rawMountOptions, int silent)
{
...
// init root inode
memset(&kstat, 0, sizeof(struct kstat) );
kstat.ino = BEEGFS_INODE_ROOT_INO;
kstat.mode = S_IFDIR | 0777; // allow access for everyone
kstat.atime = kstat.mtime = kstat.ctime = current_fs_time(sb);
kstat.uid = FhgfsCommon_getCurrentKernelUserID();
kstat.gid = FhgfsCommon_getCurrentKernelGroupID();
kstat.blksize = Config_getTuneInodeBlockSize(cfg);
kstat.nlink = 1;
// root entryInfo is always updated when someone asks for it (so we just set dummy values here)
EntryInfo_init(&entryInfo, NodeOrGroup_fromGroup(0), StringTk_strDup(""), StringTk_strDup(""),
StringTk_strDup(""), DirEntryType_DIRECTORY, 0);
rootInode = __FhgfsOps_newInode(sb, &kstat, 0, &entryInfo, &iSizeHints);
if(!rootInode || IS_ERR(rootInode) )
{
__FhgfsOps_destructFsInfo(sb);
return IS_ERR(rootInode) ? PTR_ERR(rootInode) : -ENOMEM;
}
rootDentry = d_make_root(rootInode);
if(!rootDentry)
{
__FhgfsOps_destructFsInfo(sb);
return -ENOMEM;
}
...
}
// fhgfs_client_module\source\common\toolkit\MetadataTk.c
/**
* @param outEntryInfo contained values will be kalloced (on success) and need to be kfreed with
* FhgfsInode_freeEntryMinInfoVals() later.
*/
bool MetadataTk_getRootEntryInfoCopy(App* app, EntryInfo* outEntryInfo)
{
NodeStoreEx* nodes = App_getMetaNodes(app);
NodeOrGroup rootOwner = NodeStoreEx_getRootOwner(nodes);
const char* parentEntryID = StringTk_strDup("");
const char* entryID = StringTk_strDup(META_ROOTDIR_ID_STR);
const char* dirName = StringTk_strDup("");
DirEntryType entryType = (DirEntryType) DirEntryType_DIRECTORY;
/* Even if rootOwner is invalid, we still init outEntryInfo and malloc as FhGFS
* policy says that kfree(NULL) is not allowed (the kernel allows it). */
EntryInfo_init(outEntryInfo, rootOwner, parentEntryID, entryID, dirName, entryType, 0);
return NodeOrGroup_valid(rootOwner);
}
元数据节点在初始化时，会先后在BuddyMirror和非BuddyMirror目录查找ID名为root的根目录DirEntry文件：
// fhgfs_common\source\common\storage\Metadata.h
#define META_ROOTDIR_ID_STR "root" /* initial file system entry point */
// fhgfs_meta\source\app\App.cpp
void App::initRootDir(NumNodeID localNodeNumID)
{
// try to load root dir from disk (through metaStore) or create a new one
this->metaStore = new MetaStore();
// try to reference root directory with buddy mirroring
rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, true, true);
// if that didn't work try to reference non-buddy-mirrored root dir
if (!rootDir)
{
rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);
}
if(rootDir)
{ // loading succeeded (either with or without mirroring => init rootNodeID
this->log->log(Log_NOTICE, "Root directory loaded.");
NumNodeID rootDirOwner = rootDir->getOwnerNodeID();
bool rootIsBuddyMirrored = rootDir->getIsBuddyMirrored();
// try to set rootDirOwner as root node
if((rootDirOwner != 0) && metaNodes->setRootNodeNumID(rootDirOwner, false,
rootIsBuddyMirrored) )
{ // new root node accepted (check if rootNode is localNode)
NumNodeID primaryRootDirOwner;
if (rootIsBuddyMirrored)
primaryRootDirOwner = NumNodeID(
metaBuddyGroupMapper->getPrimaryTargetID(rootDirOwner.val() ) );
else
primaryRootDirOwner = rootDirOwner;
if(localNodeNumID == primaryRootDirOwner)
{
log->log(Log_CRITICAL, "I got root (by possession of root directory)");
if (rootIsBuddyMirrored)
log->log(Log_CRITICAL, "Root directory is mirrored");
}
else
log->log(Log_CRITICAL,
"Root metadata server (by possession of root directory): " + rootDirOwner.str());
}
}
else
{ // failed to load root directory => create a new rootDir (not mirrored)
this->log->log(Log_CRITICAL,
"This appears to be a new storage directory. Creating a new root dir.");
UInt16Vector stripeTargets;
unsigned defaultChunkSize = this->cfg->getTuneDefaultChunkSize();
unsigned defaultNumStripeTargets = this->cfg->getTuneDefaultNumStripeTargets();
Raid0Pattern stripePattern(defaultChunkSize, stripeTargets, defaultNumStripeTargets);
DirInode newRootDir(META_ROOTDIR_ID_STR,
S_IFDIR | S_IRWXU | S_IRWXG | S_IRWXO, 0, 0, NumNodeID(), stripePattern, false);
this->metaStore->makeDirInode(newRootDir);
this->rootDir = this->metaStore->referenceDir(META_ROOTDIR_ID_STR, false, true);
if(!this->rootDir)
{ // error
this->log->logErr("Failed to store root directory. Unable to proceed.");
throw InvalidConfigException("Failed to store root directory");
}
}
}
元数据ID文件命名规则
每个目录或者文件都会有一个ID文件，而这个ID文件的名字组成结构为--：
// fhgfs_common\source\common\toolkit\StorageTk.h
class StorageTk
{
/**
* Generate ID for new fs entry (i.e. file or dir).
*/
static std::string generateFileID(const NumNodeID localNodeID)
{
/* note: we assume here that the clock doesn't jump backwards between restarts of
the daemon (and that there always is at least one second between restarts) and that we
don't need more than 2^32 IDs per second (sustained) */
uint64_t nextID = idCounter.increase();
// note on idCounter value: high 32bits are timestamp, low 32bits are sequential counter
/* note on switching time/counter in string representation: having the timestamp first is
bad for strcmp() and such things, which the underlying fs might need to do - because in
that order, the first characters of entryIDs/filenames would always be similar. */
uint32_t counterPart = (uint32_t) nextID;
uint32_t timestampPart = (uint32_t) (nextID >> STORAGETK_FILEID_TIMESTAMP_SHIFTBITS);
return StringTk::uintToHexStr(counterPart) + "-" + StringTk::uintToHexStr(timestampPart)
+ "-" + localNodeID.strHex();
}
}
元数据组织结构测试
清空和查看数据目录
$ rm /mnt/beegfs/* -rf
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
创建和查看空文件
$ touch /mnt/beegfs/test001
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
$ ll /beegfs_meta/meta1/dentries/38/51/root/test001 -i
3804241 -rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001
$ ll /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2 -i
3804241 -rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
$ touch /mnt/beegfs/test002
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
创建和查看一级目录
$ mkdir /mnt/beegfs/dir001
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
$ ll -i /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
6293289 -rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
$ ll -i /beegfs_meta/meta1/dentries/38/51/root/dir001
3804243 -rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001
$ getfattr -n user.fhgfs /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
getfattr: Removing leading '/' from absolute path names
# file: beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
user.fhgfs=0sBAMKAAIAAAAAAAAA7UEAANlwwlwAAAAA2XDCXAAAAABGx+RcAAAAAEbH5FwAAAAAAAAAAAAAAAABAAAADAAAADAtNUNDMjcwRDQtMgAAAAAEAAAAcm9vdAAAAAACAAAAAgAAABoAAAABAAAAAAAIAAEABAAAAAgAAAAAAAAA
$ getfattr -n user.fhgfs /beegfs_meta/meta1/dentries/38/51/root/dir001
getfattr: Removing leading '/' from absolute path names
# file: beegfs_meta/meta1/dentries/38/51/root/dir001
user.fhgfs=0sAgMQAAEAAAAMAAAAMC01Q0MyNzBENC0yAAAAAAIAAAA=
创建和查看非空文件
$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=1024
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=4096
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=16384
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}
-rw-rw-rw- 1 root root 5.0M 5月 5 16:44 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 5.5M 5月 5 16:44 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 5.5M 5月 5 16:44 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ dd if=/dev/zero of=/mnt/beegfs/test00 bs=1K count=65536
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ find /beegfs_data/data[123]/{buddymir,chunks} -type f | xargs -i ls -lh {}
-rw-rw-rw- 1 root root 21M 5月 5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22M 5月 5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22M 5月 5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
创建和查看多级目录
$ touch /mnt/beegfs/dir001/test005
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir002
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir001/dir003
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir001/dir003/dir004
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/test006
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ mkdir /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ touch /mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f
/beegfs_meta/meta1/inodes/35/5B/disposal
/beegfs_meta/meta1/inodes/38/51/root
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta2/inodes/35/5B/disposal
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta2/inodes/38/51/root
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/inodes/78/78/0-5CE4C714-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
/beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
/beegfs_meta/meta1/dentries/38/51/root/test002
/beegfs_meta/meta1/dentries/38/51/root/dir002
/beegfs_meta/meta1/dentries/38/51/root/test00
/beegfs_meta/meta1/dentries/38/51/root/dir001
/beegfs_meta/meta1/dentries/38/51/root/test001
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir010
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
/beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
/beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
/beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
$ find /beegfs_meta/meta[12]/{inodes,dentries,buddymir} /beegfs_data/data[123]/{buddymir,chunks} -type f -exec ls -l {} \;
-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/inodes/35/5B/disposal
-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/inodes/38/51/root
-rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
-rw-r--r-- 1 root root 0 5月 14 15:23 /beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
-rw-r--r-- 1 root root 0 5月 14 15:12 /beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/inodes/35/5B/disposal
-rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/inodes/38/51/root
-rw-r--r-- 1 root root 0 5月 14 15:14 /beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
-rw-r--r-- 1 root root 0 5月 14 15:15 /beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
-rw-r--r-- 2 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27098-2
-rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC2705C-2
-rw-r--r-- 2 root root 0 4月 26 10:47 /beegfs_meta/meta1/dentries/38/51/root/#fSiDs#/0-5CC27151-2
-rw-r--r-- 2 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/test002
-rw-r--r-- 1 root root 0 5月 14 15:12 /beegfs_meta/meta1/dentries/38/51/root/dir002
-rw-r--r-- 2 root root 0 4月 26 10:47 /beegfs_meta/meta1/dentries/38/51/root/test00
-rw-r--r-- 1 root root 0 4月 26 10:45 /beegfs_meta/meta1/dentries/38/51/root/dir001
-rw-r--r-- 2 root root 0 4月 26 10:44 /beegfs_meta/meta1/dentries/38/51/root/test001
-rw-r--r-- 2 root root 0 5月 5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/#fSiDs#/0-5CCEA2F5-2
-rw-r--r-- 2 root root 0 5月 5 16:47 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/test005
-rw-r--r-- 1 root root 0 5月 14 15:14 /beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
-rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/#fSiDs#/0-5CDA6D39-2
-rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/dir006
-rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
-rw-r--r-- 1 root root 0 5月 14 15:15 /beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
-rw-r--r-- 1 root root 0 5月 14 15:23 /beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
-rw-r--r-- 1 root root 0 4月 16 17:48 /beegfs_meta/meta1/buddymir/inodes/23/40/mdisposal
-rw-r--r-- 1 root root 0 4月 16 21:10 /beegfs_meta/meta2/buddymir/inodes/23/40/mdisposal
-rw-rw-rw- 1 root root 22020096 5月 5 16:46 /beegfs_data/data1/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22544384 5月 5 16:46 /beegfs_data/data2/chunks/u0/0/r/root/0-5CC27151-2
-rw-rw-rw- 1 root root 22544384 5月 5 16:46 /beegfs_data/data3/chunks/u0/0/r/root/0-5CC27151-2
测试总结分析
对于每个文件，在其dentries父目录的ID目录中有一个同名的DirEntry文件，以及#fSiDs#子目录中的以ID为名称的硬链接；
对于每个目录，在其dentries父目录的ID目录中有一个同名的DirEntry文件，以及inodes目录中的以ID为名称的DirInode文件；
如果一个目录和其父目录不在一个元数据节点，则其DirEntry文件在父目录所在节点，其DirInode文件在其被真正存储的节点；
无论是inodes还是dentry目录，里面都是有两级Hash目录，每一级有128个，根据DirEntry和DirInode的ID进行Hash。而ID文件是根据时间戳生成，所以理论上可以做到很好的均衡。而根目录的ID固定为root，所以总是在38/51这个Hash目录中。
/mnt/beegfs/dir001
/beegfs_meta/meta1/inodes/7B/4D/0-5CC270D4-2
/beegfs_meta/meta1/dentries/38/51/root/dir001
/mnt/beegfs/dir002
/beegfs_meta/meta1/inodes/45/14/0-5CDA6A5D-2
/beegfs_meta/meta1/dentries/38/51/root/dir002
/mnt/beegfs/dir001/dir003
/beegfs_meta/meta2/inodes/67/71/0-5CDA6A9C-2
/beegfs_meta/meta1/dentries/7B/4D/0-5CC270D4-2/dir003
/mnt/beegfs/dir001/dir003/dir004
/beegfs_meta/meta2/inodes/75/19/0-5CDA6B0C-5
/beegfs_meta/meta2/dentries/67/71/0-5CDA6A9C-2/dir004
/mnt/beegfs/dir001/dir003/dir004/dir005
/beegfs_meta/meta1/inodes/2/5C/0-5CDA6CEC-5
/beegfs_meta/meta2/dentries/75/19/0-5CDA6B0C-5/dir005
/mnt/beegfs/dir001/dir003/dir004/dir005/dir006
/beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
/beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
$ ll -i /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
3279957 -rw-r--r-- 1 root root 0 5月 14 15:34 /beegfs_meta/meta2/inodes/35/6F/0-5CDA6F55-2
$ ll -i /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
2362456 -rw-r--r-- 2 root root 0 5月 14 15:25 /beegfs_meta/meta1/dentries/2/5C/0-5CDA6CEC-5/test006
/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/test008
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/#fSiDs#/0-5CEFCC92-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/test008
/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007
/beegfs_meta/meta2/inodes/67/63/0-5CEFCA37-5
/beegfs_meta/meta2/dentries/35/6F/0-5CDA6F55-2/dir007
/mnt/beegfs/dir001/dir003/dir004/dir005/dir006/dir007/test009
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
/beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
1707090 -rw-r--r-- 2 root root 0 5月 30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/#fSiDs#/0-5CEFCD46-5
$ ll -i /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
1707090 -rw-r--r-- 2 root root 0 5月 30 20:32 /beegfs_meta/meta2/dentries/67/63/0-5CEFCA37-5/test009
多元数据节点的分配
在元数据节点每次创建目录时，如果有多个元数据节点，优先选择剩余空间容量多的：
// fhgfs_meta\source\net\message\storage\creating\MkDirMsgEx.cpp
std::unique_ptr<:responsestate> MkDirMsgEx::mkDirPrimary(ResponseContext& ctx)
{
...
metaCapacityPools->chooseStorageTargets(numDesiredTargets, minNumRequiredTargets,
&getPreferredNodes(), &newOwnerNodes);
...
}
/**
* @param numTargets number of desired targets
* @param minNumRequiredTargets the minimum number of targets the caller needs; ususally 1
* for RAID-0 striping and 2 for mirroring (so this parameter is intended to avoid problems when
* there is only 1 target left in the normal pool and the user has mirroring turned on).
* @param preferredTargets may be NULL
*/
void NodeCapacityPools::chooseStorageTargets(unsigned numTargets, unsigned minNumRequiredTargets,
const UInt16List* preferredTargets, UInt16Vector* outTargets)
{
SafeRWLock lock(&rwlock, SafeRWLock_READ);
if(!preferredTargets || preferredTargets->empty() )
{ // no preference => start with first pool that contains any targets
if(pools[CapacityPool_NORMAL].size() )
{
chooseStorageNodesNoPref(pools[CapacityPool_NORMAL],
numTargets, outTargets);
if(outTargets->size() >= minNumRequiredTargets)
goto unlock_and_exit;
}
/* note: no "else if" here, because we want to continue with next pool if we didn't find
enough targets for minNumRequiredTargets in previous pool */
if(pools[CapacityPool_LOW].size() )
{
chooseStorageNodesNoPref(pools[CapacityPool_LOW],
numTargets - outTargets->size(), outTargets);
if(outTargets->size() >= minNumRequiredTargets)
goto unlock_and_exit;
}
chooseStorageNodesNoPref(pools[CapacityPool_EMERGENCY],
numTargets, outTargets);
}
else
{
// caller has preferred targets, so we can't say a priori whether nodes will be found or not
// in a pool. our strategy here is to automatically allow non-preferred targets before
// touching the emergency pool.
std::set chosenTargets;
// try normal and low pool with preferred targets...
chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, false,
outTargets, chosenTargets);
if(outTargets->size() >= minNumRequiredTargets)
goto unlock_and_exit;
chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),
preferredTargets, false, outTargets, chosenTargets);
if(!outTargets->empty() )
goto unlock_and_exit;
/* note: currently, we cannot just continue here with non-empty outTargets (even if
"outTargets->size() < minNumRequiredTargets"), because we would need a mechanism to exclude
the already chosen outTargets for that (e.g. like an inverted preferredTargets list). */
// no targets yet - allow non-preferred targets before using emergency pool...
chooseStorageNodesWithPref(pools[CapacityPool_NORMAL], numTargets, preferredTargets, true,
outTargets, chosenTargets);
if(outTargets->size() >= minNumRequiredTargets)
goto unlock_and_exit;
chooseStorageNodesWithPref(pools[CapacityPool_LOW], numTargets - outTargets->size(),
preferredTargets, true, outTargets, chosenTargets);
if(!outTargets->empty() )
goto unlock_and_exit;
/* still no targets available => we have to try the emergency pool (first with preference,
then without preference) */
chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, false,
outTargets, chosenTargets);
if(!outTargets->empty() )
goto unlock_and_exit;
chooseStorageNodesWithPref(pools[CapacityPool_EMERGENCY], numTargets, preferredTargets, true,
outTargets, chosenTargets);
}
unlock_and_exit:
lock.unlock();
}
相关的配置选项和说明如下：
$ vi /etc/beegfs/beegfs-client.conf
tunePreferredMetaFile =
tunePreferredStorageFile =
# [tunePreferredMetaFile], [tunePreferredStorageFile]
# Path to a text file that contains the numeric IDs of preferred storage targets
# and metadata servers. These will be preferred when the client creates new file
# system entries. This is useful e.g. to take advantage of data locality in the
# case of multiple data centers. If unspecified, all available targets and
# servers will be used equally.
# Usage: One targetID per line for storage servers, one nodeID per line for
# metadata servers.
# Note: TargetIDs and nodeIDs can be queried with the beegfs-ctl tool.
# Default:
$ vi /etc/beegfs/beegfs-mgmtd.conf
tuneMetaDynamicPools = true
tuneMetaInodesLowLimit = 10M
tuneMetaInodesEmergencyLimit = 1M
tuneMetaSpaceLowLimit = 10G
tuneMetaSpaceEmergencyLimit = 3G
tuneStorageDynamicPools = true
tuneStorageInodesLowLimit = 10M
tuneStorageInodesEmergencyLimit = 1M
tuneStorageSpaceLowLimit = 1T
tuneStorageSpaceEmergencyLimit = 20G
# [tune{Meta,Storage}DynamicPools]
# Temporarily raise the Low/Emergency limits if the spread (difference in free
# capacity between the targets with the most and the least free space) becomes
# too large. This will move targets to a lower pool earlier if there are other
# targets with much more free capacity.
# [tune{Meta,Storage}{Space,Inodes}LowLimit]
# [tune{Meta,Storage}{Space,Inodes}EmergencyLimit]
# The free space pool thresholds. If a metadata or storage target is below a
# threshold, it will only be used to store new files and directories when no
# higher class targets are available (i.e. while there are targets in the
# "normal" pool, no targets from the "low" pool will be used.)
# Note: Preferred target settings of a client will be ignored if it helps to
# avoid using targets from the emergency class.
# Default: Space: Meta: 10G/3G; Storage: 512G/10G
# Inodes: Meta: 10M/1M; Storage: 10M/1M
# [tune{Meta,Storage}{Space,Inodes}NormalSpreadThreshold]
# [tune{Meta,Storage}{Space,Inodes}LowSpreadThreshold]
# [tune{Meta,Storage}{Space,Inodes}LowDynamicLimit]
# [tune{Meta,Storage}{Space,Inodes}EmergencyDynamicLimit]
# Only effective if tune{Meta,Storage}DynamicPools is enabled.
# Whenever the spread (see above) of the free capacities in the normal / low
# class of storage targets is above this threshold, the StorageLowLimit /
# StorageEmergencyLimit is temporarily raised to StorageLowDynamicLimit /
# StorageEmergencyDynamicLimit.
# When the {Normal,Low}SpreadThreshold values are set to 0, the value from the
# corresponding {Low,Emergency}Limit is used for the spread threshold.
# When the {Low,Emergency}DynamicLimits are set to 0, they are automatically
# assumed as two times the corresponding {Low,Emergency}(non-dynamic)Limit.
# Default: 0
寻址过程总结
管理节点在收到元数据节点的的心跳信息时，如果没有元数据Root节点，则选择ID较小的作为元数据Root节点；
客户端在Mount时，会初始化超级块，同时向管理节点询问元数据Root节点的ID，然后向元数据Root节点获取DirEntry元数据；
元数据节点初始化时，会先依次寻找buddymir/inodes/38/51/root，inodes/38/51/root来确定根目录的元数据；
根据Root节点中Root的目录的元数据信息中包含的OwnerID(确定目录所在节点)和EntryID(根据Hash确定目录路径)，就可以找到该目录的元数据位置；
该目录所有文件的DirEntry文件中包含有文件的条带信息，以及数据分布的存储节点；
该目录所有子目录的DirEntry文件的元数据中包含子目录的OwnerID(即所在节点的NodeID)和EntryID，据此找到子目录的元数据目录，不断迭代，即可找到最终的文件。