登录
当前位置： 首页 > 工具软件 > PHP 7.0-LevelDB > 使用案例 >

YCSB负载工具测试leveldb

郭兴平
2023-12-01

网上对有关YCSB怎么连接leveldb的讲解简直是太少了。我弄了好久才弄好分享下自己的经验，也为之后自己不用再查一遍。

首先下载需要的一些包：（我的系统版本是centos 7）

1.首先安装thrift：

wget http://mirrors.tuna.tsinghua.edu.cn/apache/thrift/0.12.0/thrift-0.12.0.tar.gz
tar zxvf thrift-0.12.0.tar.gz
yum install -y automake libtool flex bison pkgconfig 
yum install -y gcc-c++ libevent-devel zlib-devel python-devel ruby-devel openssl-devel

2.安装boost：（仅限于boost_1_67_0版本，因为老版本安装方式好像和这个有差别，我试过make一直报错）

wget https://sourceforge.net/projects/boost/files/boost/1.67.0/boost_1_67_0.tar.bz2
tar jxvf boost_1_67_0.tar.bz2
yum install gcc gcc-c++ bzip2 bzip2-devel bzip2-libs python-devel zlib-devel -y
sudo ./bootstrap.sh --prefix=/usr/local/include/boost
sudo ./b2 install

进入boost_1_67_0目录下的tools/build目录，执行：

sudo ./bootstrap.sh
sudo ./b2 install --prefix=/usr/local/boost
cp -rf /usr/local/include/boost/lib/* /usr/lib64 #一定要把库拷贝到/usr/lib64中，否则会报找不到头文件的错误

嘤嘤嘤，没执行cp -rf /usr/local/include/boost/lib/* /usr/lib64的操作，导致一直报以下错误了好几天，软连接也不行。。。

主要是报以下错误，现在终于弄好了，走了好多弯路。

processor/ProcessorTest.o:/usr/include/boost/test/test_tools.hpp:523: more undefined references to `boost::test_tools::tt_detail::check_impl(boost::test_tools::predicate_result const&, boost::unit_test::lazy_ostream const&, boost::unit_test::basic_cstring<char const>, unsigned long, boost::test_tools::tt_detail::tool_level, boost::test_tools::tt_detail::check_type, unsigned long, ...)' follow
g++: error: /usr/local/lib/libboost_unit_test_framework.a: No such file or directory
make[5]: *** [processor_test] Error 1
make[5]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp/test'
make[4]: *** [all] Error 2
make[4]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp/test'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib/cpp'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0/lib'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/hsk/kvstore/software/thrift-0.12.0'
make: *** [all] Error 2

设置环境变量：

vim /etc/profile之后在文件中添加
    export BOOST_INCLUDE = /usr/local/include/boost/include
    export BOOST_LIB =/usr/local/include/boost/lib
source /etc/profile

如果显示这个报错：fatal error: boost/tokenizer.hpp: No such file or directory，尝试使用

yum install boost-devel

到目前为止，boost已经安装完成。判断是否boost安装成功。创建一个cpp文件。

#include <boost/thread/thread.hpp> //包含boost头文件
#include <iostream>
#include <cstdlib>
using namespace std;

volatile bool isRuning = true;

void func1()
{
    static int cnt1 = 0;
    while(isRuning)
    {
        cout << "func1:" << cnt1++ << endl;
        sleep(1);
    }
}

void func2()
{
    static int cnt2 = 0;
    while(isRuning)
    {
        cout << "\tfunc2:" << cnt2++ << endl;
        sleep(2);
    }
}

int main()
{
    boost::thread thread1(&func1);
    boost::thread thread2(&func2);

    system("read");
    isRuning = false;

    thread2.join();
    thread1.join();
    cout << "exit" << endl;
    return 0;
}

然后执行，如果不报错则表示成功：

g++ test.cpp -g -o test -lboost_thread -lboost_system

下面回到thrift路径进行以下操作：

./bootstrap.sh
./configure --with-boost=/usr/local/include/boost
make -j20 && make install
thrift -version #验证是否成功
Thrift version 0.12.0 #显示结果表示已经安装成功

在进行make &&install 指令的时候可能会出现一些错误：

错误1：如果make error和composer有关可能是没有安装php，需要先安装pip，在这里需要安装的是pip7否则会报以下错误：

Problem 1
phpunit/phpunit 5.7.5 requires php ^5.6 || ^7.0 -> your PHP version (5.4.16) does not satisfy that requirement.

解决方案：

yum remove php*
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
yum install php70w
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
chmod +x /usr/local/bin/composer
然后在进行在有composer.ini的文件中composer install

 错误二：可能是没有安装php.xml的问题：我是使用的centos 7操作系统，如果用的是ubuntu请使用apt-get下载。

Problem 2:
Installation request for phpunit/phpunit ~4.8.36 -> satisfiable by phpunit/phpunit[4.8.36].

解决方案：

[root@localhost ~]# yum search php70w    查询可供安装的包
[root@localhost ~]# yum -y install php70w-xml.x86_64
[root@localhost ~]# systemctl restart httpd.service  
[root@localhost laravel_demo]# composer install #在含有composer.ini的路径下执行

 

3.snappy下载：

git clone https://github.com/google/snappy.git
mkdir build
cd build && cmake ../ && make
sudo cp build/libsnappy.a /usr/local/lib/

在安装snappy的时候进行cmake时有可能会显示cmake版本太老。需要进行以下操作，然后在进行cmake操作即可。

cd /home/wangrui
wget https://cmake.org/files/v3.10/cmake-3.10.2-Linux-x86_64.tar.gz
yum remove cmake
tar zxvf cmake-3.10.2-Linux-x86_64.tar.gz
cd cmake-3.10.2-Linux-x86_64
vi /etc/profile
在环境变量中添加：
export CMAKE_HOME=/home/wangrui/cmake-3.10.2-Linux-x86_64
export PATH=$PATH:$CMAKE_HOME/bin


source /etc/profile

4.levelDB安装：

git clone https://github.com/google/leveldb.git
cd leveldb
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build

然后再build目录下：

make
cd ..
cp -r include/leveldb /usr/local/include/
sudo cp build/libleveldb.a /usr/local/lib/

测试leveldb安装是否成功，创建demo.cc：

#include <cassert>
#include <iostream>
#include <string>
#include <leveldb/db.h>

int main() {
  leveldb::DB* db;
  leveldb::Options options;
  options.create_if_missing = true;
  leveldb::Status status = leveldb::DB::Open(options, "/tmp/testdb", &db);
  assert(status.ok());

  std::string key = "apple";
  std::string value = "A";
  std::string get;

  leveldb::Status s = db->Put(leveldb::WriteOptions(), key, value);

  if (s.ok()) s = db->Get(leveldb::ReadOptions(), key, &get);
  if (s.ok()) std::cout << "读取到的与(key=" << key << ")对应的(value=" << get << ")" << std::endl;
  else std::cout << "读取失败!" << std::endl;

  delete db;

  return 0;
}

命令行中输入：在makefile文件中添加-std=c++11这一行，如果版本g++不一致的话。

g++ -o demo demo.cc -pthread -lleveldb -std=c++11 -lsnappy
./demo
显示以下内容表示level安装成功：
读取到的与(key=apple)对应的(value=A)

5.下载libevent：

wget http://monkey.org/~provos/libevent-2.0.12-stable.tar.gz
tar xfvz libevent-2.0.12-stable.tar.gz
cd libevent-2.0.12-stable
./configure --prefix=/usr/local
make
sudo make install

下载mapkeeper：

git clone https://github.com/sears/mapkeeper.git
export MKROOT=/home/wangrui/downloads/mapkeeper
cd $MKROOT/thrift
make

返回到mapkeeper目录下

cd leveldb
make

make可能会报很多错误：

错误1，如果出现nullptr没有define的错误，错误显示如下：

/usr/local/include/leveldb/options.h:161:30: error: ‘nullptr’ was not declared in this scope
   const Snapshot* snapshot = nullptr;

改正1，应该在makefile文件中添加-std=c++11来解决这个问题。

错误2，如果出现LevelDbServer出现57行错误的话:

Error: LevelDbServer error: conversion from ‘boost::filesystem::path’ to non-scalar type ‘std::string {aka std::basic_string<char>}’ requested

改正2，修改第57行:

std::string mapName = boost::filesystem::path(itr->path().filename()).string();

错误3:

/usr/local/lib/libleveldb.a(table_builder.cc.o): In function `leveldb::TableBuilder::WriteBlock(leveldb::BlockBuilder*, leveldb::BlockHandle*)':
table_builder.cc:(.text+0x83d): undefined reference to `snappy::MaxCompressedLength(unsigned long)'
table_builder.cc:(.text+0x8a2): undefined reference to `snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)'
/usr/local/lib/libleveldb.a(format.cc.o): In function `leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const
&, leveldb::BlockContents*)':format.cc:(.text+0x44e): undefined reference to `snappy::GetUncompressedLength(char const*, unsigned long, unsigned long*)'
format.cc:(.text+0x5bf): undefined reference to `snappy::RawUncompress(char const*, unsigned long, char*)'
collect2: error: ld returned 1 exit status

改正3，添加/usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a对应依赖，主要是因为libleveldb.a misses Snappy when being linked which would be probably in libsnappy.a in the same directory.，同时添加-std=c++11消除版本问题:

include ../Makefile.config
EXECUTABLE = mapkeeper_leveldb
all : thrift
        g++ -Wall -o $(EXECUTABLE) *cpp -I $(THRIFT_DIR)/include/thrift -I $(THRIFT_DIR)/include \
        /usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a -lboost_thread -lboost_filesystem -lthrift -lleveldb -I ../thrift/gen-cpp \
        -L $(THRIFT_DIR)/lib \
        -L ../thrift/gen-cpp -lmapkeeper \
           -Wl,-rpath,\$$ORIGIN/../thrift/gen-cpp                       \
           -Wl,-rpath,$(THRIFT_DIR)/lib \
        -std=c++11

thrift:
        make -C ../thrift

run:
        ./$(EXECUTABLE) --sync

clean :
        - rm -rf $(THRIFT_SRC) $(EXECUTABLE) *.o

wipe:
        - rm -rf data/*

错误4:如果报错是leveldbserver.cpp的255行和259行，修改相应行数的内容，修改如下：

std::shared_ptr<LevelDbServer> handler(new LevelDbServer("data"));
std::shared_ptr<TProcessor> processor(new MapKeeperProcessor(handler));
std::shared_ptr<TServerTransport> serverTransport(new TServerSocket(port));
std::shared_ptr<TTransportFactory> transportFactory(new TFramedTransportFactory());
std::shared_ptr<TProtocolFactory> protocolFactory(new TBinaryProtocolFactory());

错误5:

/usr/bin/ld: /tmp/cch6BDEI.o: undefined reference to symbol 'pthread_condattr_setclock@@GLIBC_2.3.3'
/usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [all] Error 1

解决修改makefile文件，修改后文件内容如下：

include ../Makefile.config
EXECUTABLE = mapkeeper_leveldb
all : thrift
        g++ -Wall -o $(EXECUTABLE) *cpp -I $(THRIFT_DIR)/include/thrift -I $(THRIFT_DIR)/include \
        /usr/local/lib/libleveldb.a /usr/local/lib/libsnappy.a -lpthread -lboost_thread -lboost_system -lboost_filesystem -lthrift -lleveldb -I ../thrift/gen-cpp  \
        -L $(THRIFT_DIR)/lib \
        -L ../thrift/gen-cpp -lmapkeeper \
        -Wl,-rpath,\$$ORIGIN/../thrift/gen-cpp                       \
        -Wl,-rpath,$(THRIFT_DIR)/lib \
        -std=c++11
thrift:
        make -C ../thrift

run:
        ./$(EXECUTABLE) --sync

clean :
        -rm -rf $(THRIFT_SRC) $(EXECUTABLE) *.o

wipe:
        -rm -rf data/*

运行mapkeeper，成功运行表示成功：

./mapkeeper_leveldb 1 -d ~/mapkeeper/level/data

然后是下载ycsb加到mapkeeper/ycsb中（我试过好多版本都会make出错，我也很无奈是在没办法了，就选用特定这个0.1.4版本吧，能运行成功）：如果想找这个版本可以到我的博客中进行下载。

wget https://github.com/downloads/brianfrankcooper/YCSB/ycsb-0.1.4.tar.gz
tar xfvz ycsb-0.1.4
cd ycsb-0.1.4
bin/ycsb load mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"
bin/ycsb run mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090"

显示类似(截取一段)：

[UPDATE], 296, 0
[UPDATE], 297, 0
[UPDATE], 298, 0
[UPDATE], 299, 0
[UPDATE], 300, 0
[UPDATE], 301, 0
[UPDATE], 302, 0
[UPDATE], 303, 0
[UPDATE], 304, 0
[UPDATE], 305, 0
[UPDATE], 306, 0
[UPDATE], 307, 0
[UPDATE], 308, 0
[UPDATE], 309, 0
[UPDATE], 310, 0
[UPDATE], 311, 0
[UPDATE], 312, 0
[UPDATE], 313, 0
[UPDATE], 314, 0
[UPDATE], 315, 0
[UPDATE], 316, 0
[UPDATE], 317, 0

表示成功。

在最后的最后，我重启了服务器发现/home文件夹下什么都没有了。哭了。。。

最后发现是重启之后没有挂载硬盘，执行下面命令就好了。

mount /dev/nvme0n1 /home/hsk/kvstore

批量测试leveldb在ycsb工作负载下运行情况，编写run.sh。（一定要在每一次load和run一个工作负载文件之后删除数据库路径“rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*”下的所有文件）否则会报错。同时通过ulimit查看open file文件设置的上限，并把上限通过ulimit -n 65535指令调到64MB大小，否则会报如下错误。

insert not ok! IO error: While open a file for random read: /home/hsk/kvstore/software/mapkeeper/rocksdb/data/usertable/011619.sst: Too many open files
status: IO error: lock : /home/hsk/kvstore/software/mapkeeper/rocksdb/data/usertable/LOCK: No locks available

运行脚本如下，放在ycsb所在路径下： 

bin/ycsb load mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/1_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloada -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/1_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadb -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/2_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadb -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/2_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadc -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/3_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadc -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/3_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadd -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/4_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadd -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/4_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloade -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/5_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloade -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/5_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*
bin/ycsb load mapkeeper -s -P workloads/workloadf -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/6_load.txt 2>&1
bin/ycsb run mapkeeper -s -P workloads/workloadf -p "mapkeeper.host=127.0.0.1" -p "mapkeeper.port=9090" > /home/hsk/kvstore/test_data/YCSB_leveldb/6_run.txt 2>&1
rm -rf /home/hsk/kvstore/software/mapkeeper/leveldb/data/*

 

 类似资料：

相关阅读

性能测试和负载测试之间的区别14款好用开源的Web应用压力负载，性能测试工具推荐Java基准测试工具如何使用Selenium WebDriver执行负载测试？[关闭]Apache ab并发负载压力测试实现方法

相关文章

大疆 测试工程师测试工程师面经大疆测试工程师契约锁—测试工程师面试8.12美团测试工程师笔试

相关问答

JMeter SQLServer数据库负载测试负载测试应用。yml属性使用JMeter进行负载测试、压力测试、容量测试Quarkus。负载测试和内存利用Jmeter和Bitbucket服务器负载测试

相关文档

Camel 负载平衡中间件测试手册Linux 工具快速教程Android 测试教程OWASP 测试指南 v4.0
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.