移植librxe到riscv架构

---

编译环境:
OS: Ubuntu 20.10 LTS
x86-linux-gnu: gcc 10.3.0
riscv64-known-linux-gnu: gcc 9.2.0

---

1、获取librxe-dev源码

github链接: https://github.com/ImagineMiracle-wxn/librxe-dev

imaginemiracle:~$ git clone -b librxe-1.0.0 [e-mail]:ImagineMiracle-wxn/librxe-dev.git

2、交叉编译librxe-dev

执行configure文件配置源码

imaginemiracle:librxe-dev$ ./configure --libdir=/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/ --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/build/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu

输出如下: 有报错信息

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for riscv64-unknown-linux-gnu-strip... riscv64-unknown-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... riscv64-unknown-linux-gnu
checking how to print strings... printf
checking for style of include used by make... GNU
checking for riscv64-unknown-linux-gnu-gcc... riscv64-unknown-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by riscv64-unknown-linux-gnu-gcc... /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld
checking if the linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B
checking the name lister (/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-pc-linux-gnu file names to riscv64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld option to reload object files... -r
checking for riscv64-unknown-linux-gnu-objdump... riscv64-unknown-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for riscv64-unknown-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for riscv64-unknown-linux-gnu-ar... riscv64-unknown-linux-gnu-ar
checking for archiver @FILE support... @
checking for riscv64-unknown-linux-gnu-strip... (cached) riscv64-unknown-linux-gnu-strip
checking for riscv64-unknown-linux-gnu-ranlib... riscv64-unknown-linux-gnu-ranlib
checking command to parse /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B output from riscv64-unknown-linux-gnu-gcc object... ok
checking for sysroot... no
checking for riscv64-unknown-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking how to run the C preprocessor... riscv64-unknown-linux-gnu-gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if riscv64-unknown-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for riscv64-unknown-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if riscv64-unknown-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if riscv64-unknown-linux-gnu-gcc static flag -static works... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the riscv64-unknown-linux-gnu-gcc linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for riscv64-unknown-linux-gnu-gcc... (cached) riscv64-unknown-linux-gnu-gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... (cached) yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... (cached) none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... (cached) gcc3
checking for ibv_get_device_list in -libverbs... no
configure: error: ibv_get_device_list() not found.  librxe requires libibverbs.

2.1、解决 error: ibv_get_device_list() not found. librxe requires libibverbs.

根据错误可以知道交叉编译工具链找不到libibverbs库，需要移植libibverbs。该部分内容太多，单独写了一篇，可以参考该文章"移植libibverbs到riscv架构"。

2.2、再次配置

imaginemiracle:librxe-dev$ ./configure --libdir=/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/ --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/build/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu

输出信息如下: 仍然存在错误

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for riscv64-unknown-linux-gnu-strip... riscv64-unknown-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... riscv64-unknown-linux-gnu
checking how to print strings... printf
checking for style of include used by make... GNU
checking for riscv64-unknown-linux-gnu-gcc... riscv64-unknown-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by riscv64-unknown-linux-gnu-gcc... /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld
checking if the linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B
checking the name lister (/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-pc-linux-gnu file names to riscv64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld option to reload object files... -r
checking for riscv64-unknown-linux-gnu-objdump... riscv64-unknown-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for riscv64-unknown-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for riscv64-unknown-linux-gnu-ar... riscv64-unknown-linux-gnu-ar
checking for archiver @FILE support... @
checking for riscv64-unknown-linux-gnu-strip... (cached) riscv64-unknown-linux-gnu-strip
checking for riscv64-unknown-linux-gnu-ranlib... riscv64-unknown-linux-gnu-ranlib
checking command to parse /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B output from riscv64-unknown-linux-gnu-gcc object... ok
checking for sysroot... no
checking for riscv64-unknown-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking how to run the C preprocessor... riscv64-unknown-linux-gnu-gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if riscv64-unknown-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for riscv64-unknown-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if riscv64-unknown-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if riscv64-unknown-linux-gnu-gcc static flag -static works... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the riscv64-unknown-linux-gnu-gcc linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for riscv64-unknown-linux-gnu-gcc... (cached) riscv64-unknown-linux-gnu-gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... (cached) yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... (cached) none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... (cached) gcc3
checking for ibv_get_device_list in -libverbs... yes
checking infiniband/driver.h usability... no
checking infiniband/driver.h presence... no
checking for infiniband/driver.h... no
configure: error: <infiniband/driver.h> not found.  librxe requires libibverbs.

2.3、解决 error: <infiniband/driver.h> not found. librxe requires libibverbs.

将libibverbs源码目录下include/infiniband的所有头文件拷贝到交叉编译工具链对应的头文件目录中即可。

imaginemiracle:~$ cp -rf libibverbs/include/infiniband/* riscv64-unknown-linux-gnu/include/

2.4、重新配置librxe-dev

这次配置成功，生成Makefile文件。

imaginemiracle:librxe-dev$ ./configure --libdir=/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/ --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/build/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for riscv64-unknown-linux-gnu-strip... riscv64-unknown-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... riscv64-unknown-linux-gnu
checking how to print strings... printf
checking for style of include used by make... GNU
checking for riscv64-unknown-linux-gnu-gcc... riscv64-unknown-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by riscv64-unknown-linux-gnu-gcc... /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld
checking if the linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B
checking the name lister (/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-pc-linux-gnu file names to riscv64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld option to reload object files... -r
checking for riscv64-unknown-linux-gnu-objdump... riscv64-unknown-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for riscv64-unknown-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for riscv64-unknown-linux-gnu-ar... riscv64-unknown-linux-gnu-ar
checking for archiver @FILE support... @
checking for riscv64-unknown-linux-gnu-strip... (cached) riscv64-unknown-linux-gnu-strip
checking for riscv64-unknown-linux-gnu-ranlib... riscv64-unknown-linux-gnu-ranlib
checking command to parse /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B output from riscv64-unknown-linux-gnu-gcc object... ok
checking for sysroot... no
checking for riscv64-unknown-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking how to run the C preprocessor... riscv64-unknown-linux-gnu-gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if riscv64-unknown-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for riscv64-unknown-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if riscv64-unknown-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if riscv64-unknown-linux-gnu-gcc static flag -static works... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the riscv64-unknown-linux-gnu-gcc linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for riscv64-unknown-linux-gnu-gcc... (cached) riscv64-unknown-linux-gnu-gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... (cached) yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... (cached) none needed
checking dependency style of riscv64-unknown-linux-gnu-gcc... (cached) gcc3
checking for ibv_get_device_list in -libverbs... yes
checking infiniband/driver.h usability... yes
checking infiniband/driver.h presence... yes
checking for infiniband/driver.h... yes
checking for ibv_read_sysfs_file... yes
checking for ibv_dontfork_range... yes
checking for ibv_dofork_range... yes
checking for ibv_register_driver... yes
checking for ANSI C header files... (cached) yes
checking valgrind/memcheck.h usability... no
checking valgrind/memcheck.h presence... no
checking for valgrind/memcheck.h... no
checking for an ANSI C-conforming const... yes
checking size of long... 8
checking whether ld accepts --version-script... 
configure: creating ./config.status
config.status: creating Makefile
config.status: creating librxe.spec
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

2.5、编译并安装librxe-dev

执行make; make install编译并安装。

imaginemiracle:librxe-dev$ make; make install
/bin/bash ./libtool  --tag=CC   --mode=compile riscv64-unknown-linux-gnu-gcc -DHAVE_CONFIG_H -I.    -g -Wall -D_GNU_SOURCE -g -O2 -MT rxe.lo -MD -MP -MF .deps/rxe.Tpo -c -o rxe.lo `test -f 'src/rxe.c' || echo './'`src/rxe.c
libtool: compile:  riscv64-unknown-linux-gnu-gcc -DHAVE_CONFIG_H -I. -g -Wall -D_GNU_SOURCE -g -O2 -MT rxe.lo -MD -MP -MF .deps/rxe.Tpo -c src/rxe.c  -fPIC -DPIC -o .libs/rxe.o
In file included from src/rxe.c:59:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/include/infiniband/arch.h:135:2: warning: #warning No architecture specific defines found. Using generic implementation. [-Wcpp]
  135 | #warning No architecture specific defines found.  Using generic implementation.
      |  ^~~~~~~
src/rxe.c: In function 'convert_send_wr':
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_LOCAL_INV' not handled in switch [-Wswitch]
  578 |  switch(uwr->opcode) {
      |  ^~~~~~
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_BIND_MW' not handled in switch [-Wswitch]
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_SEND_WITH_INV' not handled in switch [-Wswitch]
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_SEND_ENABLE' not handled in switch [-Wswitch]
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_RECV_ENABLE' not handled in switch [-Wswitch]
src/rxe.c:578:2: warning: enumeration value 'IBV_WR_CQE_WAIT' not handled in switch [-Wswitch]
src/rxe.c: In function 'rxe_create_ah':
src/rxe.c:817:14: warning: passing argument 1 of 'rdma_gid2ip' from incompatible pointer type [-Wincompatible-pointer-types]
  817 |  rdma_gid2ip(&av->sgid_addr._sockaddr_in, &sgid);
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              |
      |              struct sockaddr_in *
src/rxe.c:773:48: note: expected 'struct sockaddr *' but argument is of type 'struct sockaddr_in *'
  773 | static inline int rdma_gid2ip(struct sockaddr *out, union ibv_gid *gid)
      |                               ~~~~~~~~~~~~~~~~~^~~
src/rxe.c:818:14: warning: passing argument 1 of 'rdma_gid2ip' from incompatible pointer type [-Wincompatible-pointer-types]
  818 |  rdma_gid2ip(&av->dgid_addr._sockaddr_in, &attr->grh.dgid);
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |              |
      |              struct sockaddr_in *
src/rxe.c:773:48: note: expected 'struct sockaddr *' but argument is of type 'struct sockaddr_in *'
  773 | static inline int rdma_gid2ip(struct sockaddr *out, union ibv_gid *gid)
      |                               ~~~~~~~~~~~~~~~~~^~~
libtool: compile:  riscv64-unknown-linux-gnu-gcc -DHAVE_CONFIG_H -I. -g -Wall -D_GNU_SOURCE -g -O2 -MT rxe.lo -MD -MP -MF .deps/rxe.Tpo -c src/rxe.c -o rxe.o >/dev/null 2>&1
mv -f .deps/rxe.Tpo .deps/rxe.Plo
/bin/bash ./libtool --tag=CC   --mode=link riscv64-unknown-linux-gnu-gcc -g -Wall -D_GNU_SOURCE -g -O2 -avoid-version -release     rdmav2 -Wl,--version-script=./src/rxe.map  -o src/librxe.la -rpath /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib rxe.lo  -libverbs 
libtool: link: riscv64-unknown-linux-gnu-gcc -shared  -fPIC -DPIC  .libs/rxe.o   -libverbs  -O2 -Wl,--version-script=./src/rxe.map   -Wl,-soname -Wl,librxe-rdmav2.so -o src/.libs/librxe-rdmav2.so
libtool: link: (cd "src/.libs" && rm -f "librxe.so" && ln -s "librxe-rdmav2.so" "librxe.so")
libtool: link: riscv64-unknown-linux-gnu-ar cru src/.libs/librxe.a  rxe.o
libtool: link: riscv64-unknown-linux-gnu-ranlib src/.libs/librxe.a
libtool: link: ( cd "src/.libs" && rm -f "librxe.la" && ln -s "../librxe.la" "librxe.la" )
make[1]: Entering directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/librxe-dev'
test -z "/home/imaginemiracle/Ubuntu_Download/RDMA/build/bin" || /usr/bin/mkdir -p "/home/imaginemiracle/Ubuntu_Download/RDMA/build/bin"
 /usr/bin/install -c rxe_cfg '/home/imaginemiracle/Ubuntu_Download/RDMA/build/bin'
test -z "/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib" || /usr/bin/mkdir -p "/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib"
 /bin/bash ./libtool   --mode=install /usr/bin/install -c   src/librxe.la '/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib'
libtool: install: /usr/bin/install -c src/.libs/librxe-rdmav2.so /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/librxe-rdmav2.so
libtool: install: (cd /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib && { ln -s -f librxe-rdmav2.so librxe.so || { rm -f librxe.so && ln -s librxe-rdmav2.so librxe.so; }; })
libtool: install: /usr/bin/install -c src/.libs/librxe.lai /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/librxe.la
libtool: install: /usr/bin/install -c src/.libs/librxe.a /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/librxe.a
libtool: install: chmod 644 /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/librxe.a
libtool: install: riscv64-unknown-linux-gnu-ranlib /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib/librxe.a
libtool: finish: PATH="/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin:/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/rvlinux-u_IM/toolchain/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin" ldconfig -n /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib
----------------------------------------------------------------------
Libraries have been installed in:
   /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man7" || /usr/bin/mkdir -p "/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man7"
 /usr/bin/install -c -m 644 man/rxe.7 '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man7'
test -z "/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man8" || /usr/bin/mkdir -p "/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man8"
 /usr/bin/install -c -m 644 man/rxe_cfg.8 '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man8'
test -z "/home/imaginemiracle/Ubuntu_Download/RDMA/build/etc/libibverbs.d" || /usr/bin/mkdir -p "/home/imaginemiracle/Ubuntu_Download/RDMA/build/etc/libibverbs.d"
 /usr/bin/install -c -m 644 rxe.driver '/home/imaginemiracle/Ubuntu_Download/RDMA/build/etc/libibverbs.d'
test -z "" || /usr/bin/mkdir -p ""
make[1]: Leaving directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/librxe-dev'

imaginemiracle:librxe-riscv_install$ ls
bin  etc  share
imaginemiracle:librxe-riscv_install$ tree
.
├── bin
│   └── rxe_cfg
├── etc
│   └── libibverbs.d
│       └── rxe.driver
└── share
    └── man
        ├── man7
        │   └── rxe.7
        └── man8
            └── rxe_cfg.8

7 directories, 4 files

2.5.1、可能会发生的错误

编译时可能会发生的错误，可能会出现如下几个结构体发生重复定义。

In file included from src/rxe.c:64:
src/rxe-abi.h:56:8: error: redefinition of 'struct rxe_create_cq_resp'
   56 | struct rxe_create_cq_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:61:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h:156:8: note: originally defined here
  156 | struct rxe_create_cq_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:64:
src/rxe-abi.h:61:8: error: redefinition of 'struct rxe_resize_cq_resp'
   61 | struct rxe_resize_cq_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:61:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h:160:8: note: originally defined here
  160 | struct rxe_resize_cq_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:64:
src/rxe-abi.h:66:8: error: redefinition of 'struct rxe_create_qp_resp'
   66 | struct rxe_create_qp_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:61:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h:164:8: note: originally defined here
  164 | struct rxe_create_qp_resp {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:64:
src/rxe-abi.h:72:8: error: redefinition of 'struct rxe_create_srq_resp'
   72 | struct rxe_create_srq_resp {
      |        ^~~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:61:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h:169:8: note: originally defined here
  169 | struct rxe_create_srq_resp {
      |        ^~~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:64:
src/rxe-abi.h:78:8: error: redefinition of 'struct rxe_modify_srq_cmd'
   78 | struct rxe_modify_srq_cmd {
      |        ^~~~~~~~~~~~~~~~~~
In file included from src/rxe.c:61:
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h:175:8: note: originally defined here
  175 | struct rxe_modify_srq_cmd {
      |        ^~~~~~~~~~~~~~~~~~

解决方案: 只需要将编译工具链中的相关定义注释掉即可。例如这里笔者的冲突文件为“riscv64-linux-multilib/sysroot/usr/include/rdma/rdma_user_rxe.h”，那么只需要将该文件中的这几个结构体的定义注释掉，如下:

imaginemiracle:rdma$ vim rdma_user_rxe.h
## File rdma_user_rxe.h
/*****************Alter by wxn******************/
/*struct rxe_create_cq_resp {
    struct mminfo mi;
};

struct rxe_resize_cq_resp {
    struct mminfo mi;
};

struct rxe_create_qp_resp {
    struct mminfo rq_mi;
    struct mminfo sq_mi;
};

struct rxe_create_srq_resp {
    struct mminfo mi;
    __u32 srq_num;
    __u32 reserved;
};

struct rxe_modify_srq_cmd {
    __aligned_u64 mmap_info_addr;
};
*/ /*************End Alter**************/
#endif /* RDMA_USER_RXE_H */

3、迁移librxe-dev至riscv文件系统

将安装目录的librxe相关文件拷贝至，目标文件系统的相应目录即可。

imaginemiracle:riscv-rootfs_IM$ ls
bin  dev  etc  include  init  lib  lib64  linuxrc  media  mnt  opt  proc  root  run  sbin  sys  tmp  usr  var
imaginemiracle:riscv-rootfs_IM$ cp ~/librxe-riscv_install/bin/* ./bin/
imaginemiracle:riscv-rootfs_IM$ cp ~/librxe-riscv_install/etc/* ./etc/
imaginemiracle:riscv-rootfs_IM$ cp ~/librxe-riscv_install/share/man/* ./usr/share/man/

到此，librxe-dev移植完成。