Android系统分布式编译(distcc)

1. 搭建distcc分布式编译环境（适用Android 4.4）

最好的方式是参考官网，网上其他指导书都不全：https://raw.githubusercontent.com/distcc/distcc/master/INSTALL

1.1 安装distcc（distcc client和server均需要安装）

从官网下载distcc压缩包，拷贝到服务器上，tar -zxvf 解压缩

tar -zxvf distcc-3.3.2.tar.gz
sudo apt-get install autoconf libiberty-dev python3 python3-dev
cd distcc
./autogen.sh
./configure && make && sudo make install
make installcheck
sudo update-distcc-symlinks

1.2 配置编译环境（client和server均需要配置）

server如果没有Android编译工具，则将android编译工具prebuilds目录拷贝到/opt目录下
client不用拷贝，可以创建一个软链接，指向prebuilds所在目录，sudo ln -s /home/jenkins/android/prebuilts /opt/prebuilts
将交叉编译命令增加到PATH中，且放在第一个，sudo vi /etc/profile
增加：export PATH=/usr/local/lib/distcc:/opt/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.7/bin:/opt/prebuilts/gcc/linux-x86/arm/arm-eabi-4.6/bin:$PATH
source /etc/profile
配置distcc软链接

sudo mkdir /usr/local/lib/distcc
cd /usr/local/lib/distcc
sudo ln -s ../../bin/distcc arm-linux-androideabi-ar
sudo ln -s ../../bin/distcc arm-linux-androideabi-as
sudo ln -s ../../bin/distcc arm-linux-androideabi-c++
sudo ln -s ../../bin/distcc arm-linux-androideabi-c++filt
sudo ln -s ../../bin/distcc arm-linux-androideabi-cpp
sudo ln -s ../../bin/distcc arm-linux-androideabi-elfedit
sudo ln -s ../../bin/distcc arm-linux-androideabi-g++
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcc
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcc-4.7
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcc-ar
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcc-nm
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcc-ranlib
sudo ln -s ../../bin/distcc arm-linux-androideabi-gcov
sudo ln -s ../../bin/distcc arm-linux-androideabi-gdb
sudo ln -s ../../bin/distcc arm-linux-androideabi-gdbtui
sudo ln -s ../../bin/distcc arm-linux-androideabi-gprof
sudo ln -s ../../bin/distcc arm-linux-androideabi-ld
sudo ln -s ../../bin/distcc arm-linux-androideabi-ld.bfd
sudo ln -s ../../bin/distcc arm-linux-androideabi-ld.gold
sudo ln -s ../../bin/distcc arm-linux-androideabi-nm
sudo ln -s ../../bin/distcc arm-linux-androideabi-objcopy
sudo ln -s ../../bin/distcc arm-linux-androideabi-objdump
sudo ln -s ../../bin/distcc arm-linux-androideabi-ranlib
sudo ln -s ../../bin/distcc arm-linux-androideabi-readelf
sudo ln -s ../../bin/distcc arm-linux-androideabi-run
sudo ln -s ../../bin/distcc arm-linux-androideabi-size
sudo ln -s ../../bin/distcc arm-linux-androideabi-strings
sudo ln -s ../../bin/distcc arm-linux-androideabi-strip
sudo ln -s ../../bin/distcc arm-eabi-addr2line
sudo ln -s ../../bin/distcc arm-eabi-ar
sudo ln -s ../../bin/distcc arm-eabi-as
sudo ln -s ../../bin/distcc arm-eabi-c++
sudo ln -s ../../bin/distcc arm-eabi-c++filt
sudo ln -s ../../bin/distcc arm-eabi-cpp
sudo ln -s ../../bin/distcc arm-eabi-elfedit
sudo ln -s ../../bin/distcc arm-eabi-g++
sudo ln -s ../../bin/distcc arm-eabi-gcc
sudo ln -s ../../bin/distcc arm-eabi-gcc-4.6.x-google
sudo ln -s ../../bin/distcc arm-eabi-gcov
sudo ln -s ../../bin/distcc arm-eabi-gdb
sudo ln -s ../../bin/distcc arm-eabi-gdbtui
sudo ln -s ../../bin/distcc arm-eabi-gprof
sudo ln -s ../../bin/distcc arm-eabi-ld
sudo ln -s ../../bin/distcc arm-eabi-ld.bfd
sudo ln -s ../../bin/distcc arm-eabi-nm
sudo ln -s ../../bin/distcc arm-eabi-objcopy
sudo ln -s ../../bin/distcc arm-eabi-objdump
sudo ln -s ../../bin/distcc arm-eabi-ranlib
sudo ln -s ../../bin/distcc arm-eabi-readelf
sudo ln -s ../../bin/distcc arm-eabi-run
sudo ln -s ../../bin/distcc arm-eabi-size
sudo ln -s ../../bin/distcc arm-eabi-strings
sudo ln -s ../../bin/distcc arm-eabi-strip

1.3 distcc server配置

通过命令distccd --allow 10.0.0.0/8 --daemon启动distcc服务。–allow表示允许请求的client ip段。
–log-level debug，指定日志级别，定位问题时可设置为debug
–log-file [logpath]，指定日志保存目录。
可以将命令添加到"/etc/rc.d/rc.local"以开机启动

1.4 distcc client配置

安装监控工具，sudo apt-get install distccmon-gnome
sudo vi /usr/local/etc/distcc/hosts，写入distcc server的ip地址和任务限制数，可以根据server性能来配置任务限制数，比如ip/4。127.0.0.1表示是否在本地分发编译任务。
使用distccmon-text 3命令，查看整个编译任务的进度

1.5 Android编译配置

方法一：
1、修改Android编译文件
vi build/core/combo/TARGET_linux-arm.mk
增加如下语句：
TARGET_TOOLS_PREFIX := /usr/local/lib/distcc/arm-linux-androideabi-
2、在make -jN命令后面加上CC=distcc，来触发distcc编译，修改编译脚本auto_build.sh，make -j8后面均加上 CC=distcc。
方法二：
export TARGET_TOOLS_PREFIX=/usr/local/lib/distcc/arm-linux-androideabi-
export CC=distcc
export CXX=“distcc g++”
再执行编译脚本

1.6 定位方法

在distcc server中通过lsof -i:3632或者ps -elf| grep distcc检查distccd是否正常启动
通过pkill distcc命令杀掉distccd进程
distcc日志保存在/var/log/syslog文件中
distcc --show-hosts命令查看在线的distcc server地址

1.7 借助dmucs负载均衡
https://www.cnblogs.com/silenceli/archive/2013/11/07/3412505.html

2. 实现分布式编译

2.1 server端

编译器版本为clang-4691093:即源码路径 prebuilts/clang/host/linux-x86/clang-4691093/bin,目的是在源码编译过程当中使用到prebuilts/clang/host/linux-x86/clang-4691093/bin路径下的编译器 加上distcc 前端。

2.2 client端

(1) 复制编译器: scp -rf prebuilts/ /opt/
(2) 进入源码编译器路径下 cd prebuilts/clang/host/linux-x86/clang-4691093/bin
删除 clang相关编译器 即 rm -rf clang clang++ clang-6.0
(3) 建立编译器 链接
cd prebuilts/clang/host/linux-x86/clang-4691093/bin
touch clang clang++ clang-6.0
每个文件里重新写入新的编译器地址 并加上 distcc 前端
cat clang :
distcc /opt/prebuilts/clang/host/linux-x86/clang-4691093/bin/clang “$@"catclang++:distcc/opt/prebuilts/clang/host/linux-x86/clang-4691093/bin/clang++"$@”
cat clang-6.0：
distcc /opt/prebuilts/clang/host/linux-x86/clang-4691093/clang-6.0 “$@”
(4) 编译 source build/envsetup.sh
lunch an-eng
make -j32

3 Android9自带的goma分布式编译工具

google内部工具，当前google只开放了goma client，未开放goma server，源码地址：https://chromium.googlesource.com/infra/goma/client/+/refs/heads/master

3.1 编译goma client

编译goma工具，需要安装gn和ninja编译环境
参考
下载依赖包
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH=$PATH:/path/to/depot_tools

4. 常用命令

4.1 查看cpu核数

cat /proc/cpuinfo | grep “core id” | sort | wc -l

5. FAQ

5.1 执行apt-get命令，遇到Unmet dependencies错误，无法安装外部库

sudo apt --fix-broken install
sudo apt-get update
sudo apt-get upgrade

参考： https://www.jianshu.com/p/053fbb41f0df

5.2 编译失败，报错unrecognized option '–icf=safe

/usr/lib/gcc-cross/arm-linux-gnueabi/5.4.0/../../../../arm-linux-gnueabi/bin/ld: unrecognized option '--icf=safe'

修改TARGET_linux-arm.mk中P134行对TARGET_GLOBAL_LDFLAGS的赋值，去掉–icf=safe