使用Google开源库breakpad实现错误报告功能（Windows环境）

---

layout: post
title: “使用Google开源库breakpad实现错误报告功能（Windows环境）”
subtitle: “Google大法好”
date: “2017-08-16”
author: “cj”
tags:
breakpad
c++
windows
google
bugreport

Windows环境下以前使用的CrashRpt1403错误报告系统出了点问题，问题上传端口（80）被微信服务器测试环境占用了，索性使用breakpad更新之。

目录

1. 环境搭建
   1. 下载
   2. 编译
2. 部署
   1. 生成sym并上传
      1. windows-publish
      2. linux-publish
   2. 应用程序植入breakpad
   3. 处理应用程序崩溃
      1. windows-report
      2. linux-report
   4. 邮件示例
3. 参考资料

环境搭建

下载

按照官方教程，下载源码

git clone https://chromium.googlesource.com/breakpad/breakpad

有几个第三方库默认情况没有下载，按需手动下载到breakpad/src/thirdparty中

cd breakpad
git clone https://chromium.googlesource.com/linux-syscall-support src/third_party/lss

编译

与linux环境下直接configure&&make不同，Windows下需要google另一个工具gyp生成visual studio 的sln文件。另外gyp依赖Python2.x版本，如果系统环境变量PATH中指定的python.exe已经是Python3.x版的话，需要手动修改。Python2.x与3.x共用方案网上一搜大把不再赘述。

安装gyp：

git clone https://chromium.googlesource.com/external/gyp
cd gyp
python setup.py install

生成sln解决方案：

your-path-to-gyp/gyp.bat your-path-to-breakpad/src/client/windows/breakpad_client.gyp --no-circular-check

打开breakpad_client.sln并编译所需的库文件：

• ommon.lib
• exception_handler.lib
• crash_generation_client.lib

部署

在Windows下编译好应用程序后，使用breakpad的工具dump_syms.exe生成sym符号表，并上传至服务器。当客户环境中的应用程序崩溃时，调用一个脚本将dump信息上传至服务器，服务器端处理一下并生成stack walk信息发送到自己的邮箱。思路与上一篇linux版完全一样，只不过生成sym的环境换成了Windows，并增加了web服务以便处理。

下面以分为新版本发布时的“生成sym”和崩溃发生时的“处理dmp”记录一下。

生成sym并上传

新版本发布时，自动生成sym文件并上传

windows-publish

参考了这篇文章，摘录如下：

Decoding Windows crash dumps on Linux

Windows crash dumps can be decoded the same way as Linux crash dumps. The issue is mainly getting the debugging symbols as a .sym file instead of a .pdb file.

To convert a .pdb file to a .sym file:

1. Obtain the .pdb file and put it on a Windows machine. (It may be possible to do this with Wine, YMMV.)
2. Download dump_syms.exe.
3. Run: dump_syms foo.pdb > foo.sym
   • If no error messages, then you are done.
   • If you get: CoCreateInstance CLSID_DiaSource failed (msdia80.dll unregistered?), go to step 4.
4. Get a copy of msdia80.dll and put it in c:\Program Files\Common Files\Microsoft Shared\VC\.
5. As Administrator, run: regsvr32 c:\Program Files\Common Files\Microsoft Shared\VC\msdia80.dll.
   • On success, retry step 3.
   • If you get error 0x80004005, you did not run as Administrator.

2020年1月13日23:57:56增补

新 win10 系统一直报 CoCreateInstance CLSID_DiaSource {3BFCEA48-620F-4B6B-81F7-B9AF75454C7D} failed (msdia*.dll unregistered?) Open failed 错误，按照上述方法注册了 c:\Program Files\Common Files\Microsoft Shared\VC\ 文件夹下存在的 msdia80.dll, msdia90.dll, msdia100.dll，问题依旧。

用 everything 搜索 msdia，发现一大堆，试着注册了几个，试到 "C:\Program Files (x86)\Microsoft SDKs\UWPNuGetPackages\microsoft.net.native.compiler\1.7.6\tools\Runtime\x86\msdia120.dll" 的时候成功了。

懒人的解决方案：publish.bat

@echo off
rem 引用dump_sym.exe
set dump_syms="D:\dev_libs\google\breakpad\src\tools\windows\binaries\dump_syms.exe"

rem 需要使用7z进行压缩
set the_7z="..\7-Zip\7z.exe"

rem fciv是Windows发布的一个生成文件MD5值的小工具
set fciv="fciv.exe"

rem 需要使用curl上传文件
set curl="curl.exe"

rem 生成sym
%dump_syms% ..\..\Release\your-app.pdb > your-app.sym

rem 压缩
%the_7z% a publish.7z your-app.sym ..\ChangeLog.txt
del your-app.sym

rem 获取压缩文件MD5值
for /f %%i in ('%fciv% -md5 publish.7z') do set md5=%%i

rem 获取文件版本号
set /p pub_version=<..\..\Release\VersionNo.ini

rem 上传压缩文件至服务器
%curl% -F "action=upload" -F "md5=%md5%" -F "pub_version=%pub_version%" -F "publish=@publish.7z;type=application/7z" https://www.your-domain.com/crash/publish.php

del publish.7z
echo Done!

linux-publish

publish.php

<?php

// Specify the directory where to save error reports
$file_root = "/var/www/html/crash/app/";

// This is to avoid PHP warning
date_default_timezone_set('UTC');

// Writes error code and text message and then exits
function done($return_status, $message)
{
    // Write HTTP responce code
    header("HTTP/1.0 ".$return_status." ".$message);
    // Write HTTP responce body (for backwards compatibility)
    echo $return_status." ".$message;
    exit(0);
}

// Checks that text fild doesn't contain inacceptable symbols
function checkOK($field)
{
    if (stristr($field, "\\r") || stristr($field, "\\n"))
    {
        done(450, "Invalid input parameter.");
    }
}

$md5_hash = "";    // MD5 hash for error report ZIP
$file_name = "";   // Destination file name
$pub_version = "";  // Publish version

// Check that MD5 hash exists
if(!isset($_POST['md5']))
{
    done(450, "MD5 hash is missing.");
}

// Get MD5 hash
$md5_hash = $_POST['md5'];
checkOK($md5_hash);
if(strlen($md5_hash)!=32)
{
    done(450, "MD5 hash value has wrong length.");
}

// Get CrashGUID
if(!array_key_exists("pub_version", $_POST))
{
    done(450, "Publish version missing.");  
}

$pub_version = $_POST["pub_version"];
checkOK($pub_version);
if(strlen($pub_version)==0)
{
    done(450, "Publish version has wrong length.");
}  

// Get file attachment
if(array_key_exists("publish", $_FILES))
{
    // Check upload error code
    $error_code = $_FILES["publish"]["error"];
    if($error_code!=0)
    {
        done(450, "File upload failed with code $error_code.");
    }

    // Get temporary name uploaded file is stored currently
    $tmp_file_name = $_FILES["publish"]["tmp_name"];
    checkOK($tmp_file_name);

    // Check that uploaded file data have correct MD5 hash
    $my_md5_hash = strtolower(md5_file($tmp_file_name));
    $their_md5_hash = strtolower($md5_hash);
    if($my_md5_hash!=$their_md5_hash)
    {
        done(451, "MD5 hash is invalid (yours is ".$their_md5_hash.", but mine is ".$my_md5_hash.")");
    }

    // Use crash GUID as file name
    $file_name = $file_root.$pub_version.".7z";

    // Move uploaded file to an appropriate directory
    if(!move_uploaded_file($tmp_file_name, $file_name))
    {
        done(452, "Couldn't save data to local storage");
    }

    // 调用auto_build.sh进行处理
    $output = shell_exec($file_root."auto_build.sh ".$file_name);
    done(200, $output);
}
else
{
  done(452, "File attachment missing");
}

// OK.
done(200, "Success.");

?>

auto_build.sh 解压

#!/bin/bash
if [ -z "$1" ]; 
then echo Usage:$0 [7z file]
exit
fi

cd /var/www/html/crash/app
s=$1
s=${s##*/}
name=${s%.7z}
echo name=$name
if [ -d "$name" ]; then
echo This version of 7z file had already been extracted!
exit
fi

#mkdir $name
7z x $1 -o$name
chmod 755 $name
cd $name
../build_symbols.sh your-app.pdb

• build_symbols.sh 将sym文件按照breakpad指定的方式存放

#!/bin/bash
line=$(head -n1 your-app.sym)
echo line
arr=($line)
sdir="./symbols/your-app.pdb/${arr[3]}"
mkdir -p $sdir
mv your-app.sym $sdir
echo Build symbols OK

应用程序植入breakpad

按照breakpad官方文档Windows Integration overview，应用程序中如此这般：

#include <client/windows/handler/exception_handler.h>
#ifdef _DEBUG
#pragma comment(lib, "client/windows/Debug/lib/common.lib")
#pragma comment(lib, "client/windows/Debug/lib/exception_handler.lib")
#pragma comment(lib, "client/windows/Debug/lib/crash_generation_client.lib")
#else
#pragma comment(lib, "client/windows/Release/lib/common.lib")
#pragma comment(lib, "client/windows/Release/lib/exception_handler.lib")
#pragma comment(lib, "client/windows/Release/lib/crash_generation_client.lib")
#endif // _DEBUG

bool ShowDumpResults(const wchar_t* dump_path,
    const wchar_t* minidump_id,
    void* context,
    EXCEPTION_POINTERS* exinfo,
    MDRawAssertionInfo* assertion,
    bool succeeded) {

    if (minidump_id) {
        auto crashguid = utf8::w2a(minidump_id);
        JLOG_INFO("minidump_id={}", crashguid);
        std::string cmd = get_exe_path_a() + "/tools/report.bat " + app_version + " " + crashguid;
        jlib::daemon(cmd, false, false);

        // 调用report.bat，参数为版本号、crushguid
    }

    return succeeded;
}

int main()
{
    auto handler = std::make_unique<ExceptionHandler>(
        dmp_path, // dump_path
        nullptr, // FilterCallback
        ShowDumpResults, // MinidumpCallback
        nullptr, // callback_context
        ExceptionHandler::HANDLER_ALL // handler_types
        );
}

应用程序安装后的部分文件结构为：

+-- your-app-installation-path/
    +-- 7-zip/
        +-- 7z.exe
        +-- 7z.dll
        +-- 7-zip.dll
    +-- your-app.exe
    +-- dumps/
        +-- here is the dmp files' location
    +-- tools/
        +-- curl.exe
        +-- curl-ca-bundle.crt
        +-- fciv.exe
        +-- libcurl.dll
        +-- report.bat

处理应用程序崩溃

windows-report

report.bat

@echo on
set the_7z="..\7-Zip\7z.exe"
set fciv="fciv.exe"
set curl="curl.exe"

set appversion=%1
set crashguid=%2
cd ..\dumps
copy %crashguid%.dmp crashdump.dmp
%the_7z% a -tzip %crashguid%.zip crashdump.dmp
for /f %%i in ('%fciv% -md5 %crashguid%.zip') do set md5=%%i
cd ..\tools
%curl% -F "action=upload" -F "md5=%md5%" -F "appversion=%appversion%" -F "crashguid=%crashguid%" -F "crashrpt=@..\dumps\%crashguid%.zip;type=application/zip" https://www.your-domain.com/crash/report.php

echo Done!

linux-report

report.php

<?php

// Specify the directory where to save error reports
$file_root = "/var/www/html/crash/";

// This is to avoid PHP warning
date_default_timezone_set('UTC');

// Writes error code and text message and then exits
function done($return_status, $message)
{
    // Write HTTP responce code
    header("HTTP/1.0 ".$return_status." ".$message);
    // Write HTTP responce body (for backwards compatibility)
    echo $return_status." ".$message;
    exit(0);
}

// Checks that text fild doesn't contain inacceptable symbols
function checkOK($field)
{
    if (stristr($field, "\\r") || stristr($field, "\\n"))
    {
        done(450, "Invalid input parameter.");
    }
}

$app_version = ""; // Application Version
$md5_hash = "";    // MD5 hash for error report ZIP
$file_name = "";   // Destination file name
$crash_guid = "";  // Crash GUID

if(!isset($_POST['appversion']))
{
    done(450, "AppVersion is missing.");
}
$app_version = $_POST['appversion'];

// Check that MD5 hash exists
if(!isset($_POST['md5']))
{
    done(450, "MD5 hash is missing.");
}

// Get MD5 hash
$md5_hash = $_POST['md5'];
checkOK($md5_hash);
if(strlen($md5_hash)!=32)
{
    done(450, "MD5 hash value has wrong length.");
}

// Get CrashGUID
if(!array_key_exists("crashguid", $_POST))
{
    done(450, "Crash GUID missing.");  
}

$crash_guid = $_POST["crashguid"];
checkOK($crash_guid);
if(strlen($crash_guid)!=36)
{
    done(450, "Crash GUID has wrong length.");
}  

// Get file attachment
if(array_key_exists("crashrpt", $_FILES))
{
    // Check upload error code
    $error_code = $_FILES["crashrpt"]["error"];
    if($error_code!=0)
    {
        done(450, "File upload failed with code $error_code.");
    }

    // Get temporary name uploaded file is stored currently
    $tmp_file_name = $_FILES["crashrpt"]["tmp_name"];
    checkOK($tmp_file_name);

    // Check that uploaded file data have correct MD5 hash
    $my_md5_hash = strtolower(md5_file($tmp_file_name));
    $their_md5_hash = strtolower($md5_hash);
    if($my_md5_hash!=$their_md5_hash)
    {
        done(451, "MD5 hash is invalid (yours is ".$their_md5_hash.", but mine is ".$my_md5_hash.")");
    }

    // Use crash GUID as file name 
    // /var/www/html/crash/report/$APPVERSION/$CRASHGUID.zip
    $file_name = $file_root."app/".$app_version."/".$crash_guid.".zip";

    // Move uploaded file to an appropriate directory
    if(!move_uploaded_file($tmp_file_name, $file_name))
    {
        done(452, "Couldn't save data to local storage:".$file_name);
    }

    $output = shell_exec($file_root."app/process.sh ".$app_version." ".$crash_guid." ".$_SERVER['REMOTE_ADDR']);
    done(200, $output);
}
else
{
    done(452, "File attachment missing");
}

// OK.
done(200, "Success.");

?>

process.sh

#!/bin/bash
basedir=/var/www/html/crash/app
cd $basedir

if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ];then
    echo Usage: $0 [app versin] [crash guid] [ip address]
    exit
fi

app_version=$1
crashguid=$2
ip_addr=$3
echo app_version=$app_version
echo crashguid=$crashguid
echo ip_addr=$ip_addr

if [ -d "$crashguid" ]; then
    echo This version of crashguid had already been extracted!
else
    7z x ${basedir}/$app_version/${crashguid}.zip -o${basedir}/$app_version/${crashguid}
    chmod 755 ${basedir}/$app_version/${crashguid}
fi

basedir=${basedir}/${app_version}/${crashguid}

# 这里使用了聚合的获取IP位置服务。吐个槽，干嘛都要实名注册！
# get address info by ip
ip_xml="${basedir}/ip.xml"
result_code=""
area=""
location=""
url="http://apis.juhe.cn/ip/ip2addr?ip="${ip_addr}"&dtype=xml&key=your-key"
echo url=$url
echo ip_xml=$ip_xml
curl $url > $ip_xml
result=$(cat $ip_xml)
echo result:$result

read_dom() {
    local IFS=\>
    read -d \< ENTITY CONTENT
    local RET=$?
    TAG_NAME=${ENTITY%% *}
    ATTRIBUTES=${ENTITY#* }
    return $RET
}

parse_dom_ip() {
    if [[ $TAG_NAME = "resultcode" ]]; then
        result_code=$CONTENT
        elif [[ $TAG_NAME = "area" ]]; then
        area=$CONTENT
        elif [[ $TAG_NAME = "location" ]]; then
        location=$CONTENT
    fi
}

while read_dom; do
    parse_dom_ip
done < "$ip_xml"
rm $ip_xml

if [ -z "$result_code" ] || [ "$result_code" != "200" ]; then
    echo Get address for ip $ip_addr failed!
    elif [ "$result_code" = "200" ]; then
    echo Get address for ip $ip_addr success! Area: $area, Location: $location
fi

./send_mail_.sh $app_version $crashguid $ip_addr $area $location

rm $ip_xml
echo Done!

send_mail_.sh

#!/bin/bash
basedir=/var/www/html/crash/app
to_user_file="${basedir}/mail_to.txt"
dmp2html="${basedir}/dmp2html.sh"

subject=`echo 应用程序崩溃报告`
preifx=`echo "<html lang='zh_CN'><head><meta charset='utf8'/></head><body>"`
time=`date +"%Y年%m月%d日 %H:%M:%S"`
message=""

app_version=$1
crash_guid=$2
ip_addr=$3
area=$4
location=$5

# build ip info
ip_info="<p>Client IP:"${ip_addr}"<br/>Area:"${area}"<br/>Location:"${location}"</br></p>"
echo ip_info=$ip_info

# build stack walk info
dump=${basedir}/${app_version}/${crash_guid}/crashdump.dmp
symbols=${basedir}/${app_version}/symbols
crash_dump=`minidump_stackwalk $dump $symbols &> ${dump}.txt`
crash_dump=`${dmp2html} ${dump}.txt`
rm ${dump}.txt

# build message
message=`echo "<br/>Crash Dump:<br/>"${ip_info}"<br/>"${crash_dump}"<br/></body></html>"`
message=${preifx}${time}${message}
echo 'message=' ${message}
echo $message > message.html

#send mail
while IFS= read -r line
do
    echo "Sending email to $line"
    (
        echo "From: admin";
        echo "To: $line";
        echo "Subject: ${subject}";
        echo "Content-Type: text/html";
        echo "MIME-Version: 1.0";
        echo "";
        echo "${message}";
    ) | /usr/sbin/sendmail -t
    echo "Done!"
done <"$to_user_file"

邮件示例

以下是收到的邮件部分内容：

2017年08月16日 23:43:26
Crash Dump:
Client IP:113.140.*.*
Area:陕西省西安市
Location:电信
...

2017-08-16 23:43:26: minidump.cc:4811: INFO: Minidump opened minidump /var/www/html/crash/app/1.5.42.12672/51c3e708-4c60-4334-9faf-f7f1fb9523f9/crashdump.dmp
2017-08-16 23:43:26: minidump.cc:4931: INFO: Minidump not byte-swapping minidump
2017-08-16 23:43:26: minidump.cc:5414: INFO: GetStream: type 1197932546 not present
...

Operating system: Windows NT
10.0.14393
CPU: x86
GenuineIntel family 6 model 60 stepping 3
4 CPUs

GPU: UNKNOWN

Crash reason: EXCEPTION_ACCESS_VIOLATION_WRITE
Crash address: 0x0
Process uptime: 1 seconds

Thread 0 (crashed)
0 your-app.exe!CLoginDlg::OnBnClickedOk() [logindlg.cpp : 71 + 0x0]
eip = 0x00b60ef0 esp = 0x014fb608 ebp = 0x014fb64c ebx = 0x014ff4c0
esi = 0x014ff4c0 edi = 0x00000000 eax = 0x014fb640 ecx = 0x014ff4c0
edx = 0x00000000 efl = 0x00010286
Found by: given as instruction pointer in context
1 your-app.exe!_AfxDispatchCmdMsg(CCmdTarget *,unsigned int,int,void ( CCmdTarget::*)(void),void *,unsigned int,AFX_CMDHANDLERINFO *) [cmdtarg.cpp : 77 + 0x5]
eip = 0x00c64503 esp = 0x014fb654 ebp = 0x014fb660
Found by: call frame info
2 your-app.exe!CCmdTarget::OnCmdMsg(unsigned int,int,void *,AFX_CMDHANDLERINFO *) [cmdtarg.cpp : 372 + 0x18]
eip = 0x00c64324 esp = 0x014fb668 ebp = 0x014fb698
Found by: call frame info
...

可以清楚的看到崩溃发生在了logindlg.cpp文件的71行，函数为CLoginDlg::OnBnClickedOk。

参考资料

• breakpad
• How to build google google-breakpad for windows?
• Windows Integration overview
• Decoding Crash Dumps