杂项 - GDB 调试 OpenResty
优质
小牛编辑
132浏览
2023-12-01
使用 GDB 调试 Nginx
udo gdb -q -tui # -q 安静模式启动 GDB -tui 显示代码界面
进入 GDB 运行 attach
Nginx 子进程报错如下:
issing separate debuginfos, use: debuginfo-install libgcc-4.8.5-4.el7.x86_64 zlib-1.2.7-17.el7.x86_64
直接安装即可
断点
ngx_process_events_and_timers
子进程即 worker 进程在运行后会停留在epoll_wait
处等待相应的事件发生,而这个函数调用被封装在 ngx_process_events_and_timers
中
1 breakpoint keep y 0x000000000046c8f0 in ngx_process_events_and_timers at src/event/ngx_event.c:195 breakpoint already hit 39 times2 breakpoint keep y 0x000000000046c93e in ngx_process_events_and_timers at src/event/ngx_event.c:242 breakpoint already hit 36 times3 breakpoint keep y 0x000000000046c95a in ngx_process_events_and_timers at src/event/ngx_event.c:249 breakpoint already hit 35 times4 breakpoint keep y 0x000000000046ce7d in ngx_event_process_posted at src/event/ngx_event_posted.c:335 breakpoint keep y 0x0000000000475c24 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:793 breakpoint already hit 25 times6 breakpoint keep y 0x0000000000475d28 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:9267 breakpoint keep y 0x0000000000476022 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:900 breakpoint already hit 28 times8 breakpoint keep y 0x000000000048cd4a in ngx_http_wait_request_handler at src/http/ngx_http_request.c:497 breakpoint already hit 6 times9 breakpoint keep y 0x000000000048cd6a in ngx_http_wait_request_handler at src/http/ngx_http_request.c:504 breakpoint already hit 7 times10 breakpoint keep y 0x000000000048c900 in ngx_http_process_request_line at src/http/ngx_http_request.c:945 breakpoint already hit 6 times11 breakpoint keep y 0x000000000048c912 in ngx_http_process_request_line at src/http/ngx_http_request.c:952 breakpoint already hit 6 times12 breakpoint keep y 0x000000000048ca95 in ngx_http_process_request_line at src/http/ngx_http_request.c:101113 breakpoint keep y 0x000000000048cbce in ngx_http_process_request_line at src/http/ngx_http_request.c:1027 breakpoint already hit 6 times14 breakpoint keep y 0x000000000048c5ef in ngx_http_process_request_headers at src/http/ngx_http_request.c:1322 breakpoint already hit 7 times15 breakpoint keep y 0x000000000048c7ed in ngx_http_process_request_headers at src/http/ngx_http_request.c:1342 breakpoint already hit 2 times16 breakpoint keep y 0x000000000048c826 in ngx_http_process_request_headers at src/http/ngx_http_request.c:1348 breakpoint already hit 2 times17 breakpoint keep y 0x000000000048c231 in ngx_http_process_request at src/http/ngx_http_request.c:1916 breakpoint already hit 2 times18 breakpoint keep y 0x0000000000480757 in ngx_http_handler at src/http/ngx_http_core_module.c:839 breakpoint already hit 2 times19 breakpoint keep y 0x0000000000484ced in ngx_http_core_rewrite_phase at src/http/ngx_http_core_module.c:91020 breakpoint keep y 0x0000000000485e60 in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1372 breakpoint already hit 1 time21 breakpoint keep y 0x0000000000485eac in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:138622 breakpoint keep y 0x0000000000485e7d in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1379 breakpoint already hit 1 time23 breakpoint keep y 0x0000000000504c19 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:222
Lua 请求处理
堆栈
0 __libc_writev (fd=3, vector=0x7ffe08245dd0, count=14) at ../sysdeps/unix/sysv/linux/writev.c:681 0x00000000004717e9 in ngx_writev (c=c@entry=0x7ff03d02f730, vec=vec@entry=0x7ffe08245db0) at src/os/unix/ngx_writev_chain.c:1892 0x0000000000476a7e in ngx_linux_sendfile_chain (c=0x7ff03d02f730, in=0x7ff03d001418, limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:2153 0x00000000004a7d15 in ngx_http_write_filter (r=0x7ff03cfffcf0, in=0x7ff03d001b78) at src/http/ngx_http_write_filter_module.c:2544 0x00000000004a909d in ngx_http_chunked_body_filter (r=0x7ff03cfffcf0, in=<optimized out>) at src/http/modules/ngx_http_chunked_filter_module.c:2245 0x00000000004ac7dc in ngx_http_gzip_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/modules/ngx_http_gzip_filter_module.c:3266 0x00000000004afd95 in ngx_http_ssi_body_filter (r=0x7ff03cfffcf0, in=<optimized out>) at src/http/modules/ngx_http_ssi_filter_module.c:4117 0x00000000004b2cf0 in ngx_http_charset_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/modules/ngx_http_charset_filter_module.c:6478 0x0000000000506c7c in ngx_http_lua_capture_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at ../ngx_lua-0.10.6/src/ngx_http_lua_capturefilter.c:1339 0x0000000000453395 in ngx_output_chain (ctx=ctx@entry=0x7ff03d001428, in=in@entry=0x7ffe082466d0) at src/core/ngx_output_chain.c:7410 0x00000000004b4f95 in ngx_http_copy_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/ngx_http_copy_filter_module.c:15211 0x0000000000485a37 in ngx_http_output_filter (r=r@entry=0x7ff03cfffcf0, in=in@entry=0x7ffe082466d0) at src/http/ngx_http_core_module.c:197912 0x0000000000489e33 in ngx_http_send_special (r=r@entry=0x7ff03cfffcf0, flags=flags@entry=1) at src/http/ngx_http_request.c:335813 0x00000000004ffc38 in ngx_http_lua_send_special (flags=1, r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:56914 ngx_http_lua_send_chain_link (r=0x7ff03cfffcf0, ctx=<optimized out>, in=0x0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:52315 0x0000000000501f5f in ngx_http_lua_run_thread (L=L@entry=0x40030378, r=r@entry=0x7ff03cfffcf0, ctx=ctx@entry=0x7ff03d000dc8, nrets=nrets@entry=0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:147616 0x00000000005050f0 in ngx_http_lua_content_by_chunk (L=L@entry=0x40030378, r=r@entry=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:12017 0x000000000050548d in ngx_http_lua_content_handler_file (r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:28418 0x0000000000504c21 in ngx_http_lua_content_handler (r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:22219 0x0000000000485e7f in ngx_http_core_content_phase (r=0x7ff03cfffcf0, ph=<optimized out>) at src/http/ngx_http_core_module.c:137920 0x0000000000480675 in ngx_http_core_run_phases (r=r@entry=0x7ff03cfffcf0) at src/http/ngx_http_core_module.c:85621 0x000000000048075c in ngx_http_handler (r=r@entry=0x7ff03cfffcf0) at src/http/ngx_http_core_module.c:83922 0x000000000048c249 in ngx_http_process_request (r=0x7ff03cfffcf0) at src/http/ngx_http_request.c:191623 0x000000000048cbf6 in ngx_http_process_request_line (rev=0x7ff03d069520) at src/http/ngx_http_request.c:102724 0x0000000000476029 in ngx_epoll_process_events (cycle=0x7ff03cffbce0, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:90025 0x000000000046c947 in ngx_process_events_and_timers (cycle=cycle@entry=0x7ff03cffbce0) at src/event/ngx_event.c:24226 0x0000000000473d35 in ngx_worker_process_cycle (cycle=cycle@entry=0x7ff03cffbce0, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:75327 0x0000000000472820 in ngx_spawn_process (cycle=cycle@entry=0x7ff03cffbce0, proc=0x473cf0 <ngx_worker_process_cycle>, data=0x0, name=0x6e22f5 "worker process", respawn=respawn@entry=0) at src/os/unix/ngx_process.c:19828 0x0000000000475216 in ngx_reap_children (cycle=0x7ff03cffbce0) at src/os/unix/ngx_process_cycle.c:62129 ngx_master_process_cycle (cycle=cycle@entry=0x7ff03cffbce0) at src/os/unix/ngx_process_cycle.c:17430 0x000000000044e139 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367
断点
1 breakpoint keep y 0x000000000048c420 in ngx_http_process_request_headers at src/http/ngx_http_request.c:1185 breakpoint already hit 13 times2 breakpoint keep y 0x0000000000504ba0 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:154 breakpoint already hit 13 times3 breakpoint keep y 0x0000000000504c98 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:1894 breakpoint keep y 0x0000000000504c19 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:222 breakpoint already hit 8 times5 breakpoint keep y 0x0000000000505482 in ngx_http_lua_content_handler_file at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:284 breakpoint already hit 8 times6 breakpoint keep y 0x0000000000505024 in ngx_http_lua_content_by_chunk at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:54 breakpoint already hit 8 times7 breakpoint keep y 0x00000000005050e0 in ngx_http_lua_content_by_chunk at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:120 breakpoint already hit 8 times8 breakpoint keep y 0x00000000005014d3 in ngx_http_lua_run_thread at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:1005 breakpoint already hit 22 times9 breakpoint keep y 0x0000000000501f58 in ngx_http_lua_run_thread at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:1476 breakpoint already hit 8 times10 breakpoint keep y 0x00000000004ffc4a in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:48411 breakpoint keep y 0x00000000004ffb63 in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:557 breakpoint already hit 7 times12 breakpoint keep y 0x00000000004ffb86 in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:523 breakpoint already hit 6 times13 breakpoint keep y 0x00000000004ffc2b in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:569 breakpoint already hit 6 times14 breakpoint keep y 0x0000000000489e1b in ngx_http_send_special at src/http/ngx_http_request.c:3358 breakpoint already hit 6 times15 breakpoint keep y 0x0000000000485a2b in ngx_http_output_filter at src/http/ngx_http_core_module.c:1979 breakpoint already hit 30 times16 breakpoint keep y 0x0000000000506c70 in ngx_http_lua_capture_body_filter at ../ngx_lua-0.10.6/src/ngx_http_lua_capturefilter.c:133 breakpoint already hit 30 times17 breakpoint keep y 0x00000000004b2ce1 in ngx_http_charset_body_filter at src/http/modules/ngx_http_charset_filter_module.c:647 breakpoint already hit 30 times18 breakpoint keep y 0x00000000004afd89 in ngx_http_ssi_body_filter at src/http/modules/ngx_http_ssi_filter_module.c:411 breakpoint already hit 25 times19 breakpoint keep y 0x00000000004ad41d in ngx_http_postpone_filter at src/http/ngx_http_postpone_filter_module.c:82 breakpoint already hit 25 times20 breakpoint keep y 0x00000000004ac7d0 in ngx_http_gzip_body_filter at src/http/modules/ngx_http_gzip_filter_module.c:326 breakpoint already hit 25 times21 breakpoint keep y 0x00000000004a908f in ngx_http_chunked_body_filter at src/http/modules/ngx_http_chunked_filter_module.c:224 breakpoint already hit 25 times22 breakpoint keep y 0x00000000004a7cff in ngx_http_write_filter at src/http/ngx_http_write_filter_module.c:254 breakpoint already hit 5 times23 breakpoint keep y 0x00000000004769e1 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:92 breakpoint already hit 4 times24 breakpoint keep y 0x0000000000476957 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:49 breakpoint already hit 4 times25 breakpoint keep y 0x00000000004b4f8a in ngx_http_copy_filter at src/http/ngx_http_copy_filter_module.c:152 breakpoint already hit 5 times26 breakpoint keep y 0x000000000045338b in ngx_output_chain at src/core/ngx_output_chain.c:74 breakpoint already hit 5 times27 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time28 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time29 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:2130 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time31 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time32 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time33 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time34 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time35 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time36 breakpoint keep y 0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215 breakpoint already hit 1 time
使用 GDB 调试 OpenResty
加断点
比如 ngx_http_lua_ffi_worker_pid
local ffi = require 'ffi'local C = ffi.Cffi.cdef[[int ngx_http_lua_ffi_worker_pid(void);]]local _M = { _VERSION = '0.10' }local mt = { __index = _M }function _M.new(self) local _m_name = 'weibo.com' return setmetatable({ m_name = _m_name }, mt)endfunction _M.getWorkerPid(self) return C.ngx_http_lua_ffi_worker_pid()endreturn _M
运行调用这个包的代码,则可以在 GDB 中加断点 ngx_http_lua_ffi_worker_pid
即可直接追踪 OpenResty 的大致运行过程。通过 c
继续下一步
breakpoints
ngx_http_process_request_headersngx_http_lua_content_handlerngx_http_lua_content_handler_filengx_http_lua_content_by_chunkngx_http_lua_run_threadngx_http_lua_send_chain_linkngx_http_send_specialngx_http_output_filterngx_http_lua_capture_body_filterngx_http_charset_body_filterngx_http_ssi_body_filterngx_http_postpone_filterngx_http_gzip_body_filterngx_http_chunked_body_filterngx_http_write_filterngx_linux_sendfile_chainngx_http_copy_filterngx_output_chainngx_linux_sendfile_chain