注，原文來自 https://blog.mygraphql.com/zh... 。如你看到的轉載圖片不清，請回到原文。
為何要了解 upstream/downstream 事件驅動協作下的 HTTP 反向代理流程
想不到還真堅持寫到 Part4 了。不管你信不信，每一 part 的“為何”一節，是我覺得最難寫的 :) 。如果你是第一次讀這個系列，不用擔心，每一節都是比較獨立的。
不出意外的話，你之前看到的 Envoy 簡介是這樣描述 Envoy 的特性的：
- C++ 編寫，原生底層，沒有 GC stop the world，所以效能優異
- 非同步事件驅動，多路複用，完美解決 C10k problem
- 由於單執行緒負責多連線，在大連線量時，減少了大量執行緒的記憶體開銷，和 CPU 上下文切換開銷
這些描述，當然有其合理性。但很多遠看很美麗的事物，在微距放大後，可能有很多有意思，有價值的東西。我相信只要分析得足夠細緻，總可能針對我們實際執行環境和流量特性對作一些有意義的優化。可能只是修改一個 Envoy / Kernel 的配置，也可能是修改一行 Envoy 的程式碼。或你的應用的流程相關行為，如每次寫 socket 的 buffer 資料包的大小。
而這些，都需要建立在瞭解實現細節的基礎上。除非覺得運氣或經驗值特別好，可以瞎猜。
逆向工程與雲原生現場分析 系列介紹
開始前先做個預告，逆向工程與雲原生現場分析 系列（將）包括：
- Part 1: eBPF 跟蹤 Istio/Envoy 之學步
- Part 2: eBPF 跟蹤 Istio/Envoy 之啟動、監聽與執行緒負載均衡
- Part 3: eBPF 跟蹤 Istio/Envoy 之 downstream TCP 連線 accept、TLS 握手 與 filter_chain 的選擇
- Part 4: eBPF 跟蹤 Istio/Envoy 之 upstream/downstream 事件驅動協作下的 HTTP 反向代理流程 （本文）
- Part 5: eBPF 跟蹤 Istio/Envoy 之 L3/4 Network Fitler 互動機制
- Part 6: eBPF 跟蹤 Istio/Envoy 之 http filter
- Part 7: eBPF 跟蹤 Istio/Envoy 之 http router
- Part 8: eBPF 跟蹤 Istio/Envoy 之 cluster、 connection pool 與 outbound load balance
在系列中，我將演示如何讓 bpftrace "讀" 懂執行期的由 C++ 11 編寫成的 envoy 程式中的物件資料。
為免嚇跑人，還是老套路，多圖少程式碼，不過有的圖有點點複雜。程式設計師大叔開始講故事了。?
HTTP 反向代理的總流程
整體看，Socket 事件驅動的 HTTP 反向代理總流程如下：
圖中看出，有4種事件驅動了整個流程。後面幾節會逐個分析。
為免一下子進入各個步驟細節而讓人迷途，現在先看看所有步驟的總流程：
圖：Istio/Envoy 模組協作總圖
Downstream Read Request 模組協作
圖：Downstream Read-Ready 模組協作
大概說明一下流程：
- upstream socket 可讀回撥
- Http::ConnectionManagerImpl 讀取 socket，增量放入 Http1::ConnectionImpl
- Http1::ConnectionImpl 呼叫 nghttp2 增量解釋 HTTP 請求
- 如果 nghttp2 認為已經 完整讀取了 HTTP Request 請求，則呼叫
Http::ServerConnection::onMessageCompleteBase() Http::ServerConnection::onMessageCompleteBase()首先 停止 downstream ReadReady 監聽Http::ServerConnection::onMessageCompleteBase()呼叫Http::FilterManager，發起http filter chain的 decodeHeaders 迭代流程- 一般，
http filter chain的最後一個 http filter 是Router::Filter，Router::Filter::decodeHeaders()被呼叫 Router::Filter::decodeHeaders()的邏輯就見下圖了。
圖：Downstream Request Router 模組協作
大概說明一下流程：
Router::Filter，Router::Filter::decodeHeaders()被呼叫- 根據配置的 Router 規則，匹配到 Cluster
- 如 Cluster 連線池物件不存在，則新建
- 新建
Envoy::Router::UpstreamRequest物件。 - 呼叫
Envoy::Router::UpstreamRequest::encodeHeaders(bool end_stream)， encode HTTP header - 經過一系列的負載均衡演算法，匹配到 upstream 的 host(endpoint)
發現到選定的 upstream host 的連線不足，則：
- 開啟一新的 socket fd(未連線)
- 註冊 upstream socket FD 的 WriteReady / Connected 事件。 準備在事件回撥時寫 upstream request
- 用 socket fd 發起到 upstream host 的非同步連線請求
- 關聯 downstream 與 upstream fd
Upstream Write Request 模組協作
大概說明一下流程：
- upstream socket write ready 回撥
- 發現是連線成功回撥，關聯 upstream socket 到
ConnectionPool::ActiveClient - upstream socket write ready 回撥
- 發現是連線可寫，寫入 upstream HTTP request
Upstream Read Response 模組協作
Downstream Write Response 模組協作
BPF 指令碼輸出
以上圖片，不是隻讀原始碼就得出的。是需要通過 bpf 埋點觀察：
- 記錄 downstream FD(socket 的 file descriptor)，可以認為是程式中的 socket id
- 埋點程式級和 kernel 級的相關函式入口和出口，觀察其出入參與 stack
- 關聯起 downstream FD 與 upstream FD
其中當然有很多細節，不過我不打算一個個講了，實在有需要用，想深入瞭解的同學可以聯絡我探討。下一節的 bpf 程式碼是最好的細節。
https://github.com/labilezhu/...
#### Downstream Read Request: accept downstream socket， 可見 downstream socket 的 FD=41
OS handshaked TCP:
01:22:11 4215 wrk:worker_0 172.30.207.129 58708 172.21.206.232 15006 0/4096
sys_exit_accept4 fd=41
accept4+96
Envoy::Network::IoSocketHandleImpl::accept(sockaddr*, unsigned int*)+82
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+216
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1621361178: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__110__function6__funcIZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS2_7Network21ListenerFilterMatcherEEERNS2_6Server13Configuration22ListenerFactoryContextEEUlRNSD_21ListenerFilterManagerEE_NS_9allocatorISO_EEFvSN_EEclESN_
***** elapsed=1621785490: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy10Extensions15ListenerFilters12TlsInspector25TlsInspectorConfigFactory36createListenerFilterFactoryFromProtoERKN6google8protobuf7MessageERKNS_10shared_ptrINS3_7Network21ListenerFilterMatcherEEERNS3_6Server13Configuration22ListenerFactoryContextEEUlRNSE_21ListenerFilterManagerEE_SO_EEEvDpOT_
***** elapsed=1630898092: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC1ENSt3__110shared_ptrINS2_6ConfigEEE
***** elapsed=1630908037: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterC2ENSt3__110shared_ptrINS2_6ConfigEEE
***** elapsed=1631036930: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter8onAcceptERNS_7Network23ListenerFilterCallbacksE
***** elapsed=1631045047: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter6onReadEv
***** elapsed=1631070582: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_recvfrom, fd=41, ret=89
recv+108
***** elapsed=1631085065: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6Filter16parseClientHelloEPKvm
#### Downstream Read Request: 選擇 network filter chain
***** elapsed=1631145371: tid=5327,comm=wrk:worker_0: TlsInspector*, probe=uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy10Extensions15ListenerFilters12TlsInspector6FilterD0Ev
comm:wrk:worker_0,tid:5327: Got setFilterChainName=0.0.0.0_8080, lenght=12
Envoy::StreamInfo::StreamInfoImpl::setFilterChainName(absl::string_view)+0
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
#### Downstream Read Request: 監聽 upstream FD 事件
***** elapsed=1631199672: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
Envoy::Network::ServerConnectionImpl::ServerConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+107
Envoy::Event::DispatcherImpl::createServerConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&)+70
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+307
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0 : setsockopt: level=6, fd=41, optname=1, optval=1, optlen=4.
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl9addFilterENSt3__110shared_ptrINS0_6FilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+0
std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
virtual thunk to Envoy::Network::ConnectionImpl::addFilter(std::__1::shared_ptr<Envoy::Network::Filter>)+62
std::__1::__function::__func<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0, std::__1::allocator<Envoy::Tcp::MetadataExchange::(anonymous namespace)::createFilterFactoryHelper(envoy::tcp::metadataexchange::config::MetadataExchange const&, Envoy::Server::Configuration::CommonFactoryContext&, Envoy::Tcp::MetadataExchange::FilterDirection)::$_0>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+93
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
std::__1::__function::__func<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7, std::__1::allocator<Envoy::Extensions::NetworkFilters::HttpConnectionManager::HttpConnectionManagerFilterConfigFactory::createFilterFactoryFromProtoTyped(envoy::extensions::filters::network::http_connection_manager::v3::HttpConnectionManager const&, Envoy::Server::Configuration::FactoryContext&)::$_7>, void (Envoy::Network::FilterManager&)>::operator()(Envoy::Network::FilterManager&)+353
Envoy::Server::Configuration::FilterChainUtility::buildFilterChain(Envoy::Network::FilterManager&, std::__1::vector<std::__1::function<void (Envoy::Network::FilterManager&)>, std::__1::allocator<std::__1::function<void (Envoy::Network::FilterManager&)> > > const&)+50
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
Envoy::Server::ActiveTcpListener::newConnection(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::StreamInfo::StreamInfo, std::__1::default_delete<Envoy::StreamInfo::StreamInfo> >)+774
Envoy::Server::ActiveTcpSocket::newConnection()+377
Envoy::Server::ActiveTcpSocket::continueFilterChain(bool)+107
Envoy::Server::ActiveTcpListener::onAcceptWorker(std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, bool, bool)+163
Envoy::Network::TcpListenerImpl::onSocketEvent(short)+856
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
***** elapsed=1631443267: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1631456738: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=31393726, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=100
***************************
#### Downstream Read Request: 讀 HTTP Request
***** elapsed=1631475016: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET
***** elapsed=1631513821: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=89
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1631527775: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=-11
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl6onReadEv,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+0
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl17onContinueReadingEPNS1_16ActiveReadFilterERNS0_16ReadBufferSourceE,FilterManagerImpl.this=0x559f98d38758,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98c07200,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
***** elapsed=1631671079: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
#### Downstream Read Request: downstream 的 HTTP Reuest 已經完全讀取（onMessageComplete()）。停止監聽 downstream 的 ReadReady 事件
***** elapsed=1631678652: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80002004
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Network::ConnectionImpl::readDisable(bool)+1077
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+86
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
#### Downstream Read Request: 監聽新建立的 upstream socket 的 FD=44 ， 這裡 socket 還處於未連線狀態。現在監聽 upstream socket 的 所有事件。以備後面非同步連線成功時收到回撥
***** elapsed=1632388633: tid=5327,comm=wrk:worker_0: register upstream event trigger:sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=44, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+362
Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+284
Envoy::Network::IoSocketHandleImpl::initializeFileEvent(Envoy::Event::Dispatcher&, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)+126
Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)+1026
Envoy::Network::ClientConnectionImpl::ClientConnectionImpl(Envoy::Event::Dispatcher&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+283
Envoy::Event::DispatcherImpl::createClientConnection(std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&)+75
Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterInfo const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const> const&, Envoy::Network::TransportSocketFactory&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>)+561
Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+126
non-virtual thunk to Envoy::Upstream::HostImpl::createConnection(Envoy::Event::Dispatcher&, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const>) const+21
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+112
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.add*Filter|FilterManagerImpl::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl13addReadFilterENSt3__110shared_ptrINS0_10ReadFilterEEE,FilterManagerImpl.this=0x559f98f08478,fd=41
Envoy::Network::FilterManagerImpl::addReadFilter(std::__1::shared_ptr<Envoy::Network::ReadFilter>)+0
Envoy::Http::CodecClient::CodecClient(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&)+357
Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+70
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0 : setsockopt: level=6, fd=44, optname=1, optval=1, optlen=4.
#### Downstream Read Request: 發現非同步 upstream 連線
connecting to 8080:
01:23:36 4215 wrk:worker_0 127.0.0.6 36383 172.21.206.232 8080
connect+75
Envoy::Network::IoSocketHandleImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+104
Envoy::Network::SocketImpl::connect(std::__1::shared_ptr<Envoy::Network::Address::Instance const>)+64
Envoy::Network::ClientConnectionImpl::connect()+650
Envoy::Http::CodecClient::connect()+543
Envoy::Http::CodecClientProd::CodecClientProd(Envoy::Http::CodecType, std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> >&&, std::__1::shared_ptr<Envoy::Upstream::HostDescription const>, Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&)+694
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_1>, std::__1::unique_ptr<Envoy::Http::CodecClient, std::__1::default_delete<Envoy::Http::CodecClient> > (Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase*&&)+94
Envoy::Http::FixedHttpConnPoolImpl::createCodecClient(Envoy::Upstream::Host::CreateConnectionData&)+35
Envoy::Http::ActiveClient::initialize(Envoy::Upstream::Host::CreateConnectionData&, Envoy::Http::HttpConnPoolImplBase&)+113
Envoy::Http::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&, unsigned int, unsigned int)+171
Envoy::Http::Http1::ActiveClient::ActiveClient(Envoy::Http::HttpConnPoolImplBase&)+69
std::__1::__function::__func<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0, std::__1::allocator<Envoy::Http::Http1::allocateConnPool(Envoy::Event::Dispatcher&, Envoy::Random::RandomGenerator&, std::__1::shared_ptr<Envoy::Upstream::Host const>, Envoy::Upstream::ResourcePriority, std::__1::shared_ptr<std::__1::vector<std::__1::shared_ptr<Envoy::Network::Socket::Option const>, std::__1::allocator<std::__1::shared_ptr<Envoy::Network::Socket::Option const> > > > const&, std::__1::shared_ptr<Envoy::Network::TransportSocketOptions const> const&, Envoy::Upstream::ClusterConnectivityState&)::$_0>, std::__1::unique_ptr<Envoy::ConnectionPool::ActiveClient, std::__1::default_delete<Envoy::ConnectionPool::ActiveClient> > (Envoy::Http::HttpConnPoolImplBase*)>::operator()(Envoy::Http::HttpConnPoolImplBase*&&)+40
Envoy::Http::FixedHttpConnPoolImpl::instantiateActiveClient()+35
Envoy::ConnectionPool::ConnPoolImplBase::tryCreateNewConnection(float)+1233
Envoy::ConnectionPool::ConnPoolImplBase::newStream(Envoy::ConnectionPool::AttachContext&)+2001
non-virtual thunk to Envoy::Http::HttpConnPoolImplBase::newStream(Envoy::Http::ResponseDecoder&, Envoy::Http::ConnectionPool::Callbacks&)+47
Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::newStream(Envoy::Router::GenericConnectionPoolCallbacks*)+70
Envoy::Router::Filter::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)+14120
Envoy::Http::FilterManager::decodeHeaders(Envoy::Http::ActiveStreamDecoderFilter*, Envoy::Http::RequestHeaderMap&, bool)+334
Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::RequestHeaderMap, std::__1::default_delete<Envoy::Http::RequestHeaderMap> >&&, bool)+6139
Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase()+279
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7959
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+21
Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool)+76
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
comm:wrk:worker_0,tid:5327: FilterManagerImpl.on*::uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy7Network17FilterManagerImpl16ActiveReadFilter10connectionEv,FilterManagerImpl.this=0x559f98996380,fd=41
***** elapsed=1632608064: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1632618881: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1145219, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=92
***************************
***** elapsed=1632635526: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET
***** elapsed=1632654517: tid=5327,comm=wrk:worker_0: END:EventFired
#### Upstream Write Request: 非同步 upstream 連線成功，回撥。
***** elapsed=1632659321: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET
comm:wrk:worker_0,tid:5327: ConnPoolImplBase*attachStreamToClient: uprobe:/proc/4215/root/usr/local/bin/envoy:_ZN5Envoy14ConnectionPool16ConnPoolImplBase20attachStreamToClientERNS0_12ActiveClientERNS0_13AttachContextE
Envoy::ConnectionPool::ConnPoolImplBase::attachStreamToClient(Envoy::ConnectionPool::ActiveClient&, Envoy::ConnectionPool::AttachContext&)+0
Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent(Envoy::ConnectionPool::ActiveClient&, absl::string_view, Envoy::Network::ConnectionEvent)+497
Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent)+59
Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+511
Envoy::Network::ConnectionImpl::onWriteReady()+1648
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
#### Upstream Write Request: 向 upstream socket 寫 HTTP Request
***** elapsed=1632741863: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=44, ret=262
writev+77
Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
Envoy::Network::ConnectionImpl::onWriteReady()+1876
Envoy::Network::ConnectionImpl::raiseEvent(Envoy::Network::ConnectionEvent)+529
Envoy::Network::ConnectionImpl::onWriteReady()+1648
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1632756069: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1632764448: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=129648, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=60
***************************
#### Upstream Read Response: upstream socket ReadReady 回撥
***** elapsed=1637821708: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET
#### Upstream Read Response: 讀 upstream HTTP Response
***** elapsed=1637898855: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=4511
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1637915998: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=44, ret=-11
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1638220184: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80002004
#### Upstream Read Response: 已經完整讀取 upstream HTTP Response，開始監聽 downstream 的 WriteReady 事件，準備寫 downstream HTTP Response
***** elapsed=1638231319: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_ADD, fd=41, events=0x80000005
EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)
EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)
EPOLL_CTL_ADD/MOD EdgeTrigger
epoll_ctl+14
epoll_nochangelist_add+54
evmap_io_add_+421
event_add_nolock_+603
event_add+54
Envoy::Network::ConnectionImpl::readDisable(bool)+938
Envoy::Http::Http1::StreamEncoderImpl::~StreamEncoderImpl()+112
non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onEncodeComplete()+54
Envoy::Http::Http1::StreamEncoderImpl::endEncode()+166
Envoy::Http::Http1::StreamEncoderImpl::encodeData(Envoy::Buffer::Instance&, bool)+340
Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData(Envoy::Buffer::Instance&, bool)+679
Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState)+2138
Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool)+230
Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool)+59
Envoy::Http::Http1::ClientConnectionImpl::onMessageCompleteBase()+619
Envoy::Http::Http1::ConnectionImpl::onMessageComplete()+637
Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl(http_parser_type, void*)::{lambda(http_parser*)#3}::__invoke(http_parser*)+31
http_parser_execute+7705
Envoy::Http::Http1::LegacyHttpParserImpl::execute(char const*, int)+31
Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long)+52
Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)+1151
Envoy::Http::Http1::ClientConnectionImpl::dispatch(Envoy::Buffer::Instance&)+29
Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&)+48
Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool)+21
Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&)+303
Envoy::Network::ConnectionImpl::onReadReady()+1622
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
***** elapsed=1638898463: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1638934709: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=1123611, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56
***************************
***** elapsed=1638955421: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98eb6540, fd=44, events=0x24
libevent: EV_WRITE
libevent: EV_ET
***** elapsed=1638975366: tid=5327,comm=wrk:worker_0: END:EventFired
#### Downstream Write Response: downstream 的 WriteReady 事件回撥。
***** elapsed=1638980295: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x24
libevent: EV_WRITE
libevent: EV_ET
#### Downstream Write Response: 寫 downstream HTTP Response
***** elapsed=1639253752: tid=5327,comm=wrk:worker_0: socket_write, probe=tracepoint:syscalls:sys_exit_writev, fd=41, ret=4631
writev+77
Envoy::Network::IoSocketHandleImpl::writev(Envoy::Buffer::RawSlice const*, unsigned long)+263
Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&)+107
Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool)+121
Envoy::Network::ConnectionImpl::onWriteReady()+1876
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+818
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1639293010: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1639308346: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=354486, tid2epollNrFdReady=2
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=56
***************************
***** elapsed=1643785802: tid=5327,comm=wrk:worker_0: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()
FileEventImpl*=0x559f98e26ee0, fd=41, events=0x26
libevent: EV_READ
libevent: EV_WRITE
libevent: EV_ET
***** elapsed=1643855427: tid=5327,comm=wrk:worker_0: socket_read, probe=tracepoint:syscalls:sys_exit_readv, fd=41, ret=0
readv+77
Envoy::Network::IoSocketHandleImpl::readv(unsigned long, Envoy::Buffer::RawSlice*, unsigned long)+247
Envoy::Network::IoSocketHandleImpl::read(Envoy::Buffer::Instance&, absl::optional<unsigned long>)+167
Envoy::Network::RawBufferSocket::doRead(Envoy::Buffer::Instance&)+136
Envoy::Network::ConnectionImpl::onReadReady()+753
Envoy::Network::ConnectionImpl::onFileEvent(unsigned int)+879
std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&)+65
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*)+92
0x7fffffffe000
event_base_loop+1953
Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__1::function<void ()> const&)+621
Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, absl::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*)+19
start_thread+217
***** elapsed=1643880789: tid=5327,comm=wrk:worker_0: sys_enter_epoll_ctl, epfd=10, op=EPOLL_CTL_DEL, fd=41, events=0x80000005
sys_enter_close fd=41
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98d38758
delete fd2filterManagerImpl, fd=41
delete fd2filterManagerImpl=0x559f98996380
***** elapsed=1644006858: tid=5327,comm=wrk:worker_0: END:EventFired
******* WAKE-ROUND:END Summary *******
***** elapsed=1644056452: tid=5327,comm=wrk:worker_0: sys_enter_epoll_wait, runableDuaration=289169, tid2epollNrFdReady=1
*** last_epoll_wait_args: epfd=10, events=-1741049344, maxevents=32, timeout=52
***************************
^C
@epoll_ctl_op[1]: EPOLL_CTL_ADD
@epoll_ctl_op[2]: EPOLL_CTL_DEL
@epoll_ctl_op[3]: EPOLL_CTL_MOD
@fd2sockopt[44, 6, 1, 1]: 1
@fd2sockopt[41, 6, 1, 1]: 1
@filterManagerImpl2fd[94143954060408]: 41
@filterManagerImpl2fd[94143950909952]: 41BPF 指令碼
https://github.com/labilezhu/...
#!usr/bin/bpftrace
/*
IMPORT-ENV: $ENVOY_PID
args: $1=ENVOY_PID
*/
/*
#export PODID=`sudo crictl pods | grep fortio-server | egrep '\bReady' | awk '{print $1}'`
#export CONTAINERID=`sudo crictl ps | grep $PODID | grep istio-proxy | awk '{print $1}'`
kubectl -n idm-mark exec -it fortio-server -c istio-proxy -- bash -c 'ls -l /proc/`pgrep envoy`/ns/pid'
lrwxrwxrwx 1 istio-proxy istio-proxy 0 Mar 31 09:18 /proc/15/ns/pid -> 'pid:[4026532799]'
pgrep envoy | tee /dev/fd/2 | xargs -L 1 -I '{}' sudo ls -l /proc/'{}'/ns
export PID=4215
export SCRIPT_HOME=`pwd`
export bpftrace_image=cndt-bcc-ub
docker run -it --rm --init --privileged --name bpftrace -h bpftrace \
--pid host \
--net host \
-e SCRIPT_HOME=$SCRIPT_HOME \
-e PID=$PID \
-e ENVOY_PID=$PID \
-e BT=trace-envoy-filter-router.bt \
-v /etc/localtime:/etc/localtime:ro \
-v /sys:/sys:rw \
-v /usr/src:/usr/src:rw \
-v /lib/modules:/lib/modules:ro \
-v ${SCRIPT_HOME}:${SCRIPT_HOME}:rw \
$bpftrace_image \
${SCRIPT_HOME}/warp-bt.sh $PID
*/
/*
1. BPF Map 說明:
1.1. range tid map
- @watchedWakeRound[tid]=tid
- sys_exit_accept4
- FileEventImpl*assignEvents*
- @fdFired[tid]=$fd;
- uprobe: *FileEventImpl*assignEvents*
- uretprobe: *FileEventImpl*assignEvents*
1.2. fd map
- @fd2sockpair[$fd]=@sockpair[tid]
- sys_exit_accept4
- sys_enter_close
- @fd2sockopt[$fd, $level, $optname, $optval_int] = 1
- sys_enter_setsockopt
- @filterManagerImpl2fd[arg0] = @fdFired[tid] // filterManagerImpl to downstream fd
- @fd2filterManagerImpl[$fd] = arg0
- @fd2filterManagerImpl2[$fd] = arg0
- @downFD2upFD[$downFD] = $fd; //downstream fd to upstream fd
- sys_enter_epoll_ctl
- @upFD[$fd] = $fd; //upstream fd list
- sys_enter_epoll_ctl
1.3. temp tid map
- @sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
- @sockpair_exist[tid]=1
- @watchedIo[tid]=fd
*/
#include <linux/in.h>
#include <linux/in6.h>
#include <linux/socket.h>
#include <net/sock.h>
// #include <sys/epoll.h>
// The event argument describes the object linked to the file descriptor fd. The struct epoll_event is defined as:
// struct epoll_data_t {
// int fd;
// };
// struct epoll_event {
// //a bit mask composed by ORing together zero or more of the following available event types
// uint32_t events; /* Epoll events */
// struct epoll_data_t data; /* User data variable */
// };
BEGIN
{
printf("Tracing Envoy. Hit Ctrl-C to end.\n");
printf("#define EPOLL_CTL_ADD 1\n #define EPOLL_CTL_DEL 2\n #define EPOLL_CTL_MOD 3\n");
@epoll_ctl_op[1]="EPOLL_CTL_ADD";
@epoll_ctl_op[2]="EPOLL_CTL_DEL";
@epoll_ctl_op[3]="EPOLL_CTL_MOD";
}
/*
獲取 accept 連線時的本地和對端地址。並標記本輪 wakeup 需要監控
*/
kretprobe:inet_csk_accept
/pid==$1 /
{
$sk = (struct sock *)retval;
$inet_family = $sk->__sk_common.skc_family;
if ($inet_family == AF_INET || $inet_family == AF_INET6) {
// initialize variable type:
$daddr = ntop(0);
$saddr = ntop(0);
if ($inet_family == AF_INET) {
$daddr = ntop($sk->__sk_common.skc_daddr);
$saddr = ntop($sk->__sk_common.skc_rcv_saddr);
} else {
printf("not support IPv6.\n");
return;
}
$lport = $sk->__sk_common.skc_num;
// printf("accept(), port=%d\n", $lport);
//only watch listen port 15006
if( 15006 != $lport ) {
return;
}
$dport = $sk->__sk_common.skc_dport;
$qlen = $sk->sk_ack_backlog;
$qmax = $sk->sk_max_ack_backlog;
// Destination port is big endian, it must be flipped
$dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);
printf("OS handshaked TCP:\n");
time("%H:%M:%S ");
printf("%-6d %-14s ", pid, comm);
printf("%-39s %-5d %-39s %-5d ", $daddr, $dport, $saddr,
$lport);
printf("%d/%d\n", $qlen, $qmax);
@sockpair[tid]=($sk->__sk_common.skc_daddr, $dport, $sk->__sk_common.skc_rcv_saddr, $lport);
@sockpair_exist[tid]=1;
@watchedWakeRound[tid]=tid;
}
}
/**
* 獲取新連線的 FD 。並標記本輪 libevent 回撥需要監控
*/
tracepoint:syscalls:sys_exit_accept4
/pid==$1 && @sockpair_exist[tid] /
{
$fd = args->ret;
if( $fd < 0 ) {
return;
}
printf("sys_exit_accept4 fd=%d\n", $fd);
@fd2sockpair[$fd]=@sockpair[tid];
@fdFired[tid]=$fd;
delete(@sockpair[tid]);
delete(@sockpair_exist[tid]);
printf("%s \n", ustack());
}
/**
* 記錄 FD 的 sockopt
*/
tracepoint:syscalls:sys_enter_setsockopt
/pid==$1/
{
// socket opts: https://elixir.bootlin.com/linux/v5.16.3/source/include/uapi/linux/tcp.h#L92
$level = args->level;
$fd = args->fd;
if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
$optname = args->optname;
$optval = args->optval;
$optval_int = *$optval;
$optlen = args->optlen;
// printf("\n########## setsockopt() ##########\n");
printf("comm:%-16s: setsockopt: level=%d, fd=%d, optname=%d, optval=%d, optlen=%d. \n", comm, $level, $fd, $optname, $optval_int, $optlen);
@fd2sockopt[$fd, $level, $optname, $optval_int] = 1;
}
}
/*
記錄 epoll 監聽的 FD
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_ctl/format
*/
tracepoint:syscalls:sys_enter_epoll_ctl
/pid==$1/
{
$fd=args->fd;
$epollEvents = (uint32) (args->event->events);
$op = args->op;
if( @fd2sockpair[$fd].0 ) {
printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n",
elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);
if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
}
if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
}
if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
}
printf("%s\n", ustack(30));
}
} else {
$downFD = @fdFired[tid];
if( $downFD ) {//in downstream fd event callback, register upstream event trigger
@upFD[$fd] = $fd;
@downFD2upFD[$downFD] = $fd;
printf("***** elapsed=%d: tid=%d,comm=%s: register upstream event trigger:sys_enter_epoll_ctl, epfd=%d, op=%s, fd=%d, events=0x%x\n",
elapsed, tid, comm, args->epfd, @epoll_ctl_op[$op], $fd, $epollEvents);
if( $op == 1/*EPOLL_CTL_ADD*/ || $op == 3/*EPOLL_CTL_MOD*/ ) { //add watch epoll event
if( $epollEvents & (uint32)0x001 /*EPOLLIN*/ ) {
printf("EPOLL_CTL_ADD/MOD ReadReady(EPOLLIN)\n");
}
if( $epollEvents & (uint32)0x004 /*EPOLLOUT*/ ) {
printf("EPOLL_CTL_ADD/MOD WriteReady(EPOLLOUT)\n");
}
if( $epollEvents & (1u << 31) /*EPOLLET*/ ) {
printf("EPOLL_CTL_ADD/MOD EdgeTrigger\n");
}
printf("%s\n", ustack(60));
}
}
}
}
/*
結束本輪的 wakeRound/runnableRound，並等待下一輪
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_epoll_wait/format
*/
tracepoint:syscalls:sys_enter_epoll_wait
/pid==$1/
{
if( @watchedWakeRound[tid] ) {
$runnableStartTime=@tid2Waketime[tid];
if( $runnableStartTime ) {
$runableDuaration = elapsed - $runnableStartTime;
printf("\n******* WAKE-ROUND:END Summary *******\n");
printf("***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait, runableDuaration=%d, tid2epollNrFdReady=%d\n",
elapsed, tid, comm, $runableDuaration, @tid2epollNrFdReady[tid]);
$tid_last_epoll_wait_args = @last_epoll_wait_args[tid];
if( $tid_last_epoll_wait_args.0 ) {
printf("*** last_epoll_wait_args: epfd=%d, events=%d, maxevents=%d, timeout=%d \n",
$tid_last_epoll_wait_args.0, $tid_last_epoll_wait_args.1, $tid_last_epoll_wait_args.2, $tid_last_epoll_wait_args.3);
}
printf("***************************\n\n");
}
}
delete(@tid2Waketime[tid]);
delete(@watchedWakeRound[tid]);
delete(@tid2epollNrFdReady[tid]);
//read in next sys_enter_epoll_wait
@last_epoll_wait_args[tid]=(args->epfd, args->events, args->maxevents, args->timeout);
}
/*
開始本輪 wakeRound/runnableRound
cat /sys/kernel/debug/tracing/events/syscalls/sys_exit_epoll_wait/format
*/
tracepoint:syscalls:sys_exit_epoll_wait
/pid==$1/
{
// printf("\n***** elapsed=%d: tid=%d,comm=%s: sys_enter_epoll_wait\n", elapsed, tid, comm);
// printf("epfd: 0x%08lx, events: 0x%08lx, maxevents: 0x%08lx, timeout: 0x%08lx \n",
// ((args->epfd)), ((args->events)), ((args->maxevents)), ((args->timeout)) );
@tid2Waketime[tid]=elapsed;
@tid2epollNrFdReady[tid]=args->ret;
}
/*
記錄本輪 epoll 事件觸發的 FD 級回撥開始
Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int fd, short events, void* fileEventImplThis) !!!NOT!!: FileEventImpl::assignEvents()
C++11 Lambda expressions:
event_assign( &raw_event_, base, fd_, xxx, [](evutil_socket_t, short what, void* arg)-> void {}, this );
event_assign(): https://libevent.org/doc/event_8h.html#a3e49a8172e00ae82959dfe64684eda11
event_assign ( struct event * ev,
struct event_base * base,
evutil_socket_t fd,
short events,
event_callback_fn callback,
void * callback_arg
)
https://libevent.org/doc/event_8h.html#aed2307f3d9b38e07cc10c2607322d758
typedef void(* event_callback_fn) (evutil_socket_t, short, void *)
fd An fd or signal
events One or more EV_* flags
arg A user-supplied argument.
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1/
{
$fd = arg0;
if( @fd2sockpair[$fd].0 || @upFD[$fd] ) {
$libevent_events = arg1;
printf("\n***** elapsed=%d: tid=%d,comm=%s: BEGIN:EventFired:FileEventImpl::assignEvents::eventCallback()\n", elapsed, tid, comm);
printf("FileEventImpl*=%p, fd=%d, events=0x%x\n",arg2, $fd, $libevent_events);
if( $libevent_events & (uint16)0x01 /*EV_TIMEOUT*/ ) {
printf("libevent: EV_TIMEOUT\n");
}
if( $libevent_events & (uint16)0x02 /*EV_TIMEOUT*/ ) {
printf("libevent: EV_READ\n");
}
if( $libevent_events & (uint16)0x04 /*EV_TIMEOUT*/ ) {
printf("libevent: EV_WRITE\n");
}
if( $libevent_events & (uint16)0x20 /*EV_TIMEOUT*/ ) {
printf("libevent: EV_ET\n");
}
if( $libevent_events & (uint16)0x80 /*EV_TIMEOUT*/ ) {
printf("libevent: EV_CLOSED\n");
}
printf("%s\n", kstack);
@fdFired[tid]=$fd;
@watchedWakeRound[tid]=tid;
}
}
/*
記錄本輪 epoll 事件觸發的 FD 級回撥結束
*/
uretprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FileEventImpl*assignEvents*
/pid == $1 && @fdFired[tid] /
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: END:EventFired\n", elapsed, tid, comm);
delete(@fdFired[tid]);
}
/*
埋點 socket 讀
*/
tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_readv,tracepoint:syscalls:sys_enter_recv*
/pid == $1 && (@fd2sockpair[args->fd].0 || @upFD[args->fd] ) /
{
@watchedIo[tid]=args->fd;
}
tracepoint:syscalls:sys_exit_read,tracepoint:syscalls:sys_exit_readv,tracepoint:syscalls:sys_exit_recv*
/pid == $1 && @watchedIo[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_read, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid],args->ret);
delete(@watchedIo[tid]);
printf("%s\n", ustack(20));
}
/*
埋點 socket 寫
*/
tracepoint:syscalls:sys_enter_write,tracepoint:syscalls:sys_enter_writev
/pid == $1 && ( @fd2sockpair[args->fd].0 || @upFD[args->fd] ) /
{
@watchedIo[tid]=args->fd;
}
tracepoint:syscalls:sys_exit_write,tracepoint:syscalls:sys_exit_writev
/pid == $1 && @watchedIo[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: socket_write, probe=%s, fd=%d, ret=%d\n", elapsed, tid, comm, probe, @watchedIo[tid], args->ret);
delete(@watchedIo[tid]);
printf("%s\n", ustack(20));
}
/*
列印 TlsInspector 的所有函式呼叫
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*TlsInspector*
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: TlsInspector*, probe=%s\n", elapsed, tid, comm, probe);
}
/*
列印連線確認協後的函式呼叫
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnectionSocketImpl*setRequestedApplicationProtocols*
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: ConnectionSocketImpl::setRequestedApplicationProtocols\n", elapsed, tid, comm);
// printf("%s", ustack);
}
//SslSocket::SslSocket
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocketC2ENSt3__110shared_ptrINS_3Ssl7ContextEEENS2_12InitialStateERKNS5_IKNS_7Network22TransportSocketOptionsEEENS4_8functionIFNS5_INS6_10HandshakerEEENS4_10unique_ptrI6ssl_stN4bssl8internal7DeleterISK_EEEEiPNS6_18HandshakeCallbacksEEEE
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// setTransportSocketCallbacks
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket27setTransportSocketCallbacksERNS_7Network24TransportSocketCallbacksE
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// SslSocket::doRead
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket6doReadERNS_6Buffer8InstanceE
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// SslSocket::doWrite
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10Extensions16TransportSockets3Tls9SslSocket7doWriteERNS_6Buffer8InstanceEb
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// SslSocket::onSuccess
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_N5Envoy10Extensions16TransportSockets3Tls9SslSocket9onSuccessEP6ssl_st
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
// SslSocket::connection
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZThn16_NK5Envoy10Extensions16TransportSockets3Tls9SslSocket10connectionEv
/pid == $1 && @fdFired[tid]/
{
printf("\n***** elapsed=%d: tid=%d,comm=%s: SslSocket*, probe=%s\n", elapsed, tid, comm, probe);
printf("%s\n", ustack(10));
}
/*
列印匹配到的 Network Fitler Chain 名字
void setFilterChainName(absl::string_view filter_chain_name)
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:_ZN5Envoy10StreamInfo14StreamInfoImpl18setFilterChainNameEN4absl11string_viewE
/pid == $1 && @fdFired[tid]/
{
/**
[Assembly 2: Calling convention](https://cs61.seas.harvard.edu/site/2018/Asm2/)
1. A structure argument that fits in a single machine word (64 bits/8 bytes) is passed in a single register.
Example: `struct small { char a1, a2; }`
2. A structure that fits in two to four machine words (16–32 bytes) is passed in sequential registers, as if it were multiple arguments.
Example: `struct medium { long a1, a2; }`
*/
$filterName = str(reg("si"));
$filterNameLength = reg("dx");
// printf("tid:%d: Got setFilterChainName=%s, lenght=%d\n %s \n", tid, $filterName, $filterNameLength, ustack );
printf("comm:%s,tid:%d: Got setFilterChainName=%s, lenght=%d\n", comm, tid, $filterName, $filterNameLength );
if( $filterNameLength > 0 ) {
printf("%s\n", ustack(3));
}
}
/*
埋點 FilterManagerImpl::add*Filter()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*add*Filter*,uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*FilterManagerImpl*
/pid == $1 && @fdFired[tid]/
{
if( @upFD[@fdFired[tid]] ) {
return;
}
$fd = @fdFired[tid];
@filterManagerImpl2fd[arg0] = $fd;
if( @fd2filterManagerImpl[$fd] ) {
@fd2filterManagerImpl2[$fd] = arg0;
} else {
@fd2filterManagerImpl[$fd] = arg0;
}
printf("comm:%s,tid:%d: FilterManagerImpl.add*Filter|FilterManagerImpl::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
printf("%s\n", ustack(40));
}
/*
埋點 FilterManagerImpl::on*()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*FilterManagerImpl*on*
/pid == $1 && @fdFired[tid]/
{
if( @upFD[@fdFired[tid]] ) {
return;
}
$fd = @fdFired[tid];
@filterManagerImpl2fd[arg0] = $fd;
if( @fd2filterManagerImpl[$fd] ) {
@fd2filterManagerImpl2[$fd] = arg0;
} else {
@fd2filterManagerImpl[$fd] = arg0;
}
printf("comm:%s,tid:%d: FilterManagerImpl.on*::%s,FilterManagerImpl.this=%p,fd=%d \n", comm, tid, probe, arg0, $fd);
}
/*
埋點 ConnPoolImplBase::attachStreamToClient()
*/
uprobe:/proc/${ENVOY_PID}/root/usr/local/bin/envoy:*ConnPoolImplBase*attachStreamToClient*
/pid == $1 && @fdFired[tid]/
{
printf("comm:%s,tid:%d: ConnPoolImplBase*attachStreamToClient: %s \n", comm, tid, probe);
printf("%s\n", ustack(40));
}
/*
埋點 socket connect upstream
*/
kprobe:tcp_connect
/pid == $1/
{
$sk = ((struct sock *)arg0);
$inet_family = $sk->__sk_common.skc_family;
if ($inet_family == AF_INET || $inet_family == AF_INET6)
{
if ($inet_family == AF_INET)
{
$daddr = ntop($sk->__sk_common.skc_daddr);
$saddr = ntop($sk->__sk_common.skc_rcv_saddr);
}
else
{
$daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
$saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
}
$lport = $sk->__sk_common.skc_num;
$dport = $sk->__sk_common.skc_dport;
// Destination port is big endian, it must be flipped
$dport = ($dport >> 8) | (($dport << 8) & 0x00FF00);
// only watch listen port 8080
if (8080 != $dport)
{
return;
}
printf("connecting to 8080:\n");
time("%H:%M:%S ");
printf("%-8d %-16s ", pid, comm);
printf("%-39s %-6d %-39s %-6d\n", $saddr, $lport, $daddr, $dport);
printf("%s\n", ustack(40));
}
}
/*
清理關閉的 FD 相關的 Map
cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_close/format
*/
tracepoint:syscalls:sys_enter_close
/pid==$1/
{
$fd = args->fd;
if( $fd < 0 ) {
return;
}
delete(@upFD[$fd]);
if( @fd2sockpair[$fd].0 ) {
printf("sys_enter_close fd=%d\n", $fd);
}
else {
return;
}
delete(@fd2sockpair[$fd]);
/////
$filterManagerImpl = @fd2filterManagerImpl[$fd];
if( $filterManagerImpl ) {
printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
delete( @fd2filterManagerImpl[$fd] );
}
if( @filterManagerImpl2fd[$filterManagerImpl] ) {
printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
delete(@filterManagerImpl2fd[$filterManagerImpl]);
}
$filterManagerImpl = @fd2filterManagerImpl2[$fd];
if( $filterManagerImpl ) {
printf("delete fd2filterManagerImpl, fd=%d\n", $fd );
delete( @fd2filterManagerImpl2[$fd] );
}
if( @filterManagerImpl2fd[$filterManagerImpl] ) {
printf("delete fd2filterManagerImpl=%p\n", $filterManagerImpl );
delete(@filterManagerImpl2fd[$filterManagerImpl]);
}
delete(@upFD[@downFD2upFD[$fd]]);
delete(@downFD2upFD[$fd]);
}
END
{
clear(@last_epoll_wait_args);
clear(@tid2Waketime);
clear(@tid2epollNrFdReady);
}結尾
這 part，從 socket 事件驅動的角度，瞭解了 envoy 作為反向代理的主要流程。我自己覺得是學習到東西的，你呢？
寫部落格的初憧是記錄自己。想不到也能通過部落格認識到很多志同道合，對技術認真之人（不能叫同志？）。謝謝你們，我才能堅持寫下去，也認為這樣的寫作有一定價值！
這是幾年前拍的老照片了，共勉。
