Recently I met a
context deadline exceeded error when using gRPC client to call
DialContext to a gRPC server, which was in a customer’s internal environment. After some investigation I found out the root cause, the gRPC client was behind an HTTP proxy and the proxy has no permission to access the gRPC server. It is a common trouble shooting, however I am interested in how we can force a program to use http or socks proxy, I will dive into it and compare the pros and cons among different solutions.
There exist many ways to let an application use a proxy, these ways can be classified into three types as follows, and I will talk about these methods briefly.
- Explicit environment variable to active transport feature which is builtin in the application.
- Use a hook method to hijack network calls from the application, without changing the program code.
- Use a network packet hijacking way such as the kernel netfilter module to inspect and modify network packets.
Setting environment variable
https_proxy may be the most common way to configure a proxy in Linux, however we must notice this is not a global proxy configuration on Linux, whether these two variables can enable proxy for an application depends on the application itself, which means only the application reads the environment variable and uses it explicitly as transport can make proxy available. Taking the implementation of go command line tool (go 1.16) as an example, the
impatientInsecureHTTPClient (which is used in command line tool such as
go get) always initializes a
Proxy field by
http.ProxyFromEnvironment, which is provided in net/http/transport.go, if
https_proxy is provided in environment variable, then the proxy will take effect. However, not all applications use these two env variables, such as golang standard http client doesn’t consider these two env variables by default.
Comparing with the env variables way which is limited to some programs, there exist some tools that can perform TCP redirection for more general applications, such as tsocks and proxychains-ng. Since the tsocks is not active now, I will talk about proxychains-ng mainly. Proxychains-ng is a UNIX program that hooks network-related libc functions in dynamically linked programs via a preloaded DLL (dlsym(), LD_PRELOAD) and redirects the connections through SOCKS4a/5 or HTTP proxies. Proxychains-ng works well in many scenarios (Maybe the most widely usage scenario is as a ladder to climb the GFW, but it can actually be used in production environment to redirect necessary network traffic, we did use it in some projects), but it has problem to work with a golang binary. When golang program is built with Golang Compiler (instead of the GCCGO), the binary uses the syscall wrappers of golang itself and static linking is used. So the LD_PRELOAD way for shared library way doesn’t work with golang binary. There are some articles to discuss this topic, such as proxychains-ng/issues/199 and proxychains-ng principle analytic.
In this part I will talk about graftcp, which can redirect the TCP connection made by any program [application, script, shell, etc.] to a SOCKS5 or HTTP proxy, no matter whether the target program is a dynamic executable or not. The principle of this tool is detailed described here, the core principle is to trace and modify the given program’s connect(2) by ptrace(2). The idea of this tool is amazing and it works very well under the Linux platform. However I have some concern with this method to use proxy, since it introduces the overhead of ptrace and one more network traffic between
graftcp-local server, whether the program/application performance will decrease needs to be taken into consideration.
This method often uses system firewall such as iptables to redirect specific network traffic to a transparent proxy, which will then uses HTTP or SOCKS proxy. One popular solution is to combine iptables and redsocks. Many articles have described how to deploy with redsocks. such as Linux global http proxy strategy, How to transparently use a proxy with any application (Docker) using Iptables and RedSocks. I also found an article that combines iptables and cgroup to make a transparent proxy.
In this part I have a simple benchmark to measure the performance degradation when using proxy, it is a python3 code snippet, which sends HTTP1.1 GET request to a given URL, the given URL is served with an Nginx and returns a json response simply. Besides the http proxy is served by cow. The test code and benchmark result are as follows.
|type||command||bench result||change N to 10000, bench result|
|no proxy||time ./py_read.py||0.25s user 0.18s system 84% cpu 0.503 total||2.65s user 1.36s system 72% cpu 5.505 total|
|env variable active||export http_proxy=http://127.0.0.1:7777 && time ./py_read.py||0.45s user 0.23s system 56% cpu 1.192 total||3.48s user 2.12s system 44% cpu 12.613 total|
|hook DLL function||time proxychains -f proxychains4.conf -q ./py_read.py||0.40s user 0.30s system 51% cpu 1.365 total||2.45s user 1.92s system 37% cpu 11.786 total|
|ptrace way||time ./graftcp -n ./py_read.py||0.67s user 1.51s system 0% cpu 8:42.22 total||not test|
From the benchmark above we can conclude the
environment variable active way and
DLL hook way have similar performance, the total execution time is increased by 2 times comparing to no proxy execution, which also means the throughput of network requests will decrease by half. On the other hand, the
ptrace way has very poor performance, the throughput is only 2 requests/s, and execution time increase is 1000 times. Considering this is a network request bound benchmark, I conduct another experiment to check whether there exists performance decreases when we use these ways with code logic that is not network request bound. In the following benchmark, the test code just opens a local file and write N strings into the file.
|no proxy||time ./py_write.py||0.36s user 0.05s system 90% cpu 0.449 total|
|env variable||export http_proxy=http://127.0.0.1:7777 && time ./py_write.py||0.38s user 0.03s system 90% cpu 0.453 total|
|proxychains||time proxychains -q ./py_write.py||0.36s user 0.04s system 89% cpu 0.452 total|
|graftcp||time ./graftcp -n ./py_write.py||0.40s user 0.08s system 48% cpu 1.008 total|
The benchmark result shows both the
environment variable active way and
DDL hook way have no side effect to application performance when there is no network request. But the
ptrace way still have performance overhead even there is no network request, which is reasonable, because of the overhead of ptrace.
There are many ways to use an HTTP or SOCKS proxy, but before we adopt a solution, we should not only consider whether the solution can work, but also think about more aspects, including the performance, stability and even usability. As for performance, the above benchmark can give a simple conclusion, but in the real world the usage scenario could be complex, the best way to decide which solution is the most appropriate is to test and benchmark with real workload and usage scenarios.