Dive into usage with proxy

Posted on In Symbols count in article: 6.9k Reading time ≈ 6 mins.

Recently I met a context deadline exceeded error when using gRPC client to call DialContext to a gRPC server, which was in a customer’s internal environment. After some investigation I found out the root cause, the gRPC client was behind an HTTP proxy and the proxy has no permission to access the gRPC server. It is a common trouble shooting, however I am interested in how we can force a program to use http or socks proxy, I will dive into it and compare the pros and cons among different solutions.

Ways to enable a proxy

There exist many ways to let an application use a proxy, these ways can be classified into three types as follows, and I will talk about these methods briefly.

  • Explicit environment variable to active transport feature which is builtin in the application.
  • Use a hook method to hijack network calls from the application, without changing the program code.
  • Use a network packet hijacking way such as the kernel netfilter module to inspect and modify network packets.

Configure proxy via environment variables

Setting environment variable http_proxy or https_proxy may be the most common way to configure a proxy in Linux, however we must notice this is not a global proxy configuration on Linux, whether these two variables can enable proxy for an application depends on the application itself, which means only the application reads the environment variable and uses it explicitly as transport can make proxy available. Taking the implementation of go command line tool (go 1.16) as an example, the impatientInsecureHTTPClient (which is used in command line tool such as go get) always initializes a Proxy field by http.ProxyFromEnvironment, which is provided in net/http/transport.go, if http_proxy or https_proxy is provided in environment variable, then the proxy will take effect. However, not all applications use these two env variables, such as golang standard http client doesn’t consider these two env variables by default.

Hook libc functions in dynamically linked program

Comparing with the env variables way which is limited to some programs, there exist some tools that can perform TCP redirection for more general applications, such as tsocks and proxychains-ng. Since the tsocks is not active now, I will talk about proxychains-ng mainly. Proxychains-ng is a UNIX program that hooks network-related libc functions in dynamically linked programs via a preloaded DLL (dlsym(), LD_PRELOAD) and redirects the connections through SOCKS4a/5 or HTTP proxies. Proxychains-ng works well in many scenarios (Maybe the most widely usage scenario is as a ladder to climb the GFW, but it can actually be used in production environment to redirect necessary network traffic, we did use it in some projects), but it has problem to work with a golang binary. When golang program is built with Golang Compiler (instead of the GCCGO), the binary uses the syscall wrappers of golang itself and static linking is used. So the LD_PRELOAD way for shared library way doesn’t work with golang binary. There are some articles to discuss this topic, such as proxychains-ng/issues/199 and proxychains-ng principle analytic.

Use ptrace to intercept syscall

In this part I will talk about graftcp, which can redirect the TCP connection made by any program [application, script, shell, etc.] to a SOCKS5 or HTTP proxy, no matter whether the target program is a dynamic executable or not. The principle of this tool is detailed described here, the core principle is to trace and modify the given program’s connect(2) by ptrace(2). The idea of this tool is amazing and it works very well under the Linux platform. However I have some concern with this method to use proxy, since it introduces the overhead of ptrace and one more network traffic between graftcp and graftcp-local server, whether the program/application performance will decrease needs to be taken into consideration.

Global proxy via underlying network packets inspect and modify

This method often uses system firewall such as iptables to redirect specific network traffic to a transparent proxy, which will then uses HTTP or SOCKS proxy. One popular solution is to combine iptables and redsocks. Many articles have described how to deploy with redsocks. such as Linux global http proxy strategy, How to transparently use a proxy with any application (Docker) using Iptables and RedSocks. I also found an article that combines iptables and cgroup to make a transparent proxy.

Performance overhead

In this part I have a simple benchmark to measure the performance degradation when using proxy, it is a python3 code snippet, which sends HTTP1.1 GET request to a given URL, the given URL is served with an Nginx and returns a json response simply. Besides the http proxy is served by cow. The test code and benchmark result are as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/usr/bin/python3
from urllib import request

N = 1000

def bench():
    url = 'http://bench.test/'
    for i in range(N):
        try:
            resp = request.urlopen(url).read()
        except Exception as e:
            print(e)

if __name__ == '__main__':
    bench()
type command bench result change N to 10000, bench result
no proxy time ./py_read.py 0.25s user 0.18s system 84% cpu 0.503 total 2.65s user 1.36s system 72% cpu 5.505 total
env variable active export http_proxy=http://127.0.0.1:7777 && time ./py_read.py 0.45s user 0.23s system 56% cpu 1.192 total 3.48s user 2.12s system 44% cpu 12.613 total
hook DLL function time proxychains -f proxychains4.conf -q ./py_read.py 0.40s user 0.30s system 51% cpu 1.365 total 2.45s user 1.92s system 37% cpu 11.786 total
ptrace way time ./graftcp -n ./py_read.py 0.67s user 1.51s system 0% cpu 8:42.22 total not test

From the benchmark above we can conclude the environment variable active way and DLL hook way have similar performance, the total execution time is increased by 2 times comparing to no proxy execution, which also means the throughput of network requests will decrease by half. On the other hand, the ptrace way has very poor performance, the throughput is only 2 requests/s, and execution time increase is 1000 times. Considering this is a network request bound benchmark, I conduct another experiment to check whether there exists performance decreases when we use these ways with code logic that is not network request bound. In the following benchmark, the test code just opens a local file and write N strings into the file.

1
2
3
4
5
6
7
8
#!/usr/bin/python3

N = 2000000

if __name__ == '__main__':
    with open("test.log", "w") as writter:
        for i in range(N):
            writter.write(f"{i*i}\n")
type command bench result
no proxy time ./py_write.py 0.36s user 0.05s system 90% cpu 0.449 total
env variable export http_proxy=http://127.0.0.1:7777 && time ./py_write.py 0.38s user 0.03s system 90% cpu 0.453 total
proxychains time proxychains -q ./py_write.py 0.36s user 0.04s system 89% cpu 0.452 total
graftcp time ./graftcp -n ./py_write.py 0.40s user 0.08s system 48% cpu 1.008 total

The benchmark result shows both the environment variable active way and DDL hook way have no side effect to application performance when there is no network request. But the ptrace way still have performance overhead even there is no network request, which is reasonable, because of the overhead of ptrace.

Summary

There are many ways to use an HTTP or SOCKS proxy, but before we adopt a solution, we should not only consider whether the solution can work, but also think about more aspects, including the performance, stability and even usability. As for performance, the above benchmark can give a simple conclusion, but in the real world the usage scenario could be complex, the best way to decide which solution is the most appropriate is to test and benchmark with real workload and usage scenarios.

Reference