In an older TechUp we wrote about network proxies in general, and went into more detail about the so-called reverse-proxy. The focus was on understanding what exactly such a reverse proxy does and how to configure one. In this TechUp, the well-known HAProxy was used. We made the HAProxy container-enabled and wrote a simple forwarding configuration, which we packed directly into the container image.
Although HAProxy is very widespread, there are other competing products in the cloud-native environment that prepare the proxying in a slightly different way. Most prominent is Envoy Proxy, an open source project that is also a CNCF Graduated project. Envoy is considered a service as well as an edge proxy, denoting its geographic placement in the cloud infrastructure. In both cases, Envoy acts as a reverse proxy.
As a service proxy, Envoy provides connectivity between services, which is also called an East-West connection. The classical ingress from outside to a service in the target infrastructure is also called a North-South connection. Here, the envoy is closer to the ingress, and thus closer to the edge, hence edge-proxy. These tend to be newer terms, as they only became apparent through the requirements of a cloud landscape.
In today’s TechUp we take Envoy as a proxy foundation and look at how we can use Wasm modules to write our own HTTP filter and thus extend the functionality of a reverse proxy like Envoy.
HTTP filter
As a reverse proxy, Envoy stands between the web and the backend. Thus, the reverse proxy is the network component that can influence the incoming traffic. In concrete terms, this means that a reverse proxy like Envoy can manipulate the HTTP traffic. This allows certain network- and HTTP-specific logic to be built in, which takes effect before the backend system.
Figure: HTTP transformation / mutation
Let’s take an HTTP request as an example. You can make such a request visible in the developer tools of Chrome (on MacOS simply: CMD + ALT + i) or of other common browsers. I copied out this example request below when accessing the /home/techup subpage.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
GET /solr/b-nova-techhub HTTP/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7
Connection: keep-alive
Host: solr.b-nova.com
Origin: https://b-nova.com
Referer: https://b-nova.com/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0 Safari/537.36
sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
|
A HTTP filter changes the state of an HTTP request, or also an HTTP response. In this case, it means that we remove a header from the request, modify it, or even add a new header. This is also called mutation.
Now we make a few mutations. I use a diff-like syntax here for labelling:
+ : Adds a header-: Removes a header!: Modifies an existing header
The “diff” now looks like this for us:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
GET /solr/b-nova-techhub HTTP/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7
Connection: keep-alive
Host: solr.b-nova.com
Origin: https://b-nova.com
! Referer: https://www.b-nova.com/
- Sec-Fetch-Dest: empty
- Sec-Fetch-Mode: cors
- Sec-Fetch-Site: same-site
+ X-My-New-Header: awesome-value
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0 Safari/537.36
- sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"
- sec-ch-ua-mobile: ?0
- sec-ch-ua-platform: "macOS"
|
If we assume that for each mutation there would be an HTTP filter that can make these modifications, then the result of all HTTP filter mutations would be as follows:
1
2
3
4
5
6
7
8
9
10
11
|
GET /solr/b-nova-techhub HTTP/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,de;q=0.7
Connection: keep-alive
Host: solr.b-nova.com
Origin: https://b-nova.com
Referer: https://www.b-nova.com/
X-My-New-Header: awesome-value
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
|
The ordered sequence of HTTP filters, also called cascading, is called a HTTP filter chain.
Figure: HTTP-Filter-Chain
The Proxy-Wasm ABI
An Application Binary Interface, ABI for short, is the interface that defines the data exchange between the host system and the WebAssembly module. In our case, the host system is the Envoy proxy.
Classically, the ABI allows communication between two applications. The ABI acts as the interface defined in advance, so that one can address and program the interface to this ABI.
Figure: Proxy-Wasm-ABI
For the Proxy-Wasm-ABI there is a special SDK for Rust, which provides the whole ABI comfortably in Rust. The best way to learn the ABI is to use the codebase of the SDK. For this reason, we venture a few insights into the source code in order to better understand the ABI. First of all, we look into the traits.rs source file, where we can see different hooks.
1
2
3
4
5
6
7
|
// proxy-wasm-rust-sdk/src/traits.rs
pub trait HttpContext: Context {
fn on_http_request_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> Action {
Action::Continue
}
...
}
|
fn on_http_request_headers() is a hook that, as the name suggests, is called when HTTP request headers arrive in Envoy’s filter chain. This is defined in the ABI so that nothing is done with the Action::Continue expression. Provided we re-implement and override this function, we can process our own filter logic when calling this hook in the filter chain, which will reprocess the request headers accordingly.
But what we need for this is a way to read the headers at runtime from the filter chain and rewrite them as needed. For this we find the following two functions in the same ABI definition below:
1
2
3
4
5
6
7
8
9
10
11
|
pub trait HttpContext: Context {
...
fn get_http_request_headers(&self) -> Vec<(String, String)> {
hostcalls::get_map(MapType::HttpRequestHeaders).unwrap()
}
...
fn set_http_request_headers(&self, headers: Vec<(&str, &str)>) {
hostcalls::set_map(MapType::HttpRequestHeaders, headers).unwrap()
}
...
}
|
fn get_http_request_headers() and fn set_http_request_headers() are both functions that read (get) and write (set) a map of header fields from the host call.
Thus we have all the building blocks from the ABI together to read out the header fields ourselves at runtime during a request call, to process our own desired logic, and to write the final state back into the header map, with the hope that the desired mutation of the header data arrives in the backend. So far, so good.
So, how do we proceed to write our own HTTP filter module with WASM? First, we set up a new project with Cargo. We simply call the project envoy-request-header-mutator, and give Cargo the lib flag, so that it knows that we want to compile a module and not a standalone binary:
1
2
|
$ cargo new envoy-request-header-mutator --lib
Created binary (application) `envoy-request-header-mutator` package
|
Under /src there should be a lib.rs file, which currently contains filler code (2 + 2 = 4). We will write our filter logic here:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
use proxy_wasm::traits::*;
use proxy_wasm::types::*;
use serde::{Deserialize};
use regex::Regex;
proxy_wasm::main! {{
proxy_wasm::set_log_level(LogLevel::Trace);
proxy_wasm::set_root_context(|_| -> Box<dyn RootContext> {
Box::new(HttpRequestHeaderFilterRoot {
config_stringvalue: String::new(),
})
});
}}
struct CustomFilter {
config_stringvalue: String,
}
impl Context for CustomFilter {}
#[derive(Deserialize)]
#[serde(tag = "type")]
enum Operation {
Add { header: String, value: String},
Modify { header: String, regex: String, replace_with: String},
Replace { header: String, replace_with: String },
Remove { header: String }
}
impl HttpContext for CustomFilter {.
fn on_http_request_headers(&mut self, _: usize, _: bool) -> Action {
match serde_json::from_str(self.config_stringvalue.as_str()) {
Err(e) => log::error!("Config deserialization failed. Reason: {}", e),
Ok(operation) =>
match &operation {
Operation::Add { header, value } => {
self.add_http_request_header(header, value);
log::info!("Add succeeded. Added header: {}", header);
}
Operation::Modify { header, regex, replace_with} => {
if let Some(header_field) = self.get_http_request_header(header) {
let replaced_header_field = Regex::new(regex).unwrap().replace_all(&header_field, replace_with);
self.set_http_request_header(header, Some(&replaced_header_field));
log::info!("Modify succeeded. Modified header: {}", header);
}
}
Operation::Replace { header, replace_with} => {
if let Some(_header_field) = self.get_http_request_header(header) {
self.set_http_request_header(header, Some(replace_with));
log::info!("Replace succeeded. Replaced header: {}", header);
}
}
Operation::Remove { header } => {
if let Some(_header_field) = self.get_http_request_header(header) {
self.set_http_request_header(header, None);
log::info!("Remove succeeded. Removed header: {}", header);
}
}
}
};
Action::Continue
}
}
struct CustomFilterRoot {
config_stringvalue: String,
}
impl Context for CustomFilterRoot {}
impl RootContext for CustomFilterRoot {
fn on_configure(&mut self, _: usize) -> bool {
if let Some(config_bytes) = self.get_plugin_configuration() {
self.config_stringvalue = String::from_utf8(config_bytes).unwrap()
}
true
}
fn create_http_context(&self, _: u32) -> Option<Box<dyn HttpContext>> {
Some(Box::new(CustomFilter {
config_stringvalue: self.config_stringvalue.clone(),
}))
}
fn get_type(&self) -> Option<ContextType> {
Some(ContextType::HttpContext)
}
}
|
Now we build our WASM module with wasm-pack. First you should install wasm-pack. You can do this either with a package manager of your choice (on macOS I prefer brew), with cargo install or fastest with the official installer shell script from the official wasm-pack website.
1
2
3
|
$ curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
info: downloading wasm-pack
info: successfully installed wasm-pack to `/Users/rschneider/.asdf/shims/wasm-pack`
|
If you don’t want to write the code yourself, you can also check out the corresponding GitHub repo from me and continue from its root directory.
1
|
$ git clone https://github.com/raffaelschneider/envoy-request-header-mutator.git
|
1
|
$ wasm-pack build --release
|
Here the build artefact ends up in pgk/, but you can define your own target directory with --out-dir wasm/target/your-target-directory.
Here is the envoy/envoy.yaml configuration file that configures the Envoy proxy:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
static_resources:
listeners:
- name: main
address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: web_service
http_filters:
- name: envoy.filters.http.wasm
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
config:
name: envoy-request-header-mutator
root_id: envoy-request-header-mutator
configuration:
"@type": "type.googleapis.com/google.protobuf.StringValue"
value: |
{
"type": "add",
"header": "X-My-New-Header",
"value": "awesome-value"
}
vm_config:
vm_id: vm.sentinel.envoy-request-header-mutator
runtime: envoy.wasm.runtime.v8
code:
local:
filename: /{PATH_TO_WASM_MODULE}/envoy-request-header-mutator.wasm
allow_precompiled: true
- name: envoy.filters.http.router
clusters:
- name: web_service
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: round_robin
load_assignment:
cluster_name: web_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 5678
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
|
So, now let’s first emulate our backend with a simple echo application. There are many applications that provide simple HTTP echoing. Here I have chosen HashiCorp’s http-echo. You can build the project yourself via the repository using the Go build tools, or simply use the appropriate Docker image.
Provided the Docker daemon is running, you can instantiate the http-echo image as a container as follows:
1
|
$ docker run -p 5678:5678 hashicorp/http-echo -text="This is a HTTP Response!"
|
The port mapping is 1:1, which means that port 5678, which is run by the echo application as a listener, also exposes to the outside via the same port. Thus, one can test the echo as follows:
1
2
|
$ curl localhost:5678
This is a HTTP Response!
|
This was only a test without having an Envoy as a reverse proxy in between. But now we add it to test the whole route. Envoy start with the envoy command, we give as -c flag the envoy.yaml configuration file with.
1
2
3
|
$ envoy -c envoy/envoy.yaml
[2022-08-03 14:27:47.031][14448][info][main] [external/envoy/source/server/server.cc:330] initializing epoch 0 (base id=0, hot restart version=11.104)
...
|
This is how the current constellation looks with a filter-capable Envoy reverse proxy between client and backend:
Figure: Configuration with Envoy reverse proxy
Now we want to send a new request against Envoy and see if the echo service delivers the request header. HashiCorp’s http-echo is unfortunately not able to write the request header into the response body, but there are alternatives like the ealenn/Echo-Server.
1
2
3
4
5
6
7
8
9
10
|
$ curl -i localhost:8000
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
content-length: 1264
etag: W/"4f0-vrmKg5I9fOw8rz10EbdMEUV0tf4"
date: 'Wed, 03 Aug 2022 12:29:36 GMT'
x-envoy-upstream-service-time: 23
server: envoy
{"host":{"hostname":"localhost","ip":"::ffff:10.0.2. 100","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/","protocol":"http"},"request":{"params":{"0":"/"},"query":{},"cookies":{},"body":{},"headers":{"host":"localhost:8000","user-agent":"curl/7.82. 0","accept":"*/*","x-forwarded-proto":"http","x-request-id":"b793c73e-57a8-45fd-bfb6-412e7639b653","x-my-new-header": "awesome value", "x-envoy-expected-rq-timeout-ms": "15000"}}, "environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM": "xterm",. ..}}%
|
Hooray!!! Done. The WASM module apparently acts as an HTTP filter for Envoy and enriches the incoming request with a new request header X-My-New-Header.
Conclusion
With WASM-enabled extensions, Envoy as a CNCF proxy solution is betting on a future-proof technology and thus venturing into a web-native direction. As always, I hope you were able to learn something new in this TechUp. If you like, you can check out more exciting TechUp’s in our TechHub. Stay tuned for more exciting posts on the topic! 😁
Links & Resources
Proxy-Wasm | GitHub
WebAssembly for Proxies (Rust SDK) | GitHub
WASM | Envoy Proxy Docs