I also found the docs for
consul connect to be confusing. They don't
clearly differentiate the difference between the client and server
Some declarations that are worth stating explicitly:
consul aclneeds to be setup first, see consul acl for more info
- acl and intention are used somewhat interchangeably here
- client side
consul connectproxies can only talk to other
- client side
consul connectproxies can not talk directly to a service
- the docs explaining
- I'll use the term proxy to mean
- the term service refers to the actual service (eg. redis)
- the term server proxy refers to the proxy that connects to a real service
- the term client proxy refers to the proxy that clients connect to
Having said all that, service mesh sounds like they're worth having.
Mitchell Hashimoto at least partly agrees with me.
Access Control List (acl)
When one proxy wants to talk to another one, the all important acl gods must be appeased.
There must exist an
allows a client proxy to talk to a service. Or set the
allow and ignore the whole thing.
Read about intention below.
consul connect proxy -upstream redis:6379 -service redis-client
-service flag tells consul who we are. This name is used
This client proxy needs to know which server proxy to talk to, and that's
-upstream also specifies what port to listen on which is not how I
would have done that…
The given port listens on
localhost only, each machine should run it's
own instance of a client proxy. Real clients (eg. redis-cli) connect to
this port. This makes sense if everything is running locally on a machine
but if you have a program running in a docker container then you can't
connect to “localhost” as localhost is somewhere else. So you could
-upstream:0.0.0.0:6379 which is bad for security but really good for
connectivity from docker containers. The alternative is run
inside your container along with your actual program. That gets complicated
-upstream redis:6379 means talk to the server proxy for the
service and listen for clients on port 6379.
See my notes on naming below.
There are two types of server proxy, one for a proxy for a service
that has registered itself with consul and one that hasn't. If
the service has already registered with consul I think you use
-sidecar-for but I'm not sure and haven't tried. The instructions
below are for proxy'ing a service that has not registered itself with
Here's the complete command which we'll discuss below
consul connect proxy -register -service redis \ --service-addr 127.0.0.1:7777 -listen :6377
aside, consul service names
It's very confusing (to me at least, I'm sure there's a great reason
for why they did things this way) how a server proxy registers itself
with consul. Based on the command above you'd expect
redis to get
registered but you'd be wrong.
$ consul catalog services |grep redis redis-proxy
A server proxy is registered with “-proxy” appended to the
name. When a client proxy
-upstreams to a service then “-proxy” is
appended to the given name. The
acl rules require the actual client
proxy service name but the “real” server proxy name. ie.
Now back to command flags…
We want to
-register our server proxy so that client proxy can
automatically find us.
If you specify
-register then you must also specify
-service makes sense since that's the name of the service
we want to proxy.
Is the name that client proxy should
-listen is the port we're listening on and therefore the port that
client proxy should connect to us. You can specify an ip address but it
should be the public one otherwise why bother registering for external
-listen is a server proxy flag.
This is the address and port of the actual service that we're proxy'ing.
The service should probably only be listening on
consul connect dance makes a bit less sense.
For simplicity I'm running all these commands on the same machine but
I've verified that they work across machines as well. Because the client
proxy runs on the same machine as the actual server, we have to force
redis-server to listen to a non-default port.
In four different terminals run the four following commands.
redis-server --port 7777 --bind 127.0.0.1 consul connect proxy -register -service redis --service-addr 127.0.0.1:7777 -listen :6377 consul connect proxy -upstream redis:6379 -service redis-client redis-cli -h 127.0.0.1 -p 6379 # the defaults, being explicit for this example
Some output shown and some omitted for clarity:
$ redis-server --port 7777 --bind 127.0.0.1 ... ... Port: 7777
$ consul connect proxy -register -service redis --service-addr 127.0.0.1:7777 -listen :6377 ==> Consul Connect proxy starting... Configuration mode: Flags Service: redis Public listener: :6377 => 127.0.0.1:7777 ... ... proxy: registered Consul service: redis-proxy
Note it says service
redis in the header block but
the logs. There's a “hidden” service called
redis that's not in the
catalog but can be be found by the client proxy.
$ consul connect proxy -upstream redis:6379 -service redis-client ==> Consul Connect proxy starting... Configuration mode: Flags Service: redis-proxy Upstream: redis => :6379 Public listener: Disabled
redis-client is not registered in consul since it's not a service
per-se. It's able to automatically find the redis server proxy ip
address and port.
$ redis-cli -h 127.0.0.1 -p 6379 # the defaults, being explicit for example 127.0.0.1:6379> get consul "awesome"
intention are the rules that permit client proxy to talk to server proxy.
consul intention create -deny redis-client redis consul intention delete redis-client redis consul intention create -allow redis-client redis # there isn't an intention update but maybe should be...
Here's the output where we change the rules and can see the results
immediately. Note, changing an intention doesn't affect any existing
connections, only new ones. ie: if you change the intentions in a
redis-cli will keep working.
$ redis-cli 127.0.0.1:6379> get consul "awesome" 127.0.0.1:6379> ^D $ consul intention create -deny redis-client redis Created: redis-client => redis (deny) $ redis-cli 127.0.0.1:6379> get consul Error: Server closed the connection 127.0.0.1:6379> ^D $ consul intention delete redis-client redis Intention deleted. $ consul intention create -allow redis-client redis Created: redis-client => redis (allow) $ redis-cli 127.0.0.1:6379> get consul "awesome"
So there you have it, one service mesh. Easy peasy…