I also found the docs for consul connect to be confusing. They don't clearly differentiate the difference between the client and server proxy.

Some declarations that are worth stating explicitly:

• consul acl needs to be setup first, see consul acl for more info
• acl and intention are used somewhat interchangeably here
• client side consul connect proxies can only talk to other consul connect proxies
• client side consul connect proxies can not talk directly to a service
• the docs explaining -service vs -listen vs -upstream are terrible
• I'll use the term proxy to mean consul connect process
• the term service refers to the actual service (eg. redis)
• the term server proxy refers to the proxy that connects to a real service
• the term client proxy refers to the proxy that clients connect to

Having said all that, service mesh sounds like they're worth having.

Mitchell Hashimoto at least partly agrees with me.

Access Control List (acl)

When one proxy wants to talk to another one, the all important acl gods must be appeased.

There must exist an intention that allows a client proxy to talk to a service. Or set the acl default_policy to allow and ignore the whole thing.

Read about intention below.

Client Proxy

consul connect proxy -upstream redis:6379 -service redis-client

-service

The -service flag tells consul who we are. This name is used in the acl rules.

-upstream

This client proxy needs to know which server proxy to talk to, and that's what -upstream specifies.

-upstream also specifies what port to listen on which is not how I would have done that…

The given port listens on localhost only, each machine should run it's own instance of a client proxy. Real clients (eg. redis-cli) connect to this port. This makes sense if everything is running locally on a machine but if you have a program running in a docker container then you can't connect to “localhost” as localhost is somewhere else. So you could -upstream:0.0.0.0:6379 which is bad for security but really good for connectivity from docker containers. The alternative is run consul connect inside your container along with your actual program. That gets complicated pretty fast.

So -upstream redis:6379 means talk to the server proxy for the redis service and listen for clients on port 6379.

See my notes on naming below.

Server Proxy

There are two types of server proxy, one for a proxy for a service that has registered itself with consul and one that hasn't. If the service has already registered with consul I think you use -sidecar-for but I'm not sure and haven't tried. The instructions below are for proxy'ing a service that has not registered itself with consul.

Here's the complete command which we'll discuss below

consul connect proxy -register -service redis \
    --service-addr 127.0.0.1:7777 -listen :6377

aside, consul service names

It's very confusing (to me at least, I'm sure there's a great reason for why they did things this way) how a server proxy registers itself with consul. Based on the command above you'd expect redis to get registered but you'd be wrong.

$ consul catalog services |grep redis
redis-proxy

A server proxy is registered with “-proxy” appended to the -service name. When a client proxy -upstreams to a service then “-proxy” is appended to the given name. The acl rules require the actual client proxy service name but the “real” server proxy name. ie. redis and not redis-proxy

Now back to command flags…

-register

We want to -register our server proxy so that client proxy can automatically find us.

If you specify -register then you must also specify -service and -listen. -service makes sense since that's the name of the service we want to proxy.

-service

Is the name that client proxy should -upstream to.

-listen

-listen is the port we're listening on and therefore the port that client proxy should connect to us. You can specify an ip address but it should be the public one otherwise why bother registering for external clients?

-listen is a server proxy flag.

-service-addr

This is the address and port of the actual service that we're proxy'ing. The service should probably only be listening on 127.0.0.1 otherwise this whole consul connect dance makes a bit less sense.

Full Example

For simplicity I'm running all these commands on the same machine but I've verified that they work across machines as well. Because the client proxy runs on the same machine as the actual server, we have to force redis-server to listen to a non-default port.

tl;dr

In four different terminals run the four following commands.

redis-server --port 7777 --bind 127.0.0.1
consul connect proxy -register -service redis --service-addr 127.0.0.1:7777 -listen :6377
consul connect proxy -upstream redis:6379 -service redis-client
redis-cli -h 127.0.0.1 -p 6379 # the defaults, being explicit for this example

with output

Some output shown and some omitted for clarity:

$ redis-server --port 7777 --bind 127.0.0.1
...
... Port: 7777

$ consul connect proxy -register -service redis --service-addr 127.0.0.1:7777 -listen :6377

==> Consul Connect proxy starting...
    Configuration mode: Flags
               Service: redis
       Public listener: :6377 => 127.0.0.1:7777
...
... proxy: registered Consul service: redis-proxy

Note it says service redis in the header block but redis-proxy in the logs. There's a “hidden” service called redis that's not in the catalog but can be be found by the client proxy.

$ consul connect proxy -upstream redis:6379 -service redis-client

==> Consul Connect proxy starting...
    Configuration mode: Flags
               Service: redis-proxy
              Upstream: redis => :6379
       Public listener: Disabled

The redis-client is not registered in consul since it's not a service per-se. It's able to automatically find the redis server proxy ip address and port.

$ redis-cli -h 127.0.0.1 -p 6379 # the defaults, being explicit for example
127.0.0.1:6379> get consul
"awesome"

intentions

intention are the rules that permit client proxy to talk to server proxy.

consul intention create -deny redis-client redis
consul intention delete redis-client redis
consul intention create -allow redis-client redis
# there isn't an intention update but maybe should be...

Here's the output where we change the rules and can see the results immediately. Note, changing an intention doesn't affect any existing connections, only new ones. ie: if you change the intentions in a different terminal redis-cli will keep working.

$ redis-cli
127.0.0.1:6379> get consul
"awesome"
127.0.0.1:6379> ^D
$ consul intention create -deny redis-client redis
Created: redis-client => redis (deny)
$ redis-cli
127.0.0.1:6379> get consul
Error: Server closed the connection
127.0.0.1:6379> ^D
$ consul intention delete  redis-client redis
Intention deleted.
$ consul intention create -allow redis-client redis
Created: redis-client => redis (allow)
$ redis-cli
127.0.0.1:6379> get consul
"awesome"

fin

So there you have it, one service mesh. Easy peasy…