Enough based on libvirt and publicly available services

Bonjour,

Here is the architecture proposed for the first ever Enough instance based on libvirt :tada:. It is located on a machine with a libvirt daemon running and has a single IPv4.

  • exemple.lan is created with the desired service, a forum enough --domain example.lan service create --driver libvirt forum. The certificate is based on ownca, which is the default for the libvirt driver because it is assumed that it can’t be reached from internet and LE is therefore not an option.
  • a host is created for example.org to run bind and a reverse proxy enough --domain example.lan host create --driver libvirt reverse-host
  • a glue record is added to example.org to delegate the DNS to the IPv4 of the machine
  • ~/.enough/example.org/inventory/group_vars/all/certificate.yml is modified with certificate: letsencrypt
  • a rule is added on the host to forward 80/443 to
  • a hand made playbook uses the enough-nginx role to reverse proxy forum.example.org to forum.example.lan

There may be a blocker: bind-host is hardcoded in many places to be the name of the host running the DNS for a domain. Since both example.org and example.lan run on the same libvirt hypervisor, they wil conflict. Hardcoded bind-host must be removed and replaced by a group, even if said group can only contain a single host.

Here is an example to forward port 2222 from the libvirt host to the port 22 of a given guest, based on the libvirt script from the documentation:

#!/bin/bash

# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "proxy-host" ]; then

   # Update the following variables to fit your setup
   GUEST_IP=10.23.10.234
   GUEST_PORT=22
   HOST_PORT=2222

   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /usr/sbin/iptables -D FORWARD -o virbrenough-ext -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /usr/sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /usr/sbin/iptables -I FORWARD -o virbrenough-ext -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /usr/sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

The libvirt-hook-qemu was installed and configured instead of using the above bash script (because it is a hack that does not actually work in all cases).

Cascading a reverse proxy for example.com to the example.lan reverse proxy works well. However, the website running on example.lan does not know about example.com and if it references example.lan in any way (Discourse does), things will not end well.

Not sure how to properly address this.

Going back to the libvirt script from the documentation and trying to understand how that’s supposed to work turned out ok. Here is the result, commented, for the record:

#!/bin/bash

function forward() {
   local guest_ip=$1
   local host_port=$2
   local guest_port=$3
   local context=$4
   local external_ip=$5

   local actions
   case $context in
       stopped)
	   actions=-D
	   ;;
       reconnect)
	   actions='-D -I'
	   ;;
       start)
	   actions=-I
	   ;;
   esac

   local action
   for action in $actions ; do
       #
       # external_ip:host_port is NAT'ed to guest_ip:port (either from the net or from a libvirt guest)
       #
       /usr/sbin/iptables -t nat $action PREROUTING -p tcp -d $external_ip --dport $host_port -j DNAT --to $guest_ip:$guest_port
       #
       # The default policy is DROP instead of ACCEPT, there needs to be an exception for this guest_ip:guest_port
       #
       /usr/sbin/iptables $action FORWARD -p tcp -d $guest_ip --dport $guest_port -j ACCEPT
       #
       # Masquerade the IP: it is not needed when the packet originates from the net, but it matters
       # when it originates from a libvirt guest. If not masqueraded, the libvirt guest connects to
       # external_ip and guest_ip sends the reply via the libvirt bridge because it is the shortest route
       # and the libvirt guest ignores the reply because it does not originate from external_ip.
       # 6:58:36.646616 IP 10.23.10.2.52796 > patience.easter-eggs.com.https: Flags [S], seq 2429574853, win 29200, options [mss 1460,sackOK,TS val 2954510752 ecr 0,nop,wscale 6], length 0
       # 16:58:36.646660 IP 10.23.10.2.52796 > 10.23.10.234.https: Flags [S], seq 2429574853, win 29200, options [mss 1460,sackOK,TS val 2954510752 ecr 0,nop,wscale 6], length 0
       # 16:58:36.647092 IP 10.23.10.234.https > 10.23.10.2.52796: Flags [S.], seq 3059781686, ack 2429574854, win 28960, options [mss 1460,sackOK,TS val 896856958 ecr 2954510752,nop,wscale 6], length 0
       #
       /usr/sbin/iptables -t nat $action POSTROUTING -p tcp -d $guest_ip --dport $host_port -j MASQUERADE       
       #
       # For accessing the port from the host running libvirt (the above PREROUTING/POSTROUTING are not in play)
       #
       /usr/sbin/iptables -t nat $action OUTPUT -p tcp -d $external_ip --dport $host_port -j DNAT --to $guest_ip:$guest_port

   done
}

guest=$1
action=$2

#
# The IP of the libvirt host
#
external_ip=37.9.139.14

case "$guest" in
    proxy-host)
	forward 10.23.10.234 80 80 $action $external_ip
	forward 10.23.10.234 443 443 $action $external_ip
	;;
    gitlab-host)
	forward 10.23.10.233 2222 22 $action $external_ip
	;;
esac