Setting up LXC containers to run with ISC DHCPd and BIND instead of dnsmasq, along with domain name spoofing/caching

By default, lxc-net setup containers to work along with dnsmasq, which provides both DNS and dhcpd services, name resolution and IP attribution.

Recommended setup of lxc-net includes /etc/lxc/dnsmasq.conf that only states  “dhcp-hostsfile=…” and the said dhcp-hostfiles as /etc/lxc/dnsmasq-hosts.conf with a line “hostname,IP” per host.

It works fine and there is no real reason to use anything else. Though it is obvious that lxc-net lacks a bit of modularity, since it is clearly tied, hardcoded, to dnsmasq for instance.

Except that on my main server, I already have ISC DHCPd serving IP to local area network and BIND 9 not only doing name resolution caching but also name resolution for said local area network. Not only having both dnsmasq and BIND 9 and ISC DHCPd is a bit overkill, but it requires additional config to bind them to specific interfaces to avoid conflicts.

dnsmasq shutdown

We could simply do a killall dnsmasq and comment the part in /usr/lib/x86_64-linux-gnu/lxc/lxc-net where it get started. For now, we’ll just prevent it from messing with interfaces, setting /etc/lxc/dnsmasq.conf to:

interface=lxcbr0
no-dhcp-interface=lxcbr0

Initial setup

This article assumes you already have BIND and ISC DHCPd set up for local area network (otherwise, as said, in most use cases, dnsmasq will be just fine).

If you do not have a preexisting setup but wants, nonetheless, switch to BIND 9 and ISC DHPCd, you could start with the bind setup provided in my setting up a silent/low energy consumption home server article.

This article includes dynamic clients name update. The only thing to pay attention is that this setup use 10.0.0.0/24 for local area network whereas, in the following article, 10.0.0.0/24 will be used for LXC bridge network while 192.168.1.0/24 will be dedicated to local area network.

DNS setup

I adjusted my preexisting setup (bind9 files part of my -utils-cache-spoof debian package, which I suggest you look at directly to have their current exhaustive content) based on bind9 notion of ACL (access control list) depending on which network clients belongs and, subsequently, bind9 notion of “views” that configure which zones are provided to these clients according to ACL.

The following will seems like a lot but, if you grab my debian -utils-cache-spoof package, it is actually not that much.

Since LXC bridge here is using 10.0.0.0/24 network, I have in named.conf.acl:

[...]

acl lan {
    // the cache host IP should not be part of regular lan ACL
    !10.0.0.88;
    // private IPv4 address spaces
    10.0.0.0/8;
    172.16.0.0/12;
    192.168.0.0/16;
};

acl lannocache {
   // counterpart of earlier statement: cache host needs proper unspoofed name resolution
   10.0.0.88;
};

Note that the .88 container IP is dedicated to caching (apt/steam as in my previous setup with dsniff as spoofer and my another setup using bind9 instead but outside of LXC host/container context) so it needs to be excluded from the general 10.0.0.0/8 ACL.

These ACL are in turn used in named.conf.views (Update: with latest versions of Bind9, we cannot include twice a file that as allow-update statement within, hence the …local and .local_ref):

// clients are set in named.conf.acl
include "/etc/bind/named.conf.acl";

// loopback view, for the server itself
view "loopback" {
 match-clients { loopback; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local";
 include "/etc/bind/named.conf.ads";
};

// otherwise local network area
view "lan" {
 match-clients { lan; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.cache";
 include "/etc/bind/named.conf.ads";
};

// local network area without cache, for host that will get unspoofed name resolution
// (needs to be set up one by one in named.conf.acl)
view "lannocache" {
 match-clients { lannocache; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.ads";
};


[...]

Obviously, if there was no notion of caching (and name spoofing), the setup would be even more straightforward, a single view would be enough. Nonetheless, this example shows an easy way to treat differently hosts depending whether they are LXC containers or regular LAN clients.

About the zones included (or not) in views (all files being in /etc/bind):

  • named.conf.default-zones is standard ;
  • named.conf.local is almost standard, you need to define here your local domains/network ;
  • Update:  named.conf.*_ref  is required with recent version of Bind9 to be able to use twice content for some named.conf.* in which some zone file is defined and can be updated (allow-update) : you’ll will need to use in-view feature to mimic usage of the view that previously defined it since trying another include would sprout writeable file ‘…’ already in use  ;
  • named.conf.cacheBASEIP contains list of spoofed domains, the one we want to cache, generated by named.conf.cache-rebuild.sh, BASEIP being optional;
  • named.conf.ads contains ads servers blacklist generated by update-bind-ads-block.pl ;

So basically, you need to edit /etc/bind/named.conf.local to something like:

// to store A/CNAME records for DOMAIN.EXT
zone "DOMAIN.EXT" {
 type master;
 notify no;
 file "/etc/bind/db.DOMAIN.EXT";
 allow-update { key ddns; };
};

// (we use 192.168.1.0/24 for regular LAN)
// to store PTR records (IP to name) for regular LAN 
zone "1.168.192.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.192.168.1";
 allow-update { key ddns; };
};

// (we use 10.0.0.0/24 for LXC bridge)
// to store PTR records for LXC bridge)
zone "0.0.10.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.10.0.0";
 allow-update { key ddns; };
};

Update: since recent Bind9 update, to be able to reuse these zones in another view, you’ll need to edit /etc/bind/named.conf.local_ref to something like:

// simple reference to previously defined zones for view loopback in named.conf.local
zone "DOMAIN.EXT" { in-view "loopback"; }; 
zone "1.168.192.in-addr.arpa" { in-view "loopback"; }; 
zone "0.0.10.in-addr.arpa" { in-view "loopback"; };

You also require relevant db. files: for instance db.ads pointing to loopback to filter ads/spam sources, db.cache pointing to the cache container .88 (possibly also db.cacheBASEIP) and local db. files as db.DOMAIN.EXT:

$ORIGIN .
$TTL 86400 ; 1 day
DOMAIN.EXT IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                        2823 ; serial
                        28800 ; refresh (8 hours)
                        7200 ; retry (2 hours)
                        604800 ; expire (1 week)
                        10800 ; minimum (3 hours)
                        )
          NS     server.DOMAIN.EXT.
          MX     10 server.DOMAIN.EXT.
$ORIGIN DOMAIN.EXT.
server    A      192.168.1.1
; the rest will be filled by ddns

Likewise, you should have db.192.168.1 and db.10.0.0 (obviously with 1.168.192 replaced by 0.0.10) as:

$ORIGIN .
$TTL 86400 ; 1 day
1.168.192.in-addr.arpa IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                       2803 ; serial
                       28800 ; refresh (8 hours)
                       7200 ; retry (2 hours)
                       604800 ; expire (1 week)
                       10800 ; minimum (3 hours)
                       )
           NS      server.DOMAIN.EXT.
$ORIGIN 1.168.192.in-addr.arpa.
1          PTR     server.DOMAIN.EXT.
; the rest will be filled by ddns too

And then you must run the scripts to generate named.conf.cacheBASEIP and name.conf.ads. You’ll probably need to edit /etc/bind/named.conf.cache-rebuild.sh variables according to what you are actually caching.

BIND gets updates from ISC DHCPd whenever a new clients get a lease, it is configured in name.conf.dhcp (not packaged):

include "/etc/bind/ddns.key";

controls {
 inet 127.0.0.1 allow { localhost; } keys { ddns; };
};

The ddns key was generated as documented in my setting up a silent/low energy consumption home server article as well as in Debian docs:

# dnssec-keygen -a HMAC-MD5 -b 128 -r /dev/urandom -n USER ddns

Out of the generated Kdhcp_updater.*.private, you get the content of the “Key:” statement and you put it in /etc/bind/ddns.key:

key ddns {
 algorithm HMAC-MD5;
 secret "CONTENTOFTHEKEY";
};

So this setup implies that your named.conf looks like:

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.dhcp";
include "/etc/bind/named.conf.views";

Besides, my /etc/bind/named.conf.options is generated by /etc/dhcp/dhclient-exit-hooks.d/bind so it include proper forwarders and listen-on exception.

That should cover it for BIND.

ISC DHPCd setup

In my case, I still still want IPs of LXC containers to be fixed. The syntax of /etc/lxc/dnsmasq-hosts.conf was “hostname,IP” per line which is more convenient than ISC DHCPD syntax  “host hostname { hardware ethernet MAC ADDRESS; fixed-address IP; }”.

I decided to use the same /etc/lxc/dnsmasq-hosts.conf symlinked to /etc/lxc/hosts.conf that will be used by the /etc/lxc/dhcpd-hosts.rebuild.sh (not packaged for now) script to generate /etc/dhcp/dhcpd_lxc-hosts.conf:

#!/bin/bash
# /etc/lxc/dhcpd-hosts.rebuild.sh

HOSTS=/etc/lxc/hosts.conf # similar to dnsmasq-hosts.conf: host,IP
DESTINATION=/etc/dhcp/dhcpd_lxc-hosts.conf
LXC_PATH=`lxc-config lxc.lxcpath`
cd $LXC_PATH

echo > $DESTINATION
for container in *; do
 if [ ! -d "$container" ]; then continue; fi
 if [ ! -e "$container/config" ]; then continue ; fi
 echo "host lxc-$container {" >> $DESTINATION
 echo " hardware ethernet "`cat "$container/config" | grep lxc.network.hwaddr | cut -f 2 -d "="`";" >> $DESTINATI
ON
 echo " fixed-address "`cat "$HOSTS" | grep "$container" | cut -f 2 -d ","`";" >> $DESTINATION
 echo "}" >> $DESTINATION 
done
# EOF

This primitive script will sprout out a proper ISC DHCPd host file. You have to run it each time you create a new container. Once done, we simply edit /etc/dhcp/dhcpd.conf:

# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed. We default to the
# behavior of the version 2 packages ('none', since DHCP v2 didn't
# have support for DDNS.)
ddns-updates on;
ddns-update-style interim;
ddns-domainname "DOMAIN.EXT";
ddns-rev-domainname "in-addr.arpa.";
ignore client-updates; # no touching the FQDN
include "/etc/dhcp/ddns.key";

# option definitions common to all supported networks...
option domain-name "DOMAIN.EXT";
option domain-search "DOMAIN.EXT", "ANOTHERDOMAIN.EXT";
option domain-name-servers 192.168.1.1;
option routers 192.168.1.1;

default-lease-time 600;
max-lease-time 6000;
update-static-leases on;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# LAN clients
subnet 192.168.1.0 netmask 255.255.255.0 {

 # dynamic IP depends whether the client MAC address is known
 pool {
   range 192.168.1.20 192.168.1.99;
   deny unknown-clients;
 }
 pool {
   range 192.168.1.100 192.168.1.250;
   allow unknown-clients; 
 }

 # iPXE / boot on lan
 if exists user-class and option user-class = "iPXE" {
   filename "ipxe-boot";
 } else {
   filename "undionly.kpxe";
 }
 next-server 192.168.1.1;
}

# LXC clients
subnet 10.0.0.0 netmask 255.255.255.0 {
 # use the subnet-specific router
 option routers 10.0.0.1;
 # no pool, all IP are fixed here
 # force lease time to be at least weekly
 min-lease-time 604800;
 max-lease-time 604800;
 # no boot on lan either
}

# zones
zone DOMAIN.EXT. {
 primary 127.0.0.1;
 key ddns;
}
zone 1.168.192.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}
zone 0.0.10.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}


# LAN known clients 
 host trendnetusb { hardware ethernet 00:50:b6:08:xx:xx; }
 host ugreenusb { hardware ethernet 00:0e:c6:fa:xx:xx; }

# LXC host
include "/etc/dhcp/dhcpd_lxc-hosts.conf";

That’s all. Obviously, if you want your LXC containers to get completely dynamically assigned IP, you do not even need this whole host setup. You just set a pool { } with a range of IP (and remove the specif lease time).

The cache LXC container

I wont get in much details, my my -utils-cache-apt and -utils-cache-steam debian packages should work out of the box on a LXC container, providing both the necessary nginx cache-apt and cache-steam config.

If you use resolvconf and ISC DHCP clients on LXC containers, the resolvconf to nginx resolver config script will set up /etc/nginx/conf.d/resolver.conf accordingly.

If you use udhcpc, this resolvconf script will be ignored  but the default /etc/nginx/conf.d/resolver.conf includes, in comments, proposed changes to /etc/udhcpc/default.script to generate  /etc/nginx/conf.d/resolver.conf accordingly.

Otherwise, you need to hand configure /etc/nginx/conf.d/resolver.conf

## (set resolver to something else if your local interface got
## domain names spoofed, 8.8.8.8 for Google resolver for example.
#resolver 127.0.0.1 ipv6=off; # without lxc
resolver 10.0.0.1 ipv6=off;   # within lxc

Troubleshooting

I have this setup since a while and noticed the following:

  • with ISC DHCP client within the LXC containers I get the bad udp checksums in N packets issue;  the iptables -A POSTROUTING -t mangle -p udp –dport 67 -j CHECKSUM  –checksum-fill rule set up by lxc-net is helpless; the solution i picked is to use udhcpc within all LXC containers that does not trigger the problem, with the obvious drawback that the cache container must use the edited /etc/udhcpc/default.script option since resolvconf will have no effect;
  • if ISC DHPCd and Bind9, on the LXC host, are started before or at the same time as lxc and lxc-net, they may not listen on the LXC bridge interface, possibly missing at their starting time; as result, while everything could seem properly on, LXC container would fail to get an IP assigned until you restart ISC DHPCd;  this does not occur after adding lxc lxc-net in the Should-Start: part of ISC DHCPd and Bind9 init.d scripts.
  • Update: With Bind9 recent version (notably: since Debian 9.0), if you have twice a zone defined with a file that can be updated, it wont start and logs will state something like writeable file ‘…’ already in use. The workaround, using in-view, is described in the earlier. Granted, it kills a bit the interest of using view and lead to ugly confusing setup.

Avoiding dnsmasq interference

If you are satistified and do not require dnsmasq anymore, I suggest to remove any dnsmasq package and add a symlink so dnsmasq command produces no error (when called by /usr/lib/x86_64-linux-gnu/lxc/lxc-net for instance):

ln -s /bin/true /usr/bin/dnsmasq
Advertisements

Caching steam depots on your local server with nginx and dsniff

While I usually don’t advertise non libre software for obvious reasons (that’s a stupid way to think about computing), I admit, though, that the Steam platform goes toward what I’d like to see since many years. Proprietary software platform indeed – but the business is not made out of selling overly expensive DVD-Rom once in a while but cheap soft (downloadable) copies of games (often) maintained over years. They also seem about to base a future gaming console on some sort of GNU/Linux flavor, that’s not philanthropy, that’s just the only clever way to do a cool gaming based business without getting totally dependant on another software supplier that also brand his own gaming console. Latest South Park was about the fight beetween latest Xbox and Playstation. This issue only exists when you decide to make console non compatible with usual workstation, a shortcut with so many shortcomings. Making a GNU/Linux based console, because it is good business, is obviously going in the right direction.

So I’ll allow myself a little reminder here on how not to waste your bandwidth on a local network where you have several computers having copies of the same steam game. It’s merely a simplified version of the well thought Caching Steam Downloads @ LAN’s article. Obviously, to do this, you need to have your own home server. For instance, it should work out of the box with a setup like this (this is the setup mentioned before from now on in this article).

A) HTTP setup

We first create a directory to store steam depot. It will be served with http so you need to  create something like (working with the setup mentioned before):

mkdir /srv/www/depot
chown www-data:www-data /srv/www/depot

Next, you want to setup nginx, to be able to serve as a steam content provider. Everything is based on http -no proprietary non-standard crap- so it can only go smoothly.

If you have the setup mentioned before, then  /etc/nginx/sites-available/default contains a server { } statement for general intranet. Add a new file called /etc/nginx/sites-available/steam with the following (watch out for the listen and allow statements, change it depending on your server intranet IP!):

# steam spoof/proxy
server  {
  # you want this line to be set to your server intranet IP
  listen 10.0.0.1;
  listen 127.0.0.1;
  server_name *.steampowered.com;

  access_log /var/log/nginx/steam.access.log;
  error_log /var/log/nginx/steam.error.log;

  root /srv/www/;
  resolver 127.0.0.1;
  ## use this resolver instead if your local interface got domain names spoofed 
  ##resolver 8.8.8.8;

  # restrict to local wired network
  allow 10.0.0.0/24;https://github.com/yeupou/stalag13/blob/master/etc/cron.daily/cache-steam
  allow 127.0.0.1;
  deny all;
  location /depot/ {
    try_files $uri @mirror;
  }

  location / {
    proxy_next_upstream error timeout http_404;
    proxy_pass http://$host$request_uri;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    add_header X-Mirror-Upstream-Status $upstream_status;
    add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
    add_header X-Mirror-Status $upstream_cache_status;
  }

  location @mirror {
    access_log /var/log/nginx/steam.remote.log;
    proxy_store on;
    proxy_store_access user:rw group:rw all:r;
    proxy_next_upstream error timeout http_404;
    proxy_pass http://$host$request_uri;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    add_header X-Mirror-Upstream-Status $upstream_status;
    add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
    add_header X-Mirror-Status $upstream_cache_status;
   }
}

Make it live:

cd /etc/nginx/sites-enabled && ln -s ../sites-available/steam .
invoke-rc.d nginx restart

Now nginx is able to fetch and serve steam depot files.

B) DNS setup

Now, you need your server to actually handle requests to steam content server, spoofing these servers IPs. It could be done by messing with the DNS cache server already up on the setup mentioned before but I actually find much more convenient to use dnsspoof from dsniff package with a two-line configuration than wasting time creating say bind9 unnecessarily complex db files.

So we first instead dnsspoof:

apt-get install dsniff

Here come’s the two line configuration, set in /etc/dnsspoof.conf. Obviously, here too you have to set the IP to be your server’s intranet one.

10.0.0.1     *.cs.steampowered.com
10.0.0.1     content*.steampowered.com

Then you want an init.d script. You can create an ugly /etc/init.d/dnspoof with the following (obviously, you want you ethernet ethX device to be properly set!):

/#! /bin/sh
### BEGIN INIT INFO
# Provides:          dnsspoof
# Required-Start:    bind9
# Required-Stop:     
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start dnsspoof over bind9
# Description:       Start dnsspoof over bind9
### END INIT INFO

# shitty version for testing purpose
/usr/sbin/dnsspoof -i eth1 -f /etc/dnsspoof.conf 2> /dev/null > /dev/null &

# EOF

Or, 5th December UPDATE, you can download my init script doing as follows:

cd /etc/init.d && wget https://github.com/yeupou/stalag13/raw/master/etc/init.d/dnsspoof && chmod +x dnsspoof && update-rc.d dnsspoof defaults
cd /etc/default && wget https://github.com/yeupou/stalag13/raw/master/etc/default/dnsspoof

(dont forget to edit /etc/default/dnsspoof to suit your setup, especially regarding the ethernet ethX device to be used)

Once ready, just start the spoofer:

chmod 755 /etc/init.d/dnsspoof
invoke-rc.d dnsspoof start

Now you can restart steam on your clients computers. It should work properly. You can check whether new directories appear in /srv/www/depot and monitor /var/log/nginx/steam* logs.

I’ll soon add a small script to get more meaningful info about the depots available on your server, so you can know which are what in a jiffy and remove the no longer useful willy-nilly.

December 10th update: soon is now. Get the script doing the following, if you run the setup mentioned before, it should work out of the box:

cd /etc/cron.daily && wget https://github.com/yeupou/stalag13/raw/master/etc/cron.daily/cache-steam && chmod +x cache-steam
./cache-steam

This should update  a daily depots.html file in your /srv/www/depot/.