Isn’t SRS breaking SPF itself, at least regarding spam?

Earlier on this blog, I proposed ways to implement SPF (Sender Policy Framework). I recently noticed mails forwarded by one of my servers being tagged as spam by gmail.com due to SPF checks. It means that while SPF works for my domains with near to 0 user base, no real business of forwarding, it is a nuisance for forwarding in general. So you are advised to use SRS (Sender Rewriting Scheme). Strangely enough it is not fully integrated on main servers and some implementation (Exim in Debian) are based on unmaintained library (SRS C library).

Unmaintained?

Fact is SRS is far from being nice. It makes so your own forwarding server is vouching for fowarded mails. But why would you want that?

SPF test will fail because your forwarding server is not a registered valid source for (forwarded) mails sent from domain X. SRS proposal is that your server will alter header so to forward the mail from X domain X to appear as sent from an address of your own domain for you server is a registered valid source.

I guess the logic is to make forwarders somehow responsible of filtering, not bad in principle.

But it also means that for each spam forwarders fail to identify, they’ll be tagged as spam originator. It is particulary annoying when forwarding is made on public addresses bound to attract spam. So it seems better to get a failed SPF test on every forwarded messages including valid ones than a valid SPF test on every forwarded messages including spam.

SPF without SRS breaks forwarding. But SPF with SRS, the workaround, breaks SPF itself regarding spam and will give you (your IPs, your domains) bad rep, with will make your legit mail at risk of being blacklisted, unless you apply an overly harsh policy on forwarded mails.

Annoying. I am thinking removing SPF completely, instead.  For now, I am updating my SPF records to remove any Fail statement, since there is no way for me to know whether one of my mail can legitimately be forwarded through several servers.  Funny enough, google that promotes SPF usage recommends using SoftFail over Fail. But I might even reset to Neutral.

Interesting link on topic : Mail server setup wih SRS ; Why not SPF?

Alternative: I implemented DKIM on my servers. Seems much smarter to have a server signature.

Advertisements

Using PowerDNS (server and recursor) instead of Bind9, along with domain name spoofing/caching

Recently, I made an article about how to use Bind9 with LXC containers, setup including domain name spoofing/caching. This setup was using Bind9 views so only the caching LXC container would get real IP for cached domains (like ftp.fr.debian.org or packages.devuan.org) so nginx, on this system, could mirror accordingly.

Then Debian 9.0 was released and I found out two views were no longer allowed to share writing rights to a single same zone definition file.

You would then get error like  “writeable file ‘/etc/bind/…’: already in use: /etc/bind/…”.

As sacrifial-spam-address wrote:

At first, I thought this was a bug (how can a config file line conflict
with itself?), then I realized that the conflict was between the
two views.

There does not appear to be any simple workaround. The best solution
appears to be to use the new BIND 9.10 “in-view” feature, which allows a
zone in one view to be a reference to the same zone in a different view.
When this is done, both views may share the same cache file.

The down side is that this violates one of the important principles of
programming: only specify something in one place. Instead, I have to have
a “master” definition and several “in-view” declarations referencing
the master.

I wish BIND would either deal with the problem after noticing it (by
automatically doing the equivalent of the in-view), or provide a way to
import every zone in a view, avoiding the need for a long list of in-view
declarations.

Then I fixed my setup to work with in-view, updating the article already linked. But the experience was clearly unsatisfying, adding one more layer of complexity to something already quite dense.

Plus I got some error in this new setup: it seemed that in-view, at least the way I configured, cause the different views to behave as if they share a same cache. Say, after Bind9 startup, I pinged ftp.fr.debian.org from any LXC container but the cache one, I would get the IP of the LXC cache container as it should be. But, then, if I pinged the same domain from the LXC cache container, I would still get as answer its own IP, as if there was not two different views setup. Same with the opposited test, if the first ping was from within the LXC cache container, then from any other, I would get the result wanted only for the LXC cache container.

So it lead me to the point that I had to understand better the in-view feature that in first place I did not want to use, in order to get it to behave like view did.

You got it: I found much easier to user PowerDNS instead.

PowerDNS (pdns) is composed of an authoritative DNS server plus a DNS recursor. The first one I only need on this setup for the LAN domain name (quite used: LXC containers + connected devices). The recursor is doing some caching. And can be easily scripted.

apt-get install pdns-server pdns-recursor pdns-backend-sqlite3

Often, when both the name server and the recursor are installed on the same machine, people set up the name server to listen on port 53 on the network and to pass to the recursor, listening on another port, requests it cannot handle (that it is not authoritative for and need a recursor to resolve then).

Ok, why not. Except that I want specific answer to be given depending on the querier’s IP for domains outside of the LAN, so handled by the recursor. That would not work if the recursor get queries sent over loopback device by the authoritative server.

Aside from that, just as general principle, I like better the notion of, by default, soliciting a recursor that, only when necessary, ask the local DNS server  instead of other DNS than the notion of asking a DNS server to handle queries that he is most of the time unlikely to be have authoritative answer for and that he’ll have to pass to a recursor.

So instead of the usual proposed :

  • client ->  local DNS server  -> DNS recursor if non authoritative  -> distant authoritative DNS server

It’ll be:

  • client -> DNS recursor -> authoritative DNS server (local or distant).

 

authoritative PowerDNS server

First we deal with the DNS server to server YOURDOMAIN.LAN. The sqlite3 backend should be installed and set up (or else of your liking).

By default, the sqlite3  database is in /var/lib/powerdns/pdns.sqlite3

Easiest way is to convert Bind9 zone config to set it up:

zone2sql --named-conf=/etc/bind/named.conf.local --gsqlite | sqlite3 /var/lib/powerdns/pdns.sqlite3

That’s all!

As alternative,  you can also create zone from scratch with pdnsutil:

cd /var/lib/powerdns
sqlite3 pdns.sqlite3 < /usr/share/doc/pdns-backend-sqlite3/schema.sqlite3.sql
chown pdns:pdns pdns.sqlite3
# main zone
pdnsutil create-zone YOURDOMAIN.LAN ns1.YOURDOMAIN.LAN
pdnsutil add-record YOURDOMAIN.LAN main A 192.168.1.1
pdnsutil add-record YOURDOMAIN.LAN @ MX "10 mx.YOURDOMAIN.LAN"
[...]
# first reverse zone 192.168.1
pdnsutil create-zone 1.168.192.in-addr.arpa ns1.YOURDOMAIN.LAN
pdnsutil add-record 1.168.192.in-addr.arpa 1 PTR main.YOURDOMAIN.LAN
[...]
# to be continued

 

In our previous setup, we had DNS update automated by ISC DCPDd: we want any new host on the local network to be given an IP. Nothing changed regarding ISC DHCPd, read the relevant ISC DHCPd setup part. For the record, to generate the relevant update key:

cd /etc/dhcp
dnssec-keygen -a hmac-md5 -b 256 -n USER ddns

The secret will be a string like XXXXX== within the ddns.key generated file.

Obviously, powerdns needs this data. You need to register the key+secret and give right on each zone (YOURDOMAIN.LAN plus the reverse for IP ranges, below for 192.168.1 and 10.0.0)

sqlite3 /var/lib/powerdns/pdns.sqlite3

# XXXXX== = the secret string
insert into tsigkeys (name, algorithm, secret) values ('ddns', 'hmac-md5','XXXXX==');

# find out ids of zones
select id from domains where name='YOURDOMAIN.LAN';
 2
select id from domains where name='1.168.192.in-addr.arpa';
 1
select id from domains where name='0.0.10.in-addr.arpa';
 3

 # authorized the key for each
 insert into domainmetadata (domain_id, kind, content) values (1, 'TSIG-ALLOW-DNSUPDATE', 'ddns');
 insert into domainmetadata (domain_id, kind, content) values (2, 'TSIG-ALLOW-DNSUPDATE', 'ddns');
 insert into domainmetadata (domain_id, kind, content) values (3, 'TSIG-ALLOW-DNSUPDATE', 'ddns');

Finally, you need to configure powerdns itself. You can directly edit /etc/powerdns/pdns.conf but I think easier to create a specific /etc/powerdns/pdns.d/00-pdns.conf so you do not edit the default example:

# base
 daemon=yes
 local-address=127.0.0.1
 local-port=53
 local-address-nonexist-fail=no
 local-ipv6=

# dynamic updates
 dnsupdate=yes
 allow-dnsupdate-from=127.0.0.1

Note it is an IPv4 only setup. It’ll listen only on loopback interface, since no one is supposed to contact him directly beside the recursor sitting on the same loopback.

You can restart the daemon (rc-service pdns restart  with OpenRC, else depending on your init).

PowerDNS recursor :

It is quite straighforward to configure in /etc/powerdns/recursor.conf, this one will listen on LAN addresses (not the loopback):

# restrict netmask allowed to query 
allow-from=127.0.0.1, 10.0.0.0/24, 192.168.1.0/24

daemon=yes

# for local domain, via loopback device, forward queries to the PowerDNS local authoritative server
forward-zones=YOURDOMAIN.LAN=127.0.0.1, 1.168.192.in-addr.arpa=127.0.0.1, 0.0.10.in-addr.arpa=127.0.0.1

# list of IP to listen to
local-address=10.0.0.1, 192.168.1.1
local-port=53

# that is how we will spoof/cache
lua-dns-script=/etc/powerdns/redirect.lua

So all the magic will be done in the /etc/powerdns/redirect.lua script, where the cache LXC container IP is hardcoded (that could be change in future version if necessary):

-- (requires pdns-recursor 4 at least)
-- cached servers
cached = newDS()
cachedest = "10.0.0.88"

-- ads kill list
ads = newDS()
adsdest = "127.0.0.1"

-- hand maintained black list
blacklisted = newDS()
blacklistdest = "127.0.0.1"

function preresolve(dq)
   -- DEBUG
   --pdnslog("Got question for "..dq.qname:toString().." from "..dq.remoteaddr:toString().." to "..dq.localaddr:toString(), pdns.loglevels.Error)
   
   -- handmade domains blacklist
   if(blacklisted:check(dq.qname)) then
      if(dq.qtype == pdns.A) then
	 dq:addAnswer(dq.qtype, blacklistdest)
	 return true
      end
   end
   
   -- spam/ads domains
   if(ads:check(dq.qname)) then
      if(dq.qtype == pdns.A) then
	 dq:addAnswer(dq.qtype, adsdest)
	 return true
      end
   end						
   
   -- cached domains
   if(not cached:check(dq.qname)) then
      -- not cached
      return false
   else
      -- cached: variable answer
      dq.variable = true
      -- request coming from the cache itself
      if(dq.remoteaddr:equal(newCA(cachedest))) then
	 return false
      end
      --  redirect to the cache
      if(dq.qtype == pdns.A) then
	 dq:addAnswer(dq.qtype, cachedest)
      end
   end
   
   return true
end

cached:add(dofile("/etc/powerdns/redirect-cached.lua"))
ads:add(dofile("/etc/powerdns/redirect-ads.lua"))
blacklisted:add(dofile("/etc/powerdns/redirect-blacklisted.lua"))

This script relies on three files to do its magic.

redirect-blacklisted.lua that is hand made blacklist, the default content is:

return{
--"gfe.nvidia.com",
}

redirect-cached.lua is to be generated by redirect-cached-rebuild.sh that you should edit before running, to list which domains you want to cache:

 
#!/bin/sh

DOMAINS=""

# comment this if you dont cache steam
# (note: nginx cache must also cover this)
DOMAINS="$DOMAINS cs.steampowered.com content1.steampowered.com content2.steampowered.com content3.steampowered.com content4.steampowered.com content5.steampowered.com content6.steampowered.com content7.steampowered.com content8.steampowered.com content9.steampowered.com hsar.steampowered.com.edgesuite.net akamai.steamstatic.com content-origin.steampowered.com client-download.steampowered.com steampipe.steamcontent.com steamcontent.com"
# comment this if you dont cache debian
# (useful read: https://wiki.debian.org/DebianGeoMirror )
DOMAINS="$DOMAINS cdn-fastly.deb.debian.org ftp.fr.debian.org ftp.de.debian.org ftp.debian.org security.debian.org"
# comment this if you dont cache devuan
DOMAINS="$DOMAINS packages.devuan.org amprolla.devuan.org"
# comment this if you dont cache ubuntu
DOMAINS="$DOMAINS fr.archive.ubuntu.com security.ubuntu.com"

out=redirect-cached.lua
echo "-- build by ${0}" > $out
echo "-- re-run it commenting relevant domains if you dont cache them all" >> $out
echo "return{" >> $out
for domain in $DOMAINS; do
    echo \"$domain\", >> $out
done
echo "}" >> $out
    
# EOF

 

Finally, redirect-ads.lua is to be generated by redirect-ads-rebuild.pl that you put in a weekly cronjob (following by a pdns-recursor restart):

use strict;
use Fcntl ':flock';

# disallow concurrent run
open(LOCK, "< $0") or die "Failed to ask lock. Exiting"; flock(LOCK, LOCK_EX | LOCK_NB) or die "Unable to lock. This daemon is already alive. Exiting"; open(OUT, "> redirect-ads.lua");

# You can choose between wget or curl. Both rock!
# my $snagger = "curl -q";
my $snagger = "wget -q -O - ";

# List of URLs to find ad servers.
my @urls = ("http://pgl.yoyo.org/adservers/serverlist.php?showintro=0;hostformat=one-line;mimetype=plaintext");

print OUT "return{\n";
# Grab the list of domains and add them to the realm file
foreach my $url (@urls) {
    # Open the curl command
    open(CURL, "$snagger \"$url\" |") || die "Cannot execute $snagger: $@\n";

    printf OUT ("--- Added domains on %s --\n", scalar localtime);

    while () {
	next if /^#/;
	next if /^$/;
	chomp();
	foreach my $domain (split(",")) {
	    print OUT "\"$domain\",\n";

	}
    }
}
print OUT "}\n";

So before starting the recursor, run redirect-ads-rebuild.pl and redirect-cached-rebuild.sh.

Then, after restart, everything should be up and running, with no concerns of inconsistent issues. And, as you can see for yourself, the LUA scripting possibility is as easy as extensible.

Setting up LXC containers to run with ISC DHCPd and BIND instead of dnsmasq, along with domain name spoofing/caching

By default, lxc-net setup containers to work along with dnsmasq, which provides both DNS and dhcpd services, name resolution and IP attribution.

Recommended setup of lxc-net includes /etc/lxc/dnsmasq.conf that only states  “dhcp-hostsfile=…” and the said dhcp-hostfiles as /etc/lxc/dnsmasq-hosts.conf with a line “hostname,IP” per host.

It works fine and there is no real reason to use anything else. Though it is obvious that lxc-net lacks a bit of modularity, since it is clearly tied, hardcoded, to dnsmasq for instance.

Except that on my main server, I already have ISC DHCPd serving IP to local area network and BIND 9 not only doing name resolution caching but also name resolution for said local area network. Not only having both dnsmasq and BIND 9 and ISC DHCPd is a bit overkill, but it requires additional config to bind them to specific interfaces to avoid conflicts.

dnsmasq shutdown

We could simply do a killall dnsmasq and comment the part in /usr/lib/x86_64-linux-gnu/lxc/lxc-net where it get started. For now, we’ll just prevent it from messing with interfaces, setting /etc/lxc/dnsmasq.conf to:

interface=lxcbr0
no-dhcp-interface=lxcbr0

Initial setup

This article assumes you already have BIND and ISC DHCPd set up for local area network (otherwise, as said, in most use cases, dnsmasq will be just fine).

If you do not have a preexisting setup but wants, nonetheless, switch to BIND 9 and ISC DHPCd, you could start with the bind setup provided in my setting up a silent/low energy consumption home server article.

This article includes dynamic clients name update. The only thing to pay attention is that this setup use 10.0.0.0/24 for local area network whereas, in the following article, 10.0.0.0/24 will be used for LXC bridge network while 192.168.1.0/24 will be dedicated to local area network.

DNS setup

I adjusted my preexisting setup (bind9 files part of my -utils-cache-spoof debian package, which I suggest you look at directly to have their current exhaustive content) based on bind9 notion of ACL (access control list) depending on which network clients belongs and, subsequently, bind9 notion of “views” that configure which zones are provided to these clients according to ACL.

The following will seems like a lot but, if you grab my debian -utils-cache-spoof package, it is actually not that much.

Since LXC bridge here is using 10.0.0.0/24 network, I have in named.conf.acl:

[...]

acl lan {
    // the cache host IP should not be part of regular lan ACL
    !10.0.0.88;
    // private IPv4 address spaces
    10.0.0.0/8;
    172.16.0.0/12;
    192.168.0.0/16;
};

acl lannocache {
   // counterpart of earlier statement: cache host needs proper unspoofed name resolution
   10.0.0.88;
};

Note that the .88 container IP is dedicated to caching (apt/steam as in my previous setup with dsniff as spoofer and my another setup using bind9 instead but outside of LXC host/container context) so it needs to be excluded from the general 10.0.0.0/8 ACL.

These ACL are in turn used in named.conf.views (Update: with latest versions of Bind9, we cannot include twice a file that as allow-update statement within, hence the …local and .local_ref):

// clients are set in named.conf.acl
include "/etc/bind/named.conf.acl";

// loopback view, for the server itself
view "loopback" {
 match-clients { loopback; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local";
 include "/etc/bind/named.conf.ads";
};

// otherwise local network area
view "lan" {
 match-clients { lan; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.cache";
 include "/etc/bind/named.conf.ads";
};

// local network area without cache, for host that will get unspoofed name resolution
// (needs to be set up one by one in named.conf.acl)
view "lannocache" {
 match-clients { lannocache; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.ads";
};


[...]

Obviously, if there was no notion of caching (and name spoofing), the setup would be even more straightforward, a single view would be enough. Nonetheless, this example shows an easy way to treat differently hosts depending whether they are LXC containers or regular LAN clients.

About the zones included (or not) in views (all files being in /etc/bind):

  • named.conf.default-zones is standard ;
  • named.conf.local is almost standard, you need to define here your local domains/network ;
  • Update:  named.conf.*_ref  is required with recent version of Bind9 to be able to use twice content for some named.conf.* in which some zone file is defined and can be updated (allow-update) : you’ll will need to use in-view feature to mimic usage of the view that previously defined it since trying another include would sprout writeable file ‘…’ already in use  ;
  • named.conf.cacheBASEIP contains list of spoofed domains, the one we want to cache, generated by named.conf.cache-rebuild.sh, BASEIP being optional;
  • named.conf.ads contains ads servers blacklist generated by update-bind-ads-block.pl ;

So basically, you need to edit /etc/bind/named.conf.local to something like:

// to store A/CNAME records for DOMAIN.EXT
zone "DOMAIN.EXT" {
 type master;
 notify no;
 file "/etc/bind/db.DOMAIN.EXT";
 allow-update { key ddns; };
};

// (we use 192.168.1.0/24 for regular LAN)
// to store PTR records (IP to name) for regular LAN 
zone "1.168.192.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.192.168.1";
 allow-update { key ddns; };
};

// (we use 10.0.0.0/24 for LXC bridge)
// to store PTR records for LXC bridge)
zone "0.0.10.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.10.0.0";
 allow-update { key ddns; };
};

Update: since recent Bind9 update, to be able to reuse these zones in another view, you’ll need to edit /etc/bind/named.conf.local_ref to something like:

// simple reference to previously defined zones for view loopback in named.conf.local
zone "DOMAIN.EXT" { in-view "loopback"; }; 
zone "1.168.192.in-addr.arpa" { in-view "loopback"; }; 
zone "0.0.10.in-addr.arpa" { in-view "loopback"; };

You also require relevant db. files: for instance db.ads pointing to loopback to filter ads/spam sources, db.cache pointing to the cache container .88 (possibly also db.cacheBASEIP) and local db. files as db.DOMAIN.EXT:

$ORIGIN .
$TTL 86400 ; 1 day
DOMAIN.EXT IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                        2823 ; serial
                        28800 ; refresh (8 hours)
                        7200 ; retry (2 hours)
                        604800 ; expire (1 week)
                        10800 ; minimum (3 hours)
                        )
          NS     server.DOMAIN.EXT.
          MX     10 server.DOMAIN.EXT.
$ORIGIN DOMAIN.EXT.
server    A      192.168.1.1
; the rest will be filled by ddns

Likewise, you should have db.192.168.1 and db.10.0.0 (obviously with 1.168.192 replaced by 0.0.10) as:

$ORIGIN .
$TTL 86400 ; 1 day
1.168.192.in-addr.arpa IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                       2803 ; serial
                       28800 ; refresh (8 hours)
                       7200 ; retry (2 hours)
                       604800 ; expire (1 week)
                       10800 ; minimum (3 hours)
                       )
           NS      server.DOMAIN.EXT.
$ORIGIN 1.168.192.in-addr.arpa.
1          PTR     server.DOMAIN.EXT.
; the rest will be filled by ddns too

And then you must run the scripts to generate named.conf.cacheBASEIP and name.conf.ads. You’ll probably need to edit /etc/bind/named.conf.cache-rebuild.sh variables according to what you are actually caching.

BIND gets updates from ISC DHCPd whenever a new clients get a lease, it is configured in name.conf.dhcp (not packaged):

include "/etc/bind/ddns.key";

controls {
 inet 127.0.0.1 allow { localhost; } keys { ddns; };
};

The ddns key was generated as documented in my setting up a silent/low energy consumption home server article as well as in Debian docs:

# dnssec-keygen -a HMAC-MD5 -b 128 -r /dev/urandom -n USER ddns

Out of the generated Kdhcp_updater.*.private, you get the content of the “Key:” statement and you put it in /etc/bind/ddns.key:

key ddns {
 algorithm HMAC-MD5;
 secret "CONTENTOFTHEKEY";
};

So this setup implies that your named.conf looks like:

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.dhcp";
include "/etc/bind/named.conf.views";

Besides, my /etc/bind/named.conf.options is generated by /etc/dhcp/dhclient-exit-hooks.d/bind so it include proper forwarders and listen-on exception.

That should cover it for BIND.

ISC DHPCd setup

In my case, I still still want IPs of LXC containers to be fixed. The syntax of /etc/lxc/dnsmasq-hosts.conf was “hostname,IP” per line which is more convenient than ISC DHCPD syntax  “host hostname { hardware ethernet MAC ADDRESS; fixed-address IP; }”.

I decided to use the same /etc/lxc/dnsmasq-hosts.conf symlinked to /etc/lxc/hosts.conf that will be used by the /etc/lxc/dhcpd-hosts.rebuild.sh (not packaged for now) script to generate /etc/dhcp/dhcpd_lxc-hosts.conf:

#!/bin/bash
# /etc/lxc/dhcpd-hosts.rebuild.sh

HOSTS=/etc/lxc/hosts.conf # similar to dnsmasq-hosts.conf: host,IP
DESTINATION=/etc/dhcp/dhcpd_lxc-hosts.conf
LXC_PATH=`lxc-config lxc.lxcpath`
cd $LXC_PATH

echo > $DESTINATION
for container in *; do
 if [ ! -d "$container" ]; then continue; fi
 if [ ! -e "$container/config" ]; then continue ; fi
 echo "host lxc-$container {" >> $DESTINATION
 echo " hardware ethernet "`cat "$container/config" | grep lxc.network.hwaddr | cut -f 2 -d "="`";" >> $DESTINATI
ON
 echo " fixed-address "`cat "$HOSTS" | grep "$container" | cut -f 2 -d ","`";" >> $DESTINATION
 echo "}" >> $DESTINATION 
done
# EOF

This primitive script will sprout out a proper ISC DHCPd host file. You have to run it each time you create a new container. Once done, we simply edit /etc/dhcp/dhcpd.conf:

# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed. We default to the
# behavior of the version 2 packages ('none', since DHCP v2 didn't
# have support for DDNS.)
ddns-updates on;
ddns-update-style interim;
ddns-domainname "DOMAIN.EXT";
ddns-rev-domainname "in-addr.arpa.";
ignore client-updates; # no touching the FQDN
include "/etc/dhcp/ddns.key";

# option definitions common to all supported networks...
option domain-name "DOMAIN.EXT";
option domain-search "DOMAIN.EXT", "ANOTHERDOMAIN.EXT";
option domain-name-servers 192.168.1.1;
option routers 192.168.1.1;

default-lease-time 600;
max-lease-time 6000;
update-static-leases on;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# LAN clients
subnet 192.168.1.0 netmask 255.255.255.0 {

 # dynamic IP depends whether the client MAC address is known
 pool {
   range 192.168.1.20 192.168.1.99;
   deny unknown-clients;
 }
 pool {
   range 192.168.1.100 192.168.1.250;
   allow unknown-clients; 
 }

 # iPXE / boot on lan
 if exists user-class and option user-class = "iPXE" {
   filename "ipxe-boot";
 } else {
   filename "undionly.kpxe";
 }
 next-server 192.168.1.1;
}

# LXC clients
subnet 10.0.0.0 netmask 255.255.255.0 {
 # use the subnet-specific router
 option routers 10.0.0.1;
 # no pool, all IP are fixed here
 # force lease time to be at least weekly
 min-lease-time 604800;
 max-lease-time 604800;
 # no boot on lan either
}

# zones
zone DOMAIN.EXT. {
 primary 127.0.0.1;
 key ddns;
}
zone 1.168.192.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}
zone 0.0.10.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}


# LAN known clients 
 host trendnetusb { hardware ethernet 00:50:b6:08:xx:xx; }
 host ugreenusb { hardware ethernet 00:0e:c6:fa:xx:xx; }

# LXC host
include "/etc/dhcp/dhcpd_lxc-hosts.conf";

That’s all. Obviously, if you want your LXC containers to get completely dynamically assigned IP, you do not even need this whole host setup. You just set a pool { } with a range of IP (and remove the specif lease time).

The cache LXC container

I wont get in much details, my my -utils-cache-apt and -utils-cache-steam debian packages should work out of the box on a LXC container, providing both the necessary nginx cache-apt and cache-steam config.

If you use resolvconf and ISC DHCP clients on LXC containers, the resolvconf to nginx resolver config script will set up /etc/nginx/conf.d/resolver.conf accordingly.

If you use udhcpc, this resolvconf script will be ignored  but the default /etc/nginx/conf.d/resolver.conf includes, in comments, proposed changes to /etc/udhcpc/default.script to generate  /etc/nginx/conf.d/resolver.conf accordingly.

Otherwise, you need to hand configure /etc/nginx/conf.d/resolver.conf

## (set resolver to something else if your local interface got
## domain names spoofed, 8.8.8.8 for Google resolver for example.
#resolver 127.0.0.1 ipv6=off; # without lxc
resolver 10.0.0.1 ipv6=off;   # within lxc

Troubleshooting

I have this setup since a while and noticed the following:

  • with ISC DHCP client within the LXC containers I get the bad udp checksums in N packets issue;  the iptables -A POSTROUTING -t mangle -p udp –dport 67 -j CHECKSUM  –checksum-fill rule set up by lxc-net is helpless; the solution i picked is to use udhcpc within all LXC containers that does not trigger the problem, with the obvious drawback that the cache container must use the edited /etc/udhcpc/default.script option since resolvconf will have no effect;
  • if ISC DHPCd and Bind9, on the LXC host, are started before or at the same time as lxc and lxc-net, they may not listen on the LXC bridge interface, possibly missing at their starting time; as result, while everything could seem properly on, LXC container would fail to get an IP assigned until you restart ISC DHPCd;  this does not occur after adding lxc lxc-net in the Should-Start: part of ISC DHCPd and Bind9 init.d scripts.
  • Update: With Bind9 recent version (notably: since Debian 9.0), if you have twice a zone defined with a file that can be updated, it wont start and logs will state something like writeable file ‘…’ already in use. The workaround, using in-view, is described in the earlier. Granted, it kills a bit the interest of using view and lead to ugly confusing setup.

Avoiding dnsmasq interference

If you are satistified and do not require dnsmasq anymore, I suggest to remove any dnsmasq package and add a symlink so dnsmasq command produces no error (when called by /usr/lib/x86_64-linux-gnu/lxc/lxc-net for instance):

ln -s /bin/true /usr/bin/dnsmasq

Revision control and distribution of home configuration files with Bash and git

For year, I managed different copies of home configuration files over different hosts with some revision control,  but however better are modern system like git in comparison to old CVS, it would still be quite unpractical to put your whole home directory within one single repository:

  • for obvious reason, there are only a few files that you can actually move around carelessly and put on gitlab for instance; but these files are actually nice to have there, so you can retrieve them whenever and wherever you want;
  • even if you could/would made the rest of your home directory public, most of the configuration files cannot adjust to each host they are run onto; you can obviously adjust a ~/.bashrc according to $HOSTNAME, but it gets a bit more annoying for, say, ~/.Xdefaults;

I am quite sure most people using many different hosts have all their own way to deal with that. There are too many use cases for one solution to be practical for everybody.

I already made public a small script to distributed SSH public keys, that I was using for quite a while before. Next is the script I am using now to distribute home configuration files among hosts: it needs to be added within a git repository (in my case, gitlab “rc” repository), from there, based on a pre-decided list of files or directories:

  • keep a copy of each file/directory per hostname (ex: bashrc.$HOSTNAME, config/awesome.$HOSTNAME);
  • default can be set by renaming $item.$HOSTNAME to $item.default of such file/directory (ex: bashrc.default);

It obeys to the following general rules:

  • it wont copy symlinks but their content;
  • if we only have a local file, save it in the repository;
  • if we have a local file and a repository copy, and if there is a difference, update the repository;
  • if we only have a repository copy, no local file, create the local file with a warning;

Regarding $item.default:

  • $item.default  be will used only unless a $item.$HOSTNAME exists;
  • $item.default will never be updated automatically: if the local copy based on the default is modified, then a $item.$HOSTNAME will be created instead; if it is to made default, you’ll need to rename $item.$HOSTNAME to $item.default; alternatively, you could edit $item.default first and remove the local file at once;
  • similarly, $item.default will never overwrite a local file: to use it on other hosts after an update, the local file will need to be removed;

I admit this $item.default handling is a bit cumbersome but these files update presents risk (lockout, security, etc).

If updaterc exists in the same directory, it will be sourced. It is convenient way to change the $ITEMS variable without editing the script itself.

To use it, you just need to set up and clone some git repository and, within this repository:

wget https://gitlab.com/yeupou/rc/raw/master/update.sh
chmod +x update.sh
# eventually create a custom list of items
echo 'ITEMS="bashrc config/awesome"' > updaterc
# run
./update.sh

The task can be automated by a cronjob, add the following to a call to crontab -e:

3 12 * * * ~/.rc/update.sh >/dev/null 2>/dev/null

(side note: that won’t work properly if one of your hosts is named “default”)

Build a simple kitchen terminal out of an old laptop screen and Raspberry Pi

On some occasion, it is practical to have a terminal in the kitchen, mainly to check on recipes. While a phone screen is not that great, a tablet would do. I do not have any tablet and I am not that fond of systems readily available on tablets. But I do have a few old laptops around plus a Raspberry Pi B+.

Hardware

The following RasPi.TV‘s video explains it all:

Quite straightforward, you unmount and identify your screen. So for my Dell Latitude C640, I got a Samsung LTN141X8-L02 14,1″ screen for which I easily found a controller board kit on ebay for 21,5 €.IMG_20170122_115114.jpg

Here’s the back of the said screen, with the original inverter board still attached. The kit will include another one.

Once acquired, there is not much to think about, everything just have to be plugged where it belongs according to the seller docs:

Q01415750-5.jpg

Obviously, you need to buy also a HDMI cable and a power adapter power adapter (12V, 4A).

IMG_20170301_183541.jpg

Obviously, as it is no tablet, it requires peripherals. I opted for a slim USB wireless keyboard with trackpad and some USB powered stereo speaker. These devices will be powered by the Raspberry Pi (a phone charger can be plugged to the keyboard to recharge it).

Finally, the charger and the Raspberry are plugged onto a power socket with 5V USB. It will be used to put on/off the whole.

Afterwards, I put the screen within a cheap photo frame and fixed the rest on some board.

That frame looks too fragile, though, I would recommend to build a proper one instead.

Software

1/ Raspbian desktop

I first tried some default Raspbian. Epiphany web browser is as bad as you cannot even set a default webpage without editing the .desktop files. And once it is done, it crashes on mediawiki standard page layout. Raspbian also fails to properly open videos (OMX sprout error messages, even with lot of memory attributed). Not convincing.

2/ Kodi media player

Afterwards, I went for LibreElec along with Kodi. Surprisingly, it loads movies with no problem, the interface is quite neat in general and the control with a distant web browser (port 8080 by default) is a plus. As media player, it would be nice.

IMG_20170317_145135.jpg

But it is not perfect: Kodi does not provide any proper web browser, even lacking features. They only provide some cheezy sort of said text web browser. Sort of because it is no lynx/links/elinks, it is just a strange graphical interface with low HTML layout capabilities – but, kudos, it does not crash on mediawiki, yay! Nonetheless, that is quite a blocker issue for me. Even a media player, in my opinion, should have integrated web browser. It is not a challenge to reuse gecko/khtml, whatever, to make so.

IMG_20170317_145231.jpg

3/ (tiger) VNC on top of Raspbian

So I went back on Raspbian. I found out that netsurf works fine to browse mediawiki. So just that satisfies the first requirement.

Instead of expecting to be able to finely setup Raspbian for video website, etc, I decided it might just be smarter to really think of this as terminal and so, to show some window from another computer session.

On an Devuan desktop, it is just enough to get tigervnc-scraping-server, generate a host file (for IP based control):

mkdir .vnc
echo "+IP_OF_YOUR_RASPBIAN" > .vnc/hosts

then to start it whenever you want to share your screen:

x0vncserver -HostsFile=$HOME/.vnc/hosts -SecurityTypes=None

Windows version is configured in a similar fashion.

Raspbian provides a VNC viewer graphical interface that will allow you to connect and you’ll immediately notice that TigerVNC is damned efficient and play with no problem youtube video, etc.

Ok, but VNC, while much more convenient than RDP to setup, does not care to sound forwarding.

I give some tries to PulseAudio RTP capabilities: it fails with errors like [alsa-sink-bcm2835 ALSA] module-rtp-recv.c: Sample rates too different, not adjusting (44100 vs. 90522) and when I tried to document myself about it, I found that this PulseAudio feature was bugged, flooding the network with UDP packets, a bug found in 2009 and still existing in 2017. Gosh, a feature bugged since near to a decade: back to why I try to keep away from systemd and anything made by the same crowd.

I ended up streaming audio with vlc,

cvlc -vvv pulse://`pactl list | grep "Monitor Source" | cut --delimiter ":" -f 2 | tr -d [:blank:]` --sout "#transcode{acodec=mp3,ab=128,channels=2}:standard{access=http,dst=0.0.0.0:9999/pc.mp3}" &

simply played on the Raspbian with:

mpg123 http://hostname:9999/pc.mp3

I has been summarized in a script to be run on the distant host side. I considered stream both audio and video with vlc but it  is convenient to be able to move around with VNC. This will require further testing.

Sharing graphs of multiple Munin (master) instances

Munin is a convenient monitoring tool. Even if it gets old, it is easy to set up and agrement with custom scripts.

It works with the notion of having a master munin process that will grab data from nodes (a device within the network), store it in Round-robin databases (RRD) and process the data  to generate static images and HTML pages. These sequences are split in several scripts: munin-update, munin-limits, munin-graph, munin-html.

It’s fine -overkill?- for a small local network, despite the fact RRD is a bit I/O consuming to the point it may be require to use a caching daemon like rrdcached.

It’s a different story if you want to monitor several small networks that are connected through the internet at once. Why would you? First because it might be convenient to get graphs from different networks side by side. Also because if one network disappear from the internet, data from munin might actually be meaningful, provided you can still access it.

muninex

Problem is munin updates are synchronous: any disconnect between the two would cause the data to be inconsistent. It leads  to many issues that munin-async can help with. But even though you might be able to use munin-async, one of your servers will lack a munin master: the setup will works only when both are up.

So I’m actually much more interested in having a master munin process, for each network.

How to achieve that? It is not an option to share RRD via NFS over the web. I’m also not fan of the notion of having both master munin process read through all RRD and generate graphs in parallel, re-generating exactly the same data with no value added.

I went for an alternative approach with a modified version of the munin-mergedb.pl script. We do not merge RRD trees. We simply synchronize the db files to merge and the generated graphs. So if there are graphs from another munin master process to include in the HTML output, they’ll be there. But munin master process will go undisturbed by any other process unavailability and wont have more RRD to process, more graphs to produce.

Graphs and db files replication:

On both (master munin process) hosts, you need an user dedicated to replication: here.

adduser SYNCUSER munin

This user need ssh access from one host to the other (private/public key sharing, whatever).

Directories setup:

mkdir -p /var/lib/munin-mergedb/
chown munin:munin -R /var/lib/munin-mergedb/
# the +s is very important so directory group ownership is preserved
chmod g+rws -R /var/lib/munin-mergedb/
chmod g+rws /var/lib/munin/
chmod g+rws -R /var/www/html/munin/

On one host (the one allowed to connect through ssh), synchronized two way with unison HTML files:

su - SYNCUSER --shell=/bin/bash

DISTANT_HOST=DISTANTHOST
DISTANT_PORT=22
LOCAL_HTML=/var/www/html/munin/DOMAIN
DISTANT_HTML=/var/www/html/munin/DOMAIN

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST


# step one, get directories
unison -batch -auto -ignore="Name *.html" -ignore="Name *.png" "$LOCAL_HTML" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML"
# step two, get directories img content 
cd "$LOCAL_HTML" && for DIR in *; do [ -d "$DIR" ] && unison -batch -auto -ignore="Name *.html" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR"; done

On one host (the same), synchronized one way with rsync database files:

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST

# push our db (one way action, easier with rsync)
rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$LOCAL_DB/" "$DISTANT_HOST:$DISTANT_LOCAL_DB/"
# get theirs (one way action, easier with rsync)
rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$DISTANT_HOST:$LOCAL_DB/" "$LOCAL_DISTANT_DB/"

If it works fine, set up /etc/cron.d/munin-sync:

# supposed to assist munin-mergedb.pl

DISTANT_HOST=DISTANTHOST
DISTANT_PORT=22

LOCAL_HTML=/var/www/html/munin/DOMAIN
DISTANT_HTML=/var/www/html/munin/DOMAIN

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST

# m h dom mon dow user command
# every 5 hour update dir list
01 */5 * * *  SYNCUSER unison -batch -auto -silent -log=false -ignore="Name *.html" -ignore="Name *.png" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR" 2>/dev/null

#  update content twice per hour
*/28 * * * *  SYNCUSER cd "$LOCAL_HTML" && for DIR in *; do [ -d "$DIR" ] && unison -batch -auto -silent -log=false -ignore="Name *.html" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR" 2>/dev/null; done && rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$LOCAL_DB/" "$DISTANT_HOST:$DISTANT_LOCAL_DB/" 2>/dev/null && rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$DISTANT_HOST:$LOCAL_DB/" "$LOCAL_DISTANT_DB/"2>/dev/null

Updated scripts:

Once data there, you will need munin-mergedb script to handle them, use a munin-cron script like my munin-cron-plus.pl instead of munin-cron so it actually calls munin-mergedb.pl. Plus you’ll need a fixed version of munin-graph so –host arguments are not blattlanly ignored (lacking RRD, it would fail to actually write graph for distant munin master process, but it would nonetheless delete existing graphs).

(Where these files go depends on your munin installation packaging. I have the munin processes in /usr/local/share/munin  and munin-cron-plus.pl in /usr/local/bin – it reflects the fact that original similar files are either in /usr/share/munin or /usr/bin. Beware, if you change the name of any munin process, update log rotation files otherwise you may easily fill up a disk drive, since it is kind of noisy especially when issues arise)

As conveniency, you can download these with my -utils-munin debian/devuan packages:

wget apt.rien.pl/stalag13-keyring.deb
dpkg -i apt.rien.pl/stalag13-keyring.deb
apt-get update
apt-get install stalag13-utils-munin

Once everything set up, you can test/debug it by typing:

su - munin --shell=/bin/bash

/usr/local/bin/munin-cron-plus.pl

What next?

Actually I’d welcome improvements munin-cron-plus.pl since it extract –host information in the most barbaric way. I am sure it can be done cleanly using Munin::Master::Config/else.

Then I’d welcome any insight about why munin-graph’s –host option does not works the way I’d like it. Maybe I misunderstand it’s exact purpose. The help reads:

 --host  Limit graphed hosts to . Multiple --host options
               may be supplied.

To me, it really means that it should not do anything at all to any files of hosts excluded this way. If it meant something else, maybe this should be explained.

Avoiding GPG issues while submitting to popularity-contest on Devuan

For some reason, on Devuan, popularity-contest submits fails with:

gpg: 4383FF7B81EEE66F: skipped: public key not found
gpg: /var/log/popularity-contest.new: encryption failed: public key not found

The Debian Popularity Contest being described as an attempt to map the usage of Debian packages, I think useful that it also get stats from disgrunted Debian users forced to use a fork of the same general scope.

I do not think it data transmitted in this context is really sensitive. So the simplest hack is just to set off encryption by adding to /etc/popularity-contest.conf:

ENCRYPT="no"