Pick a font able to properly render a string composed of Unicode characters with Perl

In the case of automated watermarking with randomly picked fonts within a Perl script, it is quite annoying to stumble on fonts missing many non-basic unicode characters (accents, etc). In French, you’ll likely miss the ê or ü or even é or à. In Polish, while the ł is often provided, you’ll like miss ź.

The Perl module Font::FreeType is quite convenient in this regard. The sample code here will try to find a font, within the @fonts list, able to render the $string.  It will pick the fonts randomly, one by one, and check every character of the string against the characters provided by the font. It will stop to pick the first one that actually can fully render the string:

use Font::FreeType;
use utf8; # must occur before any string definition!
use strict;

my @image_tags = "~ł ajàüd&é)=ù\$;«~źmn";
my @fonts = ("/usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf", "/usr/share/fonts/truetype/zeppelin.ttf", "/usr/share/fonts/truetype/Barrio-Regular.ttf");
my %fonts_characters;
my $watermark_font;

# we want a random font: but we also want a font that can print every character
# (not obvious with utf8)
# loop until we find a suitable one (all chars are valid, so the chars counter reached 0) or,
# worse case scenario, until we checked them all (means more suitable fonts should be added)
my $chars_to_check = length("#".@image_tags[0]);
my $fonts_to_check = scalar(@fonts);
my %fonts_checked;
while ($chars_to_check > 0 and $fonts_to_check > 0) {

 # pick a random font
 srand();
 $watermark_font = $fonts[rand @fonts];
 
 # if this font was already probed, pick another one
 next if $fonts_checked{$watermark_font};
 $fonts_checked{$watermark_font} = 1; 

 # always reset the chars counter each time we try a font
 $chars_to_check = length("#".@image_tags[0]);
 
 print "Selected font $watermark_font (to check: $fonts_to_check)\n";
 
 # if not yet already, build list of available chars with this font
 unless ($fonts_characters{$watermark_font}) {
 Font::FreeType->new->face($watermark_font)->foreach_char(
 sub {
 my $char_chr = chr($_->char_code);
 my $char_code = $_->char_code;
 $fonts_characters{$watermark_font}{$char_chr} = $char_code;
 });
 print "Slurped $watermark_font chars\n";
 }
 
 # then check if every available character of the watermark exists in this font
 for (split //, "#".@image_tags[0]) {
 print "Check $_\n";
 # breaks out if missing char
 last unless $fonts_characters{$watermark_font}{$_};
 # otherwise decrement counter of chars to check: if we reach 0, they are all valid
 # and we should get out of the font picking loop 
 $chars_to_check--;
 print "Chars still to check $chars_to_check\n";
 }
 
 # we also record there is one less font to check
 $fonts_to_check--;
 
}


print "FONT PICKED $watermark_font\n";

This code is actually included in my post-image-to-tumblr.pl script (hence the variables name).

Obviously, if no font is suitable, it’ll take the last one tested. It won’t go as far as comparing which one is the most suitable, since in the context of this script, if no fonts can fully render a tag, the only sensible course is to add more (unicode capable) fonts to the fonts/ directory.

Advertisements

Receiving and parsing DMARC reports on main and secondary mail servers

DMARC is made so aggregated reports will be sent by others servers to yours in order for you to find out if some spam source are trying to impersonate your domains. DMARC working on top of SPF and DKIM, so you need at least one of these. I already mentioned SPF here (even though my setup changed since then, since I use “v=spf1 mx a -all” as SPF DNS record now). If you have neither SPF or DKIM, I suggest you take a look at mail-tester.com first.

The destination of these reports os set in your DMARC DNS record , something like:

v=DMARC1; p=quarantine; rua=mailto:dmarc@thisdomain; sp=quarantine; ri=604800

Unfortunately, these reports are XML and frequent. There are not made to be read by human. And not so many solutions to parse and aggregate these are available; not so surprisingly since lot of customer services based on this are provided by companies.

I do not require anything overly fancy, here’s my basic setup satisfying my modest needs: I only need this to work on two twin mail servers (mxA and mxB here), no matter if one is down.  Since reports are sent by mail, so to the first server that’ll accept them, they need to be replicated from/to mxA to/from mxB.

On each server, create dmarc user. Then create home subdirectories:

adduser dmarc
su dmarc
cd 
mkdir mxA mxB

As suggested earlier, add dmarc user as recipient for the reports in the DNS record:

_dmarc 10800 IN TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@thisdomain; sp=quarantine; ri=604800"

Install the parser, filling database

On each servers, install dmarcts-report-parser.

# included in my -utils-exim deb package but you can simply clone it:
git clone https://github.com/techsneeze/dmarcts-report-parser.git dmarcts-report-parser
cp dmarcts-report-parser/dmarcts-report-parser.pl /usr/local/bin
chmod +x /usr/local/bin/dmarcts-report-parser.pl

# copy conffile (assuming we're on mxA)
cp dmarcts-report-parser/dmarcts-report-parser.conf.sample /home/dmarc/mxA/dmarcts-report-parser.conf

# requires some libraries
apt-get install libmail-imapclient-perl libmime-tools-perl libxml-simple-perl libclass-dbi-mysql-perl libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl libmail-mbox-messageparser-perl unzip

The conffile needs to be modified, most notably:

$delete_reports=1
$delete_failed = 1;

You also need a MySQL server with  dmarc database:

apt-get install mariadb-server
mysql -e "CREATE DATABASE dmarc"

From this moment, the command cd /home/dmarc/mxA/ && dmarcts-report-parser.pl -m *.mbox should already be effective. But you need to fill these mbox files. We’ll do so with a simple ~/.procmailrc as follows:

HERE=mxA
NOTHERE=mxB

DATE=`date +%Y%m%d`
DEFAULT=$HOME/$HERE/$DATE-$HERE.mbox
CLONE=$HOME/$NOTHERE/$DATE-$HERE.mbox

LOGFILE=$HOME/received-reports.log
:0c:
$CLONE

:0:
$DEFAULT

For log rotation, we create a specific ~/.logrotaterc as follows:

/home/dmarc/received-reports.log {
 weekly
 rotate 4
 compress
 missingok
}

Finally, it is just a matter of adding a cronjob so the script looks for .mbox on daily basis (and to rotate logs), crontab -e :

# feed database
15 4 * * * cd ~/mxA/ && dmarcts-report-parser.pl -m *.mbox 2>/dev/null >/dev/null
# rotate logs
0 5 * * * /usr/sbin/logrotate /home/dmarc/.logrotaterc --state /home/dmarc/.logrotate.state

Replicating data

Now, we to set up some ssh access. Only on mxA:

ssh-keygen -t rsa
cat .ssh/id_rsa.pub

The output of the cat command just have to be copied on mxB in ~/.ssh/authorized_keys

Again on mxA, we set up the cronjob doing the actual copy (and removal after copy) with crontab -e

0 3 * * * rsync --quiet --ignore-missing-args --include '*.mbox' --exclude '*' -e "ssh -p 22" mxB.thisdomain:~/mxA/* ~/mxA/ && rsync --quiet --ignore-missing-args --include '*.mbox' --exclude '*' -e "ssh -p 22" ~/mxB/ mxB.thisdomain:~/mxB/* && rm -f ~/mxB/*.mbox && ssh -p 22 mxB.thisdomain 'rm -f ~/mxA/*.mbox'

 

Viewing reports

On and http server configure to run PHP files, install dmarcts-report-parser.

cd /srv
git clone https://github.com/techsneeze/dmarcts-report-viewer.git dmarc
cd dmarc
ln -s dmarcts-report-viewer.php index.php

Eye-candy wise, it is perfectible – but the data is there.

Setting up LXC containers to run with ISC DHCPd and BIND instead of dnsmasq, along with domain name spoofing/caching

By default, lxc-net setup containers to work along with dnsmasq, which provides both DNS and dhcpd services, name resolution and IP attribution.

Recommended setup of lxc-net includes /etc/lxc/dnsmasq.conf that only states  “dhcp-hostsfile=…” and the said dhcp-hostfiles as /etc/lxc/dnsmasq-hosts.conf with a line “hostname,IP” per host.

It works fine and there is no real reason to use anything else. Though it is obvious that lxc-net lacks a bit of modularity, since it is clearly tied, hardcoded, to dnsmasq for instance.

Except that on my main server, I already have ISC DHCPd serving IP to local area network and BIND 9 not only doing name resolution caching but also name resolution for said local area network. Not only having both dnsmasq and BIND 9 and ISC DHCPd is a bit overkill, but it requires additional config to bind them to specific interfaces to avoid conflicts.

dnsmasq shutdown

We could simply do a killall dnsmasq and comment the part in /usr/lib/x86_64-linux-gnu/lxc/lxc-net where it get started. For now, we’ll just prevent it from messing with interfaces, setting /etc/lxc/dnsmasq.conf to:

interface=lxcbr0
no-dhcp-interface=lxcbr0

Initial setup

This article assumes you already have BIND and ISC DHCPd set up for local area network (otherwise, as said, in most use cases, dnsmasq will be just fine).

If you do not have a preexisting setup but wants, nonetheless, switch to BIND 9 and ISC DHPCd, you could start with the bind setup provided in my setting up a silent/low energy consumption home server article.

This article includes dynamic clients name update. The only thing to pay attention is that this setup use 10.0.0.0/24 for local area network whereas, in the following article, 10.0.0.0/24 will be used for LXC bridge network while 192.168.1.0/24 will be dedicated to local area network.

DNS setup

I adjusted my preexisting setup (bind9 files part of my -utils-cache-spoof debian package, which I suggest you look at directly to have their current exhaustive content) based on bind9 notion of ACL (access control list) depending on which network clients belongs and, subsequently, bind9 notion of “views” that configure which zones are provided to these clients according to ACL.

The following will seems like a lot but, if you grab my debian -utils-cache-spoof package, it is actually not that much.

Since LXC bridge here is using 10.0.0.0/24 network, I have in named.conf.acl:

[...]

acl lan {
    // the cache host IP should not be part of regular lan ACL
    !10.0.0.88;
    // private IPv4 address spaces
    10.0.0.0/8;
    172.16.0.0/12;
    192.168.0.0/16;
};

acl lannocache {
   // counterpart of earlier statement: cache host needs proper unspoofed name resolution
   10.0.0.88;
};

Note that the .88 container IP is dedicated to caching (apt/steam as in my previous setup with dsniff as spoofer and my another setup using bind9 instead but outside of LXC host/container context) so it needs to be excluded from the general 10.0.0.0/8 ACL.

These ACL are in turn used in named.conf.views (Update: with latest versions of Bind9, we cannot include twice a file that as allow-update statement within, hence the …local and .local_ref):

// clients are set in named.conf.acl
include "/etc/bind/named.conf.acl";

// loopback view, for the server itself
view "loopback" {
 match-clients { loopback; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local";
 include "/etc/bind/named.conf.ads";
};

// otherwise local network area
view "lan" {
 match-clients { lan; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.cache";
 include "/etc/bind/named.conf.ads";
};

// local network area without cache, for host that will get unspoofed name resolution
// (needs to be set up one by one in named.conf.acl)
view "lannocache" {
 match-clients { lannocache; };
 include "/etc/bind/named.conf.default-zones";
 include "/etc/bind/named.conf.local_ref";
 include "/etc/bind/named.conf.ads";
};


[...]

Obviously, if there was no notion of caching (and name spoofing), the setup would be even more straightforward, a single view would be enough. Nonetheless, this example shows an easy way to treat differently hosts depending whether they are LXC containers or regular LAN clients.

About the zones included (or not) in views (all files being in /etc/bind):

  • named.conf.default-zones is standard ;
  • named.conf.local is almost standard, you need to define here your local domains/network ;
  • Update:  named.conf.*_ref  is required with recent version of Bind9 to be able to use twice content for some named.conf.* in which some zone file is defined and can be updated (allow-update) : you’ll will need to use in-view feature to mimic usage of the view that previously defined it since trying another include would sprout writeable file ‘…’ already in use  ;
  • named.conf.cacheBASEIP contains list of spoofed domains, the one we want to cache, generated by named.conf.cache-rebuild.sh, BASEIP being optional;
  • named.conf.ads contains ads servers blacklist generated by update-bind-ads-block.pl ;

So basically, you need to edit /etc/bind/named.conf.local to something like:

// to store A/CNAME records for DOMAIN.EXT
zone "DOMAIN.EXT" {
 type master;
 notify no;
 file "/etc/bind/db.DOMAIN.EXT";
 allow-update { key ddns; };
};

// (we use 192.168.1.0/24 for regular LAN)
// to store PTR records (IP to name) for regular LAN 
zone "1.168.192.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.192.168.1";
 allow-update { key ddns; };
};

// (we use 10.0.0.0/24 for LXC bridge)
// to store PTR records for LXC bridge)
zone "0.0.10.in-addr.arpa" {
 type master;
 notify no;
 file "/etc/bind/db.10.0.0";
 allow-update { key ddns; };
};

Update: since recent Bind9 update, to be able to reuse these zones in another view, you’ll need to edit /etc/bind/named.conf.local_ref to something like:

// simple reference to previously defined zones for view loopback in named.conf.local
zone "DOMAIN.EXT" { in-view "loopback"; }; 
zone "1.168.192.in-addr.arpa" { in-view "loopback"; }; 
zone "0.0.10.in-addr.arpa" { in-view "loopback"; };

You also require relevant db. files: for instance db.ads pointing to loopback to filter ads/spam sources, db.cache pointing to the cache container .88 (possibly also db.cacheBASEIP) and local db. files as db.DOMAIN.EXT:

$ORIGIN .
$TTL 86400 ; 1 day
DOMAIN.EXT IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                        2823 ; serial
                        28800 ; refresh (8 hours)
                        7200 ; retry (2 hours)
                        604800 ; expire (1 week)
                        10800 ; minimum (3 hours)
                        )
          NS     server.DOMAIN.EXT.
          MX     10 server.DOMAIN.EXT.
$ORIGIN DOMAIN.EXT.
server    A      192.168.1.1
; the rest will be filled by ddns

Likewise, you should have db.192.168.1 and db.10.0.0 (obviously with 1.168.192 replaced by 0.0.10) as:

$ORIGIN .
$TTL 86400 ; 1 day
1.168.192.in-addr.arpa IN SOA server.DOMAIN.EXT. root.DOMAIN.EXT. (
                       2803 ; serial
                       28800 ; refresh (8 hours)
                       7200 ; retry (2 hours)
                       604800 ; expire (1 week)
                       10800 ; minimum (3 hours)
                       )
           NS      server.DOMAIN.EXT.
$ORIGIN 1.168.192.in-addr.arpa.
1          PTR     server.DOMAIN.EXT.
; the rest will be filled by ddns too

And then you must run the scripts to generate named.conf.cacheBASEIP and name.conf.ads. You’ll probably need to edit /etc/bind/named.conf.cache-rebuild.sh variables according to what you are actually caching.

BIND gets updates from ISC DHCPd whenever a new clients get a lease, it is configured in name.conf.dhcp (not packaged):

include "/etc/bind/ddns.key";

controls {
 inet 127.0.0.1 allow { localhost; } keys { ddns; };
};

The ddns key was generated as documented in my setting up a silent/low energy consumption home server article as well as in Debian docs:

# dnssec-keygen -a HMAC-MD5 -b 128 -r /dev/urandom -n USER ddns

Out of the generated Kdhcp_updater.*.private, you get the content of the “Key:” statement and you put it in /etc/bind/ddns.key:

key ddns {
 algorithm HMAC-MD5;
 secret "CONTENTOFTHEKEY";
};

So this setup implies that your named.conf looks like:

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.dhcp";
include "/etc/bind/named.conf.views";

Besides, my /etc/bind/named.conf.options is generated by /etc/dhcp/dhclient-exit-hooks.d/bind so it include proper forwarders and listen-on exception.

That should cover it for BIND.

ISC DHPCd setup

In my case, I still still want IPs of LXC containers to be fixed. The syntax of /etc/lxc/dnsmasq-hosts.conf was “hostname,IP” per line which is more convenient than ISC DHCPD syntax  “host hostname { hardware ethernet MAC ADDRESS; fixed-address IP; }”.

I decided to use the same /etc/lxc/dnsmasq-hosts.conf symlinked to /etc/lxc/hosts.conf that will be used by the /etc/lxc/dhcpd-hosts.rebuild.sh (not packaged for now) script to generate /etc/dhcp/dhcpd_lxc-hosts.conf:

#!/bin/bash
# /etc/lxc/dhcpd-hosts.rebuild.sh

HOSTS=/etc/lxc/hosts.conf # similar to dnsmasq-hosts.conf: host,IP
DESTINATION=/etc/dhcp/dhcpd_lxc-hosts.conf
LXC_PATH=`lxc-config lxc.lxcpath`
cd $LXC_PATH

echo > $DESTINATION
for container in *; do
 if [ ! -d "$container" ]; then continue; fi
 if [ ! -e "$container/config" ]; then continue ; fi
 echo "host lxc-$container {" >> $DESTINATION
 echo " hardware ethernet "`cat "$container/config" | grep lxc.network.hwaddr | cut -f 2 -d "="`";" >> $DESTINATI
ON
 echo " fixed-address "`cat "$HOSTS" | grep "$container" | cut -f 2 -d ","`";" >> $DESTINATION
 echo "}" >> $DESTINATION 
done
# EOF

This primitive script will sprout out a proper ISC DHCPd host file. You have to run it each time you create a new container. Once done, we simply edit /etc/dhcp/dhcpd.conf:

# The ddns-updates-style parameter controls whether or not the server will
# attempt to do a DNS update when a lease is confirmed. We default to the
# behavior of the version 2 packages ('none', since DHCP v2 didn't
# have support for DDNS.)
ddns-updates on;
ddns-update-style interim;
ddns-domainname "DOMAIN.EXT";
ddns-rev-domainname "in-addr.arpa.";
ignore client-updates; # no touching the FQDN
include "/etc/dhcp/ddns.key";

# option definitions common to all supported networks...
option domain-name "DOMAIN.EXT";
option domain-search "DOMAIN.EXT", "ANOTHERDOMAIN.EXT";
option domain-name-servers 192.168.1.1;
option routers 192.168.1.1;

default-lease-time 600;
max-lease-time 6000;
update-static-leases on;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# LAN clients
subnet 192.168.1.0 netmask 255.255.255.0 {

 # dynamic IP depends whether the client MAC address is known
 pool {
   range 192.168.1.20 192.168.1.99;
   deny unknown-clients;
 }
 pool {
   range 192.168.1.100 192.168.1.250;
   allow unknown-clients; 
 }

 # iPXE / boot on lan
 if exists user-class and option user-class = "iPXE" {
   filename "ipxe-boot";
 } else {
   filename "undionly.kpxe";
 }
 next-server 192.168.1.1;
}

# LXC clients
subnet 10.0.0.0 netmask 255.255.255.0 {
 # use the subnet-specific router
 option routers 10.0.0.1;
 # no pool, all IP are fixed here
 # force lease time to be at least weekly
 min-lease-time 604800;
 max-lease-time 604800;
 # no boot on lan either
}

# zones
zone DOMAIN.EXT. {
 primary 127.0.0.1;
 key ddns;
}
zone 1.168.192.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}
zone 0.0.10.in-addr.arpa. {
 primary 127.0.0.1;
 key ddns;
}


# LAN known clients 
 host trendnetusb { hardware ethernet 00:50:b6:08:xx:xx; }
 host ugreenusb { hardware ethernet 00:0e:c6:fa:xx:xx; }

# LXC host
include "/etc/dhcp/dhcpd_lxc-hosts.conf";

That’s all. Obviously, if you want your LXC containers to get completely dynamically assigned IP, you do not even need this whole host setup. You just set a pool { } with a range of IP (and remove the specif lease time).

The cache LXC container

I wont get in much details, my my -utils-cache-apt and -utils-cache-steam debian packages should work out of the box on a LXC container, providing both the necessary nginx cache-apt and cache-steam config.

If you use resolvconf and ISC DHCP clients on LXC containers, the resolvconf to nginx resolver config script will set up /etc/nginx/conf.d/resolver.conf accordingly.

If you use udhcpc, this resolvconf script will be ignored  but the default /etc/nginx/conf.d/resolver.conf includes, in comments, proposed changes to /etc/udhcpc/default.script to generate  /etc/nginx/conf.d/resolver.conf accordingly.

Otherwise, you need to hand configure /etc/nginx/conf.d/resolver.conf

## (set resolver to something else if your local interface got
## domain names spoofed, 8.8.8.8 for Google resolver for example.
#resolver 127.0.0.1 ipv6=off; # without lxc
resolver 10.0.0.1 ipv6=off;   # within lxc

Troubleshooting

I have this setup since a while and noticed the following:

  • with ISC DHCP client within the LXC containers I get the bad udp checksums in N packets issue;  the iptables -A POSTROUTING -t mangle -p udp –dport 67 -j CHECKSUM  –checksum-fill rule set up by lxc-net is helpless; the solution i picked is to use udhcpc within all LXC containers that does not trigger the problem, with the obvious drawback that the cache container must use the edited /etc/udhcpc/default.script option since resolvconf will have no effect;
  • if ISC DHPCd and Bind9, on the LXC host, are started before or at the same time as lxc and lxc-net, they may not listen on the LXC bridge interface, possibly missing at their starting time; as result, while everything could seem properly on, LXC container would fail to get an IP assigned until you restart ISC DHPCd;  this does not occur after adding lxc lxc-net in the Should-Start: part of ISC DHCPd and Bind9 init.d scripts.
  • Update: With Bind9 recent version (notably: since Debian 9.0), if you have twice a zone defined with a file that can be updated, it wont start and logs will state something like writeable file ‘…’ already in use. The workaround, using in-view, is described in the earlier. Granted, it kills a bit the interest of using view and lead to ugly confusing setup.

Avoiding dnsmasq interference

If you are satistified and do not require dnsmasq anymore, I suggest to remove any dnsmasq package and add a symlink so dnsmasq command produces no error (when called by /usr/lib/x86_64-linux-gnu/lxc/lxc-net for instance):

ln -s /bin/true /usr/bin/dnsmasq

Revision control and distribution of home configuration files with Bash and git

For year, I managed different copies of home configuration files over different hosts with some revision control,  but however better are modern system like git in comparison to old CVS, it would still be quite unpractical to put your whole home directory within one single repository:

  • for obvious reason, there are only a few files that you can actually move around carelessly and put on gitlab for instance; but these files are actually nice to have there, so you can retrieve them whenever and wherever you want;
  • even if you could/would made the rest of your home directory public, most of the configuration files cannot adjust to each host they are run onto; you can obviously adjust a ~/.bashrc according to $HOSTNAME, but it gets a bit more annoying for, say, ~/.Xdefaults;

I am quite sure most people using many different hosts have all their own way to deal with that. There are too many use cases for one solution to be practical for everybody.

I already made public a small script to distributed SSH public keys, that I was using for quite a while before. Next is the script I am using now to distribute home configuration files among hosts: it needs to be added within a git repository (in my case, gitlab “rc” repository), from there, based on a pre-decided list of files or directories:

  • keep a copy of each file/directory per hostname (ex: bashrc.$HOSTNAME, config/awesome.$HOSTNAME);
  • default can be set by renaming $item.$HOSTNAME to $item.default of such file/directory (ex: bashrc.default);

It obeys to the following general rules:

  • it wont copy symlinks but their content;
  • if we only have a local file, save it in the repository;
  • if we have a local file and a repository copy, and if there is a difference, update the repository;
  • if we only have a repository copy, no local file, create the local file with a warning;

Regarding $item.default:

  • $item.default  be will used only unless a $item.$HOSTNAME exists;
  • $item.default will never be updated automatically: if the local copy based on the default is modified, then a $item.$HOSTNAME will be created instead; if it is to made default, you’ll need to rename $item.$HOSTNAME to $item.default; alternatively, you could edit $item.default first and remove the local file at once;
  • similarly, $item.default will never overwrite a local file: to use it on other hosts after an update, the local file will need to be removed;

I admit this $item.default handling is a bit cumbersome but these files update presents risk (lockout, security, etc).

If updaterc exists in the same directory, it will be sourced. It is convenient way to change the $ITEMS variable without editing the script itself.

To use it, you just need to set up and clone some git repository and, within this repository:

wget https://gitlab.com/yeupou/rc/raw/master/update.sh
chmod +x update.sh
# eventually create a custom list of items
echo 'ITEMS="bashrc config/awesome"' > updaterc
# run
./update.sh

The task can be automated by a cronjob, add the following to a call to crontab -e:

3 12 * * * ~/.rc/update.sh >/dev/null 2>/dev/null

(side note: that won’t work properly if one of your hosts is named “default”)

Using networked filesystems hosted by LXC containers with Samba

For more than a decade, I used NFS on my home server to share files. I did not consider using Samba for anything but to provide Windows access to shares. NFSv3 then NFSv4 suited me, allowing per host/IP write access policy. The only main drawback was very crude handling of NFS server downtime: X sessions would be half-frozen, requiring restart to be usable once again.

However, I moved recently my servers to LXC (which I’ll probably document a bit later) and NFS server on Debian, as you can guess from nfs-kernel-server package’s name, is kernel-based: not only it apparently defeats the purpose of LXC containers to actually have a server within a container tied to the kernel, but it does not seems to really work reliably. I managed to get it running, but it had to be run on both the master host and within the container. Even then, depending which started first could make the shares unavailable to hosts.

I checked a few articles over the web (https://superuser.com/questions/515080/alternative-to-nfs-or-better-configuration-instable-network-simple-to-set-up, http://serverfault.com/questions/372151/nas-performance-nfs-vs-samba-vs-glusterfs etc) and it looked that, as of today, you can expect decent performances from Samba, as much as of NFS. That could possibly be proven wrong if I was using massively NFS, writing a lot through networked file systems, opening a big number of files simultaneously, moving big files around a lot, but I have really simple requirements: no latency when browsing directories, no latency when playing 720p/1080p videos and that’s about it.

I had already a restricted write access directory per user, via Samba, but I use it only on lame systems as temporary area: on proper systems, I use SSH/scp/rsync/git to manipulate/save files.

Dropping NFS, I have now quite a simple setup, here are relevant parts of my /etc/samba/smb.conf:

[global]

## Browsing/Identification ###

# What naming service and in what order should we use to resolve host names
# to IP addresses
 name resolve order = lmhosts host wins bcast


#### Networking ####

# The specific set of interfaces / networks to bind to
# This can be either the interface name or an IP address/netmask;
# interface names are normally preferred
 interfaces = eth0

# Only bind to the named interfaces and/or networks; you must use the
# 'interfaces' option above to use this.
# It is recommended that you enable this feature if your Samba machine is
# not protected by a firewall or is a firewall itself. However, this
# option cannot handle dynamic or non-broadcast interfaces correctly.
 bind interfaces only = true


#### File names ####

# remove characters forbidden on Windows
mangled names = no

# charsets
dos charset = iso8859-15
unix charset = UTF8


####### Authentication #######

# "security = user" is always a good idea. This will require a Unix account
# in this server for every user accessing the server. See
# /usr/share/doc/samba-doc/htmldocs/Samba3-HOWTO/ServerType.html
# in the samba-doc package for details.
 security = user

# Private network
 hosts allow = 192.168.1.


# You may wish to use password encryption. See the section on
# 'encrypt passwords' in the smb.conf(5) manpage before enabling.
 encrypt passwords = true

# If you are using encrypted passwords, Samba will need to know what
# password database type you are using. 
 passdb backend = tdbsam

obey pam restrictions = yes

guest account = nobody
 invalid users = root bin daemon adm sync shutdown halt mail news uucp operator www-data sshd Debian-exim debian-transmission
 map to guest = bad user

# This boolean parameter controls whether Samba attempts to sync the Unix
# password with the SMB password when the encrypted SMB password in the
# passdb is changed.
 unix password sync = yes


#======================= Share Definitions =======================


realm = ...


[commun]
comment = Commun
path = /srv/common
browseable = yes
writable = yes
public = yes
guest ok = yes
valid users = @smbusers
force group = smbusers
create mode = 0660
directory mode = 0770
force create mode = 0660
force directory mode = 0770

[tmpthisuser]
comment = Données protégées
path = /srv/users/thisuser
browseable = yes
writable = yes
public = yes
valid users = thisuser
create mode = 0600
directory mode = 0700
force create mode = 0600
force directory mode = 0700
guest ok = no

 

I installed package libpam-smbpass and edited /etc/pam.d/samba as follow:

@include common-auth
@include common-account
@include common-session-noninteractive
@include common-password

For this setup to work, you need every user allowed to connect:

  • to be member of group smbusers – including nobody (or whatever the guest account is) ;
  • to have a unix password set ;
  • to be known to samba (smbpasswd -e thisuser or option -a).

If you are not interested in per user access restricted area, only nobody account will need to be taken care of.

And, obviously, files and directories ownership and modes must be set accordingly:

cd /srv/common
# (0770/drwxrwx---) GID : (nnnnn/smbusers)
find . -type d -print0 | xargs -0 chmod 770 -v
find . -type f -print0 | xargs -0 chmod 660 -v
cd /srv/users
# (0700/drwx------) UID : ( nnnn/ thisuser) GID : ( nnnn/ thisuser)
find . -type d -print0 | xargs -0 chmod 700 -v
find . -type f -print0 | xargs -0 chmod 600 -v
# main directories, in addition, need sticky bit some future directory get proper modes
chmod 2770 /srv/common/*
chmod 2700 /srv/users/*

To access this transparently over GNU/Linux systems, just add in /etc/fstab:

//servername/commun /mountpoint cifs guest,uid=nobody,gid=users,iocharset=utf8 0 0

This assumes that any users entitled to access files belongs to users group. If not, update accordingly.

With this setup, there is no longer any IP based specific write access set but, over years, I found out it was quite useless for my setup.

The only issue I have is with files with colon within  (“:”). Due to MS Windows limitations, CIFS list these files but access is made impossible. The easier fix I found was to actually rename these files (not a problem due to the nature of the files served) through a cronjob /etc/cron.hourly/uncolon :

#!/bin/bash
# a permanent cifs based fix would be welcomed
find "/srv" -name '*:*' -exec rename 's/://g' {} +

but I’d be interested in better options.

 

 

Build a simple kitchen terminal out of an old laptop screen and Raspberry Pi

On some occasion, it is practical to have a terminal in the kitchen, mainly to check on recipes. While a phone screen is not that great, a tablet would do. I do not have any tablet and I am not that fond of systems readily available on tablets. But I do have a few old laptops around plus a Raspberry Pi B+.

Hardware

The following RasPi.TV‘s video explains it all:

Quite straightforward, you unmount and identify your screen. So for my Dell Latitude C640, I got a Samsung LTN141X8-L02 14,1″ screen for which I easily found a controller board kit on ebay for 21,5 €.IMG_20170122_115114.jpg

Here’s the back of the said screen, with the original inverter board still attached. The kit will include another one.

Once acquired, there is not much to think about, everything just have to be plugged where it belongs according to the seller docs:

Q01415750-5.jpg

Obviously, you need to buy also a HDMI cable and a power adapter power adapter (12V, 4A).

IMG_20170301_183541.jpg

Obviously, as it is no tablet, it requires peripherals. I opted for a slim USB wireless keyboard with trackpad and some USB powered stereo speaker. These devices will be powered by the Raspberry Pi (a phone charger can be plugged to the keyboard to recharge it).

Finally, the charger and the Raspberry are plugged onto a power socket with 5V USB. It will be used to put on/off the whole.

Afterwards, I put the screen within a cheap photo frame and fixed the rest on some board.

That frame looks too fragile, though, I would recommend to build a proper one instead.

Software

1/ Raspbian desktop

I first tried some default Raspbian. Epiphany web browser is as bad as you cannot even set a default webpage without editing the .desktop files. And once it is done, it crashes on mediawiki standard page layout. Raspbian also fails to properly open videos (OMX sprout error messages, even with lot of memory attributed). Not convincing.

2/ Kodi media player

Afterwards, I went for LibreElec along with Kodi. Surprisingly, it loads movies with no problem, the interface is quite neat in general and the control with a distant web browser (port 8080 by default) is a plus. As media player, it would be nice.

IMG_20170317_145135.jpg

But it is not perfect: Kodi does not provide any proper web browser, even lacking features. They only provide some cheezy sort of said text web browser. Sort of because it is no lynx/links/elinks, it is just a strange graphical interface with low HTML layout capabilities – but, kudos, it does not crash on mediawiki, yay! Nonetheless, that is quite a blocker issue for me. Even a media player, in my opinion, should have integrated web browser. It is not a challenge to reuse gecko/khtml, whatever, to make so.

IMG_20170317_145231.jpg

3/ (tiger) VNC on top of Raspbian

So I went back on Raspbian. I found out that netsurf works fine to browse mediawiki. So just that satisfies the first requirement.

Instead of expecting to be able to finely setup Raspbian for video website, etc, I decided it might just be smarter to really think of this as terminal and so, to show some window from another computer session.

On an Devuan desktop, it is just enough to get tigervnc-scraping-server, generate a host file (for IP based control):

mkdir .vnc
echo "+IP_OF_YOUR_RASPBIAN" > .vnc/hosts

then to start it whenever you want to share your screen:

x0vncserver -HostsFile=$HOME/.vnc/hosts -SecurityTypes=None

Windows version is configured in a similar fashion.

Raspbian provides a VNC viewer graphical interface that will allow you to connect and you’ll immediately notice that TigerVNC is damned efficient and play with no problem youtube video, etc.

Ok, but VNC, while much more convenient than RDP to setup, does not care to sound forwarding.

I give some tries to PulseAudio RTP capabilities: it fails with errors like [alsa-sink-bcm2835 ALSA] module-rtp-recv.c: Sample rates too different, not adjusting (44100 vs. 90522) and when I tried to document myself about it, I found that this PulseAudio feature was bugged, flooding the network with UDP packets, a bug found in 2009 and still existing in 2017. Gosh, a feature bugged since near to a decade: back to why I try to keep away from systemd and anything made by the same crowd.

I ended up streaming audio with vlc,

cvlc -vvv pulse://`pactl list | grep "Monitor Source" | cut --delimiter ":" -f 2 | tr -d [:blank:]` --sout "#transcode{acodec=mp3,ab=128,channels=2}:standard{access=http,dst=0.0.0.0:9999/pc.mp3}" &

simply played on the Raspbian with:

mpg123 http://hostname:9999/pc.mp3

I has been summarized in a script to be run on the distant host side. I considered stream both audio and video with vlc but it  is convenient to be able to move around with VNC. This will require further testing.

Add proportional label/watermark to images with ImageMagick

There is nothing exceptional here, there are many documented ways to achieve this.

Still, follows the most efficient way I found to do so on any images within a directory, after testing quite a few: I wanted to be able to add a small tag on top, size being propertional to the image (that excluded many solutions based on -pointsize), on static images as well as animated ones.

#!/bin/bash
rm -f *marked*

for pic in *.jpg *.png *.gif; do
   ext="${pic##*.}"
   # identify is messed up with gif (give width per frame), use convert
   # instead
   width=$(convert $pic -print "%w" /dev/null)
   height=$(expr 5 \* $width \/ 100)

   # first build a watermark so we can easily manipulate it is size afterwards
   convert -size ${width}x${height} -gravity NorthEast -stroke "#454545" -fill "#c8c8c8" -background transparent -strokewidth 1 -font LiberationSerif-Bold.ttf label:"$pic" ${pic//.$ext}-mark.png
 
   # then add it
   convert $pic -coalesce -gravity NorthEast null: ${pic//.$ext}-mark.png -layers composite -layers optimize ${pic//.$ext}-marked.$ext

   rm ${pic//.$ext}-mark.png
done

Within a perl script, it would be something like (beware, most variables were set earlier in the script):

    # build watermark (for now minimal checks, assume files are regular) 
    my ($watermark_h, $watermark) = tempfile(SUFFIX => ".png", 
					     UNLINK => $unlink_temp);
    binmode($watermark_h);
    my $watermark_width = $$image_info{"ImageWidth"};
    my $watermark_height = $watermark_proportion * $watermark_width / 100;
    system($convert,
	   "-size", $watermark_width."x".$watermark_height,
	   "-gravity", "NorthEast",
	   "-stroke", "#454545",
	   "-fill", "#c8c8c8",
	   "-background", "transparent",
	   "-strokewidth", "1",
	   "-font", $watermark_font,
	   "label:#".@image_tags[0],
	   $watermark);

    # add watermark
    system($convert,
	   $altered_image,
	   "-coalesce",
	   "-gravity", "NorthEast",
	   "null:",
	   $watermark,
	   "-layers", "composite",
	   "-layers", "optimize",
	   $altered_image);    
}

Yeah, so that does mean that my script post-image-to-tumblr.pl managing a tumblr posts-queue locally with #Tags was updated to add these #Tags as watermark/label. Note that it won’t alter the original file, only the one posted.

New options can be added in ~/.tumblrrc:

# no watermark/label
no_watermark 

# specific font
watermark_font=/path/to/font

Note that it requires ImageMagick to be installed.