Revision control and distribution of home configuration files with Bash and git

For year, I managed different copies of home configuration files over different hosts with some revision control,  but however better are modern system like git in comparison to old CVS, it would still be quite unpractical to put your whole home directory within one single repository:

  • for obvious reason, there are only a few files that you can actually move around carelessly and put on gitlab for instance; but these files are actually nice to have there, so you can retrieve them whenever and wherever you want;
  • even if you could/would made the rest of your home directory public, most of the configuration files cannot adjust to each host they are run onto; you can obviously adjust a ~/.bashrc according to $HOSTNAME, but it gets a bit more annoying for, say, ~/.Xdefaults;

I am quite sure most people using many different hosts have all their own way to deal with that. There are too many use cases for one solution to be practical for everybody.

I already made public a small script to distributed SSH public keys, that I was using for quite a while before. Next is the script I am using now to distribute home configuration files among hosts: it needs to be added within a git repository (in my case, gitlab “rc” repository), from there, based on a pre-decided list of files or directories:

  • keep a copy of each file/directory per hostname (ex: bashrc.$HOSTNAME, config/awesome.$HOSTNAME);
  • default can be set by renaming $item.$HOSTNAME to $item.default of such file/directory (ex: bashrc.default);

It obeys to the following general rules:

  • it wont copy symlinks but their content;
  • if we only have a local file, save it in the repository;
  • if we have a local file and a repository copy, and if there is a difference, update the repository;
  • if we only have a repository copy, no local file, create the local file with a warning;

Regarding $item.default:

  • $item.default  be will used only unless a $item.$HOSTNAME exists;
  • $item.default will never be updated automatically: if the local copy based on the default is modified, then a $item.$HOSTNAME will be created instead; if it is to made default, you’ll need to rename $item.$HOSTNAME to $item.default; alternatively, you could edit $item.default first and remove the local file at once;
  • similarly, $item.default will never overwrite a local file: to use it on other hosts after an update, the local file will need to be removed;

I admit this $item.default handling is a bit cumbersome but these files update presents risk (lockout, security, etc).

If updaterc exists in the same directory, it will be sourced. It is convenient way to change the $ITEMS variable without editing the script itself.

To use it, you just need to set up and clone some git repository and, within this repository:

wget https://gitlab.com/yeupou/rc/raw/master/update.sh
chmod +x update.sh
# eventually create a custom list of items
echo 'ITEMS="bashrc config/awesome"' > updaterc
# run
./update.sh

The task can be automated by a cronjob, add the following to a call to crontab -e:

3 12 * * * ~/.rc/update.sh >/dev/null 2>/dev/null

(side note: that won’t work properly if one of your hosts is named “default”)

Build a simple kitchen terminal out of an old laptop screen and Raspberry Pi

On some occasion, it is practical to have a terminal in the kitchen, mainly to check on recipes. While a phone screen is not that great, a tablet would do. I do not have any tablet and I am not that fond of systems readily available on tablets. But I do have a few old laptops around plus a Raspberry Pi B+.

Hardware

The following RasPi.TV‘s video explains it all:

Quite straightforward, you unmount and identify your screen. So for my Dell Latitude C640, I got a Samsung LTN141X8-L02 14,1″ screen for which I easily found a controller board kit on ebay for 21,5 €.IMG_20170122_115114.jpg

Here’s the back of the said screen, with the original inverter board still attached. The kit will include another one.

Once acquired, there is not much to think about, everything just have to be plugged where it belongs according to the seller docs:

Q01415750-5.jpg

Obviously, you need to buy also a HDMI cable and a power adapter power adapter (12V, 4A).

IMG_20170301_183541.jpg

Obviously, as it is no tablet, it requires peripherals. I opted for a slim USB wireless keyboard with trackpad and some USB powered stereo speaker. These devices will be powered by the Raspberry Pi (a phone charger can be plugged to the keyboard to recharge it).

Finally, the charger and the Raspberry are plugged onto a power socket with 5V USB. It will be used to put on/off the whole.

Afterwards, I put the screen within a cheap photo frame and fixed the rest on some board.

That frame looks too fragile, though, I would recommend to build a proper one instead.

Software

1/ Raspbian desktop

I first tried some default Raspbian. Epiphany web browser is as bad as you cannot even set a default webpage without editing the .desktop files. And once it is done, it crashes on mediawiki standard page layout. Raspbian also fails to properly open videos (OMX sprout error messages, even with lot of memory attributed). Not convincing.

2/ Kodi media player

Afterwards, I went for LibreElec along with Kodi. Surprisingly, it loads movies with no problem, the interface is quite neat in general and the control with a distant web browser (port 8080 by default) is a plus. As media player, it would be nice.

IMG_20170317_145135.jpg

But it is not perfect: Kodi does not provide any proper web browser, even lacking features. They only provide some cheezy sort of said text web browser. Sort of because it is no lynx/links/elinks, it is just a strange graphical interface with low HTML layout capabilities – but, kudos, it does not crash on mediawiki, yay! Nonetheless, that is quite a blocker issue for me. Even a media player, in my opinion, should have integrated web browser. It is not a challenge to reuse gecko/khtml, whatever, to make so.

IMG_20170317_145231.jpg

3/ (tiger) VNC on top of Raspbian

So I went back on Raspbian. I found out that netsurf works fine to browse mediawiki. So just that satisfies the first requirement.

Instead of expecting to be able to finely setup Raspbian for video website, etc, I decided it might just be smarter to really think of this as terminal and so, to show some window from another computer session.

On an Devuan desktop, it is just enough to get tigervnc-scraping-server, generate a host file (for IP based control):

mkdir .vnc
echo "+IP_OF_YOUR_RASPBIAN" > .vnc/hosts

then to start it whenever you want to share your screen:

x0vncserver -HostsFile=$HOME/.vnc/hosts -SecurityTypes=None

Windows version is configured in a similar fashion.

Raspbian provides a VNC viewer graphical interface that will allow you to connect and you’ll immediately notice that TigerVNC is damned efficient and play with no problem youtube video, etc.

Ok, but VNC, while much more convenient than RDP to setup, does not care to sound forwarding.

I give some tries to PulseAudio RTP capabilities: it fails with errors like [alsa-sink-bcm2835 ALSA] module-rtp-recv.c: Sample rates too different, not adjusting (44100 vs. 90522) and when I tried to document myself about it, I found that this PulseAudio feature was bugged, flooding the network with UDP packets, a bug found in 2009 and still existing in 2017. Gosh, a feature bugged since near to a decade: back to why I try to keep away from systemd and anything made by the same crowd.

I ended up streaming audio with vlc,

cvlc -vvv pulse://`pactl list | grep "Monitor Source" | cut --delimiter ":" -f 2 | tr -d [:blank:]` --sout "#transcode{acodec=mp3,ab=128,channels=2}:standard{access=http,dst=0.0.0.0:9999/pc.mp3}" &

simply played on the Raspbian with:

mpg123 http://hostname:9999/pc.mp3

I has been summarized in a script to be run on the distant host side. I considered stream both audio and video with vlc but it  is convenient to be able to move around with VNC. This will require further testing.

Sharing graphs of multiple Munin (master) instances

Munin is a convenient monitoring tool. Even if it gets old, it is easy to set up and agrement with custom scripts.

It works with the notion of having a master munin process that will grab data from nodes (a device within the network), store it in Round-robin databases (RRD) and process the data  to generate static images and HTML pages. These sequences are split in several scripts: munin-update, munin-limits, munin-graph, munin-html.

It’s fine -overkill?- for a small local network, despite the fact RRD is a bit I/O consuming to the point it may be require to use a caching daemon like rrdcached.

It’s a different story if you want to monitor several small networks that are connected through the internet at once. Why would you? First because it might be convenient to get graphs from different networks side by side. Also because if one network disappear from the internet, data from munin might actually be meaningful, provided you can still access it.

muninex

Problem is munin updates are synchronous: any disconnect between the two would cause the data to be inconsistent. It leads  to many issues that munin-async can help with. But even though you might be able to use munin-async, one of your servers will lack a munin master: the setup will works only when both are up.

So I’m actually much more interested in having a master munin process, for each network.

How to achieve that? It is not an option to share RRD via NFS over the web. I’m also not fan of the notion of having both master munin process read through all RRD and generate graphs in parallel, re-generating exactly the same data with no value added.

I went for an alternative approach with a modified version of the munin-mergedb.pl script. We do not merge RRD trees. We simply synchronize the db files to merge and the generated graphs. So if there are graphs from another munin master process to include in the HTML output, they’ll be there. But munin master process will go undisturbed by any other process unavailability and wont have more RRD to process, more graphs to produce.

Graphs and db files replication:

On both (master munin process) hosts, you need an user dedicated to replication: here.

adduser SYNCUSER munin

This user need ssh access from one host to the other (private/public key sharing, whatever).

Directories setup:

mkdir -p /var/lib/munin-mergedb/
chown munin:munin -R /var/lib/munin-mergedb/
# the +s is very important so directory group ownership is preserved
chmod g+rws -R /var/lib/munin-mergedb/
chmod g+rws /var/lib/munin/
chmod g+rws -R /var/www/html/munin/

On one host (the one allowed to connect through ssh), synchronized two way with unison HTML files:

su - SYNCUSER --shell=/bin/bash

DISTANT_HOST=DISTANTHOST
DISTANT_PORT=22
LOCAL_HTML=/var/www/html/munin/DOMAIN
DISTANT_HTML=/var/www/html/munin/DOMAIN

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST


# step one, get directories
unison -batch -auto -ignore="Name *.html" -ignore="Name *.png" "$LOCAL_HTML" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML"
# step two, get directories img content 
cd "$LOCAL_HTML" && for DIR in *; do [ -d "$DIR" ] && unison -batch -auto -ignore="Name *.html" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR"; done

On one host (the same), synchronized one way with rsync database files:

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST

# push our db (one way action, easier with rsync)
rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$LOCAL_DB/" "$DISTANT_HOST:$DISTANT_LOCAL_DB/"
# get theirs (one way action, easier with rsync)
rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$DISTANT_HOST:$LOCAL_DB/" "$LOCAL_DISTANT_DB/"

If it works fine, set up /etc/cron.d/munin-sync:

# supposed to assist munin-mergedb.pl

DISTANT_HOST=DISTANTHOST
DISTANT_PORT=22

LOCAL_HTML=/var/www/html/munin/DOMAIN
DISTANT_HTML=/var/www/html/munin/DOMAIN

LOCAL_DB=/var/lib/munin
DISTANT_LOCAL_DB=/var/lib/munin-mergedb/THISHOST
LOCAL_DISTANT_DB=/var/lib/munin-mergedb/DISTANTHOST

# m h dom mon dow user command
# every 5 hour update dir list
01 */5 * * *  SYNCUSER unison -batch -auto -silent -log=false -ignore="Name *.html" -ignore="Name *.png" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR" 2>/dev/null

#  update content twice per hour
*/28 * * * *  SYNCUSER cd "$LOCAL_HTML" && for DIR in *; do [ -d "$DIR" ] && unison -batch -auto -silent -log=false -ignore="Name *.html" "$LOCAL_HTML/$DIR" "ssh://$DISTANT_HOST:$DISTANT_PORT/$DISTANT_HTML/$DIR" 2>/dev/null; done && rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$LOCAL_DB/" "$DISTANT_HOST:$DISTANT_LOCAL_DB/" 2>/dev/null && rsync -a --include='datafile*' --include='limits*' --exclude='*' -e "ssh -p $DISTANT_PORT" "$DISTANT_HOST:$LOCAL_DB/" "$LOCAL_DISTANT_DB/"2>/dev/null

Updated scripts:

Once data there, you will need munin-mergedb script to handle them, use a munin-cron script like my munin-cron-plus.pl instead of munin-cron so it actually calls munin-mergedb.pl. Plus you’ll need a fixed version of munin-graph so –host arguments are not blattlanly ignored (lacking RRD, it would fail to actually write graph for distant munin master process, but it would nonetheless delete existing graphs).

(Where these files go depends on your munin installation packaging. I have the munin processes in /usr/local/share/munin  and munin-cron-plus.pl in /usr/local/bin – it reflects the fact that original similar files are either in /usr/share/munin or /usr/bin. Beware, if you change the name of any munin process, update log rotation files otherwise you may easily fill up a disk drive, since it is kind of noisy especially when issues arise)

As conveniency, you can download these with my -utils-munin debian/devuan packages:

wget apt.rien.pl/stalag13-keyring.deb
dpkg -i apt.rien.pl/stalag13-keyring.deb
apt-get update
apt-get install stalag13-utils-munin

Once everything set up, you can test/debug it by typing:

su - munin --shell=/bin/bash

/usr/local/bin/munin-cron-plus.pl

What next?

Actually I’d welcome improvements munin-cron-plus.pl since it extract –host information in the most barbaric way. I am sure it can be done cleanly using Munin::Master::Config/else.

Then I’d welcome any insight about why munin-graph’s –host option does not works the way I’d like it. Maybe I misunderstand it’s exact purpose. The help reads:

 --host  Limit graphed hosts to . Multiple --host options
               may be supplied.

To me, it really means that it should not do anything at all to any files of hosts excluded this way. If it meant something else, maybe this should be explained.

Avoiding GPG issues while submitting to popularity-contest on Devuan

For some reason, on Devuan, popularity-contest submits fails with:

gpg: 4383FF7B81EEE66F: skipped: public key not found
gpg: /var/log/popularity-contest.new: encryption failed: public key not found

The Debian Popularity Contest being described as an attempt to map the usage of Debian packages, I think useful that it also get stats from disgrunted Debian users forced to use a fork of the same general scope.

I do not think it data transmitted in this context is really sensitive. So the simplest hack is just to set off encryption by adding to /etc/popularity-contest.conf:

ENCRYPT="no"

Preventing filenames with semicolon “:” to be garbled by Samba

In some cases, Samba garble file names, as backward compatibility with old Microsoft Windows system that cannot handle long filenames or filenames with specific characters. It would then be shown with the form XXXXX~X.ext

You can switch off this mecanism:

In /etc/samba/smb.conf, in [Global], add:

mangle case = no
mangled names = no

Then simply restart Samba (invoke-rc.d samba restart).

Files then will be listed with the real name. Not sure Microsoft Windows will however allow you to open the files.

Using GnuPG/PGP on multiple devices

I started using GnuPG in 2002. I dont usually do stuff that requires heavy privacy so I dont care much for it. From time to time, I just encrypt some useless crap so if anyday I had serious stuff to encrypt it would not look obviously suspicious.

Things is most of the people I communicate with are not using GnuPG and are probably not about to.

There is also an obvious issue with GnuPG is how to share key among computers/clients. How to decrypt messages with your phone or webmail? Copy the private key everywhere? It might just be worse than having no security at all.

I dont use GnuPG much, especially since I created my key in 2002 and don’t even know how secure this key is still now. I need it nonetheless to sign stuff like packages. Confronted to the problem of having to copy the key by hand on one more laptop,  I considered dropping my current set  and, inspired by this example of  primary key/subkeys model and debian’s one, to have a primary key secure somewhere and give a short-lived subkey per device.

But, it fixes not much of GnuPG problems, and implies lot of annoying not automated work, not satisfying. And anyway, if en/decrypt can only work for one subkey. So one subkey per device is not really working.

To make the process less painy, on a box being made from time to time available over network, I did as follow:

I created a primary key running gpg --expert --gen-key (cannot sign, cannot encrypt) with 4y expiry. (more entropy with rngd -r /dev/urandom).

I added with adduid and trust save the relevant additional addresses running gpg --expert --edit-key myemail

I created a sign and a encrypt subkey with no expiry (considering that they ll be revoked on the fly from the primary whenever it make sense).

I made up gpg-grabsub.sh that prompt for the hostname of the box hosting the keyring, will import the ring and remove the primary key from it, leaving just the necessary keys to sign and encrypt.

This script could probably be used in a chain (box secured from the net -> script run on gate server -> script run on a end client). It requires further testing.