Converting video files to H.265/HEVC, no washed out colors and all streams mapped, with ffmpeg

I had a few 1080p video files using AV1 codec. Not sure why, if it is a player issue or hardware issue, nonetheless, my (slightly aging) laptop was having a hard time playing these while playing with no effort at all H.265/HEVC 2160p videos. The following command converts all mkv files in a folder to H.265/HEVC, not washing out colors and keep all streams:

for av in *.mkv; do ffmpeg -i "$av" -c:v libx265 -color_trc smpte2084 -color_primaries bt2020 -crf 22 -map 0 -map_metadata 0 -map_chapters 0 -c:a copy -c:s copy "${av%.*}"-x265.mkv ; done

In my case, it results in slightly larger files but, and that was the point, these play on the laptop with no noticeable CPU-usage:

1,2G 'XX1 AV1 Opus [AV1D].mkv'
1,1G 'XX1 AV1 Opus [AV1D]-x265.mkv'
830M 'XX2 AV1 Opus [AV1D].mkv'
951M 'XX2 AV1 Opus [AV1D]-x265.mkv'

Sorting and moving automatically videos from download to storage directory

In the spirit the earlier scripts to clean up ogg/mp3 collection (tags, filenames) with lltag, the following script is a proposal to automatically move, from download directory to storage directory, the video files that deserved to be kept. This is especially useful when both directories are on different physical drives and, as such, take a while, and typically with implies heavy IO usage at the moment you are actually using the computer that host the drivers.

The idea is to put a simple mark on directories or files, in the download area, that should be move to the storage area, assuming storage area contains top directories to sort files. The mark is arbitrary: ##Mark##, Mark matching a top directory within the download area.

For example, say we have in the download area the file Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.avi, associated with a subtitle file Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.en.srt, within a directory named Vabank II. It must go in storage area top directory Action Espionnage.

The way the script sort-download-area.pl works requires only the Vabank II or Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.avi to be renamed to include ##Action Espionnage##. And, obviously, to make it more practical, if can be also ##Action##, ##action##, ##Espionnage##, ##AE##, ##ActionEspionnage##, and others aliases as long as they are not confusing regarding other top directories of the download area.

Then running sort-download-area.pl --download /mnt/download --storage /mnt/storage (assuming these are the relevant directories) will take care of moving the found video and text/subtitles files (based on mime-type and filenames). The old directory will remain, the script won’t take any risk to erase any data by itself.

It can be run with --debug option to make a dry-run, to check if everything is in order, list possible marks, etc. If run as root, it will take care of changing mode and ownership to match the relevant download area top directory.

Once a satisfying setup is in place (assuming the script is in /usr/local/bin), it is enough to add a /etc/cron.daily/sort-download-area like:

#!/bin/sh
/usr/local/bin/sort-download-area.pl  --download /mnt/download --storage /mnt/storage

Here the current version of the sort-download-area.pl (but you are advised to always take the latest gitlab version) :

#!/usr/bin/perl

use strict "vars";
use Fcntl ':flock';
use POSIX qw(strftime);
use File::Find;
use File::Basename;
use File::Path qw(make_path);
use File::Copy qw(move);
use File::MimeInfo;
use File::Slurp qw(read_dir);
use Getopt::Long;
use Term::ANSIColor qw(:constants);

# config:
my $user = "nobody";
my $group = "nobody";
my ($download, $storage);
my $debug = 0;
my ($getopt, $help);

# get standard opts with getopt
eval {
    $getopt = GetOptions("debug" => \$debug,
			 "help" => \$help,
			 "download|d:s" => \$download,
			 "storagedir:s" => \$storage);
};

if ($help) {
    print STDERR <<EOF;
Usage: $0 [OPTIONS]

    -d DIR, --download DIR   (mandatory) path to the download/input area
    -s DIR, --storage DIR    (mandatory) path to the storage/output area

    --debug                  Dry-run debug test


Author: yeupou\@gnu.org
       https://yeupou.wordpress.com/

EOF
    exit(1);    
}

unless ($download and $storage) {
    die "Both --download INDIR and --storage OUTDIR must be provided.\nExiting";
}
unless (-d $download and -d $storage) {
    die "Both $download (--download) and $storage (--storage) must exists.\nExiting";
}

sub debug {
    return unless $debug;
    print $_[1] if $_[1]; 
    print $_[0];
    print RESET if $_[1];
    print "\n";
}

########################################################################
## run

# silently forbid concurrent runs
# (http://perl.plover.com/yak/flock/samples/slide006.html)
open(LOCK, "< $0") or die "Failed to ask lock. Exit";
flock(LOCK, LOCK_EX | LOCK_NB) or exit;


####
#### Find out current possible storage top-dirs
#### (with their respective uid/gid)
# value equal to the real-top dir
my %storage_topdir;
# uid/gid of the real-top dir
my %storage_topdir_uid;
my %storage_topdir_gid;
# keep a list of confusing marks
my %storage_topdir_confusingmark;


debug("\n\nStorage ($storage) top-dirs:\n", ON_CYAN);

for my $dir (read_dir($storage)) {
    next unless -d "$storage/$dir";
    next if ($dir =~ /^\./);   # skip hidden dirs

    # store top dir details
    $storage_topdir{$dir} = $dir;
    $storage_topdir_uid{$dir} = (lstat "$storage/$dir")[4];
    $storage_topdir_gid{$dir} = (lstat "$storage/$dir")[5];    
    debug("\t$storage_topdir{$dir}", GREEN);
    debug("\t($storage_topdir_uid{$dir}:$storage_topdir_gid{$dir})");

    # store also top dir useful aliases (end user might want to use shortcuts)
    # but no checks will be made in case of confusing aliases (ie two top dirs shortened in the name way)
    # for instance, Action Espionnage would also accept:
    #           action espionnage (lowercased)
    #		ActionEspionnage (removal of non-word chars)
    #		actionespionnage (lowercased removal of non-word chars)
    #		AE (only capital letters)
    #           ea (lowercase only capital letters)
    #           Action (single word apart)
    #           action (lowercased single word apart)
    #           Espionnage (single word apart)
    #           espionnage (lowercased single word apart)
    

    # alias as lowercased : WesteRn eq western
    my $alias = lc($dir);
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (lowercased)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tlowercased alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
    }
    # alias with space in place of any non word characters
    $alias = $dir;
    $alias =~ s/[^[:alnum:]]//g;
    
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (removal of non-word chars)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tremoval of non-word chars alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
	
	# same lowercased
	$alias = lc($alias);
	if ($alias ne $dir) {
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (lowercased removal of non-word chars)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tlowercased removal of non-word chars alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	}
    }
    # alternatively, only keep the capitalized letters
    $alias = $dir;
    $alias =~ s/[^[:alnum:]]//g;
    $alias =~ s/[^[:upper:]]//g;
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (only capital letters)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tonly capital letter alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
	# same lowercased     
	$alias = lc($alias);
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (lowercased only capital letter alias)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tlowercased only capital letter alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
    }
    # finally, if several worlds compose a string, try to register each
    # (this is where it is most likely to find confusing aliases)
    if (split(" ", $dir) > 1) {
	foreach my $word (split(" ", $dir)) {
	    $alias = $word;
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (single word apart)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tsingle word apart alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	    $alias = lc($alias);
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (lowercased single word apart)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tlowercased single word apart alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	}
    }
        
}


####
#### Find out any file or directory that we should be moving
#### (do not start moving files unless we checked everything)

# build an hash of files to move
# (with a secondary hash to keep track of the storage topdir) 
my %tomove;
my %tomove_topdir;


debug("\n\nDownload ($download) files:\n", ON_CYAN);

sub wanted {
    # $File::Find::dir is the current directory name,
    # $_ is the current filename within that directory
    # $File::Find::name is the complete pathname to the file.

    # check if we have a ##STRING## inside
    my $mark;
    $mark = $1 if $File::Find::name =~ m/##(.*)##/;

    # none found, skipping
    next unless $mark;

    # string refers to non-existant directory, skipping
    unless ($storage_topdir{$mark}) {
	debug("Mark $mark found for $File::Find::name while no such storage directory exists in $storage", ON_RED);
	# this is an issue that requires manual handling, print ont STDERR
	print STDERR ("Mark $mark found for $File::Find::name while no such storage directory exists in $storage\n");
	next;
    }

    # take into account only videos and text files
    my $suffix;
    $suffix = $1 if $_ =~ /([^\.]*)$/;
    my ($mime_type,$mime_type2) = split("/", mimetype($File::Find::name));
    if ($mime_type ne "video" and
	$mime_type ne "text") {
	# second pass to allow even more text files based on extension
	# (subtitles : srt sub ssa ass idx txt smi)
		
	unless ($suffix eq "srt" or
		$suffix eq "sub" or
		$suffix eq "txt" or
		$suffix eq "ssa" or
		$suffix eq "ass" or
		$suffix eq "idx" or
		$suffix eq "smi") {
	    debug("\tskip $_ ($mime_type/$mime_type2 type)");
	    next;		
	}
    }

    my $destination_dir = "$storage/$storage_topdir{$mark}";
    my $destination_file = $_;
    $destination_file =~ s/##(.*)##//g;
    $destination_file =~ s/^\s*//;
    $destination_file =~ s/\s*$//;

    # now handle the special S00E00 case of series, like 30 Rock (2006) - S05E16 or 30 Rock S05E16
    my ($season, $before_season, $show);
    $before_season = $1 and $season = $2 if $_ =~ m/^(.*)S(\d\d)\ ?E\d\d[^\d]/i;
    if ($season) {
	# there is a season, we must determine the show name
	#    30 Rock (2006) - S05E16 => 30 Rock
	# end user must pay attention to have consistent names
	$show = $1 if $before_season  =~ m/^([\w|\s|\.|\'|\,]*)/g;
	# dots often are used in place of white spaces
	$show =~ s/\./ /g;    
	# keep only spaces in shows name, nothing else
	$show =~ s/[^[:alnum:]|\ ]//g;    
	$show =~ s/^\s*//;
	$show =~ s/\s*$//;
	# capitalize first letter
	$show =~ s/\b(\w)/\U$1/g;
	
	# if we managed to find the show name, then set up the specific series tree
	last unless $show;
	debug("found show: $show", MAGENTA);
	$destination_dir = "$storage/$storage_topdir{$mark}/$show/S$season";	    	
    }

    
    # if we reach this point, everything seems in order, plan the move
    debug("plan -> $destination_dir/$destination_file");
    $tomove{$File::Find::name} = "$destination_dir/$destination_file";
    $tomove_topdir{$File::Find::name} = $storage_topdir{$mark};


    # additionally, if we deal with a video, look for any possibly related file to add also that would not have been picked
    # otherwise
    if ($mime_type eq "video") {
	my $other_files_path = $File::Find::name;
	$other_files_path =~ s/\.$suffix$//g;

	debug("glob $other_files_path*");
	my @other_files =
	    glob('$other_files_path*.srt'),
	    glob('$other_files_path*.sub'),
	    glob('$other_files_path*.txt'),
	    glob('$other_files_path*.ssa'),
	    glob('$other_files_path*.ass'),
	    glob('$other_files_path*.idx'),
	    glob('$other_files_path*.smi');
	foreach my $file (@other_files) {
	    debug("plan -> $destination_dir/$file");
	    $tomove{"$File::Find::dir/$file"} = "$destination_dir/$file";
	    $tomove_topdir{"$File::Find::name/$file"} = $storage_topdir{$mark};
	}		    
    }

    debug();
      
}
find(\&wanted, $download);

####
#### Actually move files now
####

debug("\n\nMove from download ($download) to storage ($storage):\n", ON_CYAN);

foreach my $file (sort keys %tomove) {

    debug(basename($file), YELLOW);

    my $uid = $storage_topdir_uid{$tomove_topdir{$file}};
    my $gid = $storage_topdir_gid{$tomove_topdir{$file}};    

    # create directory if needed
    my $dir = dirname($tomove{$file});
    unless (-e $dir) {
	make_path($dir, { chmod => 0770, user => $uid, group => $gid }) unless $debug;
	debug("make_path $dir (chmod => 0770, user => $uid, group => $gid)");
    }

    # then move the file (chown if root)
    # avoid overwriting, add number in the end, no extension saving
    my $copies;
    if (-e $tomove{$file}) {
	while (-e "$tomove{$file}.$copies") {
	    $copies++;
	    # stop at 10, makes no sense to keep more than that amount of copies
	    last if $copies > 9;
	}
    }    
    $tomove{$file} .= ".$copies" if $copies;
    move($file, $tomove{$file}) unless $debug;
    chown($uid, $gid, $tomove{$file}) unless $debug or $< ne 0;
    debug("$file -> $tomove{$file}");

    debug();	  
}

# EOF

Signing the open letter in favor of RMS, even though he probably should not head FSF?

Six years ago, I posted an article related to my (limited) direct experience with Richard Matthew Stallman, which I concluded by: although he values his freedom and values freedom in general, working with him, even in a very distant way, is just a matter of subordination; he’d make a credible science-fiction character: distopian guru, the Pied Piper of MIT.

You would assume that I would approve his removal from Free Software Foundation. But no: I was not expecting this to be based on the current trendy totalitarian philosophy. No one should be happy that someone is prevented to do his work due to his identity or political and philosophical opinions – or, worse, how he is depicted by an angry mob no matter what he actually thinks or said.

In regard of Free Software, RMS is as important as Winston Churchill was regarding UK’s position during World War II. He built the philosophical base of what Free Software is. It would not be, or in a completely different form, without him. And you cannot claim to promote of something “meant to serve everyone regardless of their age, ability or disability, gender identity, sex, ethnicity, nationality, religion or sexual orientation” when you actually exactly do the contrary. This way of thinking when you make a list of people always right other always wrong, when you silence the one that are wrong, matches totalitarian ideologies, not freedom.

So I really don’t care about RMS position at FSF, it is probably for the best that he is no longer in his autocratic position. I surely don’t care about his personal opinion about this or that topic unrelated to software. I guess some other people might think likewise. But it is not a reason to keep silent toward ideological violence.

If you like to rethink all these dark events in history, that are never black or white, you’ll consider that the issue is not that much about the main protagonists, following their path whether they’ll turn out to be criminals or freedom fighters. No, the issue is regarding the bystanders, that will see questionable things being done but won’t comment, because it does not affect them really, because they felt no connection to the one attacked or, because they felt maybe it is was on some other level deserved. But history judgment is harsh on them, nonetheless.

“We ask for contributors to free software projects to take a stand against bigotry and hate within their [FSF] projects”, they wrote. Bigotry and hate are terms that can easily be turned to describe them, or easy to manipulate in every direction. When you silence people, there is hate. When you create a work environment in which people are silenced due to their opinions, there is bigotry. I believe that the sane way to regulate society is called rights of man: can be punished, silenced, only if they have been proven of breaking legitimate laws by a legitimate court. And that led me to sign the letter in favor of RMS, even though I do not think he should not be at the head of FSF. We should not accept a society of oppression, no matter in which name, especially not in the name of greater good because that’s always the one invoked to do the worse. We thought ideologies were dead. No, they are as dangerous as ever.

PS : since GNOME Foundation is heavily involved and claim acting in regard of Free Sofware credibility, it is easy to point out they are not exactly known for that. Regarding Mozilla, RedHat, and similar companies, etc, hum, if they really want to howl with the wolves, maybe some day no one will care to promote their work instead of “don’t be Evil”-company.

PPS: seems that some people that want RMS eviction also advise to blacklist, recruitment-wise, anyone that signed the letter in favor of RMS (a, b, c, etc). The whole process is definitely quite disgusting, besides being completely stupid.

PPPS: some other made named-based statistics to guess which ethnicity or genre is voting for or against. I do not think there is any progress in essentialism and surely we cannot call democrats people in favor of racial or sexual-based voting rights (except by erasing concept of modern citizenship).

Moving a live encrypted system from one hard disk to another

This a short memento based on earlier articles Moving a live system from one hard disk to another and http://Single passphrase to boot Devuan GNU/Linux with multiple encrypted partitions. This is useful when you start to move over your systems partitions from HDD to SSD, that nowadays are clearly worth their cheap price.

This article is made for Devuan GNU/Linux but should not be distro specific – you just might want to replace devuan string in later command by something else.

We start by setting some variables depending on the relevant drives. Any doubt about which drive is what, running lsblk should help.

# new NVMe disk 
NDISK=/dev/nvme0n1
NPART_PREFIX=p

# or alternatively for a SATA new disk:
# NDISK=/dev/sdb
# NPART_PREFIX=

LUKS_PREFIX=250g21

(AHCI SATA SSD are faster than HDD, but AHCI itself will be the bottleneck, so I’d suggest to install a NVMe SSD if your mainboard allows).

# key necessary to mount all partitions with a singlepassphrase

key=/boot/klucz
if [ ! -e $key ]; then dd if=/dev/urandom of=$key bs=1024 count=4 && chmod 400 $key ; fi

Next step is to replicate the disk structure. While this article is BIOS-boot based, it should go along UEFI:

parted $NDISK

(parted shell)
mklabel gpt
mkpart biosreserved ext2 1049kB  50,3MB
toggle 1 bios_grub
mkpart efi fat32 50,3MB  500MB
toggle 2 msftdata
mkpart swap linux-swap 500MB   16,0GB
toggle 3 swap
mkpart root ext4 16,0GB 250GB
print

Model: KINGSTON SA2000M8250G (nvme)
Disk /dev/nvme1n1: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name          Flags
 1      1049kB  50,3MB  49,3MB                  biosreserved  bios_grub
 2      50,3MB  500MB   450MB                   efi           msftdata
 3      500MB   16,0GB  15,5GB  linux-swap(v1)  swap          swap
 4      16,0GB  250GB   234GB   ext4            root

quit

We build the new system partition (luks1 is still mandatory with grub, but that won’t be true forever we can suppose):

cryptsetup luksFormat --type luks1 "$NDISK$NPART_PREFIX"4
cryptsetup luksOpen "$NDISK$NPART_PREFIX"4 "$LUKS_PREFIX"devuan64
cryptsetup  luksAddKey "$NDISK$NPART_PREFIX"4 $key
mkfs.ext4 /dev/mapper/"$LUKS_PREFIX"devuan64 -L "$LUKS_PREFIX"devuan64

mkdir /tmp/"$LUKS_PREFIX"devuan64
mount /dev/mapper/"$LUKS_PREFIX"devuan64 /tmp/"$LUKS_PREFIX"devuan64

ignore="backups home dev lost+found media proc run sys tmp"
for dir in $ignore; do touch /$dir.ignore ; done && for dir in /*; do if [ -d $dir ]; then if [ ! -e $dir.ignore ]; then /usr/bin/rsync --archive --one-file-system --delete $dir /tmp/"$LUKS_PREFIX"devuan64/ ; else if [ ! -e /tmp/"$LUKS_PREFIX"devuan64/$dir ]; then mkdir /tmp/"$LUKS_PREFIX"devuan64/$dir; fi ; rm $dir.ignore ; fi ; fi ; done

We update required system files:

echo " " >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "# "`date` >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "$LUKS_PREFIX"devuan64 UUID=`blkid -s UUID -o value "$NDISK$NPART_PREFIX"4` $key luks,tries=3,discard >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "$LUKS_PREFIX"swap `find -L /dev/disk -samefile "$NDISK$NPART_PREFIX"3 | grep by-id | tail -1` /dev/urandom cipher=aes-xts-plain64,size=256,swap,discard >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab

# comment out old lines
nano /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab


echo " " >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "# "`date` >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "/dev/mapper/"$LUKS_PREFIX"devuan64	/		ext4	errors=remount-ro		0 1" >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "/dev/mapper/"$LUKS_PREFIX"swap	none		swap	sw		0 0" >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab

# comment out old lines
nano /tmp/"$LUKS_PREFIX"devuan64/etc/fstab



echo "Make sure this is in grub config:"
echo
echo GRUB_CMDLINE_LINUX=\"rd.luks.key=$key:UUID=`blkid "$NDISK$NPART_PREFIX"4 -s UUID -o value`\"
echo GRUB_ENABLE_CRYPTODISK=\"y\"
echo GRUB_PRELOAD_MODULES=\"luks cryptodisk lvm\"

# update grub config
nano /tmp/"$LUKS_PREFIX"devuan64/etc/default/grub

Last step is to install the boot loader on the new disk:

mount --bind /dev /tmp/"$LUKS_PREFIX"devuan64/dev
mount --bind /sys /tmp/"$LUKS_PREFIX"devuan64/sys
mount -t proc /proc /tmp/"$LUKS_PREFIX"devuan64/proc
chroot /tmp/"$LUKS_PREFIX"devuan64

grub-mkdevicemap
update-initramfs -u

# need to be retyped since it not in chroot environment
NDISK=/dev/nvme0n1

grub-install $NDISK
grub-mkconfig > /boot/grub/grub.cfg

That’s all.

Using PowerDNS (server and recursor) with DNSSEC and domain name spoofing/caching

I updated my earlier PowerDNS (server and recursor) setup along with domain name spoofing/caching. This is a short update to allow DNSSEC usage and unlimited list of destinations from spoof/cache list. Files described here can be found on gitlab.

DNSSEC

Adding DNSSEC support can done easily by creating /etc/powerdns/recursor.d/10-dnssec.conf :

# dnssec	DNSSEC mode: off/process-no-validate 
#                    (default)/process/log-fail/validate
dnssec=validate

#################################
# dnssec-log-bogus	Log DNSSEC bogus validations
dnssec-log-bogus=no

Every local zone must be excluded by adding to /etc/powerdns/recursor.lua :

addNTA("10.10.10.in-addr.arpa", "internal zone")

New redirect.lua renewal script

Earlier version provided a static /etc/powerdns/redirect.lua which was depending on with redirect-cached.lua, redirect-ads.lua and redirect-blacklisted.lua, which contained lists of domains to either blacklist (meaning: redirected to loopback) or spoof.

Now, the script redirect-rebuild.pl use the configuration redirect-spooflist.conf to generate redirect.lua. The ads blacklist part is unchanged.

The configuration syntax is as follow:

# IP:	domain domain 

# redirect thisdomain.lan and thisother.lan to 192.168.0.1,
# except if 192.168.0.1 is asking 
192.168.0.1: thisdomain.lan thisother.lan 

# redirect anotherthisdomain.lan and anotherthisother.lan to 10.0.0.1,
# even if 10.0.0.1 is asking 
10.0.0.1+:    anotherthisdomain.lan anotherthisother.lan 

# you can use 127.0.0.1: to blacklist domains

It is enough to run the redirect-rebuild.pl script and restart the recursor:

use strict;
use Fcntl ':flock';

my $spooflist = "redirect-spooflist.conf";
my $ads_lua = "redirect-ads.lua";
my $ads_pl = "redirect-ads-rebuild.pl";
my $main_lua = "redirect.lua";

# disallow concurrent run
open(LOCK, "< $0") or die "Failed to ask lock. Exiting";
flock(LOCK, LOCK_EX | LOCK_NB) or die "Unable to lock. This daemon is already alive. Exiting";

# first check if we have a ads list to block
# if not, run the local script to build izt
unless (-e $ads_lua) {
    print "$ads_lua missing\n";
    print "run $ads_pl\n" and do "./$ads_pl" if -x $ads_pl;
}

my %cache;
# read conf
open(LIST, "< $spooflist");
while (<LIST>) {
    next if /^#/;
    next unless s/^(.*?):\s*//;
    $cache{$1} = [ split ];
}
close(LIST);

# build lua
open(NEWCONF, "> $main_lua");
printf NEWCONF ("-- Generated on %s by $0\n", scalar localtime);
print NEWCONF '-- IPv4 only script

-- ads kill list
ads = newDS()
adsdest = "127.0.0.1"
ads:add(dofile("/etc/powerdns/redirect-ads.lua"))

-- spoof lists
';

foreach my $ip (keys %cache) {
    # special handling of IP+, + meaning we spoof even to the destination host
    my $name = $ip;
    $name =~ s/(\.|\+)//g;  
    print NEWCONF "spoof$name = newDS()\n";
    print NEWCONF "spoof$name:add{", join(", ", map "\"$_\"", sort@{$cache{$ip}}), "}\n";
    $ip =~ s/(\+)//g;
    print NEWCONF "spoofdest$name = \"$ip\"\n";
}

print NEWCONF '
function preresolve(dq)
   -- DEBUG
   --pdnslog("Got question for "..dq.qname:toString().." from "..dq.remoteaddr:toString().." to "..dq.localaddr:toString(), pdns.loglevels.Error)
   
   -- spam/ads domains
   if(ads:check(dq.qname)) then
     if(dq.qtype == pdns.A) then
       dq:addAnswer(dq.qtype, adsdest)
       return true
     end
   end
    ';

foreach my $ip (keys %cache) {
    my $always = 0;
    $always = 1 if ($ip =~ s/(\+)//g);     # + along with IP means always spoof no matter who is asking
    my $name = $ip;
    $name =~ s/\.//g;

    print NEWCONF '
   -- domains spoofed to '.$ip.'
   if(spoof'.$name.':check(dq.qname)) then';
    print NEWCONF '
     dq.variable = true
     if(dq.remoteaddr:equal(newCA(spoofdest'.$name.'))) then
       -- request coming from the spoof/cache IP itself, no spoofing
       return false
     end' unless $always;
    print NEWCONF '   
     if(dq.qtype == pdns.A) then
       -- redirect to the spoof/cache IP
       dq:addAnswer(dq.qtype, spoofdest'.$name.')
       return true
     end
   end
	';
}

print NEWCONF '
   return false
end
';
close(NEWCONF);


# EOF

Testing Roccat Kain mouse failure with xev

I’m using Roccat mouse since a while. Confortable to hold, precise enough, I was quite satisfied. Except they dont live so long. One had a left button failure. Another one, new, died instantly after a small drop of coffee. I drop a lot of coffee on such devices, it is the first time I managed to instakill a mouse like this. Nonetheless, I bought even more of them, ROCCAT Kain 122, two at once.

One mouse mousewheel was behaving obviously erratically: scrolling down was inconsistent. Running xev is enough to catch the issue. You’ll find out that “button 4” is the action activated for mousewheel going up (with a specific serial that is irrelevant here, “button 5” going down). So it is enough to run:

xev | grep "button 4"

and to turn the mousewheel down to notice a few erratic “button 4” pressed pop-in up.

Worse, it did so too with the second mouse for which the bug was not obvious. Even worse, it looks like this hardware issue exists at least since a year. So, to sum up, they still ship a buggy series of mouses, bug you might not immediately notice. Despite their other good points, I’ll avoid this brand for now.

Generating static galleries with nanogallery2

As wrote earlier, I now use nanogallery2 to share pictures due to nextcloud removal of the earlier gallery app and the fact that the replacement app called Photos does not match my needs.

It is copied directly from the cloud LXC container, with rsync and ssh, to another server that only serves static files via HTTPS – with access restricted through usual basic authentication. It means that I am not giving any access to the cloud infrastructure to the HTTPS server.

nanogallery2 have associated “providers” to easily set up a gallery (directly from a local directory or distant storage). But that implies on-the-fly index and thumbnails generation by a PHP CGI server while I prefer the server to be serving only static files.

So I made a small perl script called dir2nanog.pl to build a single index and thumbnails, using subdirectories as distinct galleries:

#!/usr/bin/perl
#
# Copyright (c) 2020 Mathieu Roy <yeupou--gnu.org>
#      https://yeupou.wordpress.com
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place,d Suite 330, Boston, MA  02111-1307
#   USA


use strict "vars";
use File::Find;
use File::Basename;
use Image::ExifTool;
use Image::Magick;
use Fcntl ':flock';
use CGI qw(:standard escapeHTML);

my $debug = 1;
my $path = $ARGV[0];
my $topdirstring = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ";   # hopefully no such directory exists
my @topimages;
my %subdirsimages;
my %comment;
my %gps;
my %model;
my %focalength;
my %flash;
my %exposure;
my %iso;
my $images_types = "png|gif|jpg|jpeg";
my @mandatory = ("https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.core.min.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.min.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/nanogallery2.woff.min.css",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/nanogallery2.min.css"); 

my @mandatoryfont = ("https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/font/ngy2_icon_font.woff",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/font/ngy2_icon_font.woff2");



# silently forbid concurrent runs
# (http://perl.plover.com/yak/flock/samples/slide006.html)
open(LOCK, "< $0") or die "Failed to ask lock. Exit";
flock(LOCK, LOCK_EX | LOCK_NB) or exit;

### FUNCTIONS
sub thumbnail_dimensions {
        my ($x, $y, $ratio, $height, $width);
        ($x, $y) = @_;
        $ratio = $x / $y;	
	$width = 200;
	$height = int($width / $ratio);
        return $width, $height;
}

sub thumbnail_name {
    my ($name,$path) = fileparse($_[0]);
    return "$path.thumbnail.$name";
}

sub wanted {
    # skip directories and non-images in general
    next if -d $_;
    next unless lc($_) =~ /\.($images_types)$/i;
    # skip thumbnails
    next if $_ =~ /^\.thumbnail\./i;

    # create thumbnail if missing
    unless (-e thumbnail_name($File::Find::name)) {
	my ($image, $error, $x, $y, $size, $format, $width, $height);
	
        # https://www.perlmonks.org/?node_id=209235
        $image = Image::Magick->new;
        $error = $image->Read($File::Find::name);
	if (!$error) {
	    ($x, $y, $size, $format) = $image->Ping($File::Find::name);
	    ($width, $height) = thumbnail_dimensions($x, $y);
	    $image->Scale(width=>$width, height=>$height);
	    $error = $image->Write(thumbnail_name($File::Find::name));
        }
    }
    
    # identify URL, path removed 
    my $url = substr($File::Find::name, length($path)+1, length($File::Find::name));    

    # name will be based on file creation (using comment would require utf8 checks)
    # according to exif data, not real filesystem info
    my $exifTool = new Image::ExifTool;
    my %exifTool_options = (DateFormat => '%x');
    my $info = $exifTool->ImageInfo($File::Find::name, ("CreateDate","GPSLatitude","GPSLongitude","GPSDateTime", "Make", "Model", "Software", "FocalLength", "Flash", "Exposure", "ISO"), \%exifTool_options);
    $comment{$url} = $info->{"CreateDate"};
    $comment{$url} = $info->{"GPSDateTime"} if $info->{"GPSDateTime"};
    
    if ($info->{"GPSLatitude"} and $info->{"GPSLongitude"}) {
	$gps{$url} = $info->{"GPSLatitude"}." ".$info->{"GPSLongitude"};
	$gps{$url} =~ s/\sdeg/°/g;
    }

    $model{$url} = $info->{"Make"}." " if $info->{"Make"};
    $model{$url} .= $info->{"Model"}." " if $info->{"Model"};
    $model{$url} .= "(".$info->{"Software"}.")" if $info->{"Software"};

    $focalength{$url} = $info->{"FocalLength"} if $info->{"FocalLength"};
    $flash{$url} = $info->{"Flash"} if $info->{"Flash"};
    $exposure{$url} = $info->{"Exposure"} if $info->{"Exposure"};
    $iso{$url} = $info->{"ISO"} if $info->{"ISO"};

    
    # top directory image
    push(@{$subdirsimages{$topdirstring}}, $url) and next if ($File::Find::dir eq $path);    

    # other images
    # identify subdir by removing the path and keeping only the resulting top directory
    my ($subdir,) = split("/", substr($File::Find::dir, length($path)+1, length($File::Find::dir)));
    push(@{$subdirsimages{$subdir}}, $url);
        
    return;
}


### RUN

chdir($path) or die "Unable to enter $path, exiting";

# list images in top dir and subdirectories
find(\&wanted, $path);

# create the output
# first check if mandatory files are there
foreach my $file (@mandatory) {
    next if -e "$path/".basename($file);
    system("/usr/bin/wget", $file);
}
mkdir("$path/font") unless -e  "$path/font";
chdir("$path/font");
foreach my $file (@mandatoryfont) {
    next if -e "$path/font/".basename($file);
    system("/usr/bin/wget", $file);
}
chdir($path);
    
# create index
open(INDEX, "> $path/index.html");
print INDEX '<!DOCTYPE html
	PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta name="generator" content="dir2nanog.pl" />
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <!-- jQuery -->
    <script type="text/javascript"  src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
    <!-- nanogallery2 -->
    <link href="nanogallery2.min.css" rel="stylesheet" type="text/css" />
    <script  type="text/javascript" src="jquery.nanogallery2.min.js"></script>
    <!-- other -->
    <link type="text/css" rel="stylesheet" href="style.css" />
    <title>'.basename($path).'</title>
  </head>
  <body>
';


my $galleriescount = 0;

for (reverse(sort(keys(%subdirsimages)))) {
    
    # top dir gallery has no specific ID or title
    if ($galleriescount > 0) {	
	print INDEX '    <h1>'.$_."</h1>\n";
	print INDEX '    <div id="nanogallery2'.$_;
    } else {
	print INDEX '    <div id="nanogallery2';
    }

    # add gallery setup
    print INDEX '" data-nanogallery2 = \'{
      "viewerToolbar":   {
        "display":    true,
        "standard":   "label, infoButton",
        "minimized":  "label, infoButton"
      },
      "viewerTools":     {
        "topLeft":    "pageCounter, playPauseButton",
        "topRight":   "downloadButton, fullscreenButton, closeButton"
      },
      "thumbnailWidth": "auto",
      "galleryDisplayMode": "pagination",
      "galleryMaxRows": 3,
      "thumbnailHoverEffect2": "borderLighter"
    }\'>
';

    # add image list
    for (sort(@{$subdirsimages{$_}})) {
	my $extra;
	$extra .= ' data-ngexiflocation="'.escapeHTML($gps{$_}).'"' if $gps{$_};
	$extra .= ' data-ngexifmodel="'.escapeHTML($model{$_}).'"' if $model{$_};
	$extra .= ' data-ngexiffocallength="'.escapeHTML($focalength{$_}).'"' if $focalength{$_};
	$extra .= ' data-ngexifflash="'.escapeHTML($flash{$_}).'"' if $flash{$_};
	$extra .= ' data-ngexifexposure="'.escapeHTML($exposure{$_}).'"' if $exposure{$_};
	$extra .= ' data-ngexifiso="'.escapeHTML($iso{$_}).'"' if $iso{$_};

	# if no comment set, check if if the filename might be YYMMDD_
	$comment{$_} = "$3/$2/20$1" if (!$comment{$_} and /(\d{2})(\d{2})(\d{2})_[^\\]*$/);  	
	
	print INDEX '      <a href="'.$_.'" data-ngThumb="'.thumbnail_name($_).'"'.$extra.'>'.escapeHTML($comment{$_})."</a>\n";
    }
print INDEX '
	</div>
';
    $galleriescount++;
}

# finish index	
	print INDEX
	'</body>
</html>';

close(INDEX);

# EOF 

To run it:

chmod +x /usr/local/bin/dir2nanog.pl
/usr/local/bin/dir2nanog.pl /var/www/html/pictures/

I have the following crontab for the dedicated user:

*/15 * * * *     export LC_ALL=fr_FR.UTF-8 && /usr/local/bin/dir2nanog.pl /var/www/html/pictures/thistopic

Dealing with Nextcloud broken oc_flow_operations table after 19 upgrade

I finally upgraded my Nextcloud server, using nanogallery2 instead of relying on the new brand Photos app, since it does not provide a simple way to share a gallery and is clearly not about to be resolved (the developers that in first place considered there was no problem to fix finally said that people were too harsh try to convince them otherwise so they finally lost interest in fixing it). Well, in any case, using static files for this purpose is much more efficient.

During the upgrade, I hit issue with the oc_flow_operations table that is mentioned in numerous reports (the whole first page of google search).

I first fixed it by, as suggested by some users, adding a missing column. That allowed to finish the upgrade but something was still off, with the logfile growing to a few MB every minutes.

Finally, recreating the whole table fixed it for me:

cat /etc/mysql/debian.cnf | grep password
mysql -udebian-sys-maint -p
connect nextcloud;
DROP table oc_flow_operations ;
CREATE table oc_flow_operations (id   int(11) NOT NULL  AUTO_INCREMENT PRIMARY KEY, class  varchar(256), name     varchar(256) NOT NULL, checks   longtext, operation longtext,entity  varchar(256),events longtext);
su www-data -s /bin/bash
php occ maintenance:repair

Fetching mails from gmail.com with lieer/notmuch instead of fetchmail

Google is gradually making traditional IMAPS access to gmail.com impossible, in it’s usual opaque way. It is claimed to be a security plus, though, if your data is already on gmail.com, it means that you are already accepting that it can and is spied on, so whether the extra fuss is justified is not so obvious.

Nonetheless, I have a few secondary old boxes on gmail, that I’d like to still extract mails from, not to miss any, without bothering connecting to with a web browser. But the usual fetchmail setup is no longer reliable.

The easiest alternative is to use lieer, notmuch and procmail together.

apt install links notmuch lieer procmail

# set up as regular user
su enduser

boxname=mygoogleuser
mkdir -p ~/mail/$boxname
notmuch
notmuch new
cd ~/mail/$boxname
gmi init $boxname@gmail.com

# at this moment, you'll get an URL to connect to gmail.com 
# with a web browser
# 
# if it started links instead, exit cleanly (no control-C)
# to get the URL
#
# that will then return an URL to a localhost:8080
# to no effect if you are on a distant server
# in this case, just run, in an another terminal
links "localhost:8080/..."

The setup should be ok. You can check and toy a bit with the setup:

gmi sync
notmuch new
notmuch search tag:unread
notmuch show --format=mbox thread:000000000000009c

Then you should mark all previous messages as read:

for thread in `notmuch search --output=threads tag:unread`; do echo "$thread" && notmuch tag -unread "$thread" ; done
gmi push

Finally, we set up the mail forward (in my case, the fetch action is done in a dedicated container, so it is forwarded to ‘user’ by SMTP to the destination host, but procmail allows any setup) and fetchmail script: each new unread mail is forwarded and then marked as read:

echo ':0
! user' > ~/.procmail

echo '#!/bin/bash

BOXES=""

# no concurrent run
if pidof -o %PPID -x "fetchmail.sh">/dev/null; then
        exit 1
fi

for box in $BOXES; do
    cd ~/mail/$box/
    # get data
    gmi pull --quiet
    notmuch new --quiet >/dev/null
    for thread in `notmuch search --output=threads tag:unread`; do
	# send unread through procmail
	notmuch show --format=mbox "$thread" | procmail
	# mark as read
	notmuch tag -unread "$thread"
    done
    # send data
    gmi push --quiet
    cd
done

# EOF' > ~/fetchmail.sh
chmod +x ~/fetchmail.sh

Then it is enough to call the script (with BOXES=”boxname” properly set) and to include it in cronjob, like using crontab -e`

notmuch does not remove mails.

Downgrading Nextcloud 18.0.3.0 to 17.0.5

I upgraded Nextcloud to the latest save without realizing the gallery app has been replaced by an half-baked “photos” app, completely useless to share pictures in any relevant way to me, in addition to a bug with ublock origin making the whole “sharing” interface disappearing.

Rolling back is not as easy at it seems: the “occ” php app check with your config version if it matches the software version, and if not matching just plainly refuses to work with:

An unhandled exception has been thrown:
OC\HintException: [0]: Downgrading is not supported and is likely to cause unpredictable issues (from 18.0.3.0 to 17.0.5.0) ()

So you need to update this config/config.php. Then, playing with occ app:list, occ app:remove and occ app:install only you can get back to a working install.

update: since then, nothing changed, nextcloud 20 has been released and it looks like the developers have no plan to deal with this issue. Since users were pissed at their refusal to consider there is an issue to fix, now they lost their interest in fixing the issue that they did not want to consider in first place.