Switching from SpamAssassin+Bogofilter+Exim to Rspamd+Exim+Dovecot

For more than ten years, I used SpamAssassin and Bogofilter along with Exim to filter spams, along with SFP and Greylisting directly within Exim.

Why changing?

I must say that almost no spam reached me unflagged for years. Why changing anything then?

First, I have more users and the system was not really multiuser-aware. For instance, the bayesian filter training cronjob had configured SPAMDIR, etc.

Second, my whole setup was based on using specific transports and routers in exim to send mails first to bogofilter, then to spamassassin. It means that filtering is done after SMTP-time, when the mail has been already accepted. You filter but do not discourage or block spam sources.

Rspamd?

General
Written inC/LuaPerlC
Process modelevent drivenpre-forked poolLDA and pre-forked
MTA integrationmilter, LDA, custommilter, custom (Amavis)LDA
Web interfaceembedded3rd party
Languages supportfull, UTF-8 conversion/normalisation, lemmatizationnaïve (ASCII lowercase)naïve
Scripting supportLua APIPerl plugins
LicenceApache 2Apache 2GPL
Development statusvery activeactiveabandoned

Rspamd seems activitely developed and easy to integrate not only with Exim, the SMTP, but also with Dovecot, which is use as IMAPS server.

Instead of having:

Exim SMTP accept with greylist -> bogofilter -> spamassassin -> procmail -> dovecot 

The idea is to have:

Exim SMTP accept with greylist and rspamd -> dovecot with sieve filtering 

It blocks rejects/discard spam earlier and makes filtering easier in a multiuser environment (sieve is not dangerous, unlike procmail, and can be managed by clients, if desirable)

My new setup is contained in my rien-mx package: the initial greylist system is still there.

Exim

What matters most is acl_check_rcpt definition (already used in previous version) and new acl_check_data definition.:

### acl/41_rien-check_data_spam
#################################
# based on https://rspamd.com/doc/integration.html
# -  using CHECK_DATA_LOCAL_ACL_FILE included in the acl_check_data instead a creating a new acl
# - and scan all the messages no matter the source:
#    because some might be forwarded by smarthost client, requiring scanning with no defer/reject

## process earlier scan

# find out if a (positive) spam level is already set
warn
  condition = ${if match{$h_X-Spam-Level:}{\N\*|\+\N}}
  set acl_m_spamlevel = $h_X-Spam-Level:
warn
  condition = ${if match{$h_X-Spam-Bar:}{\N\*|\+\N}}
  set acl_m_spamlevel = $h_X-Spam-Bar:
warn
  condition = ${if match{$h_X-Spam_Bar:}{\N\*|\+\N}}
  set acl_m_spamlevel = $h_X-Spam_Bar:

# discard high probability spam identified by earlier scanner
# (probably forwarded by a friendly server, since it is unlikely that a spam source would shoot
# itself in the foot, no point to generate bounces)
discard
  condition = ${if >={${strlen:$acl_m_spamlevel}}{15}}
  log_message = discard as high-probability spam announced

# at least make sure X-Spam-Status is set if relevant
warn
  condition = ${if and{{ !def:h_X-Spam-Status:}{ >={${strlen:$acl_m_spamlevel}}{6} }}}
  add_header = X-Spam-Status: Yes, earlier scan ($acl_m_spamlevel)

# accept content from relayed hosts with no spam check
# unless registered in final_from_hosts (they are outside the local network)
accept
  hosts = +relay_from_hosts
  !hosts = ${if exists{CONFDIR/final_from_hosts}\
		      {CONFDIR/final_from_hosts}\
		      {}}

# rename earlier reports and score
warn
  condition = ${if def:h_X-Spam-Report:}
  add_header = X-Spam-Report-Earlier: $h_X-Spam-Report:
warn
  condition = ${if def:h_X-Spam_Report:}
  add_header = X-Spam-Report-Earlier: $h_X-Spam_Report:
warn
  condition = ${if def:h_X-Spam-Score:}
  add_header = X-Spam-Score-Earlier: $h_X-Spam-Score:
warn
  condition = ${if def:h_X-Spam_Score:}
  add_header = X-Spam-Score-Earlier: $h_X-Spam_Score:


# scan the message with rspamd
warn spam = nobody:true
# This will set variables as follows:
# $spam_action is the action recommended by rspamd
# $spam_score is the message score (we unlikely need it)
# $spam_score_int is spam score multiplied by 10
# $spam_report lists symbols matched & protocol messages
# $spam_bar is a visual indicator of spam/ham level

# remove foreign headers except spam-status, because it better to have twice than none 
warn
  remove_header = x-spam-bar : x-spam_bar : x-spam-score : x-spam_score : x-spam-report : x-spam_report : x-spam_score_int : x-spam_action : x-spam-level
  
# add spam-score and spam-report header
# (possible to add condition to add header rspamd recommend:
#   condition  = ${if eq{$spam_action}{add header})
warn
  add_header = X-Spam-Score: $spam_score
  add_header = X-Spam-Report: $spam_report

# add x-spam-status header if message is not ham
# do not match when $spam_action is empty (e.g. when rspamd is not running)
warn
  ! condition  = ${if match{$spam_action}{^no action\$|^greylist\$|^\$}}
  add_header = X-Spam-Status: Yes

# add x-spam-bar header if score is positive
warn
  condition = ${if >{$spam_score_int}{0}}
  add_header = X-Spam-Bar: $spam_bar

## delay/discard/deny depending on the scan
  
# use greylisting with rspamd
# (unless coming from authenticated or relayed host)
defer message    = Please try again later
   condition  = ${if eq{$spam_action}{soft reject}}
   !hosts = ${if exists{CONFDIR/final_from_hosts}\
		       {CONFDIR/final_from_hosts}\
		       {}}
   !authenticated = *
   log_message  = greylist $sender_host_address according to soft reject spam filtering

# high probability spam get silently discarded if 
# coming from authenticated or relayed host
discard
   condition  = ${if eq{$spam_action}{reject}}
   hosts = ${if exists{CONFDIR/final_from_hosts}\
		       {CONFDIR/final_from_hosts}\
		       {}}
   log_message  = discard as high-probability spam from final from host

discard
   condition  = ${if eq{$spam_action}{reject}}
   authenticated = *
   log_message  = discard as high-probability spam from authentificated
   
# refuse high probability spam from other sources
deny  message    = Message discarded as high-probability spam
   condition  = ${if eq{$spam_action}{reject}}
   log_message	= reject mail from $sender_host_address as high-probability spam

These two will take to send through rspamd and accept/reject/discard mails.

A dovecot_lmtp transport is also necessary:

dovecot_lmtp:   
  debug_print = "T: dovecot_lmtp for $local_part@$domain"   
  driver = lmtp   
  socket = /var/run/dovecot/lmtp   
  #maximum number of deliveries per batch, default 1   
  batch_max = 200   
  # remove suffixes/prefixes   
  rcpt_include_affixes = false 

There are also other internal files, especially in conf.d/main. For instance. If you want to follow my setup, you are encouraged to download the whole mx/etc/exim folder at least. Most files have comments, easy to find out if they are relevant or not. Or you can just copy/paste relevant settings into etc/conf.d/main/10_localsettings, like for instance:

# path of rspamd 
spamd_address = 127.0.0.1 11333 variant=rspamd 

# data acl definition 
CHECK_DATA_LOCAL_ACL_FILE =  /etc/exim4/conf.d/acl/41_rien-check_data_spam

# memcache traditional greylioting
GREY_MINUTES  = 0.4
GREY_TTL_DAYS = 25
# we greylist servers, so we keep it to the minimum required to cross-check with SPF
#   sender IP, sender domain
GREYLIST_ARGS = {${quote:$sender_host_address}}{${quote:$sender_address_domain}}{GREY_MINUTES}{GREY_TTL_DAYS}

Other files exim4/conf.d/ are useful for other local features a bit outside the scope of this article (business per target email aliases, specific handling of friendly relays, SMTP forward to specific authenticated SMPT for specific domains when sending mails).

Dovecot

This assumes that dovecot already works (with all components installed). Nonetheless, you need to edit LTMP delivery by editing /etc/dovecot/conf.d/20-lmtp.conf as follow:

# to be added
lmtp_proxy = no
lmtp_save_to_detail_mailbox = no
lmtp_rcpt_check_quota = no
lmtp_add_received_header = no 

protocol lmtp {
  # Space separated list of plugins to load (default is global mail_plugins).
  mail_plugins = $mail_plugins
  # remove domain from user name
  auth_username_format = %n
}

You also need to edit /etc/dovecot/conf.d/90-sieve.conf:

# to be added

 # editheader is restricted to admin global sieve
 sieve_global_extensions = +editheader

 # run global sieve (sievec must ran manually every time they are updated)
 sieve_before = /etc/dovecot/sieve.d/

You also need to edit /etc/dovecot/conf.d/20-imap.conf:

protocol imap {   
  mail_plugins = $mail_plugins imap_sieve
}

You also need to edit /etc/dovecot/conf.d/90-plugin.conf:

plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment

  imapsieve_mailbox4_name = Spam
  imapsieve_mailbox4_causes = COPY APPEND
  imapsieve_mailbox4_before = file:/usr/local/lib/dovecot/report-spam.sieve

  imapsieve_mailbox5_name = *
  imapsieve_mailbox5_from = Spam
  imapsieve_mailbox5_causes = COPY
  imapsieve_mailbox5_before = file:/usr/local/lib/dovecot/report-ham.sieve

  imapsieve_mailbox3_name = Inbox
  imapsieve_mailbox3_causes = APPEND
  imapsieve_mailbox3_before = file:/usr/local/lib/dovecot/report-ham.sieve

  sieve_pipe_bin_dir = /usr/local/lib/dovecot/
}

You need custom scripts to train dovecot: both shell and sieve filters. /usr/local/lib/dovecot/report-ham.sieve:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-ham.sh" [ "${username}" ];

/usr/local/lib/dovecot/report-spam.sieve:

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"];

if environment :matches "imap.user" "*" {
  set "username" "${1}";
}

pipe :copy "sa-learn-spam.sh" [ "${username}" ];

/usr/local/lib/dovecot/sa-learn-ham.sh

#!/bin/sh
exec /usr/bin/rspamc learn_ham

/usr/local/lib/dovecot/sa-learn-spam.sh

#!/bin/sh 
exec /usr/bin/rspamc learn_spam

Then you need a /etc/dovecot/sieve.d similar as mine to put all site-wide sieve scripts. Mine are shown as example of what can be done easily with sieve. Regarding spam, they will only flag spam. End user sieve filter will matter:

#; -*-sieve-*-
require ["editheader", "regex", "imap4flags", "vnd.dovecot.pipe", "copy"];

# simple flagging for easy per-user sorting
# chained, so only a single X-Sieve-Mark is possible

## flag Spam
if anyof (
	  header :regex "X-Spam-Status" "^Yes",
	  header :regex "X-Spam-Flag" "^YES",
	  header :regex "X-Bogosity" "^Spam",
	  header :regex "X-Spam_action" "^reject")
{
  # flag for the mail client
  addflag "Junk";
  # header for further parsing
  addheader "X-Sieve-Mark" "Spam";
  # autolearn
  pipe :copy "sa-learn-spam.sh";
}
## sysadmin
elsif address :localpart ["from", "sender"] ["root", "netdata", "mailer-daemon"]
{
  addheader "X-Sieve-Mark" "Sysadmin";
} 
## social network
elsif address :domain :regex ["to", "from", "cc"] ["^twitter\.",
						   "^facebook\.",
						   "^youtube\.",
						   "^mastodon\.",
						   "instagram\."]
{
  addheader "X-Sieve-Mark" "SocialNetwork";
}
## computer related
elsif address :domain :regex ["to", "from", "cc"] ["debian\.",
						   "devuan\.",
						   "gnu\.",
						   "gitlab\.",
						   "github\."]
{
  addheader "X-Sieve-Mark" "Cpu";
}

Each time scripts are modified in this folder, sievec must be run by root (because otherwise sieve script are compiled by current user, which cannot write in /etc for obvious reasons):

sievec -D /etc/dovecot/sieve.d
sievec -D /usr/local/lib/dovecot

Finally, as example of final user sieve script (to put in ~/.dovecot.sieve:

#; -*-sieve-*-
require ["fileinto", "regex", "vnd.dovecot.pipe", "copy"];

if header :is "X-Sieve-Mark" "Spam"
{
   # no care for undisclosed recipients potential false positive
  if address :contains ["to", "cc", "bcc"] ["undisclosed recipients", "undisclosed-recipients"]
		{
		  discard;
		  stop;
		}

  # otherwise just put in dedicated folder		
  fileinto "Spam";
  stop;
}

Rspamd

Rspamd was installed by devuan/debian package (not clear to me why Rspamd people discourage using these packages on their website, lacking context). It work out of the box.

I also installed clamav, razor and redis. Rspamd require lot of small tuning, check the folder /etc/rspamd

To get razor running, it pass request through /etc/xinetd.d/razor:

service razor
{
#	disable		= yes
	type		= UNLISTED
	socket_type     = stream
	protocol	= tcp
	wait		= no
	user		= _rspamd
	bind            = 127.0.0.1
	only_from	= 127.0.0.1
	port		= 11342
	server		= /usr/local/bin/razord
}

going along with wrapper script /usr/local/bin/razord:

#!/bin/sh
/usr/bin/razor-check && echo -n "spam" || echo -n "ham"

It is configured to work in the same way with pyzor but so far it does not work (not clear to me why – seems also an IPv6 issue, see below).

I noticed issues with IPv6: so far my mail servers are still IPv4 only and Rspamd nonetheless tries sometimes to connect on IPv6. I solved issue by commenting ::1 localhost in /etc/hosts.

Results

So far it works as expected (except the issues IPv4 vs IPv6 and pyzor). Rspamd required a bit more work than expected, but once it is going, it seems good.

Obviously, in the process, I lost the benefit of the well trained Bogofilter, but I hope soon enough Rspamd own bayesian filters will kick in.

In my setup there are extra files related to replicating over multiple servers that I might cover in another article (replication of email, sieve users filter through nextcloud and redis shared database via stunnel). The switch to Rspamd+Exim+Dovecot made this replication of multiples servers much better.

UPDATE: pipe :copy vs execute :pipe

Using pipe :copy in sieve script is actually causing issues. Sieve pipe is a disposition-type action, it is intended to deliver the message, similarly to a fileinto or redirect command. As such, if the command return failure, sieve filter stop. That is not desirable, if we use rspamd with learn_condition (defined in statistic.conf) to avoid multiple learning of the same file, etc. It would lead to such error in logs and sieve scripts prematurely ended:

[dovecot log]
Apr  9 21:08:10 mx dovecot: lmtp(userx)<31807><11nYLZrZUWI/fAAA4k3FvQ>: program exec:/usr/local/lib/dovecot/sa-learn-spam.sh (31810): Terminated with non-zero exit code 1
Apr  9 21:08:10 mx dovecot: lmtp(userx)<31807><11nYLZrZUWI/fAAA4k3FvQ>: Error: sieve: failed to execute to program `sa-learn-spam.sh': refer to server log for more information.
Apr  9 21:08:10 mx dovecot: lmtp(userx)<31807><11nYLZrZUWI/fAAA4k3FvQ>: sieve: msgid=<20220409190707.0A81E808EB@xxxxxxxxxxxxxxxxxxx>: stored mail into mailbox 'INBOX'
Apr  9 21:08:10 mx dovecot: lmtp(userx)<31807><11nYLZrZUWI/fAAA4k3FvQ>: Error: sieve: Execution of script /etc/dovecot/sieve.d/20_rien-mark.sieve failed, but implicit keep was successful

[rspamd log]
2022-04-09 21:08:10 #4264(controller) <eb2984>; csession; rspamd_stat_classifier_is_skipped: learn condition for classifier bayes returned: already in class spam; probability 93.38%; skip classifier
2022-04-09 21:08:10 #4264(controller) <eb2984>; csession; rspamd_task_process: learn error: all learn conditions denied learning spam in default classifier

We got an implicit keep, with an already known and identified spam forcefully sent to INBOX due to learning failure since it was already known.

Using execute :pipe instead solves the issue and match what we really want: the spam/ham learning process is extra step, it is neither involved in the filtering or delivery of the message. Its failure or success is irrelevant to the delivery process.

Using execute, the non-zero error return code from the executed script will be logged too, but without any other effect, especially not stopping sieve further processing:

[dovecot log]
Apr 10 15:12:09 mx dovecot: lmtp(userx)<3450><iqoWCKnXUmJ6DQAA4k3FvQ>: program exec:/usr/local/lib/dovecot/sa-learn-spam.sh (3451): Terminated with non-zero exit code 1
Apr 10 15:12:09 mx dovecot: lmtp(userx)<3450><iqoWCKnXUmJ6DQAA4k3FvQ>: sieve: msgid=<6252b1ee.1c69fb81.8a4e2.f847@xxxxxxxxxxxx>: fileinto action: stored mail into mailbox 'Spam'

[rspamd log]
2022-04-10 15:12:09 #9025(controller) <820d3e>; csession; rspamd_task_process: learn error: all learn conditions denied learning spam in default classifier

Check dovecot related files for up-to-date example/version:/etc/dovecot /usr/local/lib/dovecot

Checking potential mail server issues with SASL/PAM and Rainloop

I recently encountered the two following issues that exist more or less out of the box.

SASL/PAM: beware of empty passwords

On my mail servers, authenticated SMTP exists for the domains users. It works using sasl2-bin through PAM.

If you happen to have empty password for some unix accounts on this server, like:

thisuser::17197:0:99999:7:::

You would except the lack of password (like the root account can be on LXC container, after creation) to prevent login. It won’t, it will accept a blank password as valid authenticated login.

In /etc/pam.d/common-auth you can find an explanation:

auth [success=1 default=ignore] pam_unix.so nullok

it will effectively make (at least on current Devuan/Debian stable) the server an open relay for thisuser.

Some people reported it already and there are spambots around that specifically use root with blank password (without doing beforehand any other stupid attempt to misuse the server).

Here is a workaround, simply making sure no user got a blank password field in /etc/shadow :

usermod -p '!' thisuser

Rainloop: unmaintained, existing security issues

It you use the webmail Rainloop, be aware that is seems to be no longer maintained, with several attempt by differents people to warn the team about security issues left unanswered.

While we can hope that nothing tragic happening in life of the people behind the software causing this odd silence, the best course of action for now seems to be switching to the Snappymail fork, which includes security fixes and that is actively developed.

Setting up a cheap 5.1 surround sound setup reusing hardware

Initial 4.0 setup

I had some sort of custom 4.0 (4 speaker, 0 subwoofer) surround setup, aka quadraphonic, associated through Sweex 7.1 external USB sound card connected to a laptop running Devuan GNU/Linux – video output going through HDMI to a projector.

This surround was composed of a an old Denon PMA-715R amplifier (65 W per channel 8Ω) coupled with bookshelf loudspeakers Infinity reference 1 mkII (50W 6Ω) for the two front speakers, and an Edifier M3280BT 2.1 sound system (8W per channel + 20W subwoofer) for the rear speakers.

Yes, it meant that the Edifier subwoofer was actually unused and it could have been used to make a 4.1 setup. But it made sense to keep the best hardware for the front.

It also meant that it was connected to the Sweex in 5.1 mode (only designed to do 7.1/5.1/2.1 – not 4.0), using only two plugs (2 rear, 2 front = 4.0) out of the 3 required (front/bass = 1.1, 4.0+1.1 = 5.1). So it meant also using the extra-stereo filter in Smplayer, otherwise sound would sometimes be incomplete, especially with movies with a complex 5.1 flow.

Not perfect but good enough for years.

5.1 setup

The gradual death of the old Denon PMA-715R amplifier changed that.

A full surround setup is expensive and would leave me with unused hardware, especially the Infinity loudspeakers.

I bought two Ruizhi 2x50W + 100W amps for 30 euros each (well, I could have bought the simpler Ruizhi 2x50W for 10 euros each, but I was not entirely decided about the final setup at that moment), 12V DC adapters to go along, and two Edifier P12 20W speakers.

I doubt these Ruizhi can actually do 2x50W proper but if they do only half it will be enough.

The setup is now: front left+right = Infinity 50W + Ruizhi 50W ; rear left+right = Edifier P12 20W + Ruizhi 50W ; center+bass : Edifier M3280BT 8Wx2+20W.

It would be possible to use bluetooth with all these (Ruizhi and M3208BT) but I kept if simple with cables.

Such setup from scratch cost around 30 * 2 (Ruizhi) + 80 (P12) + 60 (M3280BT is no longer sold but similar models can be found between 50 to 90 euros, like Edifier M1370 or XB6BT) + 150 (passive 50/80W loudspeakers can be found from 100 to 200 euros) + cables =~ 400 euros for the whole. It is not entirely sure it is worth it: very cheap 5.1 setups, around 200 euros, would be as good and models around 400 euros might be better.

Reusing existing hardware, as in my case, we are down to 140 euros.

Preventing stored files deletion without assigning to root or using chattr

To prevent accidental or malicious files deletion (for instance a local collection of images, or a collection of movies on a Samba server), one option is to grant the directory to root or use chattr to make these files immutables (which also require root privileges).

That works. But any further modification would then require root privileges.

The proposed approach is, instead, to change ownership of files that reached a certain age (one week, one month or one year) to a dedicated “read-only” user, in a way that usual users can still add new files in the collection directories but no longer remove the old ones to safekeep.

This is not opposed to backups or filesystem snapshots, it is a step to prevent data loss instead of curing it.

Say on your video storage library on a samba server, files added by guest users are forcibly assigned to nobody:smbusers. You would then create a dedicated nobody-ro user and would configure in /etc/read-only-this.conf the library path to be handled by the read-only-this.pl script. Run by a daily cronjob, the read-only-this.pl script would reassign all files older than say one week to nobody-ro:smbusers, with no write group privilege. Directories would get the special T sticky bit so samba guest users would still be able to add new files but not remove old ones.

It would be possible to actually allow nobody-ro to log in through samba, shell or whatever scripts, to enable file removal or directory reorganisation. But the video storage library is protected from mistakes from regular users or malicious deletion by scripts using regular users accounts.

(Note that the read-only-this.pl script cares only for video/image/audio/documents mime types – the mime-type selection might later be added as configuration option)

It is included in the rien-common package.

Bye linuxfr.org

Today, someone posted a message on linuxfr wondering who are the users from 1999 still active on the website (created in 1998 – but first with another account database). 31 apparently. My own was created in December 1999, last post in 2010. After 22 years and 11 years without posting, while still from time to time checking if there might be an article of interest to me, it is safe to assume this account is now purposeless.

(update: maybe the message posted on linuxfr made people think, the 1999 active accounts list is now down to 28)

Converting video files to H.265/HEVC, no washed out colors and all streams mapped, with ffmpeg

I had a few 1080p video files using AV1 codec. Not sure why, if it is a player issue or hardware issue, nonetheless, my (slightly aging) laptop was having a hard time playing these while playing with no effort at all H.265/HEVC 2160p videos. The following command converts all mkv files in a folder to H.265/HEVC, not washing out colors and keep all streams:

for av in *.mkv; do ffmpeg -i "$av" -c:v libx265 -color_trc smpte2084 -color_primaries bt2020 -crf 22 -map 0 -map_metadata 0 -map_chapters 0 -c:a copy -c:s copy "${av%.*}"-x265.mkv ; done

In my case, it results in slightly larger files but, and that was the point, these play on the laptop with no noticeable CPU-usage:

1,2G 'XX1 AV1 Opus [AV1D].mkv'
1,1G 'XX1 AV1 Opus [AV1D]-x265.mkv'
830M 'XX2 AV1 Opus [AV1D].mkv'
951M 'XX2 AV1 Opus [AV1D]-x265.mkv'

Sorting and moving automatically videos from download to storage directory

In the spirit the earlier scripts to clean up ogg/mp3 collection (tags, filenames) with lltag, the following script is a proposal to automatically move, from download directory to storage directory, the video files that deserved to be kept. This is especially useful when both directories are on different physical drives and, as such, take a while, and typically with implies heavy IO usage at the moment you are actually using the computer that host the drivers.

The idea is to put a simple mark on directories or files, in the download area, that should be move to the storage area, assuming storage area contains top directories to sort files. The mark is arbitrary: ##Mark##, Mark matching a top directory within the download area.

For example, say we have in the download area the file Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.avi, associated with a subtitle file Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.en.srt, within a directory named Vabank II. It must go in storage area top directory Action Espionnage.

The way the script sort-download-area.pl works requires only the Vabank II or Vabank II, czyli riposta (1985) DVDRip XviD AC3-BR.avi to be renamed to include ##Action Espionnage##. And, obviously, to make it more practical, if can be also ##Action##, ##action##, ##Espionnage##, ##AE##, ##ActionEspionnage##, and others aliases as long as they are not confusing regarding other top directories of the download area.

Then running sort-download-area.pl --download /mnt/download --storage /mnt/storage (assuming these are the relevant directories) will take care of moving the found video and text/subtitles files (based on mime-type and filenames). The old directory will remain, the script won’t take any risk to erase any data by itself.

It can be run with --debug option to make a dry-run, to check if everything is in order, list possible marks, etc. If run as root, it will take care of changing mode and ownership to match the relevant download area top directory.

Once a satisfying setup is in place (assuming the script is in /usr/local/bin), it is enough to add a /etc/cron.daily/sort-download-area like:

#!/bin/sh
/usr/local/bin/sort-download-area.pl  --download /mnt/download --storage /mnt/storage

Here the current version of the sort-download-area.pl (but you are advised to always take the latest gitlab version) :

#!/usr/bin/perl

use strict "vars";
use Fcntl ':flock';
use POSIX qw(strftime);
use File::Find;
use File::Basename;
use File::Path qw(make_path);
use File::Copy qw(move);
use File::MimeInfo;
use File::Slurp qw(read_dir);
use Getopt::Long;
use Term::ANSIColor qw(:constants);

# config:
my $user = "nobody";
my $group = "nobody";
my ($download, $storage);
my $debug = 0;
my ($getopt, $help);

# get standard opts with getopt
eval {
    $getopt = GetOptions("debug" => \$debug,
			 "help" => \$help,
			 "download|d:s" => \$download,
			 "storagedir:s" => \$storage);
};

if ($help) {
    print STDERR <<EOF;
Usage: $0 [OPTIONS]

    -d DIR, --download DIR   (mandatory) path to the download/input area
    -s DIR, --storage DIR    (mandatory) path to the storage/output area

    --debug                  Dry-run debug test


Author: yeupou\@gnu.org
       https://yeupou.wordpress.com/

EOF
    exit(1);    
}

unless ($download and $storage) {
    die "Both --download INDIR and --storage OUTDIR must be provided.\nExiting";
}
unless (-d $download and -d $storage) {
    die "Both $download (--download) and $storage (--storage) must exists.\nExiting";
}

sub debug {
    return unless $debug;
    print $_[1] if $_[1]; 
    print $_[0];
    print RESET if $_[1];
    print "\n";
}

########################################################################
## run

# silently forbid concurrent runs
# (http://perl.plover.com/yak/flock/samples/slide006.html)
open(LOCK, "< $0") or die "Failed to ask lock. Exit";
flock(LOCK, LOCK_EX | LOCK_NB) or exit;


####
#### Find out current possible storage top-dirs
#### (with their respective uid/gid)
# value equal to the real-top dir
my %storage_topdir;
# uid/gid of the real-top dir
my %storage_topdir_uid;
my %storage_topdir_gid;
# keep a list of confusing marks
my %storage_topdir_confusingmark;


debug("\n\nStorage ($storage) top-dirs:\n", ON_CYAN);

for my $dir (read_dir($storage)) {
    next unless -d "$storage/$dir";
    next if ($dir =~ /^\./);   # skip hidden dirs

    # store top dir details
    $storage_topdir{$dir} = $dir;
    $storage_topdir_uid{$dir} = (lstat "$storage/$dir")[4];
    $storage_topdir_gid{$dir} = (lstat "$storage/$dir")[5];    
    debug("\t$storage_topdir{$dir}", GREEN);
    debug("\t($storage_topdir_uid{$dir}:$storage_topdir_gid{$dir})");

    # store also top dir useful aliases (end user might want to use shortcuts)
    # but no checks will be made in case of confusing aliases (ie two top dirs shortened in the name way)
    # for instance, Action Espionnage would also accept:
    #           action espionnage (lowercased)
    #		ActionEspionnage (removal of non-word chars)
    #		actionespionnage (lowercased removal of non-word chars)
    #		AE (only capital letters)
    #           ea (lowercase only capital letters)
    #           Action (single word apart)
    #           action (lowercased single word apart)
    #           Espionnage (single word apart)
    #           espionnage (lowercased single word apart)
    

    # alias as lowercased : WesteRn eq western
    my $alias = lc($dir);
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (lowercased)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tlowercased alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
    }
    # alias with space in place of any non word characters
    $alias = $dir;
    $alias =~ s/[^[:alnum:]]//g;
    
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (removal of non-word chars)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tremoval of non-word chars alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
	
	# same lowercased
	$alias = lc($alias);
	if ($alias ne $dir) {
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (lowercased removal of non-word chars)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tlowercased removal of non-word chars alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	}
    }
    # alternatively, only keep the capitalized letters
    $alias = $dir;
    $alias =~ s/[^[:alnum:]]//g;
    $alias =~ s/[^[:upper:]]//g;
    if ($alias ne $dir) {
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (only capital letters)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tonly capital letter alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
	# same lowercased     
	$alias = lc($alias);
	unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
	    debug("\t\t$alias (lowercased only capital letter alias)");
	    $storage_topdir{$alias} = $dir;
	} else {
	    debug("\t\tlowercased only capital letter alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
	    $storage_topdir_confusingmark{$alias} = 1;
	    delete($storage_topdir{$alias});
	}
    }
    # finally, if several worlds compose a string, try to register each
    # (this is where it is most likely to find confusing aliases)
    if (split(" ", $dir) > 1) {
	foreach my $word (split(" ", $dir)) {
	    $alias = $word;
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (single word apart)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tsingle word apart alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	    $alias = lc($alias);
	    unless ($storage_topdir{$alias} or $storage_topdir_confusingmark{$alias}) {
		debug("\t\t$alias (lowercased single word apart)");
		$storage_topdir{$alias} = $dir;
	    } else {
		debug("\t\tlowercased single word apart alias ($alias) is confusing regarding earlier items, skipping", ON_RED);
		$storage_topdir_confusingmark{$alias} = 1;
		delete($storage_topdir{$alias});
	    }
	}
    }
        
}


####
#### Find out any file or directory that we should be moving
#### (do not start moving files unless we checked everything)

# build an hash of files to move
# (with a secondary hash to keep track of the storage topdir) 
my %tomove;
my %tomove_topdir;


debug("\n\nDownload ($download) files:\n", ON_CYAN);

sub wanted {
    # $File::Find::dir is the current directory name,
    # $_ is the current filename within that directory
    # $File::Find::name is the complete pathname to the file.

    # check if we have a ##STRING## inside
    my $mark;
    $mark = $1 if $File::Find::name =~ m/##(.*)##/;

    # none found, skipping
    next unless $mark;

    # string refers to non-existant directory, skipping
    unless ($storage_topdir{$mark}) {
	debug("Mark $mark found for $File::Find::name while no such storage directory exists in $storage", ON_RED);
	# this is an issue that requires manual handling, print ont STDERR
	print STDERR ("Mark $mark found for $File::Find::name while no such storage directory exists in $storage\n");
	next;
    }

    # take into account only videos and text files
    my $suffix;
    $suffix = $1 if $_ =~ /([^\.]*)$/;
    my ($mime_type,$mime_type2) = split("/", mimetype($File::Find::name));
    if ($mime_type ne "video" and
	$mime_type ne "text") {
	# second pass to allow even more text files based on extension
	# (subtitles : srt sub ssa ass idx txt smi)
		
	unless ($suffix eq "srt" or
		$suffix eq "sub" or
		$suffix eq "txt" or
		$suffix eq "ssa" or
		$suffix eq "ass" or
		$suffix eq "idx" or
		$suffix eq "smi") {
	    debug("\tskip $_ ($mime_type/$mime_type2 type)");
	    next;		
	}
    }

    my $destination_dir = "$storage/$storage_topdir{$mark}";
    my $destination_file = $_;
    $destination_file =~ s/##(.*)##//g;
    $destination_file =~ s/^\s*//;
    $destination_file =~ s/\s*$//;

    # now handle the special S00E00 case of series, like 30 Rock (2006) - S05E16 or 30 Rock S05E16
    my ($season, $before_season, $show);
    $before_season = $1 and $season = $2 if $_ =~ m/^(.*)S(\d\d)\ ?E\d\d[^\d]/i;
    if ($season) {
	# there is a season, we must determine the show name
	#    30 Rock (2006) - S05E16 => 30 Rock
	# end user must pay attention to have consistent names
	$show = $1 if $before_season  =~ m/^([\w|\s|\.|\'|\,]*)/g;
	# dots often are used in place of white spaces
	$show =~ s/\./ /g;    
	# keep only spaces in shows name, nothing else
	$show =~ s/[^[:alnum:]|\ ]//g;    
	$show =~ s/^\s*//;
	$show =~ s/\s*$//;
	# capitalize first letter
	$show =~ s/\b(\w)/\U$1/g;
	
	# if we managed to find the show name, then set up the specific series tree
	last unless $show;
	debug("found show: $show", MAGENTA);
	$destination_dir = "$storage/$storage_topdir{$mark}/$show/S$season";	    	
    }

    
    # if we reach this point, everything seems in order, plan the move
    debug("plan -> $destination_dir/$destination_file");
    $tomove{$File::Find::name} = "$destination_dir/$destination_file";
    $tomove_topdir{$File::Find::name} = $storage_topdir{$mark};


    # additionally, if we deal with a video, look for any possibly related file to add also that would not have been picked
    # otherwise
    if ($mime_type eq "video") {
	my $other_files_path = $File::Find::name;
	$other_files_path =~ s/\.$suffix$//g;

	debug("glob $other_files_path*");
	my @other_files =
	    glob('$other_files_path*.srt'),
	    glob('$other_files_path*.sub'),
	    glob('$other_files_path*.txt'),
	    glob('$other_files_path*.ssa'),
	    glob('$other_files_path*.ass'),
	    glob('$other_files_path*.idx'),
	    glob('$other_files_path*.smi');
	foreach my $file (@other_files) {
	    debug("plan -> $destination_dir/$file");
	    $tomove{"$File::Find::dir/$file"} = "$destination_dir/$file";
	    $tomove_topdir{"$File::Find::name/$file"} = $storage_topdir{$mark};
	}		    
    }

    debug();
      
}
find(\&wanted, $download);

####
#### Actually move files now
####

debug("\n\nMove from download ($download) to storage ($storage):\n", ON_CYAN);

foreach my $file (sort keys %tomove) {

    debug(basename($file), YELLOW);

    my $uid = $storage_topdir_uid{$tomove_topdir{$file}};
    my $gid = $storage_topdir_gid{$tomove_topdir{$file}};    

    # create directory if needed
    my $dir = dirname($tomove{$file});
    unless (-e $dir) {
	make_path($dir, { chmod => 0770, user => $uid, group => $gid }) unless $debug;
	debug("make_path $dir (chmod => 0770, user => $uid, group => $gid)");
    }

    # then move the file (chown if root)
    # avoid overwriting, add number in the end, no extension saving
    my $copies;
    if (-e $tomove{$file}) {
	while (-e "$tomove{$file}.$copies") {
	    $copies++;
	    # stop at 10, makes no sense to keep more than that amount of copies
	    last if $copies > 9;
	}
    }    
    $tomove{$file} .= ".$copies" if $copies;
    move($file, $tomove{$file}) unless $debug;
    chown($uid, $gid, $tomove{$file}) unless $debug or $< ne 0;
    debug("$file -> $tomove{$file}");

    debug();	  
}

# EOF

Signing the open letter in favor of RMS, even though he probably should not head FSF?

Six years ago, I posted an article related to my (limited) direct experience with Richard Matthew Stallman, which I concluded by: although he values his freedom and values freedom in general, working with him, even in a very distant way, is just a matter of subordination; he’d make a credible science-fiction character: distopian guru, the Pied Piper of MIT.

You would assume that I would approve his removal from Free Software Foundation. But no: I was not expecting this to be based on the current trendy totalitarian philosophy. No one should be happy that someone is prevented to do his work due to his identity or political and philosophical opinions – or, worse, how he is depicted by an angry mob no matter what he actually thinks or said.

In regard of Free Software, RMS is as important as Winston Churchill was regarding UK’s position during World War II. He built the philosophical base of what Free Software is. It would not be, or in a completely different form, without him. And you cannot claim to promote of something “meant to serve everyone regardless of their age, ability or disability, gender identity, sex, ethnicity, nationality, religion or sexual orientation” when you actually exactly do the contrary. This way of thinking when you make a list of people always right other always wrong, when you silence the one that are wrong, matches totalitarian ideologies, not freedom.

So I really don’t care about RMS position at FSF, it is probably for the best that he is no longer in his autocratic position. I surely don’t care about his personal opinion about this or that topic unrelated to software. I guess some other people might think likewise. But it is not a reason to keep silent toward ideological violence.

If you like to rethink all these dark events in history, that are never black or white, you’ll consider that the issue is not that much about the main protagonists, following their path whether they’ll turn out to be criminals or freedom fighters. No, the issue is regarding the bystanders, that will see questionable things being done but won’t comment, because it does not affect them really, because they felt no connection to the one attacked or, because they felt maybe it is was on some other level deserved. But history judgment is harsh on them, nonetheless.

“We ask for contributors to free software projects to take a stand against bigotry and hate within their [FSF] projects”, they wrote. Bigotry and hate are terms that can easily be turned to describe them, or easy to manipulate in every direction. When you silence people, there is hate. When you create a work environment in which people are silenced due to their opinions, there is bigotry. I believe that the sane way to regulate society is called rights of man: can be punished, silenced, only if they have been proven of breaking legitimate laws by a legitimate court. And that led me to sign the letter in favor of RMS, even though I do not think he should not be at the head of FSF. We should not accept a society of oppression, no matter in which name, especially not in the name of greater good because that’s always the one invoked to do the worse. We thought ideologies were dead. No, they are as dangerous as ever.

PS : since GNOME Foundation is heavily involved and claim acting in regard of Free Sofware credibility, it is easy to point out they are not exactly known for that. Regarding Mozilla, RedHat, and similar companies, etc, hum, if they really want to howl with the wolves, maybe some day no one will care to promote their work instead of “don’t be Evil”-company.

PPS: seems that some people that want RMS eviction also advise to blacklist, recruitment-wise, anyone that signed the letter in favor of RMS (a, b, c, etc). The whole process is definitely quite disgusting, besides being completely stupid.

PPPS: some other made named-based statistics to guess which ethnicity or genre is voting for or against. I do not think there is any progress in essentialism and surely we cannot call democrats people in favor of racial or sexual-based voting rights (except by erasing concept of modern citizenship).

Moving a live encrypted system from one hard disk to another

This a short memento based on earlier articles Moving a live system from one hard disk to another and http://Single passphrase to boot Devuan GNU/Linux with multiple encrypted partitions. This is useful when you start to move over your systems partitions from HDD to SSD, that nowadays are clearly worth their cheap price.

This article is made for Devuan GNU/Linux but should not be distro specific – you just might want to replace devuan string in later command by something else.

We start by setting some variables depending on the relevant drives. Any doubt about which drive is what, running lsblk should help.

# new NVMe disk 
NDISK=/dev/nvme0n1
NPART_PREFIX=p

# or alternatively for a SATA new disk:
# NDISK=/dev/sdb
# NPART_PREFIX=

LUKS_PREFIX=250g21

(AHCI SATA SSD are faster than HDD, but AHCI itself will be the bottleneck, so I’d suggest to install a NVMe SSD if your mainboard allows).

# key necessary to mount all partitions with a singlepassphrase

key=/boot/klucz
if [ ! -e $key ]; then dd if=/dev/urandom of=$key bs=1024 count=4 && chmod 400 $key ; fi

Next step is to replicate the disk structure. While this article is BIOS-boot based, it should go along UEFI:

parted $NDISK

(parted shell)
mklabel gpt
mkpart biosreserved ext2 1049kB  50,3MB
toggle 1 bios_grub
mkpart efi fat32 50,3MB  500MB
toggle 2 msftdata
mkpart swap linux-swap 500MB   16,0GB
toggle 3 swap
mkpart root ext4 16,0GB 250GB
print

Model: KINGSTON SA2000M8250G (nvme)
Disk /dev/nvme1n1: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name          Flags
 1      1049kB  50,3MB  49,3MB                  biosreserved  bios_grub
 2      50,3MB  500MB   450MB                   efi           msftdata
 3      500MB   16,0GB  15,5GB  linux-swap(v1)  swap          swap
 4      16,0GB  250GB   234GB   ext4            root

quit

We build the new system partition (luks1 is still mandatory with grub, but that won’t be true forever we can suppose):

cryptsetup luksFormat --type luks1 "$NDISK$NPART_PREFIX"4
cryptsetup luksOpen "$NDISK$NPART_PREFIX"4 "$LUKS_PREFIX"devuan64
cryptsetup  luksAddKey "$NDISK$NPART_PREFIX"4 $key
mkfs.ext4 /dev/mapper/"$LUKS_PREFIX"devuan64 -L "$LUKS_PREFIX"devuan64

mkdir /tmp/"$LUKS_PREFIX"devuan64
mount /dev/mapper/"$LUKS_PREFIX"devuan64 /tmp/"$LUKS_PREFIX"devuan64

ignore="backups home dev lost+found media proc run sys tmp"
for dir in $ignore; do touch /$dir.ignore ; done && for dir in /*; do if [ -d $dir ]; then if [ ! -e $dir.ignore ]; then /usr/bin/rsync --archive --one-file-system --delete $dir /tmp/"$LUKS_PREFIX"devuan64/ ; else if [ ! -e /tmp/"$LUKS_PREFIX"devuan64/$dir ]; then mkdir /tmp/"$LUKS_PREFIX"devuan64/$dir; fi ; rm $dir.ignore ; fi ; fi ; done

We update required system files:

echo " " >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "# "`date` >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "$LUKS_PREFIX"devuan64 UUID=`blkid -s UUID -o value "$NDISK$NPART_PREFIX"4` $key luks,tries=3,discard >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab
echo "$LUKS_PREFIX"swap `find -L /dev/disk -samefile "$NDISK$NPART_PREFIX"3 | grep by-id | tail -1` /dev/urandom cipher=aes-xts-plain64,size=256,swap,discard >> /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab

# comment out old lines
nano /tmp/"$LUKS_PREFIX"devuan64/etc/crypttab


echo " " >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "# "`date` >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "/dev/mapper/"$LUKS_PREFIX"devuan64	/		ext4	errors=remount-ro		0 1" >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab
echo "/dev/mapper/"$LUKS_PREFIX"swap	none		swap	sw		0 0" >> /tmp/"$LUKS_PREFIX"devuan64/etc/fstab

# comment out old lines
nano /tmp/"$LUKS_PREFIX"devuan64/etc/fstab



echo "Make sure this is in grub config:"
echo
echo GRUB_CMDLINE_LINUX=\"rd.luks.key=$key:UUID=`blkid "$NDISK$NPART_PREFIX"4 -s UUID -o value`\"
echo GRUB_ENABLE_CRYPTODISK=\"y\"
echo GRUB_PRELOAD_MODULES=\"luks cryptodisk lvm\"

# update grub config
nano /tmp/"$LUKS_PREFIX"devuan64/etc/default/grub

Last step is to install the boot loader on the new disk:

mount --bind /dev /tmp/"$LUKS_PREFIX"devuan64/dev
mount --bind /sys /tmp/"$LUKS_PREFIX"devuan64/sys
mount -t proc /proc /tmp/"$LUKS_PREFIX"devuan64/proc
chroot /tmp/"$LUKS_PREFIX"devuan64

grub-mkdevicemap
update-initramfs -u

# need to be retyped since it not in chroot environment
NDISK=/dev/nvme0n1

grub-install $NDISK
grub-mkconfig > /boot/grub/grub.cfg

That’s all.

Using PowerDNS (server and recursor) with DNSSEC and domain name spoofing/caching

update, October 2021: I consider stopping using PowerDNS. I do not want to rely on unreliable people (debian bug #997054 and debian bug #997056).

I updated my earlier PowerDNS (server and recursor) setup along with domain name spoofing/caching. This is a short update to allow DNSSEC usage and unlimited list of destinations from spoof/cache list. Files described here can be found on gitlab.

DNSSEC

Adding DNSSEC support can done easily by creating /etc/powerdns/recursor.d/10-dnssec.conf :

# dnssec	DNSSEC mode: off/process-no-validate 
#                    (default)/process/log-fail/validate
dnssec=validate

#################################
# dnssec-log-bogus	Log DNSSEC bogus validations
dnssec-log-bogus=no

Every local zone must be excluded by adding to /etc/powerdns/recursor.lua :

addNTA("10.10.10.in-addr.arpa", "internal zone")

New redirect.lua renewal script

Earlier version provided a static /etc/powerdns/redirect.lua which was depending on with redirect-cached.lua, redirect-ads.lua and redirect-blacklisted.lua, which contained lists of domains to either blacklist (meaning: redirected to loopback) or spoof.

Now, the script redirect-rebuild.pl use the configuration redirect-spooflist.conf to generate redirect.lua. The ads blacklist part is unchanged.

The configuration syntax is as follow:

# IP:	domain domain 

# redirect thisdomain.lan and thisother.lan to 192.168.0.1,
# except if 192.168.0.1 is asking 
192.168.0.1: thisdomain.lan thisother.lan 

# redirect anotherthisdomain.lan and anotherthisother.lan to 10.0.0.1,
# even if 10.0.0.1 is asking 
10.0.0.1+:    anotherthisdomain.lan anotherthisother.lan 

# you can use 127.0.0.1: to blacklist domains

It is enough to run the redirect-rebuild.pl script and restart the recursor:

use strict;
use Fcntl ':flock';

my $spooflist = "redirect-spooflist.conf";
my $ads_lua = "redirect-ads.lua";
my $ads_pl = "redirect-ads-rebuild.pl";
my $main_lua = "redirect.lua";

# disallow concurrent run
open(LOCK, "< $0") or die "Failed to ask lock. Exiting";
flock(LOCK, LOCK_EX | LOCK_NB) or die "Unable to lock. This daemon is already alive. Exiting";

# first check if we have a ads list to block
# if not, run the local script to build izt
unless (-e $ads_lua) {
    print "$ads_lua missing\n";
    print "run $ads_pl\n" and do "./$ads_pl" if -x $ads_pl;
}

my %cache;
# read conf
open(LIST, "< $spooflist");
while (<LIST>) {
    next if /^#/;
    next unless s/^(.*?):\s*//;
    $cache{$1} = [ split ];
}
close(LIST);

# build lua
open(NEWCONF, "> $main_lua");
printf NEWCONF ("-- Generated on %s by $0\n", scalar localtime);
print NEWCONF '-- IPv4 only script

-- ads kill list
ads = newDS()
adsdest = "127.0.0.1"
ads:add(dofile("/etc/powerdns/redirect-ads.lua"))

-- spoof lists
';

foreach my $ip (keys %cache) {
    # special handling of IP+, + meaning we spoof even to the destination host
    my $name = $ip;
    $name =~ s/(\.|\+)//g;  
    print NEWCONF "spoof$name = newDS()\n";
    print NEWCONF "spoof$name:add{", join(", ", map "\"$_\"", sort@{$cache{$ip}}), "}\n";
    $ip =~ s/(\+)//g;
    print NEWCONF "spoofdest$name = \"$ip\"\n";
}

print NEWCONF '
function preresolve(dq)
   -- DEBUG
   --pdnslog("Got question for "..dq.qname:toString().." from "..dq.remoteaddr:toString().." to "..dq.localaddr:toString(), pdns.loglevels.Error)
   
   -- spam/ads domains
   if(ads:check(dq.qname)) then
     if(dq.qtype == pdns.A) then
       dq:addAnswer(dq.qtype, adsdest)
       return true
     end
   end
    ';

foreach my $ip (keys %cache) {
    my $always = 0;
    $always = 1 if ($ip =~ s/(\+)//g);     # + along with IP means always spoof no matter who is asking
    my $name = $ip;
    $name =~ s/\.//g;

    print NEWCONF '
   -- domains spoofed to '.$ip.'
   if(spoof'.$name.':check(dq.qname)) then';
    print NEWCONF '
     dq.variable = true
     if(dq.remoteaddr:equal(newCA(spoofdest'.$name.'))) then
       -- request coming from the spoof/cache IP itself, no spoofing
       return false
     end' unless $always;
    print NEWCONF '   
     if(dq.qtype == pdns.A) then
       -- redirect to the spoof/cache IP
       dq:addAnswer(dq.qtype, spoofdest'.$name.')
       return true
     end
   end
	';
}

print NEWCONF '
   return false
end
';
close(NEWCONF);


# EOF