Downgrading Nextcloud 18.0.3.0 to 17.0.5

I upgraded Nextcloud to the latest save without realizing the gallery app has been replaced by an half-baked “photos” app, completely useless to share pictures in any relevant way to me, in addition to a bug with ublock origin making the whole “sharing” interface disappearing.

Rolling back is not as easy at it seems: the “occ” php app check with your config version if it matches the software version, and if not matching just plainly refuses to work with:

An unhandled exception has been thrown:
OC\HintException: [0]: Downgrading is not supported and is likely to cause unpredictable issues (from 18.0.3.0 to 17.0.5.0) ()

So you need to update this config/config.php. Then, playing with occ app:list, occ app:remove and occ app:install only you can get back to a working install.

Resize completely a wordpress.com blog’s media gallery

I found no convenient way to resize a whole media gallery on wordpress.com, with the free plan which does not allow to install plugins. Aside from that, I find strange wordpress itself still does not prevent duplicates media using checksum or else.

I had a blog with a media gallery reaching the limit of upload on a free plan. And it contained tons of very high-res pictures that actually could be downsized without posing any problem.

I found no convenient way to replace images (with the free plan and no plugins). If you reupload the same file, after deleting it, it will get an extra suffix -1/-2, etc: wordpress clearly keep the deleted media names in the database and prevent it to be reused so it is a no go.

The only solution I found was to:

  • export both post and media files;
  • delete all files and post;
  • run scripts to update images files and xml export/import files;
  • reimport everything with new filenames.

It is not perfect, some data will be lost, namely old galleries and some post front image choice, etc. Follows the first approach (1, 2, 3) and the second one (1, 2+3, 4) that tries to do smarter things (but is then more likely to break soon).

1. Renaming and downsizing the images

# to run in the exported media directory (extracted in a single directory)    
date=`date +%H%M%S`
backup=`mktemp --directory --tmpdir=$PWD -t $date-XXX.bak`
    
for file in  *.png *.jpg *.jpeg; do
	# skip if not a file
	if [ ! -f "$file" ]; then continue; fi

	# rename:
	newfile="BLOGNAME.wordpress.com-"$file
	echo "$file => $newfile"
	
	# limit to 1600 in max size  - to check up to decent size amount for the full
	convert "$file" -resize 1600x1600\> "$newfile"
	mv "$file" $backup/
	
done

2. Updating post export/import with new images filenames

#!/usr/bin/perl
# to be saved as as perl script file, 
# edit (especial THISBLOG and 2020/03, YYYY/MM of the new upload that will be added automatically in the URL of the media file during upload)
# and then 
# run against the exports files like
# chmod +x ./thiscript.pl
# ./thiscript.pl 

# note: wordpress.com rename -- in - during upload

use strict;


open(IN, "< $ARGV[0]");
open(OUT, "> edited.$ARGV[0]");
while (<IN>) {
    s@\.files\.wordpress.com\/\d{4}\/\d{2}\/@.files.wordpress.com/2020/03/THISBLOG.wordpress.com-re.up-@ig;
    print OUT $_;
}
close(IN);
close(OUT);

3. Checking if images appears in posts

#!/usr/bin/perl
#
# make sure every image downloaded actually exist in posts
# also to download and run against the xml import/export files and the media directory
# ./thiscript.pl THISBLOG.wordpress.2020-03-29.001.xml #../images_updated

use strict;
use warnings;
use Regexp::Common qw/URI/;
use File::Basename;
use File::Copy;

my %images;

open(IN, "< $ARGV[0]");
while (<IN>) {
    my $url;
    next unless /($RE{URI}{HTTP}{-keep})/;
    $url = $1;
    next unless $url =~ /THISBLOG\.files\./;
    my ($basename, $parentdir, $extension) = fileparse($url, qr/\.[^.]*$/);

    if ($extension  =~ /^\.(png|jpg|jpeg|gif)$/i) {
	#print "$basename$extension ($url)\n";
	$images{"$basename$extension"} = "$basename$extension"
	    unless $images{"$basename$extension"};
    } else {
	#print "IGNORE $basename$extension ($url)\n";
    }

}

while (my($image, ) = sort(each(%images))) {
    print "$image\n";
    if ($ARGV[1]) {
	if ((-d "$ARGV[1]") and (-e "$ARGV[1]/$image")) {
	    mkdir("$ARGV[1]/valide") unless -d "$ARGV[1]/valide";
	    copy("$ARGV[1]/$image", "$ARGV[1]/valide/") or print "failed to copy $image to $ARGV[1]/valide/\n";
	}
    }
}

This scripts are primitive (sure, even the blog name and upload YYYY/MM was hardcoded). Since platforms like wordpress.com often changes, this might no longer works at another time. Many pages on the Internet claims you can simply erase and reupload image with the same name: this clearly no longer works. This page could save you some trial-and-error process to give a solution that works as of today.

Note also that some wordpress.com theme have front page image and other selections that wont be carried over.

2+3. Updated script to update xml and check images

I wrote the following script for a smoother process. Being more complex, it is more likely to be fragile also. It still miss handling/removing some specific outdated wp:metadata. This one list duplicates, unused and missing files.

#!/usr/bin/perl
#
# ./renew-url+checkimages.pl FOLDER_OF_XML FOLDER_OF_IMAGES (no subdirs)

use strict;
use warnings;
use Regexp::Common qw/URI/;
use File::Basename;
use File::Copy;
use File::Slurp;
use Cwd 'abs_path';
use MIME::Base64;
use URI;

my $blog = "zukowka";
my $newprefix = "zukowka.wordpress.com-re.up-";
my $upload_yyyymm = "2020/03"; # time of the new upload
my $images_types = "png|jpg|jpeg|gif";

my %oldurl_newurl;       # {original} = new  
my %imagebase64_newurl;    # {base64} = new_url


die "$0 FOLDER_OF_XML FOLDER_OF_IMAGES (no subdirs)" unless $ARGV[0] and$ARGV[1];



# get dir full path 
my $xmldir;
$xmldir = abs_path($ARGV[0])
    if $ARGV[0];
# get dir full path 
my $imgdir;
$imgdir = abs_path($ARGV[1])
    if $ARGV[1];


my ($unused,$dupes,$missing,$mapped);
open(DUPES, "> $0.duplicatedimages");  # same image found with different names/URL
open(MISSING, "> $0.missingimages");   # file listed in xml not found in the image directory
open(UNUSED, "> $0.unusedimages");   #  file found in the image directory but not listed
open(MAPPING, "> $0.mapping");


# test access to dir
print "XML dir (ARG1): $xmldir
images dir (ARG2): $imgdir\n";
chdir($xmldir) if -d $xmldir or die "unable to enter $xmldir, exiting"; 
chdir($imgdir) if -d $imgdir or die "unable to enter $imgdir, exiting";

# - slurp all urls in import files
# - check if the image listed exist, store with base64 so we keep only one
chdir($xmldir);
while (defined(my $file = glob("*.xml"))) {
    print "### slurp $file ###\n";
    open(IN, "< $file");
    while (<IN>) {
	while (/($RE{URI}{HTTP}{-scheme => qr@https?@}{-keep})/g) {
	    # remove the http/https and possible args to keep the most portable url 
	    my $uri = URI->new($1);
	    my $url = $uri->authority.$uri->path;

	    
	    # images URL always start with $blog.files.wordpress.com
	    next unless $url =~ /^$blog\.files\./;
	    
	    my ($basename, $parentdir, $extension) = fileparse($url, qr/\.[^.]*$/);
	    my $newurl = "$blog.files.wordpress.com/$upload_yyyymm/$newprefix$basename$extension";
	    
	    # skip if the url was already mapped
	    if ($oldurl_newurl{$url}) {
		#print "SEEN ALREADY (skipping) $url\n";
		next;
	    }
	    
	    # work only on images
	    next unless lc($extension) =~ /^\.($images_types)$/i;
	    
	    # check if the relevant image exists in the image folder
	    my $newimage = "$imgdir/$newprefix$basename$extension";
	    unless (-e $newimage) { 
		print "MISSING (skipping) $newimage\n";
		print MISSING "$newimage\n";
		$missing++;
		next;
	    }
	    
	    # get image base64
	    my $base64 = encode_base64(read_file($newimage));
	    
	    # find out if this exact image is already known
	    if ($imagebase64_newurl{$base64}) {
		# already known, will point to the first one found
		print "DUPES $newprefix$basename$extension\n";
		print DUPES "$newprefix$basename$extension:\n\t$url => $imagebase64_newurl{$base64}\n";
		$dupes++;
		
		# URLs will point to the first image found
		#   full form like http://blog.files.wordpress.com/YYYY/MM/file.jpg
		$oldurl_newurl{$url} = $imagebase64_newurl{$base64};
		#   short form like http://blog.wordpress.com/file/
		my ($base64_basename, $base64_parentdir, $base64_extension) = fileparse($imagebase64_newurl{$base64}, qr/\.[^.]*$/);
		$oldurl_newurl{"$blog.wordpress.com/$basename/"} = "$blog.wordpress.com/$base64_basename/" if
		    "$blog.wordpress.com/$basename/" ne "$blog.wordpress.com/$base64_basename/";
		
	    } else {
		# store base64 with full form url to
		$imagebase64_newurl{$base64} = $newurl;
		
		# URLs will point to the first image found
		#   full form like http://blog.files.wordpress.com/YYYY/MM/file.jpg
		$oldurl_newurl{$url} = $newurl;
		#   short form like http://blog.wordpress.com/file/
		$oldurl_newurl{"$blog.wordpress.com/$basename/"} = "$blog.wordpress.com/$newprefix$basename/" if
		    "$blog.wordpress.com/$basename/" ne "$blog.wordpress.com/$newprefix$basename/";
			
	    }
	}
    }
    close(IN);
}

# store mappings
my %used;
while (my($oldurl,$newurl) = sort(each(%oldurl_newurl))) {
    print MAPPING "$oldurl => $newurl\n";
    my ($basename, $parentdir, $extension) = fileparse($newurl, qr/\.[^.]*$/);
    $used{"$basename$extension"} = 1;
    $mapped++;
}


# build import xml 
chdir($xmldir);
mkdir("$xmldir/renew") unless -d "$xmldir/renew";
while (defined(my $file = glob("*.xml"))) {
    print "### rewrite $file ###\n";
    open(OUT, "> renew/renewed-$file");
    open(IN, "< $file");
    while (<IN>) {
	my $line = $_;
	# check every url
	while (/($RE{URI}{HTTP}{-scheme => qr@https?@}{-keep})/g) {
	    # remove the http/https and possible args to keep the most portable url 
	    my $uri = URI->new($1);
	    my $url = $uri->authority.$uri->path;
	    
	    # update if mapping registered
	    if ($oldurl_newurl{$url}) {
		$line =~ s/$url/$oldurl_newurl{$url}/g;
		#print "$url -> $oldurl_newurl{$url}\n"
	    }
	}
	print OUT $line;
    }
    close(OUT);
    close(IN);	
}

# finally list useless images
chdir($imgdir);
mkdir("$xmldir/renew") unless -d "$xmldir/renew";
while (defined(my $file = glob("*"))) {    
    # work only on images
    my ($basename, $parentdir, $extension) = fileparse($file, qr/\.[^.]*$/);
    next unless lc($extension) =~ /^\.($images_types)$/i;

    # check if registered yet
    next if $used{$file};

    # if we reach this point, this media is unknown
    $unused++;
    print UNUSED $file, "\n";
}

close(MAPPING);
close(DUPES);
close(MISSING);
close(UNUSED);


print "=============================
$mapped mapped URL
$missing missing files (!)
$dupes duplicated files/links
$unused unused files\n";

# EOF

Grabbing new images post_id

There are used in gallery in the form . To use this script, you must compare your new XML produced by the previous script and new XML export made by wordpress.com AFTER uploading the new images.

The point is to get post_id from newly uploaded images and to map them to the old removed images post_id.


#!/usr/bin/perl
#
# ./update-post_id.pl FOLDER_OF_XML_UPDATED FOLDER_OF_XML_AFTER_IMAGE_REUPLOAD
#
#
# galleries are } -->
# refering to <wp:post_id>27</wppost_id> of image attachements.
#
# reuploaded files have new post_id along with new metadata hardcoded in the database
# 	<guid isPermaLink="false">http://XX.files.wordpress.com/2020/03/XXX-img_20160814_125558.jpg</guid>
#       <wp:post_type>attachment</wp:post_type>
# should match
#       <wp:attachment_url>https://XX.files.wordpress.com/2020/03/XXX-img_20160814_125558.jpg</wp:attachment_url>


use strict;
use warnings;
use Regexp::Common qw/URI/;
use File::Basename;
use File::Copy;
use File::Slurp;
use Cwd 'abs_path';
use MIME::Base64;
use URI;
use XML::LibXML;

die "$0 FOLDER_OF_XML_UPDATED FOLDER_OF_XML_AFTER_IMAGE_REUPLOAD" unless $ARGV[0] and$ARGV[1];



# get dir full path 
my $xmldir;
$xmldir = abs_path($ARGV[0])
    if $ARGV[0];
# get dir full path 
my $xmlafterreuploaddir;
$xmlafterreuploaddir = abs_path($ARGV[1])
    if $ARGV[1];

# test access to dir
print "XML dir (ARG1): $xmldir
XML after reupload dir (ARG2): $xmlafterreuploaddir\n";
chdir($xmldir) if -d $xmldir or die "unable to enter $xmldir, exiting"; 
chdir($xmlafterreuploaddir) if -d $xmlafterreuploaddir or die "unable to enter $xmlafterreuploaddir, exiting";


my %guid_postid;  # {guid} = postid



chdir($xmlafterreuploaddir);
# get postid after reupload
while (defined(my $file = glob("*.xml"))) {
    print "### read $file ###\n";
    my $dom = XML::LibXML->load_xml(location=>$file);

    foreach my $e ($dom->findnodes('//item')) {	
	#	print $e->to_literal();

	# only care for attachement type post
	next unless $e->findvalue('./wp:post_type') eq 'attachment';
	
	# store new post_id with the guid as key 	
	$guid_postid{$e->findvalue('./guid')} = $e->findvalue('./wp:post_id');
    }   
}


my %old2new_postid; # {old} = new


chdir($xmldir);
# get postid in first export
while (defined(my $file = glob("*.xml"))) {
    print "### read $file ###\n";
    my $dom = XML::LibXML->load_xml(location=>$file);

    foreach my $e ($dom->findnodes('//item')) {
	# only care for attachement type post
	next unless $e->findvalue('./wp:post_type') eq 'attachment';

	# ignore if this guid was not found/replaced
	next unless $guid_postid{$e->findvalue('./guid')};

	# map postids
	$old2new_postid{$e->findvalue('./wp:post_id')} = $guid_postid{$e->findvalue('./guid')};	
	print $e->findvalue('./wp:post_id')." -> ".$guid_postid{$e->findvalue('./guid')}."\n";
    }   
}


# finally, with this mapping, edit xml gallery entries:
# build import xml 
chdir($xmldir);
mkdir("$xmldir/newpostid") unless -d "$xmldir/newpostid";
while (defined(my $file = glob("*.xml"))) {
    print "### rewrite $file ###\n";
    open(OUT, "> newpostid/$file");
    open(IN, "< $file");
    while (<IN>) {
	my $line = $_;
	# older galleries
	# 	
	while (/\*)".*]/g) {
	    print "$1\n";
	    my $original = $1;
	    my @new_ids;
	    foreach my $id (split(",", $original)) {
		if ($old2new_postid{$id}) {
		    push(@new_ids, $old2new_postid{$id});
		} else {
		    # if not found, push back the original one
		    push(@new_ids, $id);
		}
	    }
	    my $new = join(",", @new_ids);
	    print " => $new\n";

	    $line =~ s/\,"columns":2} -->	
	while (/\<\!\-\- wp\:gallery \{\"ids\"\:\[([\d|,]*)\].*\}/g) {
	    print "$1\n";
	    my $original = $1;
	    my @new_ids;
	    foreach my $id (split(",", $original)) {
		if ($old2new_postid{$id}) {
		    push(@new_ids, $old2new_postid{$id});
		} else {
		    # if not found, push back the original one
		    push(@new_ids, $id);
		}
	    }
	    my $new = join(",", @new_ids);
	    print " => $new\n";

	    $line =~ s/\<\!\-\- wp\:gallery \{\"ids\"\:\[$original\]/<!-- wp:gallery {"ids":[$new]/g;
	    print $line;
	}

	print OUT $line;
    }
    close(OUT);
    close(IN);	
}


# EOF

SPF-aware greylisting with Exim and memcache

This is a followup of my 2011’s article avoiding Spams with SPF and greylisting within Exim. What changed since then? I actually am not more harrassed by spam that I was earlier on. It works. I am spam free since a decade now. No, but, however, several importants mail providers have a tendancy to send mail through multiples SMTPs, so many it took a while for any of them to do at least two attempt. So some mails takes ages to pass the greylist.

Contemplating the idea to use opensmtpd, I incidentally found an interesting proposal to mix greylisting of IP with SPF-validated domains.

The idea is that you greylist either an SMTP IP or a domain including any SMTP IP approved by SPF.

I updated the memcached-exim.pl script previously used and described. It was simplified because I dont think useful to actually make greylist per sender and recipient, only per IP or domain. Now it either only greylist IP, if not validated by SPF, or the domain and IP on success (to save a few SPF further test).

I dont think it should have any noticeable impact on the server behavior. SPF is anyway checked, so it is meaningless since there is local caching DNS on my mail servers.

The earlier /etc/exim4/memcached.conf is actually no longer required (defaults are enough). You still need exim configuration counterparts:  /etc/exim4/conf.d/main/00_stalag13-config_0greylist and /etc/exim4/conf.d/acl/26_stalag13-config_check_rcpt.

Delisting an Exim4 server from Office365 ban list

Ever tried to get delisted from Office365 ban list, for whatever reason you might try to get (new IP for a server that was abused in the past or else, you won’t know since they wont tell – and it even looks like they probably dont even really know)?

It is a funny process, because it involves receive a mail from their servers, a mail that will probably be flagged as spam, with clues so big that it might be blocked at SMTP time.

With Exim4, you’ll probably get in the log something like:

2019-08-20 22:18:09 1i0Aa1-0004Hu-8h H=mail-eopbgr740042.outbound.protection.outlook.com (NAM01-BN3-obe.outbound.protection.outlook.com) [40.107.74.42] X=TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256 CV=no F=<no-reply@microsoft.com> rejected after DATA: maximum allowed line length is 998 octets, got 3172

Long story short (this length test is not welcomed by all users), add /etc/exim4/conf.d/main/00_localoptions add

IGNORE_SMTP_LINE_LENGTH_LIMIT=1

and then restart the server.

Try delisting and check your spam folder. You should get now the relevant mail. Whatever we think about the lenght limit test of Exim4 (based on RCF, isn’t it?), you still end up with a mail sent by Office365 like this:

X-Spam-Flag: YES
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.2 required=3.4 tests=BASE64_LENGTH_79_INF,
	HTML_IMAGE_ONLY_08,HTML_MESSAGE,MIME_HTML_ONLY,MIME_HTML_ONLY_MULTI,
	MPART_ALT_DIFF,SPF_HELO_PASS,SPF_PASS autolearn=no autolearn_force=no
	version=3.4.2
X-Spam-Report: 
	* -0.0 SPF_PASS SPF: sender matches SPF record
	* -0.0 SPF_HELO_PASS SPF: HELO matches SPF record
	*  0.7 MIME_HTML_ONLY BODY: Message only has text/html MIME parts
	*  0.7 MPART_ALT_DIFF BODY: HTML and text parts are different
	*  0.0 HTML_MESSAGE BODY: HTML included in message
	*  1.8 HTML_IMAGE_ONLY_08 BODY: HTML: images with 400-800 bytes of
	*      words
	*  2.0 BASE64_LENGTH_79_INF BODY: base64 encoded email part uses line
	*      length greater than 79 characters
	*  0.0 MIME_HTML_ONLY_MULTI Multipart message only has text/html MIME
	*      parts

Considering the context, it screams incompetence.

No-fuss setting user-specific locales (for instance for XFCE with ligthdm or slim)

End of 2018, you’d think, by now, that locales setup should not be a concern. But, still, in the case of user-specific configuration, mismatching the system locale (granted, that must not so be so common), I got various odd results. Like lightdm not setting anything no matter what you select on the login window. Or, worse, half-assed setup, with LANG being set and then unset, of LANGUAGE being not set but still expected by some apps, with a desktop with no option to configure it like XFCE.

After a few tests, turns out that user .xsessionrc works perfecly, independantly from desktop environment or desktop login manager:

echo "export LANG=fr_FR.UTF-8
export LANGUAGE=fr_FR.UTF-8" >> ~/.xsessionrc

with french (fr_FR) selected here.

Typing SSH passphrase(s) only once per session

Here’s a very simple way to type SSH passphrases only once. This simple function, to be added in your ~/.bashrc, will make sure that ssh-agent will always be called before ssh, once per session, so you do not have to type your ssh passphrase more than once:

function sshwithauthsock {
 if [ ! -S ~/.ssh/ssh_auth_sock ]; then
   eval `ssh-agent`
   ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
 fi
 export SSH_AUTH_SOCK=~/.ssh/ssh_auth_sock
 ssh-add -l > /dev/null || ssh-add
 "$@" 
}

alias ssh='sshwithauthsock ssh'
alias scp='sshwithauthsock scp'

Check for possibly updated version directly in my repository.

 

Checking Western Digital Green load cycle per hour / Intellipark issues

I got a few Western Digital Green hard disk. I’ve read they have been rebranded blue now. It was supposed to be hard disk with long consumption, possibly lower speed due to low rotation. Low rotation, you would assume: longer-life span, since usually, mechanical devices lives longer when running slower.

But when you do realize that these Green have the shortest warranty possible (2 years against 3 or 5 for others), you wonder.

And then, when you have a hard disk that starts to fails, you learn stuff like these Western Digital Green having a 8 seconds timeout to park the drive (yeah, like in old DOS era, when you where using park before shutting off your computer). I assume it is to save energy but it takes no genious to evaluate the result if your system writes every 10 seconds, which is not un unlikely scenario.

I am not talking theory, I do have a failing Western Digital Green 2Tb (WDC WD20EZRX-22D8PB0) that is just 2 years and a few months.

With different cables and different mainboards, power supply units, etc, it sprouts:

 [ 3996.054577] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 3996.054580] ata7.00: irq_stat 0x40000001
[ 3996.054585] ata7.00: failed command: READ DMA EXT
[ 3996.054595] ata7.00: cmd 25/00:08:00:88:e0/00:00:e8:00:00/e0 tag 17 dma 4096 in
 res 51/04:08:00:88:e0/00:00:e8:00:00/e0 Emask 0x1 (device error)
[ 3996.054598] ata7.00: status: { DRDY ERR }
[ 3996.054600] ata7.00: error: { ABRT }
[ 3996.055191] ata7.00: failed to enable AA (error_mask=0x1)
[ 3996.056015] ata7.00: failed to enable AA (error_mask=0x1)

So what about this wdidle3 timeout and resulting Load_Cycle?

# hdparm -J /dev/sdd
/dev/sdd:
 wdidle3 = 8.0 secs

# smartctl /dev/sdd -a | grep Load_Cycle
193 Load_Cycle_Count 0x0032 116 116 000 Old_age Always - 253474

253474 for recent hard disk? I’ve read the life expectancy is usually between 300000 and 1000000 load cycle count. But as reference, I’ll check my other hard drives on the workstation I put the disk to test:

# DISK="a b c d e"
# TMP=`mktemp` && for disk in $DISK; do smartctl -xa /dev/sd$disk > $TMP ; grep "Device Model" $TMP ; hdparm -J /dev/sd$disk 2>/dev/null| grep wdidle ; grep Power_On_Hours $TMP ; grep Load_Cycle_Count $TMP ; Count=`grep Load_Cycle_Count $TMP | grep -oE '[^ ]+$'` ; Hours=`grep Power_On_Hour $TMP | sed "s/\s[(][^)]*[)]//g" | grep -oE '[^ ]+$'` ; if [ x$Hours != x ]; then echo `echo print $Count/$Hours. | perl` load cycles per hour ; echo ; fi ; done
Device Model: WDC WD5000AZRX-00A8LB0
 wdidle3 = 128 ??
 9 Power_On_Hours -O--CK 075 075 000 - 18587
193 Load_Cycle_Count -O--CK 121 121 000 - 239519
12.8863721956206 load cycles per hour

Device Model: ST2000DX002-2DV164
 wdidle3 = 1 ??
 9 Power_On_Hours -O--CK 094 094 000 - 5385
193 Load_Cycle_Count -O--CK 099 099 000 - 3617
0.671680594243268 load cycles per hour

Device Model: WDC WD20EZRX-22D8PB0
 wdidle3 = 8.0 secs
 9 Power_On_Hours -O--CK 090 090 000 - 7672
193 Load_Cycle_Count -O--CK 116 116 000 - 253490
33.0409280500521 load cycles per hour

Device Model: WDC WD2001FASS-00W2B0
 wdidle3 = 128 ??
 9 Power_On_Hours -O--CK 038 038 000 - 45726
193 Load_Cycle_Count -O--CK 073 073 000 - 382183
8.35811135896427 load cycles per hour

Depends obviously of the purpose of the hard disk. Still, the affected Western Digital Green, with its 33  load cycles per hour stands out, in the wrong sense. At this rate, the first disk would reach 613000 load cycles instead of 239519 by now, likely a goner already.  And the last one would be around 1509000, a goner definitely too!

Then on a  server:

Device Model: ST4000DM005-2DP166
 wdidle3 = 1 ??
 9 Power_On_Hours -O--CK 090 090 000 - 9091 (43 85 0)
193 Load_Cycle_Count -O--CK 100 100 000 - 400
0.0439995600044 load cycles per hour

Device Model: WDC WD40EFRX-68WT0N0
 wdidle3 = 300 secs (or 13.8 secs for older drives)
 9 Power_On_Hours -O--CK 062 062 000 - 28038
193 Load_Cycle_Count -O--CK 200 200 000 - 653
0.0232898209572723 load cycles per hour

We have read too an infamous Western Digital, but not a Green, so the widle3 is much less extreme.

What about on a laptop (Lenovo 20017 IdeaPad Y550 ) ?

Device Model: WDC WD5000BEVT-22ZAT0
 wdidle3 = 8.0 secs
 9 Power_On_Hours -O--CK 041 041 000 - 43353
193 Load_Cycle_Count -O--CK 001 001 000 - 889112
20.508661453648 load cycles per hour

Gasp! But wait, isn’t it a Western Digital Blue – so, Green rebranded?

Questioned about this kind of issue, it seems that Western Digital claims “we’ve not seen the drives fail over high load/unload counts”. It may be right, maybe the problem is something else. But that the only odd thing noticeable to me. And I am apparently not the only one questioning Western Digital statements, if not challenging them.

As you can see, I got a few disk from this brand and must say even the Western Digital knowledge base entry titled “The Load/Unload counter for S.M.A.R.T Attribute 193 continues to increase under some distributions of the Linux Operating system and some Windows applications”  is not what I expect as customer. They do not question their 8 seconds timer, which is questionable – I do not care about their very own opinion about how often a system should or should not write to a disk.  They claim the issue “artificially increases the number of load-unload cycles”. There is nothing artificial, it simply does increase. They say it is no problem because they are “within design margins (drive has been validated to 1 million load/unload cycles without issue)”. But my test shows that it is out of proportions in any case, for no real added benefits.

I have to admit the issue is not new. But if you do not especially pay attention to hard drives in general, why would you be aware of it.

What to make out of this?

First, on the laptop, I’ll disable this widle3:

# apt install idle3-tools 
# idle3ctl -g /dev/sda
Idle3 timer set to 80 (0x50)
# idle3ctl -d /dev/sda

Myself, I think I’ll stay clear of Western Digital all together.

 

 

 

 

Booting two Devuan GNU/Linux installed on encrypted partitions on a single disk

Followup of previous article (Single passphrase to boot Devuan GNU/Linux with multiple encrypted partitions), I found out that if you have two clone system, both on encrypted partitions, on the same hard disk, grub/os-prober as of today fails to automatically configure boot for the clone.

It the concept of having such clone system odd? Not really if you think of laptop that you use for two completely unrelated activities (work and out of the work?), that you do not want to mix at all.

I spent quite a time trying to understand why the clone system was ignored by os-prober and all, even though the partition it was on was mounted.

In the end, I decided it was easier to actually clone the config built for the running system, adjusting the UUID of partitions than to look further.

Here is my /etc/grub.d/11_linux_cryptoclone wrapper for /etc/grub.d/10_linux:

#! /bin/bash
set -e

# require GRUB_LINUX_CLONE_MAPPER_NAME to be set
# for instance to XY if the relevant fs is /dev/mapper/XY
# along with relevant grub parameters
#GRUB_ENABLE_CRYPTODISK=y
#GRUB_PRELOAD_MODULES="luks cryptodisk lvm"
#GRUB_LINUX_CLONE_MAPPER_NAME=XY

. /etc/default/grub
[ x"$GRUB_LINUX_CLONE_MAPPER_NAME" == x ] && exit

# setup
CLONE=$GRUB_LINUX_CLONE_MAPPER_NAME # only necessary to edit
CLONEUUID=`blkid /dev/mapper/$CLONE -o value -s UUID`
CLONENAME="Devuan GNU/Linux (on $CLONE)"
CLONECRYPTOUUID=`grep $CLONE /etc/crypttab | awk '{print $2}' | cut -f 2 -d =`
CLONECRYPTOUUIDGRF=`echo $CLONECRYPTOUUID | tr -d -`

ORIG=`df / --output=source | tail -1 | tr -d /dev/mapper`
ORIGUUID=`blkid /dev/mapper/$ORIG -o value -s UUID`
ORIGNAME="Devuan GNU/Linux"
ORIGCRYPTOUUID=`grep $ORIG /etc/crypttab | awk '{print $2}' | cut -f 2 -d =`
ORIGCRYPTOUUIDGRF=`echo $ORIGCRYPTOUUID | tr -d -`

# produce arranged conffile
>&2 echo "$ORIG -> $CLONE:"
`dirname "$0"`/10_linux | sed s/$ORIGCRYPTOUUID/$CLONECRYPTOUUID/ig | sed s/$ORIGCRYPTOUUIDGRF/$CLONECRYPTOUUIDGRF/ig | sed s/$ORIGUUID/$CLONEUUID/ig | sed s@"$ORIGNAME"@"$CLONENAME"@ig
>&2 echo " $ORIGCRYPTOUUID -> $CLONECRYPTOUUID"
>&2 echo " $ORIGUUID -> $CLONEUUID"

# make sure there is proper kernel and initrd installed
MCLONE=0
CLONEDIR=`grep /dev/mapper/$CLONE /etc/mtab | awk '{print $2}'`
if [ "x$CLONEDIR" == "x" ]; then
 MCLONE=1
 mount /dev/mapper/$CLONE
 CLONEDIR=`grep /dev/mapper/$CLONE /etc/mtab | awk '{print $2}'`
fi
for file in /boot/config-* /boot/init* /boot/vmlinuz-* /boot/System.map-*; do
 [ ! -e "$CLONEDIR/$file" ] && >&2 echo " $CLONE $file missing!"
done
>&2 echo " (remember $CLONE needs properly built initramfs)" 
if [ $MCLONE == 1 ]; then umount /dev/mapper/$CLONE ; fi

# EOF

As said it requires you to add GRUB_LINUX_CLONE_MAPPER_NAME=XY in /etc/default/grub, XY  being the /dev/mapper/XY of the clone system.

It expect the clone system to be similarly set up: it needs to have proper initramfs for the same kernel.

It also expect this clone system to be accessible and set in /etc/crypttab et /etc/fstab, since it needs to be able to find clone UUIDs which should not come as a surprise because if it would have to be if os-prober was to find it anyway.

Once done, you can simply run

update-grub

Single passphrase to boot Devuan GNU/Linux with multiple encrypted partitions

These days, considering the amount of data are stored on an average computer and how easy is it to get access to it once you get physical access, running such computer without any form of encryption seem unsound. Especially since it is reasonably easy to set up a en encrypted system and does not seems to imply much overhead.

When you have an old setup you are fine with, using numerous partitions or systems, it can be inconvenient, though. For instance  if you have to type a long specific passphrase 5 times when booting your computer.

There are a few things I found useful to make my life easier. Obviously, any shortcut security wise means less security. It is help to you to decide whether the risk is worth it or not depending on what kind of data you carry, what kind of attackers you expect, etc. This is part 1.

Single passphrase to boot GNU/Linux with multiple encrypted partitions

One obviously approach to type a single passphrase to boot a system is to have the boot loader files on a regular partition and the rest on a single encrypted partition. In GNU/Linux case, you would have /boot on a specific non-encrypted partition. Fact is anyone with access to your computer can easily replace your kernel or initramfs with a malicious one and you would not notice.

So I think non-encrypted /boot is as much of the table as would be a non-encrypted swap partition.

So for the boot manager grub to load, /boot need to be readable: the passphrase will be required here. The idea is that from this moment on, a keyfile will be used instead of passphrase to load any other partition.

I guess there is not much point to describe in detail the crypt setup itself (I followed the many guides out there). For each partition you want:

# 1. you create the partition with parted/fdisk

# 2. you format it as encrypted
cryptsetup luksFormat /dev/sdX1

# 2b. you record it in crypttab
# <target name> <source device> <key file> <options>
echo "Name1 UUID=`blkid -s UUID -o value /dev/sdX1` /boot/k/ka luks,tries=3" >> /etc/crypttab

# 3. you open the encrypt-formatted partition
cryptsetup luksOpen /dev/sdX1 Name1

# 4. you format the resulting /dev/mapper... to a regular filesystem
mkfs.ext4 /dev/mapper/Name1 -L Name1

# 4b. you record it in fstab (adjusting the mount point!)
# <file system> <mount point> <type> <options> <dump> <pass> 
echo "/dev/mapper/Name1 / ext4 errors=remount-ro 0 1" >> /etc/fstab

# you are set, you can mount the partition, 
#  and install the system/copy the system there

The swap  require specific treatment. Provided you know one with partition is the current  unencrypted swap is (here sdX?), this is enough:

# update crypttab
echo "SW `find -L /dev/disk -samefile /dev/sdX? | grep by-id | tail -1` /dev/urandom swap" >> /etc/crypttab

# update fstab
echo "/dev/mapper/SW  none   swap sw 0 0" >> /etc/fstab

Noticed the /boot/k/ka? That’s the unlocking key. You can use whatever other filename, just be consistent.

# generate some
dd if=/dev/urandom of=/boot/k/ka bs=1024 count=4
chmod 400 /boot/k/ka

# and obviously, add it to any luks formatted partition:
for part in `blkid | grep crypto_LUKS | awk {'print $1 '} | tr -d :$`; do cryptsetup -v luksAddKey $part /boot/k/ka; done

Then, you need a proper initramfs:

# dracut works almost out of the box
apt install dracut

# set up a few things
echo 'omit_dracutmodules+="systemd systemd-initrd dracut-system"' > /etc/dracut.conf.d/00-systemd.conf
echo 'add_dracutmodules+="crypt lvm"
install_items+="/sbin/e2fsck /sbin/cryptsetup /boot/k/ka"' > /etc/dracut.conf.d/99-luks.conf

# (re)build ramfs
dracut --force

# make sure there are no old initrd leftovers, that would confuse grub
rm /boot/initrd* -f

 

Then, you need to edit grub (version 2!) config:

# first obtain the UUID of crypted partition (not the /dev/mapper/... one) 
# that hold the /boot partition. 
# (it was Name1 earlier but obviously it depends to the real name you gave)
grep Name1 /etc/crypttab

# now edit /etc/default, with XXXXXXXXXXXX being the UUID value
# you just found.
GRUB_CMDLINE_LINUX="rd.luks.key=/boot/k/ka:UUID=XXXXXXXXXXXX"
GRUB_ENABLE_CRYPTODISK=y
GRUB_PRELOAD_MODULES="luks cryptodisk lvm"

# and now update-grub (install grub if not done yet)
update-grub

It took me a while to find the proper rd.luks.key value, no docs I read were clear about it. Many give the impression that putting rd.luks.key=/keyfile or rd.luks.key=/keyfile:/ would be enough since the key is actually on the same partition as grub.cfg. But no.

That is all. Rebooting now, you should be asked for the passphrase before getting the grub menu. And then boot process should be uninterrupted.