Generating static galleries with nanogallery2

As wrote earlier, I now use nanogallery2 to share pictures due to nextcloud removal of the earlier gallery app and the fact that the replacement app called Photos does not match my needs.

It is copied directly from the cloud LXC container, with rsync and ssh, to another server that only serves static files via HTTPS – with access restricted through usual basic authentication. It means that I am not giving any access to the cloud infrastructure to the HTTPS server.

nanogallery2 have associated “providers” to easily set up a gallery (directly from a local directory or distant storage). But that implies on-the-fly index and thumbnails generation by a PHP CGI server while I prefer the server to be serving only static files.

So I made a small perl script called dir2nanog.pl to build a single index and thumbnails, using subdirectories as distinct galleries:

#!/usr/bin/perl
#
# Copyright (c) 2020 Mathieu Roy <yeupou--gnu.org>
#      https://yeupou.wordpress.com
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 59 Temple Place,d Suite 330, Boston, MA  02111-1307
#   USA


use strict "vars";
use File::Find;
use File::Basename;
use Image::ExifTool;
use Image::Magick;
use Fcntl ':flock';
use CGI qw(:standard escapeHTML);

my $debug = 1;
my $path = $ARGV[0];
my $topdirstring = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ";   # hopefully no such directory exists
my @topimages;
my %subdirsimages;
my %comment;
my %gps;
my %model;
my %focalength;
my %flash;
my %exposure;
my %iso;
my $images_types = "png|gif|jpg|jpeg";
my @mandatory = ("https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.core.min.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/jquery.nanogallery2.min.js",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/nanogallery2.woff.min.css",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/nanogallery2.min.css"); 

my @mandatoryfont = ("https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/font/ngy2_icon_font.woff",
		 "https://raw.githubusercontent.com/nanostudio-org/nanogallery2/master/dist/css/font/ngy2_icon_font.woff2");



# silently forbid concurrent runs
# (http://perl.plover.com/yak/flock/samples/slide006.html)
open(LOCK, "< $0") or die "Failed to ask lock. Exit";
flock(LOCK, LOCK_EX | LOCK_NB) or exit;

### FUNCTIONS
sub thumbnail_dimensions {
        my ($x, $y, $ratio, $height, $width);
        ($x, $y) = @_;
        $ratio = $x / $y;	
	$width = 200;
	$height = int($width / $ratio);
        return $width, $height;
}

sub thumbnail_name {
    my ($name,$path) = fileparse($_[0]);
    return "$path.thumbnail.$name";
}

sub wanted {
    # skip directories and non-images in general
    next if -d $_;
    next unless lc($_) =~ /\.($images_types)$/i;
    # skip thumbnails
    next if $_ =~ /^\.thumbnail\./i;

    # create thumbnail if missing
    unless (-e thumbnail_name($File::Find::name)) {
	my ($image, $error, $x, $y, $size, $format, $width, $height);
	
        # https://www.perlmonks.org/?node_id=209235
        $image = Image::Magick->new;
        $error = $image->Read($File::Find::name);
	if (!$error) {
	    ($x, $y, $size, $format) = $image->Ping($File::Find::name);
	    ($width, $height) = thumbnail_dimensions($x, $y);
	    $image->Scale(width=>$width, height=>$height);
	    $error = $image->Write(thumbnail_name($File::Find::name));
        }
    }
    
    # identify URL, path removed 
    my $url = substr($File::Find::name, length($path)+1, length($File::Find::name));    

    # name will be based on file creation (using comment would require utf8 checks)
    # according to exif data, not real filesystem info
    my $exifTool = new Image::ExifTool;
    my %exifTool_options = (DateFormat => '%x');
    my $info = $exifTool->ImageInfo($File::Find::name, ("CreateDate","GPSLatitude","GPSLongitude","GPSDateTime", "Make", "Model", "Software", "FocalLength", "Flash", "Exposure", "ISO"), \%exifTool_options);
    $comment{$url} = $info->{"CreateDate"};
    $comment{$url} = $info->{"GPSDateTime"} if $info->{"GPSDateTime"};
    
    if ($info->{"GPSLatitude"} and $info->{"GPSLongitude"}) {
	$gps{$url} = $info->{"GPSLatitude"}." ".$info->{"GPSLongitude"};
	$gps{$url} =~ s/\sdeg/°/g;
    }

    $model{$url} = $info->{"Make"}." " if $info->{"Make"};
    $model{$url} .= $info->{"Model"}." " if $info->{"Model"};
    $model{$url} .= "(".$info->{"Software"}.")" if $info->{"Software"};

    $focalength{$url} = $info->{"FocalLength"} if $info->{"FocalLength"};
    $flash{$url} = $info->{"Flash"} if $info->{"Flash"};
    $exposure{$url} = $info->{"Exposure"} if $info->{"Exposure"};
    $iso{$url} = $info->{"ISO"} if $info->{"ISO"};

    
    # top directory image
    push(@{$subdirsimages{$topdirstring}}, $url) and next if ($File::Find::dir eq $path);    

    # other images
    # identify subdir by removing the path and keeping only the resulting top directory
    my ($subdir,) = split("/", substr($File::Find::dir, length($path)+1, length($File::Find::dir)));
    push(@{$subdirsimages{$subdir}}, $url);
        
    return;
}


### RUN

chdir($path) or die "Unable to enter $path, exiting";

# list images in top dir and subdirectories
find(\&wanted, $path);

# create the output
# first check if mandatory files are there
foreach my $file (@mandatory) {
    next if -e "$path/".basename($file);
    system("/usr/bin/wget", $file);
}
mkdir("$path/font") unless -e  "$path/font";
chdir("$path/font");
foreach my $file (@mandatoryfont) {
    next if -e "$path/font/".basename($file);
    system("/usr/bin/wget", $file);
}
chdir($path);
    
# create index
open(INDEX, "> $path/index.html");
print INDEX '<!DOCTYPE html
	PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta name="generator" content="dir2nanog.pl" />
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <!-- jQuery -->
    <script type="text/javascript"  src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
    <!-- nanogallery2 -->
    <link href="nanogallery2.min.css" rel="stylesheet" type="text/css" />
    <script  type="text/javascript" src="jquery.nanogallery2.min.js"></script>
    <!-- other -->
    <link type="text/css" rel="stylesheet" href="style.css" />
    <title>'.basename($path).'</title>
  </head>
  <body>
';


my $galleriescount = 0;

for (reverse(sort(keys(%subdirsimages)))) {
    
    # top dir gallery has no specific ID or title
    if ($galleriescount > 0) {	
	print INDEX '    <h1>'.$_."</h1>\n";
	print INDEX '    <div id="nanogallery2'.$_;
    } else {
	print INDEX '    <div id="nanogallery2';
    }

    # add gallery setup
    print INDEX '" data-nanogallery2 = \'{
      "viewerToolbar":   {
        "display":    true,
        "standard":   "label, infoButton",
        "minimized":  "label, infoButton"
      },
      "viewerTools":     {
        "topLeft":    "pageCounter, playPauseButton",
        "topRight":   "downloadButton, fullscreenButton, closeButton"
      },
      "thumbnailWidth": "auto",
      "galleryDisplayMode": "pagination",
      "galleryMaxRows": 3,
      "thumbnailHoverEffect2": "borderLighter"
    }\'>
';

    # add image list
    for (sort(@{$subdirsimages{$_}})) {
	my $extra;
	$extra .= ' data-ngexiflocation="'.escapeHTML($gps{$_}).'"' if $gps{$_};
	$extra .= ' data-ngexifmodel="'.escapeHTML($model{$_}).'"' if $model{$_};
	$extra .= ' data-ngexiffocallength="'.escapeHTML($focalength{$_}).'"' if $focalength{$_};
	$extra .= ' data-ngexifflash="'.escapeHTML($flash{$_}).'"' if $flash{$_};
	$extra .= ' data-ngexifexposure="'.escapeHTML($exposure{$_}).'"' if $exposure{$_};
	$extra .= ' data-ngexifiso="'.escapeHTML($iso{$_}).'"' if $iso{$_};

	# if no comment set, check if if the filename might be YYMMDD_
	$comment{$_} = "$3/$2/20$1" if (!$comment{$_} and /(\d{2})(\d{2})(\d{2})_[^\\]*$/);  	
	
	print INDEX '      <a href="'.$_.'" data-ngThumb="'.thumbnail_name($_).'"'.$extra.'>'.escapeHTML($comment{$_})."</a>\n";
    }
print INDEX '
	</div>
';
    $galleriescount++;
}

# finish index	
	print INDEX
	'</body>
</html>';

close(INDEX);

# EOF 

To run it:

chmod +x /usr/local/bin/dir2nanog.pl
/usr/local/bin/dir2nanog.pl /var/www/html/pictures/

I have the following crontab for the dedicated user:

*/15 * * * *     export LC_ALL=fr_FR.UTF-8 && /usr/local/bin/dir2nanog.pl /var/www/html/pictures/thistopic

Dealing with Nextcloud broken oc_flow_operations table after 19 upgrade

I finally upgraded my Nextcloud server, using nanogallery2 instead of relying on the new brand Photos app, since it does not provide a simple way to share a gallery and is clearly not about to be resolved (the developers that in first place considered there was no problem to fix finally said that people were too harsh try to convince them otherwise so they finally lost interest in fixing it). Well, in any case, using static files for this purpose is much more efficient.

During the upgrade, I hit issue with the oc_flow_operations table that is mentioned in numerous reports (the whole first page of google search).

I first fixed it by, as suggested by some users, adding a missing column. That allowed to finish the upgrade but something was still off, with the logfile growing to a few MB every minutes.

Finally, recreating the whole table fixed it for me:

cat /etc/mysql/debian.cnf | grep password
mysql -udebian-sys-maint -p

connect nextcloud;
DROP table oc_flow_operations ;
CREATE table oc_flow_operations (id   int(11) NOT NULL  AUTO_INCREMENT PRIMARY KEY, class  varchar(256), name     varchar(256) NOT NULL, checks   longtext, operation longtext,entity  varchar(256),events longtext);

su www-data -s /bin/bash
php occ maintenance:repair

Fetching mails from gmail.com with lieer/notmuch instead of fetchmail

Google is gradually making traditional IMAPS access to gmail.com impossible, in it’s usual opaque way. It is claimed to be a security plus, though, if your data is already on gmail.com, it means that you are already accepting that it can and is spied on, so whether the extra fuss is justified is not so obvious.

Nonetheless, I have a few secondary old boxes on gmail, that I’d like to still extract mails from, not to miss any, without bothering connecting to with a web browser. But the usual fetchmail setup is no longer reliable.

The easiest alternative is to use lieer, notmuch and procmail together.

apt install links notmuch lieer procmail

# set up as regular user
su enduser

boxname=mygoogleuser
mkdir -p ~/mail/$boxname
notmuch
notmuch new
cd ~/mail/$boxname
gmi init $boxname@gmail.com

# at this moment, you'll get an URL to connect to gmail.com 
# with a web browser
# 
# if it started links instead, exit cleanly (no control-C)
# to get the URL
#
# that will then return an URL to a localhost:8080
# to no effect if you are on a distant server
# in this case, just run, in an another terminal
links "localhost:8080/..."
# or  run `gmi --noauth_local_webserver auth`  and use the URL
# on another browser on another computer

The setup should be ok. You can check and toy a bit with the setup:

gmi sync
notmuch new
notmuch search tag:unread
notmuch show --format=mbox thread:000000000000009c

Then you should mark all previous messages as read:

for thread in `notmuch search --output=threads tag:unread`; do echo "$thread" && notmuch tag -unread "$thread" ; done
gmi push

Finally, we set up the mail forward (in my case, the fetch action is done in a dedicated container, so it is forwarded to ‘user’ by SMTP to the destination host, but procmail allows any setup) and fetchmail script: each new unread mail is forwarded and then marked as read:

echo ':0
! user' > ~/.procmail

echo '#!/bin/bash

BOXES=""

# no concurrent run
if pidof -o %PPID -x "fetchmail.sh">/dev/null; then
        exit 1
fi

for box in $BOXES; do
    cd ~/mail/$box/
    # get data
    gmi pull --quiet
    notmuch new --quiet >/dev/null
    for thread in `notmuch search --output=threads tag:unread`; do
	# send unread through procmail
	notmuch show --format=mbox "$thread" | procmail
	# mark as read
	notmuch tag -unread "$thread"
    done
    # send data
    gmi push --quiet
    cd
done

# EOF' > ~/fetchmail.sh
chmod +x ~/fetchmail.sh

Then it is enough to call the script (with BOXES=”boxname” properly set) and to include it in cronjob, like using crontab -e`

notmuch does not remove mails.