Apt-get this /scratch (apt/steam cache, exim greylist, owncloud, tumblr daily post, etc)

 

I conveniently use, since more than 10 years, one debian package to distribute over different computers some scripts, notably all the ones published here and on github (except rawdog and pxe setup – but that could change). While it’s practical for me, it’s not for anyone else since the package content is overly random.

Someone using the apt/steam cache as described on this blog can easily end up with out-of-date scripts – unless he fetch updates from git directly.

I decided to split this big package into multiples and improve the conffiles so they are more or less zeroconf.

The result is:

  • -keyring: Keyring used to signed packages along with repository set-up
  • -utils-cache-apt: APT downloads cache setup
  • -utils-cache-steam: Steam downloads cache setup
  • -utils-cloud: Owncloud setup and apps
  • -utils-exim: Exim greylist, bogofilter and spamassassin setup
  • -utils-torrent: Manage transmission via NFS/Samba
  • -utils-tumblr: Automated image post on tumblr
  • -utils-webmail: Roundcube setup and plugins
  • and some others packages that these depends on.

 

The easiest way to get anything is to first install the keyring package and then directly apt-get:

  	# wget http://apt.attique.org/stalag13-keyring.deb
	# dpkg -i stalag13-keyring.deb
	# apt-get update
	# apt-get install stalag13-...

 

Advertisements

Caching debian/etc (apt) repositories on your local server with nginx and dsniff

It’s quite easy to set up a debian mirror. But having a mirror on a local server is rather overkill in a scenario where you simply regularly have say 3 boxes running some Debian testing amd64, 1 box running the same on arch i686 and 2 other boxes on Ubuntu. Well, it’s more caching than mirroring that you’ll want, as transparently (with no client side setup) as possible.

And that’s overly easy to do with nginx, similarly to Steam depot caching. No, really, just do the same!

So, assuming nginx and dnsspoof are already up and running -if not, really follow the link about steam cache- you want to:

– create the apt folders…

mkdir -p /srv/www/apt/debian /srv/www/apt/debian-security /srv/www/apt/ubuntu
chown www-data:www-data -R /srv/www/apt
cd /srv/www
ln -s /srv/www/apt/debian .
ln -s /srv/www/apt/debian-security .
ln -s /srv/www/apt/ubuntu .

– update nginx by adding a /etc/nginx/sites-available/apt (and a symlink in /etc/nginx/sites-enabled/) with:

# apt spoof/proxy
server  {
  listen 80;
  server_name ftp.fr.debian.org security.debian.org fr.archive.ubuntu.com security.ubuntu.com;

  access_log /var/log/nginx/apt.access.log;
  error_log /var/log/nginx/apt.error.log;

  root /srv/www/;
  resolver 127.0.0.1;

  allow 10.0.0.0/24;
  allow 127.0.0.1;
  deny all;

  location /debian/pool/ {
    try_files $uri @mirror;
  }

  location /debian-security/pool/ {
    try_files $uri @mirror;
  }

  location /ubuntu/pool/ {
    try_files $uri @mirror;
  }

  location / {
    proxy_next_upstream error timeout http_404;
    proxy_pass http://$host$request_uri;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    add_header X-Mirror-Upstream-Status $upstream_status;
    add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
    add_header X-Mirror-Status $upstream_cache_status;
  }

  location @mirror {
    access_log /var/log/nginx/apt.remote.log;
    proxy_store on;
    proxy_store_access user:rw group:rw all:r;
    proxy_next_upstream error timeout http_404;
    proxy_pass http://$host$request_uri;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
    add_header X-Mirror-Upstream-Status $upstream_status;
    add_header X-Mirror-Upstream-Response-Time $upstream_response_time;
    add_header X-Mirror-Status $upstream_cache_status;
   }
}

– add the new domains to be spoofed in /etc/dnsspoof.conf:

10.0.0.1	ftp.fr.debian.org
10.0.0.1     security.debian.org
10.0.0.1	fr.archive.ubuntu.com
10.0.0.1     security.ubuntu.com

Then you have to restart both nginx and dnsspoof. Obviously, the domains have to match the sources you have configured in /etc/apt/sources.list[.d] – should be the nearest hosts to your location.

And since you do not want to keep a complete archive, you need to add a cronjob to remove outdated files, like this /etc/cron.weekly/apt-cache:

#!/bin/sh
# cleanup apt mirrors:

# remove any file that has not been accessed in the last 30 days 
find /srv/www/apt -type f -atime +30 -print0 | xargs -0 --no-run-if-empty rm

# remove any empty dir (except the main ones)
find /srv/www/apt -mindepth 2 -type d -empty -print0 | xargs -0  --no-run-if-empty rm -r

That’s it. You may notice that it actually only caches the pools, not the packages lists. It could work with the whole repositories, not just the pools, but I noticed someĀ  failures with packages lists checks from time to time and it’s not worth the trouble.

Keeping the dpkg installed software database clean

The system on my workstation was installed in 2008, December. Actually, I installed Debian AMD64 version over an i386 version on the same box, which was installed around 2003.

Debian ships tools that makes it easy to keep a clean system. For instance, debfoster allows to easily get rid of all no-longer necessary libraries and al: you just have to select the important pieces and it will remove any software that is not required by one of these. And apt-get, nowadays, just like deborphan used to, even warns you when some software is no longer required and provides you with the autoremove command line argument that do the job automatically.

(debfoster is, supposedly, deprecated, like apt-get is in favor of aptitude. Well, I like debfoster)

That being said, if I run dpkg --list | grep ^r | nl | tail -n 1 on this box, after only one year, I get 617 lines about removed software I do not care about. Mostly, they were kept in the dpkg database because I (or me using the system) modified their conffiles. The following will clean this: for package in `dpkg --list | grep ^r | cut -f 3 -d " "`; do dpkg --purge $package; done && debfoster

Being warned of pending packages upgrades with apt-warn

I started using GNU/Linux with RedHat 5.2. It came with plenty of packages (GNOME 0.20, Linux 2.0.36, etc) and I was quite happy to deal with RPM (RedHat Package Manager, hum) telling me which package is required to install another one, which package contains which files. You simply had to go to RPMFind.net to get missing packages. If no package was available, you could write a clean RPM spec to build one or use checkinstall to build RPMs on the fly when doing make install. It was more than ten years ago, still, nowadays Microsoft Windows XP (sorry, I never used Vista/7) have no clean packaging system that I know of; you have a clumsy list of installed software (InstallShield, whatever it means), no clear idea of dependancies, you can remove pieces of software required by other still installed software and there are plenty of installed pieces of software that you have no way of clearly listing.

At that time, I had a Pentium II 350 MHz and a Pentium 200 MMX as workstations and a Pentium 133 MHz as home server. I, soon enough, had the idea to write a script to produce a list of installed packages readable over intranet and so I published a BASH-based script to output an HTML view of a RPM database called pdbv, standing for Package DataBase View, the first version 1.0.0 being released in June 2002. On the Gna! project page, when listing pros and cons of pdbv, the first pro that came up was “it does not require lucid/gtk+/qt or other big libs”: nowadays, GTK+ and Qt probably no longer strike the mind of anyone as “big (bloated) libraries” and I assume Lucid is no longer even installed on most GNU/Linux systems. Later, I rewrote pdbv in Perl which made if was faster and lighter. Here are demos of pdbv: pdbv 1.x with French locale, pdbv 2.x.

As you can see browing pdbv’s demos, it obviously supports also dpkg (Debian Package, duh). I gradually switched over Debian GNU/Linux for two reasons: apt-get and the branching stable/testing/unstable. Apt-get was the end of wasting time on RPMFind. Debian stable offers astonishing stability for servers while testing/unstable provides brand new desktop software in a timely fashion.

Nowadays, I spend less time dealing with computers and I no longer rely much on pdbv. Due to lack of support (I guess I’m to blame; but KPackage or Synaptic are surely more useful to endusers anyway), it will be removed from Debian at its next stable release (it is still in Debian lenny but no longer in testing). I no longer care much about which software is installed, I use debfoster to keep clean my systems (I know, just like apt-get, debfoster is deprecated in favor of aptitude, but I cannot help using it instead).

However I’d like to know which upgrades are pending. For this reason (and I’m quite sure I’m reinventing something that already exists, but I failed to find it and I wanted it my way), I wrote a small script called apt-warn that will run apt-get update and then warn you of pending updates (only if it has not warned you already about them). It requires Apt::Pkg. It is supposed to be installed a cronjob in /etc/cron.daily. Running on my workstation this morning, it outputs:


Follows 4 newly updated package(s) that you could upgrade on bender:
hicolor-icon-theme (0.11-1 -> 0.12-1)
sudo (1.7.2p7-1 -> 1.7.4p4-2)
xserver-common (2:1.7.7-4 -> 2:1.7.7-6)
xserver-xorg-core (2:1.7.7-4 -> 2:1.7.7-6)

Follows 5 recently updated package(s) that you also could upgrade:
autopoint (0.18.1.1-1 -> 0.18.1.1-2)
gettext (0.18.1.1-1 -> 0.18.1.1-2)
gettext-base (0.18.1.1-1 -> 0.18.1.1-2)
login (1:4.1.4.2-1 -> 1:4.1.4.2+svn3283-1)
passwd (1:4.1.4.2-1 -> 1:4.1.4.2+svn3283-1)

Autopoint, gettext, login and passwd pending upgrades were already warned about yesterday. A second run will return no output since there is no other available upgrade not already warned about.