Accessing video/audio from a computer with a Freebox (using UPnP)

Nowadays, you would think having a network device like a TV box accessing data from your unix-based system easy going. I mean, we have NFS, Samba all-right, how hard could that be to devise an interface to access at least one of the many network shares, provided a user/password or based on it’s IP?

But no, it won’t work that way. It’s supposedly too complex, so instead, people promotes zeroconf and such, stuff supposed to work out-of-the-box that actually may not work at all. For instance, to access movies/music from your computer with your Freebox HD (TV box) v5, and I assume it’s the same with many similar boxes from others ISP, you can forget about using NFS/SMB/http or whatever protocol you already had working or think easy to put in motion. No, you’ll have to use UPnP, standing for Universal Plug and Play, words that truly often refers to Plug and Pray instead.

From the computer…

So, let’s go getting our hands dirty. The setup I’m working with is quite simple: a Freebox HD v5, a single computer with a single user having some videos, some with subtitles (mostly .srt), and audio files. It’s basic but it took me a while to figure it out.

I did test plenty of UPnP servers. I tried Mediatomb but it did not work – plus the gothic interface seemed weird. I tried XBMC and it worked nicely but only to show empty folders, and no obvious way to have it up without the CPU consuming interface. Then I installed MythTV and I did not even understood how it is supposed to work in regard of UPnP.

So I tried minidlna, the lightweight one I avoided from the start because it’s not known to properly support subtitles files. And, tada!, it actually works almost out-of-the-box. Yeah, almost. That’s the funny thing, even if you claim to aim zeroconf, when it comes to share files, at some point you’re still in need to list what you actually wanna share. Whatever. So I did apt-got minidlna. Then I edited /etc/default/minidlna:

START_DAEMON="yes"
USER="thisguy"
GROUP="thisguy"

As there’s only one legit user on this box, I wanted the daemon to be able to access his files with no specific consideration for privileges/ownership. I implied doing then:

chown thisguy.thisguy /var/lib/minidlna -Rv

Then I modified /etc/minidlna.conf as follows:

media_dir=V,/home/thisguy/Films
media_dir=A,/home/thisguy/Musique
media_dir=P,/home/thisguy/Photos

log_level=info
network_interface=eth0
friendly_name=thisserver
inotify=yes

… to the box

I restarted the daemon (and made sure it’s included in /etc/rc2.d). And that’s all (I modified also the firewall setup but I’m not sure that’s relevant considering that UPnP implies that it advertises itself to other devices when it’s up and not the other way around – so firewall is an issue only if you have one that blocks connections from the inside to the outside).

By that’s all, I meant: it was enough to get access to movies on the Freebox HD for the first time; but not with the subtitles.

I googled around: the minidlna version I had was supposed to properly give access to .srt along with videos. The Freebox HD itself supports .srt files with the same name of the video, when you access video over an USB device. But apparently plenty of implementations of UPnP have no consideration for subtitle files and the Freebox’s one is probably one such. So having separate .srt or .sub or whatever is a no-go.

Then, I gave a try to Matroska files (.mkv) despite the fact that I always had bad experience with it. Most notably, I usually implies videos costing tons of CPU time to decode and render and videos players usually fail to properly keep video in sync with audio – yeah, that’s really nasty. But Matroska allows to embed subtitles in the file without touching the video stream, which is neat. So I did that. Long story short, Matroska files, 9 times out of 10, freeze the Freebox HD: and I’m talking about Matroska files that are not bigger than the original .avi that run well on the very same Freebox, and I’m talking about Matroska files that run well on the computer with VLC or mplayer. So that’s a no-go too.

So I ended with the worse solution: altering original files, with mencoder to incrust subtitles within. Yeah, it’s kind of definitive and if you don’t want to spend hours of CPU time to do it, it implies quality loss. But, at least, it works. So here it goes, I wrote the following script to ease the process, assuming that video files along with .srt where originally on an USB device called HERMES and then copied to thisguys home:

#!/bin/bash

DEST=/home/thisguy/Films
ORIG=/media/HERMES

# go thru the list of videos
find $ORIG -name "*.avi" -or -name "*.mpg" -or -name "*.mp4" |
while read video; do
  # find out basename
  basename=${video%.*}
  format=${video##*.}
  endname=`basename "$basename"`-WS.$format

  echo $endname

  # use french subtitles in priority over english
  subtitle=0
  if [ -r "$basename"_en.srt ]; then subtitle="$basename"_en.srt; fi
  if [ -r "$basename"_fr.srt ]; then subtitle="$basename"_fr.srt; fi
  if [ -r "$basename".srt ]; then subtitle="$basename".srt; fi

  # no valid subtitle found at this point? skip the video
  if [ ! -r "$subtitle" ]; then continue; fi

  # now create to relevant directory if missing
  enddir=`dirname "$basename" | sed "s@${ORIG}@${DEST}@g;"`
  if [ ! -d "$enddir" ]; then mkdir -pv "$enddir"; fi

  # proceed only if the file is missing
  if [ -r "$enddir/$endname" ]; then continue; fi

  # if we reach this, go for it
  mencoder "$video" -subpos 92 -sub "$subtitle" -o "$enddir/$endname" -oac copy -ovc lavc

done
# EOF

Yeah. Plug and play my ass.

Advertisements

Automounting NFS shares using if-up.d/if-down.d

I enjoy NFS since many years. But, with laptops, by essence that are not always connected to the same local netwok, it’s quite a pain in the ass.

Editing /etc/fstab each time you connect is not really an option. AutoFS seemed a great idea. It took me a while to get the damn thing running and it failed to work after a reboot. After spending plenty of time googling around, fooling around, I eventually reached the conclusion that I was not able to set it up in a reliable fashion. So I dropped the idea.

Then I had some hopes in regard of the DHCP client. It provides hooks, with plenty of variables useful to determine to which network you are connected to. I gave it a try. Worked well: a script in dhclient-exit-hooks.d to mount NFS shares after the interface is brought up on the LAN, the counterpart in dhclient-enter-hooks.d to umount NFS shares just before the interface is going down.

Then I realized that one of the two laptops supposed to make use of this script is running Ubuntu -not mine. And Ubuntu use by default a very very nasty software called NetworkManager, the kind of well thought user interface that stores configuration in anything but the standard stuff that worked finely before it even existed. Yeah, it literally makes a litter of /etc/network/interface. So, no, obviously, handling properly /etc/dhcp/dhclient-*-hooks.d/ scripts is not an option for NetworkManager, it’s so much better to reinvent the wheel with a poorly designed (And What’s The Deal With These Upper Cases?) /etc/NetworkManager/dispatcher.d.

Plenty of people complained already about obvious limits of NetworkManager. Sure, there’s is room for improvements and it’s better to contribute than just to rant. But considering the kind of replies provided to NetworkManager people about bugs reports (cause, really, not handling dhcp-*-hooks.d is a regression), I think I’ll pass. Funny links though: “After discussing with a few folks we found that pre-up will not come back … please provide detailed infos for your use-case as we have to find other means to achieve this.” (hum, they… found that useful working features will not come back but havent found any better alternative yet ?), “if the resolvconf abilities are not enough you can also stuff in a NM dispatcher.d script (see: /etc/NetworkManager/dispatcher.d/)” (please, have fun writing new scripts to replace the ones that worked just fine). In fact, when developers deal with such issue like this “Changed in network-manager (Fedora): status: Confirmed → Won’t Fix”, best is just to find a workaround that absolutely not relies on their stuff that is sure to be broken some other day – no doubt that if a new trend comes, they’ll ask you to one more time rewrite scripts just to do the same frickin thing you were able to do years ago with simple dhclient-*-hooks.d.

So I finally came up with /etc/network/if-up.d and /etc/network/if-down.d scripts. Its quite standard and, oh!, NetworkManager got a “dispatcher” that run-parts on this dir. The obvious drawback is the fact it cannot be used to properly unmount the NFS shares because it’s unclear whether NetworkManager will run if-down.d before or after having brought down the network interface and, also, because it’s way to more painy to determine whether the loss of the current interface means loosing the relevant network where shares are (if you loose the Wifi, clearly, you may still be properly connected to the LAN). And I’m off trying to guess how behave and how will behave in 6 month this piece of software.

Instead of hardcoding the list of NFS shares in one more script, considering that initscripts already provides a well-thought /etc/network/if-up.d/mountnfs, I figured I would simply rely on /etc/fstab. My /etc/network/if-up.d/01prepmountnfs (that must run before initscript’s mountnfs) simply goes through /etc/fstab, looks for NFS shares that are in noauto mode (so, not configured to be mounted automatically when the box starts), find out if the server exists on the current LAN. If so, it removes the noauto option and then initscript’s mountnfs does its magic. On Ubuntu, there’s no /etc/network/if-up.d/mountnfs, but the following is enough to replace it:


echo "#!/bin/sh
mount -a" > /etc/network/if-up.d/mountnfs
chmod a+x /etc/network/if-up.d/mountnfs

The /etc/network/if-down.d/unprepmountnfs counterpart only reverts /etc/fstab to its previous state. Yes, if you loose connection to the NFS server, your X session will probably get frozen. For the reasons previously stated, for now it will have to do.

Switching from NFSv3 to NFSv4

Today, I switched over NFSv4. I guess there published it for some reason and people claim it could increase file transfert rate by 20%.

In my case, to get it working properly, I…

Modified /etc/default/nfs-kernel-server on server side to have

NEED_SVCGSSD=no

Modified /etc/default/nfs-common on both clients and server side to have

NEED_STATD=no
NEED_IDMAPD=yes
NEED_GSSD=no

Modified /etc/exports on server side to have something starting by


/server 192.168.1.1/24(ro,fsid=0,no_subtree_check,async)

/server/temp 192.168.1.1/24(rw,nohide,no_subtree_check,async,all_squash,anonuid=65534,anongid=65534)

[...]

It forces you to set a root for the NFS server, in my case /server (which I had already in my NFSv3 scenario, so…), aka fsid=0.
You also need to specify nohide for any exports.

Modified /etc/fstab on clients side to set mount type to nfs4 and to remove the /server part from the paths, no longer necessary as path are relatives to fsid=0 which is /server. It gives entries like:

[...]
gate:/temp /stockage/temp nfs4 nolock 0 0
[...]

I had an export which was a symlink to somewhere in /home. NFSv4 is stricter than NFSv3 and there is no way to export something outside from fsid=0. So I made a bind, adding to /etc/fstab on server side:

[...]
/home/torrent/watch /server/torrent/watch none bind 0 0
[...]

After restarting nfs-kernel-server on the server side and nfs-common on both sides, umount NFS partitions and doing a mount -a on the client side, everything seems fine.

Minimalistic BitTorrent client over NFS/Samba

Not quite AJAX

While current trends in music/movie industry will surely encourage development of a new generation of peer-to-peer softwares, the same way they made CD-burners cheap in a less than a decade, I’m still quite happy with BitTorrent.

I used torrentflux for quite some time. Shipped with Debian, installed on my local home server, accessible to any box on the network over https, even if it’s interface is not exactly eye candy, it works. I just had to configure web browsers to access http://server/torrentflux/index.php?url_upload=$ each time they hit a .torrent file. But even if web interface may be powerful, user-friendly, I resent torrentflux for having me to click plenty of time (at least two times just to start a download), after login in.

I took a look at rTorrent. It works by looking into a directory for new .torrent then load them automatically. Wonderful. Sadly, you have to log in over SSH and then manually select over a text user interface which download you want to actually start.

I liked the idea of dragging’n’dropping .torrent in one directory. It can be done over NFS or Samba, with no additional login. I have those already set up on my server. Next step is to handle queue management with the same directory.

I came up with the idea of using a command line BitTorrent client through a script that would watch the damn NFS/Samba directory. It would :
– notice and register new .torrents dropped
– allow to forcefully pause/remove any designated torrent
– allow to forcefully pause all downloads
– warn by mail whenever a download is completed and unregister the relevant torrent

So I wrote such script so it would handle transmission daemon as shipped by debian stable. It looks for file in a given directory named after the following syntax:
– $file.torrent = torrent to be added
– $realfile.hash = torrent being processed (delete it to remove the torrent)
– $realfile.hash- = torrent paused
– $realfile.hash+ = torrent (supposedly) completed and already removed
– all- = pause all

Here’s the HOWTO:

apt-get install tranmissioncli screen
adduser torrent
echo "torrent: youruser" >> /etc/aliases

su torrent
cd ~/
mkdir watch download
exit

mkdir -p /server
ln -s /home/torrent /server

Obtain uid/gid of torrent necessary below:

cat /etc/passwd | grep torrent

Here I get 1003/1003.

Edit /etc/exports to set up NFS access (this assumes your NFS server is already set up), add:

# every box on the network get rw access to rtorrent
/home/torrent/download 192.168.1.1/24(rw,sync,all_squash,anonuid=1003,anongid=1003)
/home/torrent/watch 192.168.1.1/24(rw,sync,all_squash,anonuid=1003,anongid=1003)

On each NFS client, add in /etc/fstab (you must create mount points):

server:/home/torrent/download /mnt/torrent/download nfs rw,nolock 0 0
server:/home/torrent/watch /mnt/torrent/watch nfs rw,nolock 0 0

Edit /etc/samba/smb.conf to set up Samba access (this assumes your Samba server is already set up, add:

[Download]
path = /home/torrent/download
browseable = yes
public = yes
valid users = youruser
force user = torrent
force group = torrent
writable = yes

[Watch]
path = /home/torrent/watch
browseable = yes
public = yes
valid users = youruser
force user = torrent
force group = torrent
writable = yes

Restart NFS/Samba servers, mount networked file system on the clients.

Add a startup script for transmission-daemon, edit it if need be (daemon configuration is done here), fire it up:

cd /etc/init.d/
wget http://yeupou.free.fr/torrent/init.d/torrent
update-rc.d torrent defaults 80
/etc/init.d/torrent start

At any time, you can check the current daemon process with screen:

screen -r torrent

Add torrent-watch.pl in /usr/local/bin or /usr/bin (anywhere in $PATH):

cd /usr/local/bin
wget http://yeupou.free.fr/torrent/torrent-watch.pl
chmod a+x torrent-watch.pl

Check that it runs properly. Drag’n’drop any .torrent in /home/torrent/watch and run:

su torrent
torrent-watch.pl
cat status

If everything is ok, add in /etc/cron.d/torrent:

* * * * * torrent cd ~/watch && /usr/local/bin/torrent-watch.pl

And /etc/logrotate.d/torrent:

/home/torrent/watch/log {
weekly
missingok
rotate 2
nocompress
notifempty
}

You’re done.