Converting video files to H.265/HEVC, no washed out colors and all streams mapped, with ffmpeg

I had a few 1080p video files using AV1 codec. Not sure why, if it is a player issue or hardware issue, nonetheless, my (slightly aging) laptop was having a hard time playing these while playing with no effort at all H.265/HEVC 2160p videos. The following command converts all mkv files in a folder to H.265/HEVC, not washing out colors and keep all streams:

for av in *.mkv; do ffmpeg -i "$av" -c:v libx265 -color_trc smpte2084 -color_primaries bt2020 -crf 22 -map 0 -map_metadata 0 -map_chapters 0 -c:a copy -c:s copy "${av%.*}"-x265.mkv ; done

In my case, it results in slightly larger files but, and that was the point, these play on the laptop with no noticeable CPU-usage:

1,2G 'XX1 AV1 Opus [AV1D].mkv'
1,1G 'XX1 AV1 Opus [AV1D]-x265.mkv'
830M 'XX2 AV1 Opus [AV1D].mkv'
951M 'XX2 AV1 Opus [AV1D]-x265.mkv'

Accessing video/audio from a computer with a Freebox (using UPnP)

Nowadays, you would think having a network device like a TV box accessing data from your unix-based system easy going. I mean, we have NFS, Samba all-right, how hard could that be to devise an interface to access at least one of the many network shares, provided a user/password or based on it’s IP?

But no, it won’t work that way. It’s supposedly too complex, so instead, people promotes zeroconf and such, stuff supposed to work out-of-the-box that actually may not work at all. For instance, to access movies/music from your computer with your Freebox HD (TV box) v5, and I assume it’s the same with many similar boxes from others ISP, you can forget about using NFS/SMB/http or whatever protocol you already had working or think easy to put in motion. No, you’ll have to use UPnP, standing for Universal Plug and Play, words that truly often refers to Plug and Pray instead.

From the computer…

So, let’s go getting our hands dirty. The setup I’m working with is quite simple: a Freebox HD v5, a single computer with a single user having some videos, some with subtitles (mostly .srt), and audio files. It’s basic but it took me a while to figure it out.

I did test plenty of UPnP servers. I tried Mediatomb but it did not work – plus the gothic interface seemed weird. I tried XBMC and it worked nicely but only to show empty folders, and no obvious way to have it up without the CPU consuming interface. Then I installed MythTV and I did not even understood how it is supposed to work in regard of UPnP.

So I tried minidlna, the lightweight one I avoided from the start because it’s not known to properly support subtitles files. And, tada!, it actually works almost out-of-the-box. Yeah, almost. That’s the funny thing, even if you claim to aim zeroconf, when it comes to share files, at some point you’re still in need to list what you actually wanna share. Whatever. So I did apt-got minidlna. Then I edited /etc/default/minidlna:

START_DAEMON="yes"
USER="thisguy"
GROUP="thisguy"

As there’s only one legit user on this box, I wanted the daemon to be able to access his files with no specific consideration for privileges/ownership. I implied doing then:

chown thisguy.thisguy /var/lib/minidlna -Rv

Then I modified /etc/minidlna.conf as follows:

media_dir=V,/home/thisguy/Films
media_dir=A,/home/thisguy/Musique
media_dir=P,/home/thisguy/Photos

log_level=info
network_interface=eth0
friendly_name=thisserver
inotify=yes

… to the box

I restarted the daemon (and made sure it’s included in /etc/rc2.d). And that’s all (I modified also the firewall setup but I’m not sure that’s relevant considering that UPnP implies that it advertises itself to other devices when it’s up and not the other way around – so firewall is an issue only if you have one that blocks connections from the inside to the outside).

By that’s all, I meant: it was enough to get access to movies on the Freebox HD for the first time; but not with the subtitles.

I googled around: the minidlna version I had was supposed to properly give access to .srt along with videos. The Freebox HD itself supports .srt files with the same name of the video, when you access video over an USB device. But apparently plenty of implementations of UPnP have no consideration for subtitle files and the Freebox’s one is probably one such. So having separate .srt or .sub or whatever is a no-go.

Then, I gave a try to Matroska files (.mkv) despite the fact that I always had bad experience with it. Most notably, I usually implies videos costing tons of CPU time to decode and render and videos players usually fail to properly keep video in sync with audio – yeah, that’s really nasty. But Matroska allows to embed subtitles in the file without touching the video stream, which is neat. So I did that. Long story short, Matroska files, 9 times out of 10, freeze the Freebox HD: and I’m talking about Matroska files that are not bigger than the original .avi that run well on the very same Freebox, and I’m talking about Matroska files that run well on the computer with VLC or mplayer. So that’s a no-go too.

So I ended with the worse solution: altering original files, with mencoder to incrust subtitles within. Yeah, it’s kind of definitive and if you don’t want to spend hours of CPU time to do it, it implies quality loss. But, at least, it works. So here it goes, I wrote the following script to ease the process, assuming that video files along with .srt where originally on an USB device called HERMES and then copied to thisguys home:

#!/bin/bash

DEST=/home/thisguy/Films
ORIG=/media/HERMES

# go thru the list of videos
find $ORIG -name "*.avi" -or -name "*.mpg" -or -name "*.mp4" |
while read video; do
  # find out basename
  basename=${video%.*}
  format=${video##*.}
  endname=`basename "$basename"`-WS.$format

  echo $endname

  # use french subtitles in priority over english
  subtitle=0
  if [ -r "$basename"_en.srt ]; then subtitle="$basename"_en.srt; fi
  if [ -r "$basename"_fr.srt ]; then subtitle="$basename"_fr.srt; fi
  if [ -r "$basename".srt ]; then subtitle="$basename".srt; fi

  # no valid subtitle found at this point? skip the video
  if [ ! -r "$subtitle" ]; then continue; fi

  # now create to relevant directory if missing
  enddir=`dirname "$basename" | sed "s@${ORIG}@${DEST}@g;"`
  if [ ! -d "$enddir" ]; then mkdir -pv "$enddir"; fi

  # proceed only if the file is missing
  if [ -r "$enddir/$endname" ]; then continue; fi

  # if we reach this, go for it
  mencoder "$video" -subpos 92 -sub "$subtitle" -o "$enddir/$endname" -oac copy -ovc lavc

done
# EOF

Yeah. Plug and play my ass.

Videos misc: saving space by converting .avi, .mpg or .mkv to .mp4, resetting subtitles

These days, while according to neutral sources movies industry has never been so juicy so it made obviously necessary to restrict freedom in France in the name of its survival, I thought nice to share two small scripts handy when dealing with videos on your harddrive.

Surely, you wouldn’t store videos downloaded over the internet that you haven’t paid for. I guess that’s immoral since in 2009 the US box office top $10bn for the first time in history, during worldwide economic downfall . An advertisement paid by these guys that made these $10bn, while bankrupcy was really an option for major financial institutions and eviction just the same for poor tenants, said it is piracy – while it is still hard for me to envision how it relates to these events occuring on regular basis in Malaysia, Cameroon, Red Sea, etc.

And if you are not concerned by the moral issue (a communist like Jesus Christ, aren’t you?), maybe you are afraid to get caught anyway. Well, it is unlikely that police would come to your house with a search warrant looking for piracy evidence. Mostly because there is no such thing in French law as a search warrant. Indeed, police is entitled to enter in your house in some cases: in three cases only. First, there is what is named commission rogatoire, an order given to a policemen by a judge to do something specific in his name such as search in your house. Looks obviously like an US/UK common law search warrant but it is not: sure it gives the same rights to the police, but it is not an usual procedure in France as it applies only to criminal investigations (information judiciaire), not for trivial misdemeanour/regulatory offences. Second, in the case of a enquête de crime ou délit flagrant (felony or misdemeanour punishable by jail time that just occurred), police can enter your house without your consent. Three, in what is actually the only case that would allow police to enter your house for a regulatory offence (which what this piracy is actually more or less about) is the enquête préliminaire – funny, in this case, police requires your (written) consent to enter. If guess that if I had this kind of piracy evidence at home and the police coming to my doorstep asking to enter in an enquête préliminaire, I would probably not consent.

And I would not even dare to bring the issue of the fine endorsed by this HADOPI law. It is said that, as a friendly reminder of your place you should not have forgotten so easily (customer, yes, that’s you – nothing else – even if you do no harm, it is not up to you to proceed otherwise as you’d wish), by Law, your Internet access will be discarded while you’ll still pay for it. Well, French penal Law states also that “Nul n’est responsable que de son propre fait”, meaning that you can only be punished for your own doings. Sure, there are exceptions (boss that somehow forced employee to misbehave, etc). But none that I think of such as the case of two persons living in the same house and being punished together for the actions of only one of them without the knowledge and consent of the other. By principle, this idea is outlaw, a regression of two thousands years of penal law, dropping us back in the days when you were entitled to take possessions (by force if necessary) of the belongings of a spartian you’ve just met because, as athenian (or whatever), you were recently spoiled by another spartian, no matter that they had no ties aside their citizenship.

1. So there is a script called dir2x264.sh that I wrote for the purpose of saving harddisk space by cleanly converting .avi and .mpg files to an x264-encoded .mp4 file.

It could surely be tuned – I noticed issues when dealing with .mkv files. So far it uses mencoder (mplayer’s encoder) with lavc as audio codec and x264 as video codec. So obviously it requires mencoder, with support of these two (usual) codecs. It implies ffmpeg as well.

To use it, go in the directory where you have avi or mpg files, put the script in there and call it (it will always process all the files in the current `pwd`).

$ cd ~/myvideos
$ chmod +x dir2x264.sh # (if not made executable already)
$ ./dir2x264.sh

It will log work being done in dir2x264log, to easily evaluate the harddisk space saved.

2. In case you cannot find (Have you tried SMPlayer?) the exact correct subtitle file for your video but found one that is just fine except there is a delay between the sound/image and the subtitle, the subtitle_reset.pl could do the trick for you. It depends on Subtitles.pm (libsubtitles-perl in debian).

This script takes two command line arguments: –file and –time (in seconds, positive value or negative), so the usage is quite obvious.
It will make a backup of the original file. If you run it several times to finely adjust your file, it will always restart from the backup file, unless removed obviously.

Note that however this script will not help if the matter is that the video and subtitles file frame-rate differs. You may want to give a try to subs, a script that is now shipped with Subtitles.pm (that was not there or that I missed when I first wrote this one).

July 31, 2010 Update: I thought I found a workaround the Matroska sound issue (Too many audio packets in the buffer). As mencoder is unable to properly convert them, after googling around, I gave a try to the idea of extracting apart the audio and video stream and to put them back together afterwards with mencoder. It actually now is able to produce a working mp4 file with the two streams but it seems that the audio is out of synch with the video. SMPlayer can correct the delay while running, though. So it is better than before but it is still buggy.

August 11, 2010 Update: I noticed several issues with mencoder-produced mp4. I modified the script to use only ffmpeg (except the mkv-specific part that I have not modified).