The Case of the Mysterious Linker

On a few of my Fedora 19 machines, I’ve had odd linker problems. Apps that used to work, like k3b’s ripper, Audacity, etc. would fail to run with a linker error, such as:

audacity: error while loading shared libraries: libopenjpeg.so.3: cannot open shared object file: No such file or directory

I checked the packages, and rpmfusion did the proper rebuilds when Fedora changed the version number of libopenjpeg from 3 to 1 (ouch) last year. But rpm told me I had the correct version installed:

$ rpm -q openjpeg-libs
openjpeg-libs-1.5.1-5.fc19.x86_64

I tried rebuilding audacity from source (but couldn’t find source reference to libopenjpeg), deleting and rebuilding ld.so.cache, prelink.cache all to no avail. A clue came in when I found the LD_DEBUG=all environment variable and ran audacity under it. It gave nice output like:

     29608:     file=libopenjpeg.so.3 [0];  needed by /lib64/libavcodec.so.52 [0]
     29608:     find library=libopenjpeg.so.3 [0]; searching
     29608:      search cache=/etc/ld.so.cache
     29608:      search path=/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64                (system search path)
     29608:       trying file=/lib64/tls/x86_64/libopenjpeg.so.3
     29608:       trying file=/lib64/tls/libopenjpeg.so.3
     29608:       trying file=/lib64/x86_64/libopenjpeg.so.3
     29608:       trying file=/lib64/libopenjpeg.so.3
     29608:       trying file=/usr/lib64/tls/x86_64/libopenjpeg.so.3
     29608:       trying file=/usr/lib64/tls/libopenjpeg.so.3
     29608:       trying file=/usr/lib64/x86_64/libopenjpeg.so.3
     29608:       trying file=/usr/lib64/libopenjpeg.so.3
     29608:

Ah, ha, it’s libavcodec that’s requiring libopenjpeg:

$ rpm -qf /usr/lib64/libavcodec.so.52
ffmpeg-compat-0.6.7-1.fc19.x86_64

So, this is part of the ffmpeg (oldversion) package. yum reinstalling that didn’t help.

But, wait, why is ffmpeg in /lib64? rpm -qf confirms that no package owns it (this may have been due in part to an ugly upgrade to f19). So, is it in /usr/lib64 where one would expect?

$ ls /usr/lib64/libavcodec.so.52* -l
lrwxrwxrwx 1 root root      22 Oct 22 17:06 /usr/lib64/libavcodec.so.52 -> libavcodec.so.52.121.0
-rwxr-xr-x 1 root root 6219152 Jul  1  2011 /usr/lib64/libavcodec.so.52.121.0
-rwxr-xr-x 1 root root 4746744 Oct  1 17:33 /usr/lib64/libavcodec.so.52.72.2

Oh, my.

$ rpm -qf /usr/lib64/libavcodec.so.52* 
ffmpeg-compat-0.6.7-1.fc19.x86_64
file /usr/lib64/libavcodec.so.52.121.0 is not owned by any package
ffmpeg-compat-0.6.7-1.fc19.x86_64

OK, then, to slay the beasts, he thinksts:

$ sudo rm /usr/lib64/libavcodec.so.52.121.0 /lib64/libavcodec.so.52

Now, we should be good to go, right?

audacity: relocation error: /lib64/libavformat.so.52: symbol ff_raw_pix_fmt_tags, version LIBAVCODEC_52 not defined in file libavcodec.so.52 with link time reference

Argh. After some googling about, an old ld lib looked like a candidate problem.

$ ls /usr/lib64/ld-* -l
-rwxr-xr-x 1 root root 162472 Nov 16 20:34 /usr/lib64/ld-2.17.so
lrwxrwxrwx 1 root root     10 Dec 30 00:41 /usr/lib64/ld-linux-x86-64.so.2 -> ld-2.17.so
lrwxrwxrwx 1 root root     20 Dec 30 01:09 /usr/lib64/ld-lsb-x86-64.so -> ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root     20 Sep  2 17:37 /usr/lib64/ld-lsb-x86-64.so.3 -> ld-linux-x86-64.so.2

Crikey – so.2 and .so.3 mixed in. Must be multiple versions left after an upgrade.

$ rpm -qf /usr/lib64/ld-* 
glibc-2.17-20.fc19.x86_64
glibc-2.17-20.fc19.x86_64
file /usr/lib64/ld-lsb-x86-64.so is not owned by any package
redhat-lsb-core-4.1-14.fc19.x86_64

$ sudo rm /usr/lib64/ld-lsb-x86-64.so

and then went looking for other libraries that could be in the same situation:

$ rpm -qf  /usr/lib64/*so | grep package

And there were a bunch of them. So, in the end, I did:

# rpm -qf /usr/lib64/*.so* | grep package | cut -f 2 -d ' ' | xargs rm

And now all of the apps work again. This will be a new go-to step for a difficult distro update.