The Case of the Mysterious Linker

On a few of my Fedora 19 machines, I’ve had odd linker problems. Apps that used to work, like k3b’s ripper, Audacity, etc. would fail to run with a linker error, such as:

audacity: error while loading shared libraries: cannot open shared object file: No such file or directory

I checked the packages, and rpmfusion did the proper rebuilds when Fedora changed the version number of libopenjpeg from 3 to 1 (ouch) last year. But rpm told me I had the correct version installed:

$ rpm -q openjpeg-libs

I tried rebuilding audacity from source (but couldn’t find source reference to libopenjpeg), deleting and rebuilding, prelink.cache all to no avail. A clue came in when I found the LD_DEBUG=all environment variable and ran audacity under it. It gave nice output like:

     29608: [0];  needed by /lib64/ [0]
     29608:     find [0]; searching
     29608:      search cache=/etc/
     29608:      search path=/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64                (system search path)
     29608:       trying file=/lib64/tls/x86_64/
     29608:       trying file=/lib64/tls/
     29608:       trying file=/lib64/x86_64/
     29608:       trying file=/lib64/
     29608:       trying file=/usr/lib64/tls/x86_64/
     29608:       trying file=/usr/lib64/tls/
     29608:       trying file=/usr/lib64/x86_64/
     29608:       trying file=/usr/lib64/

Ah, ha, it’s libavcodec that’s requiring libopenjpeg:

$ rpm -qf /usr/lib64/

So, this is part of the ffmpeg (oldversion) package. yum reinstalling that didn’t help.

But, wait, why is ffmpeg in /lib64? rpm -qf confirms that no package owns it (this may have been due in part to an ugly upgrade to f19). So, is it in /usr/lib64 where one would expect?

$ ls /usr/lib64/* -l
lrwxrwxrwx 1 root root      22 Oct 22 17:06 /usr/lib64/ ->
-rwxr-xr-x 1 root root 6219152 Jul  1  2011 /usr/lib64/
-rwxr-xr-x 1 root root 4746744 Oct  1 17:33 /usr/lib64/

Oh, my.

$ rpm -qf /usr/lib64/* 
file /usr/lib64/ is not owned by any package

OK, then, to slay the beasts, he thinksts:

$ sudo rm /usr/lib64/ /lib64/

Now, we should be good to go, right?

audacity: relocation error: /lib64/ symbol ff_raw_pix_fmt_tags, version LIBAVCODEC_52 not defined in file with link time reference

Argh. After some googling about, an old ld lib looked like a candidate problem.

$ ls /usr/lib64/ld-* -l
-rwxr-xr-x 1 root root 162472 Nov 16 20:34 /usr/lib64/
lrwxrwxrwx 1 root root     10 Dec 30 00:41 /usr/lib64/ ->
lrwxrwxrwx 1 root root     20 Dec 30 01:09 /usr/lib64/ ->
lrwxrwxrwx 1 root root     20 Sep  2 17:37 /usr/lib64/ ->

Crikey – so.2 and .so.3 mixed in. Must be multiple versions left after an upgrade.

$ rpm -qf /usr/lib64/ld-* 
file /usr/lib64/ is not owned by any package

$ sudo rm /usr/lib64/

and then went looking for other libraries that could be in the same situation:

$ rpm -qf  /usr/lib64/*so | grep package

And there were a bunch of them. So, in the end, I did:

# rpm -qf /usr/lib64/*.so* | grep package | cut -f 2 -d ' ' | xargs rm

And now all of the apps work again. This will be a new go-to step for a difficult distro update.