Discussion:
Processed: Not just a dpkg bug
Debian Bug Tracking System
2005-08-17 16:03:16 UTC
Permalink
reassign 317082 libc6-dev,dpkg-dev
Bug#317082: libc6-s390x: missing depends on lib64gcc1
Bug reassigned from package `dpkg-dev' to `libc6-dev,dpkg-dev'.
thanks
Stopping processing here.

Please contact me if you need assistance.

Debian bug tracking system administrator
(administrator, Debian Bugs database)
Andreas Jochens
2005-08-17 18:26:46 UTC
Permalink
For those following, the problem is that people are building 64-bit
libraries on a 32-bit platform (or the other way around) as part of the
package build. dpkg-shlibdeps uses plain old "ldd" to find out the
dependencies of a binary or shared library, but the ldd on the system
won't be able to identify the impostor libraries.
dpkg_shlibdeps -p lib64z1 -lbuild-tree/zlib-1.2.3
/usr/bin/ldd: line 161: /lib64/ld-linux-x86-64.so.2: cannot execute binary file
dpkg-shlibdeps: failure: ldd on `debian/lib64z1/usr/lib64/libz.so.1.2.3' gave error exit status 1
dh_shlibdeps: command returned error code 256
In this example, the 32-bit i386 ldd on my system can't read the amd64
binaries that have been generated.
Please 'apt-get install amd64-libs' and try this again. 'ldd' should
work then.

The current 'ldd' already supports biarch.

However, to work for 64-bit binaries, 'ldd' needs the 64-bit linker
'/lib64/ld-linux-x86-64.so.2'. This linker is currently in 'amd64-libs'
(this may change to 'libc6-amd64' later).

This means that the problem can be solved by always installing
'amd64-libs' (or 'libc6-amd64') before dpkg-shlibdeps is used for
a 64-bit binary.

Generally, a package which creates 64-bit binaries on i386 will have to
Build-Depend on 'amd64-libs-dev' or 'libc6-dev-amd64' anyway, because
it needs to link to a 64-bit libc. In this case there is no problem.

In the case of the 'glibc' package itself, I think a Build-Depends
on 'amd64-libs-dev [i386]' (or 'libc6-dev-amd64 [i386]') should be
added.

The same applies to the other biarch cases (powerpc, sparc, s390).

This should fix the bug.

Regards
Andreas Jochens
Andreas Jochens
2005-08-17 20:05:42 UTC
Permalink
No, that doesn't solve the problem. How are you supposed to invoke a
64-bit linker for a bi-arch build being done on a 32-bit buildd?
I guess you will generally have many more issues than this one when you
try to build 64-bit packages on a 32-bit buildd (e.g. compiling and
running 64-bit programs from configure scripts, running 'make check' or
'make test' targets, using binaries which have been built by the package
itself etc.)

In the end it will be much easier to require a 64-bit machine to be
used to build 32/64-bit biarch packages instead of trying to circumvent
all these issues.

Regards
Andreas Jochens
Steve Langasek
2005-08-17 19:52:39 UTC
Permalink
Post by Andreas Jochens
For those following, the problem is that people are building 64-bit
libraries on a 32-bit platform (or the other way around) as part of the
package build. dpkg-shlibdeps uses plain old "ldd" to find out the
dependencies of a binary or shared library, but the ldd on the system
won't be able to identify the impostor libraries.
dpkg_shlibdeps -p lib64z1 -lbuild-tree/zlib-1.2.3
/usr/bin/ldd: line 161: /lib64/ld-linux-x86-64.so.2: cannot execute binary file
dpkg-shlibdeps: failure: ldd on `debian/lib64z1/usr/lib64/libz.so.1.2.3' gave error exit status 1
dh_shlibdeps: command returned error code 256
In this example, the 32-bit i386 ldd on my system can't read the amd64
binaries that have been generated.
Please 'apt-get install amd64-libs' and try this again. 'ldd' should
work then.
The current 'ldd' already supports biarch.
However, to work for 64-bit binaries, 'ldd' needs the 64-bit linker
'/lib64/ld-linux-x86-64.so.2'. This linker is currently in 'amd64-libs'
(this may change to 'libc6-amd64' later).
This means that the problem can be solved by always installing
'amd64-libs' (or 'libc6-amd64') before dpkg-shlibdeps is used for
a 64-bit binary.
No, that doesn't solve the problem. How are you supposed to invoke a
64-bit linker for a bi-arch build being done on a 32-bit buildd?
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
***@debian.org http://www.debian.org/
GOTO Masanori
2005-08-18 05:32:41 UTC
Permalink
At Wed, 17 Aug 2005 17:00:23 +0100,
I don't think this is just a dpkg-dev bug, these "bi-arch" systems need
to provide ldd or an equivalent that can read either form of shared
library that it would support.
objdump isn't a solution either, while it sometimes can read the other
shared library, it doesn't provide the linker search patch which is of
critical importance to get this stuff right.
So I'm including libc6-dev (for want of a better package) in this bug,
as we first need ldd on these bi-arch systems (or something similar) for
dpkg-shlibdeps to use before we can fix that!
ldd is very hard to handle 64bit binaries on 32bit systems without
breaking glibc, kernel and various technical beautiful concepts.

Please look at /usr/bin/ldd. It just passes LD_* variable to dynamic
linker. When dynamic linker is invoked from kernel elf module, it
parses LD_* environment variable and put the library path message to
stdout. So ldd is just wrapper to take notice to dynamic linker.

The actual dynamic linker path is specified in each binaries. For
example, do "strings /bin/ls |grep ld" on both i386 and amd64. You'll
find 64bit binaries designate 64bit dynamic loader that is placed in
/lib64. And, dynamic loader itself is built with "64bit code" - so we
cannot invoke even dynamic loader.


OK, now back to the discussion. The above means "we cannot use ldd to
search pathes". OTOH, Ryan suggested me to look at dpkg-shlibdeps in
dpkg-cross package as I described. It's something adhoc fix, but I
think we have no other way to support biarch nicely.

Regards,
-- gotom
GOTO Masanori
2005-08-18 05:47:46 UTC
Permalink
At Wed, 17 Aug 2005 22:05:42 +0200,
Post by Andreas Jochens
I guess you will generally have many more issues than this one when you
try to build 64-bit packages on a 32-bit buildd (e.g. compiling and
running 64-bit programs from configure scripts, running 'make check' or
'make test' targets, using binaries which have been built by the package
itself etc.)
In the end it will be much easier to require a 64-bit machine to be
used to build 32/64-bit biarch packages instead of trying to circumvent
all these issues.
Yes, that's one solution. However, instead, I would like to propose
supporting 32bit and 64bit binaries as separated architectures.

For example, think about ppc32 and ppc64. My proposal is to have both
Debian/dists/sid/main/binary-powerpc and
Debian/dists/sid/main/binary-ppc64. It's similar to the different
architecture between i386 and amd64 - but ppc32 and ppc64 are not
distinguished so much.

If a package has (ex:) "Features: biarch" in debian/control (like
Tollef's proposal), ppc64 buildd picks up to build this package. If
it does not biarch capable package, it should not be built. It
reduces much disk spaces on mirror servers, and the 90% of
non-biarch-needed (binaries and libraries) packages do not need to
consider about biarch problems. To install both 32/64 bit packages,
we need to consider about the file duplication in /usr/share and
/usr/bin - but fortunatelly the proposal of Scott's dpkg modification,
FILTERS and CLASSES, probably fix this kind of problems.

It's very simple way and we don't modify a lot of packages. If you
guys like this idea, I'll write the proposal to debian-devel lists.
Or is this idea already proposed?

Regards,
-- gotom
Andreas Jochens
2005-08-18 08:24:14 UTC
Permalink
Post by GOTO Masanori
At Wed, 17 Aug 2005 22:05:42 +0200,
Post by Andreas Jochens
In the end it will be much easier to require a 64-bit machine to be
used to build 32/64-bit biarch packages instead of trying to circumvent
all these issues.
Yes, that's one solution. However, instead, I would like to propose
supporting 32bit and 64bit binaries as separated architectures.
For example, think about ppc32 and ppc64. My proposal is to have both
Debian/dists/sid/main/binary-powerpc and
Debian/dists/sid/main/binary-ppc64. It's similar to the different
architecture between i386 and amd64 - but ppc32 and ppc64 are not
distinguished so much.
If a package has (ex:) "Features: biarch" in debian/control (like
Tollef's proposal), ppc64 buildd picks up to build this package. If
it does not biarch capable package, it should not be built. It
reduces much disk spaces on mirror servers, and the 90% of
non-biarch-needed (binaries and libraries) packages do not need to
consider about biarch problems. To install both 32/64 bit packages,
we need to consider about the file duplication in /usr/share and
/usr/bin - but fortunatelly the proposal of Scott's dpkg modification,
FILTERS and CLASSES, probably fix this kind of problems.
It's very simple way and we don't modify a lot of packages. If you
guys like this idea, I'll write the proposal to debian-devel lists.
Or is this idea already proposed?
I certainly like this idea.

There is already an inofficial buildd for the ppc64 architecture
running for 'unstable'. The respective ppc64 package archive is located at

deb http://debian-ppc64.alioth.debian.org/gcc4 unstable main

and almost 95% of all source packages from 'unstable' have already been
compiled for the ppc64 architecture.

A small number of packages still need some minor patches
to make this work, notably 'glibc' (see #301438 - the patch in that
bug is outdated, but I could supply an updated patch for this).

With this setup, the ppc64/powerpc situation would become very similar
to the amd64/i386 situation and the same solution for multiarch/biarch
support could be used for both cases.

Regards
Andreas Jochens
GOTO Masanori
2005-08-18 10:38:25 UTC
Permalink
At Thu, 18 Aug 2005 10:24:14 +0200,
Post by Andreas Jochens
There is already an inofficial buildd for the ppc64 architecture
running for 'unstable'. The respective ppc64 package archive is located at
deb http://debian-ppc64.alioth.debian.org/gcc4 unstable main
and almost 95% of all source packages from 'unstable' have already been
compiled for the ppc64 architecture.
Indeed - if we put this thought forward, we'll reach to the whole
native ppc64 support of all packages.

BTW I think supporting all binary packages as 64bit native does not
have much sense - most packages do not use such large memory space,
and even they run slower than 32bit binaries. This is because I would
like to introduce (ex:) Features flags.
Post by Andreas Jochens
A small number of packages still need some minor patches
to make this work, notably 'glibc' (see #301438 - the patch in that
bug is outdated, but I could supply an updated patch for this).
I didn't put your patch because the native ppc64 support was not
adopted, from the discussion that was done in debian-powerpc lists.
If your patch does not have any impact, please send your latest
version to me.
Post by Andreas Jochens
With this setup, the ppc64/powerpc situation would become very similar
to the amd64/i386 situation and the same solution for multiarch/biarch
support could be used for both cases.
Yes. The current partial support for biarch imposes various
modifications on package maintainers. Such method will not be
spereaded - debian is the large voluntary organization, reducing cost
of package maintainer's load is the primal principle.

Regards,
-- gotom
Steve Langasek
2005-08-19 02:26:45 UTC
Permalink
Post by GOTO Masanori
At Wed, 17 Aug 2005 22:05:42 +0200,
Post by Andreas Jochens
I guess you will generally have many more issues than this one when you
try to build 64-bit packages on a 32-bit buildd (e.g. compiling and
running 64-bit programs from configure scripts, running 'make check' or
'make test' targets, using binaries which have been built by the package
itself etc.)
In the end it will be much easier to require a 64-bit machine to be
used to build 32/64-bit biarch packages instead of trying to circumvent
all these issues.
Yes, that's one solution. However, instead, I would like to propose
supporting 32bit and 64bit binaries as separated architectures.
For example, think about ppc32 and ppc64. My proposal is to have both
Debian/dists/sid/main/binary-powerpc and
Debian/dists/sid/main/binary-ppc64. It's similar to the different
architecture between i386 and amd64 - but ppc32 and ppc64 are not
distinguished so much.
If a package has (ex:) "Features: biarch" in debian/control (like
Tollef's proposal), ppc64 buildd picks up to build this package. If
it does not biarch capable package, it should not be built. It
reduces much disk spaces on mirror servers, and the 90% of
non-biarch-needed (binaries and libraries) packages do not need to
consider about biarch problems. To install both 32/64 bit packages,
we need to consider about the file duplication in /usr/share and
/usr/bin - but fortunatelly the proposal of Scott's dpkg modification,
FILTERS and CLASSES, probably fix this kind of problems.
It's very simple way and we don't modify a lot of packages. If you
guys like this idea, I'll write the proposal to debian-devel lists.
Yes, this sounds like a much better solution to me: it's more robust so
long as there are still many developers running the 32-bit variants of
the biarch systems, it makes better use of archive space, and it appears
to get us at least a little bit closer to multiarch, which I still
believe should be the long-term goal for such systems.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
***@debian.org http://www.debian.org/
Nikita V. Youshchenko
2005-08-21 15:42:54 UTC
Permalink
objdump isn't a solution either, while it sometimes can read the other
shared library, it doesn't provide the linker search patch which is of
critical importance to get this stuff right.
Hmm...
But what for is that search path?
As far as I understand, dpkg-shlibdeps should just get NEEDED sonames, and
match those against available shlibs files.

And that is what code actually does. Path information is used only if
something is not found in this way - that means, package that provides the
library does not provide shlibs data, which is a bug anyway.

So looks it is safe to remove any path processing from dpkg-slibdeps, and
use only objdump. If something breaks, it should be fixed elsewhere (i.e.
proper shlibs data added to packages).

But anyway, this is not enough: 32bit and 64bit versions of a library are
likely to have same sonames, so they can't be distinguished within the
current shlibs file format. Complete fix should probably add one more
field to shlibs files - the binary format.

So a possible way to fix this is:
- add binary format field to slibs file format [for backward-compatability,
the field should be optional and default to 'normal' binary format for the
architecture; packages providing such extended shlibs data should conflict
with dpkg-dev versions that don't support it]
- fix dpkg-shlibdeps: don't call ldd; use objdump both to extract NEEDED
and to get binary format; add extended shlibs format handling;
- fix any packages that happen to depend on [just removed] dpkg-shlibdeps
support of no shlibs information.
Frank Lichtenheld
2006-01-18 23:14:35 UTC
Permalink
Let's revive this discussion.
Post by Nikita V. Youshchenko
objdump isn't a solution either, while it sometimes can read the other
shared library, it doesn't provide the linker search patch which is of
critical importance to get this stuff right.
Hmm...
But what for is that search path?
As far as I understand, dpkg-shlibdeps should just get NEEDED sonames, and
match those against available shlibs files.
And that is what code actually does. Path information is used only if
something is not found in this way - that means, package that provides the
library does not provide shlibs data, which is a bug anyway.
This isn't quite true I think. The current dpkg-shlibdeps code works
like this:

1) use "ldd <binary>" to find the paths to the linked libraries
2) use "objdump -p <binary>" to actually check which of this libraries
are listed as NEEDED (Are there cases where ldd lists
libraries that are not NEEDED?)
3) check the shlibs.local and shlibs.overides files for matches
4) for remaining libraries search the corresponding package with
dpkg --search <path>
Here the ldd information is used!
5) check the shlibs files of the packages identified
Post by Nikita V. Youshchenko
From reading the source code of dpkg-cross' version of dpkg-shlibdeps
it works like this:

c1) like 2)
c2) like 3)
c3) determine the path to the library by just prepending $crossroot or
$crossroot64 to it (both variables which can be set by the user)
c4) like 4)
c5) like 5)

I don't see how the current way this works can deal with several
library directories like /lib /usr/lib /usr/X11R6/lib. But maybe I'm
missing something?
Post by Nikita V. Youshchenko
So looks it is safe to remove any path processing from dpkg-slibdeps, and
use only objdump. If something breaks, it should be fixed elsewhere (i.e.
proper shlibs data added to packages).
If we don't use the path information from ldd there are several ways to
go:
1) use "dpkg --search" but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump isn't
actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
2) we could try to use the ldconfig cache to make to work of ldd for
ourself.
Questions: - Is this really an advantage? Or has the cache the same
problems ldd has?
3) we could parse /etc/ld.so.conf and search for the libraries ourself.
this is essentially a more general version of the current way
dpkg-cross works AFAICS
4) we could just parse _all_ shlib files, perhaps even generating a cache
from them.
Questions: - Seems a complex solution (if it uses caching). Is any of
the simpler solutions feasible?
- How slow is this?

Comments and corrections welcome.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Frank Lichtenheld
2006-01-18 23:37:35 UTC
Permalink
Post by Frank Lichtenheld
If we don't use the path information from ldd there are several ways to
1) use "dpkg --search" but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump isn't
actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
Note that this method is used in the patch for #285857 which I'm
currently evaluating for inclusion. So it seems the answer to
question 1 is "no" (or else the patch in #285857 is useless)
and the answer to question 2 is most likely
"it is listed in the shlibs file of the corresponding package" which
might have been the sense of Nikita's explanation of "buggy packages
that don't list their libs in their shlibs will need to be fixed" that
I just don't got.

But now I definetly need some sleep before I can wind my head around
more of that dpkg-shlibdeps code...

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Russ Allbery
2006-01-19 04:00:55 UTC
Permalink
Post by Frank Lichtenheld
This isn't quite true I think. The current dpkg-shlibdeps code works
1) use "ldd <binary>" to find the paths to the linked libraries
2) use "objdump -p <binary>" to actually check which of this libraries
are listed as NEEDED (Are there cases where ldd lists
libraries that are not NEEDED?)
Yes. ldd will list all shared libraries pulled in by a binary, regardless
of whether they're NEEDED by the binary itself or just NEEDED by one of
the shared libraries it uses. For example:

windlord:~> ldd /usr/bin/remctl
linux-gate.so.1 => (0xffffe000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0xb7f8e000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7e57000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0xb7ddf000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0xb7dbb000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0xb7db8000)
libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0xb7db3000)
libresolv.so.2 => /lib/tls/libresolv.so.2 (0xb7da0000)
/lib/ld-linux.so.2 (0xb7fb3000)
windlord:~> objdump -p /usr/bin/remctl | grep NEEDED
NEEDED libgssapi_krb5.so.2
NEEDED libc.so.6
windlord:~> objdump -p /usr/lib/libgssapi_krb5.so.2 | grep NEEDED
NEEDED libkrb5.so.3
NEEDED libk5crypto.so.3
NEEDED libcom_err.so.2
NEEDED libkrb5support.so.0
NEEDED libresolv.so.2
NEEDED libc.so.6
--
Russ Allbery (***@debian.org) <http://www.eyrie.org/~eagle/>
Daniel Jacobowitz
2006-01-20 03:22:32 UTC
Permalink
Post by Russ Allbery
Post by Frank Lichtenheld
This isn't quite true I think. The current dpkg-shlibdeps code works
1) use "ldd <binary>" to find the paths to the linked libraries
2) use "objdump -p <binary>" to actually check which of this libraries
are listed as NEEDED (Are there cases where ldd lists
libraries that are not NEEDED?)
Yes. ldd will list all shared libraries pulled in by a binary, regardless
of whether they're NEEDED by the binary itself or just NEEDED by one of
And also LD_PRELOAD'd libraries, et cetera (which may include
fakeroot).
--
Daniel Jacobowitz
CodeSourcery
Steve Langasek
2006-01-20 11:47:35 UTC
Permalink
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
So looks it is safe to remove any path processing from dpkg-slibdeps, and
use only objdump. If something breaks, it should be fixed elsewhere (i.e.
proper shlibs data added to packages).
If we don't use the path information from ldd there are several ways to
1) use "dpkg --search" but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump isn't
actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
FWIW, the most robust way to handle this probably looks something like:

foreach $dir (@searchpath) {
$var = `dpkg --search $file`
do_stuff;
last if found;
}

That would seem to address the Hurd case (lib is available multiple times in
the search path due to symlinks, but only one of these files is known by
dpkg), while also eliminating collisions with similarly named files that are
not in the library search path at all (e.g., objects intended for use with
LD_PRELOAD or something).

The only requirement is that dpkg have an internal representation of the
library search path for the object type -- part of which comes from
/etc/ld.so.conf, part of which is hard-coded in ld.so. Oh... and then
there's RPATH...
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
***@debian.org http://www.debian.org/
Frank Lichtenheld
2006-01-22 21:14:16 UTC
Permalink
tags 317082 patch
tags 317082 - moreinfo
thanks
Post by Steve Langasek
Post by Frank Lichtenheld
1) use "dpkg --search" but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump isn't
actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
[...]
Post by Steve Langasek
The only requirement is that dpkg have an internal representation of the
library search path for the object type -- part of which comes from
/etc/ld.so.conf, part of which is hard-coded in ld.so. Oh... and then
there's RPATH...
I've implemented this option. Patch and new script (since the patch is
garbled with a little code clean-up I did while going through the
script) are attached.

Comments and/or testing welcome.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Frank Lichtenheld
2006-01-24 14:51:06 UTC
Permalink
Post by Frank Lichtenheld
I've implemented this option. Patch and new script (since the patch is
garbled with a little code clean-up I did while going through the
script) are attached.
Comments and/or testing welcome.
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Frank Lichtenheld
2006-01-24 15:34:43 UTC
Permalink
Post by Frank Lichtenheld
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...
This option wasneeded when package builds a binary that uses libraries
built from the same source package. In this case, libraries may not be
available outside of package build directory; so ldd called from
dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH.
Don't yet know how this situation is handled in your patched
dpkg-shlibdeps...
Hmm, but dpkg-shlibdeps already tries to take care of this by using
all local shlibs files. Can someone point me to a package that really
misbuilds without -l ? This would really help me to understand the
real problem.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Frank Lichtenheld
2006-01-24 16:00:22 UTC
Permalink
dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH won't
point to directories with local libraries.
That's not true. ldd will just happily print "libfoo.so.1 => not found"
and exit with exit code 0. So this doesn't terminate dpkg-shlibdeps.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Nikita V. Youshchenko
2006-01-24 15:43:02 UTC
Permalink
Post by Frank Lichtenheld
Post by Frank Lichtenheld
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...
This option wasneeded when package builds a binary that uses libraries
built from the same source package. In this case, libraries may not be
available outside of package build directory; so ldd called from
dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH.
Don't yet know how this situation is handled in your patched
dpkg-shlibdeps...
Hmm, but dpkg-shlibdeps already tries to take care of this by using
all local shlibs files. Can someone point me to a package that really
misbuilds without -l ? This would really help me to understand the
real problem.
dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH won't
point to directories with local libraries.

defined($c= open(P,"-|")) || syserr("cannot fork for ldd");
if (!$c) { exec("ldd","--",$exec[$i]); syserr("cannot exec ldd"); }
while (<P>) {
if (m,^\s+(\S+)\s+=>\s+(\S+)\s+\(0x.+\)?$,) {
$so2path{$1} = $2;
}
}
close(P); $? && subprocerr("ldd on \`$exec[$i]'");

If I understand this code correctly, it will go to subprocerr() and die if
ldd returns non-zero.

Any package that uses local libs will be affected. E.g. zlib (if built in
environment where zlib is not yet installed).
Steve Langasek
2006-01-25 00:35:54 UTC
Permalink
Post by Frank Lichtenheld
Post by Frank Lichtenheld
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...
This option wasneeded when package builds a binary that uses libraries
built from the same source package. In this case, libraries may not be
available outside of package build directory; so ldd called from
dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH.
Don't yet know how this situation is handled in your patched
dpkg-shlibdeps...
Hmm, but dpkg-shlibdeps already tries to take care of this by using
all local shlibs files. Can someone point me to a package that really
misbuilds without -l ? This would really help me to understand the
real problem.
If you don't handle the -l, you won't be able to resolve a full path for
these libs. If you have a package in this situation that's biarch and you
have local libs for both the 32bit and the 64bit targets, how will you know
which shlibs file to use if you don't look up the full library path and
examine the lib for target compatibility?
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
***@debian.org http://www.debian.org/
Frank Lichtenheld
2006-01-25 11:14:29 UTC
Permalink
[CCing Joey Hess. As debhelper maintainer he might want to add something
to the discussion and I don't know if he reads it already]
Post by Steve Langasek
If you don't handle the -l, you won't be able to resolve a full path for
these libs. If you have a package in this situation that's biarch and you
have local libs for both the 32bit and the 64bit targets, how will you know
which shlibs file to use if you don't look up the full library path and
examine the lib for target compatibility?
Yeah, that's indeed a problem. But that isn't solved by the current
implementation either. When I think about it there is now way the
-l option (if pointing to a directory that is not known to dpkg)
changes anything about the build currently since the local shlibs
information is considered before using the ldd paths. And the ldd paths
are only used in conjunction with dpkg --search ...
The only thing it does is supressing the error "couldn't find path for
X".

So -l is usefull in one case currently:
- If it points to an installed package that has a library which is
listed in its shlibs file but which is not in the normal systems
library paths (which maybe later is accessed via a wrapper that
sets LD_LIBRARY_PATH, e.g. like /usr/bin/firefox)
(does anyone use it this way?)
This one should be easy to fix and I will do it before uploading the
current dpkg to unstable.

And we could make it usefull in one other case:
- If there are several "local" packages that provide the same library
let the user choose one of them.

The 32bit vs. 64bit problem should be solved by checking the library's
binary format though. I will also fix this before uploading to unstable.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Joey Hess
2006-01-25 18:18:55 UTC
Permalink
Post by Frank Lichtenheld
Yeah, that's indeed a problem. But that isn't solved by the current
implementation either. When I think about it there is now way the
-l option (if pointing to a directory that is not known to dpkg)
changes anything about the build currently since the local shlibs
information is considered before using the ldd paths. And the ldd paths
are only used in conjunction with dpkg --search ...
The only thing it does is supressing the error "couldn't find path for
X".
You're right.

BTW, dpkg-shlibdeps -l does not seem to be documented in its manual page
or help text ATM.
--
see shy jo
Frank Lichtenheld
2006-01-25 20:10:54 UTC
Permalink
Post by Joey Hess
BTW, dpkg-shlibdeps -l does not seem to be documented in its manual page
or help text ATM.
I was talking about dh_shlibdeps -l and the LD_LIBRARY_PATH handling in
dpkg-shlibdeps. dpkg-shlibdeps has indeed no -l option. Sorry if that
was confusing.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Nikita V. Youshchenko
2006-01-24 15:05:43 UTC
Permalink
Post by Frank Lichtenheld
Post by Frank Lichtenheld
I've implemented this option. Patch and new script (since the patch is
garbled with a little code clean-up I did while going through the
script) are attached.
Comments and/or testing welcome.
Comment to myself: The current patch probably breaks dh_shlibdeps
-l option because it doesn't honor LD_LIBRARY_PATH. Can someone
tell me a package which really needs this option? The ones that
I tested so far seem to build equally fine without it...
This option wasneeded when package builds a binary that uses libraries
built from the same source package. In this case, libraries may not be
available outside of package build directory; so ldd called from
dpkg-shlibdeps won't find them witout LD_LIBRARY_PATH.

Don't yet know how this situation is handled in your patched
dpkg-shlibdeps...
Nikita V. Youshchenko
2006-01-20 08:57:28 UTC
Permalink
Hello people.
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
objdump isn't a solution either, while it sometimes can read the
other shared library, it doesn't provide the linker search patch
which is of critical importance to get this stuff right.
Hmm...
But what for is that search path?
As far as I understand, dpkg-shlibdeps should just get NEEDED sonames,
and match those against available shlibs files.
And that is what code actually does. Path information is used only if
something is not found in this way - that means, package that provides
the library does not provide shlibs data, which is a bug anyway.
This isn't quite true I think. The current dpkg-shlibdeps code works
1) use "ldd <binary>" to find the paths to the linked libraries
2) use "objdump -p <binary>" to actually check which of this libraries
are listed as NEEDED (Are there cases where ldd lists
libraries that are not NEEDED?)
3) check the shlibs.local and shlibs.overides files for matches
4) for remaining libraries search the corresponding package with
dpkg --search <path>
Here the ldd information is used!
5) check the shlibs files of the packages identified
Looks so. I was mistaken.
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
From reading the source code of dpkg-cross' version of dpkg-shlibdeps
c1) like 2)
c2) like 3)
c3) determine the path to the library by just prepending $crossroot or
$crossroot64 to it (both variables which can be set by the user)
c4) like 4)
c5) like 5)
I don't see how the current way this works can deal with several
library directories like /lib /usr/lib /usr/X11R6/lib. But maybe I'm
missing something?
In cross-compile environment created by dpkg-cross, all libraries are
installed in the same directory.

As for this directory being user-configurable, I believe it's a misfeature
came from old versions of dpkg-cross, which should be eventually fixed.
While it's there, depending on -arch-cross packages is unreliable, which
is something I wish to avoid.
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
So looks it is safe to remove any path processing from dpkg-slibdeps,
and use only objdump. If something breaks, it should be fixed
elsewhere (i.e. proper shlibs data added to packages).
If we don't use the path information from ldd there are several ways to
1) use "dpkg --search" but only with the library name from objdump, not
with the full path.
Questions: - Are there cases where the library name from objdump
isn't actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
2) we could try to use the ldconfig cache to make to work of ldd for
ourself.
Questions: - Is this really an advantage? Or has the cache the same
problems ldd has?
I sometimes have to work with MontaVista toolchains. They to provide
cross-ldd tool that just implements the same library-searching logic for
non-native binaries as dynamic libker implements for native ones.
I don't know if this thing is free etc, but it could be easilly implemented
from scratch if we decide we need one.
Post by Frank Lichtenheld
3) we could parse /etc/ld.so.conf and search for the libraries ourself.
this is essentially a more general version of the current way
dpkg-cross works AFAICS
4) we could just parse _all_ shlib files, perhaps even generating a
cache from them.
Questions: - Seems a complex solution (if it uses caching). Is any of
the simpler solutions feasible?
- How slow is this?
I believe this is *the* correct way.

If should be *much* faster than calling dpkg -S - just because dpkg -S
actually loads all *.list files, which is many times larger than *.shlibs
files.
As I side effect, this will allow to remove LD_LIBRARY_PATH thing from
dh_shlibdeps, which gives some unwanted effects when cross-compiling
packages (like finding cross version of libz.so.1 in LD_LIBRARY_PATH and
refusing to load native binaries that link against libz.so.1).

The only question is that is will *require* packages to provide valid
shlibs.local files. We may make an experiment: try to build entire archive
using dpkg-shlibdeps that is based on shlibs file checking only, and see
how many packages get different dependences.

One more question that came to mind right now: is it possible that more
than one installed package provides shlibs infomation for the same soname?
From the other hand, what is the wanted behaviour in this case?
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
1) use "dpkg --search" but only with the library name from objdump,
not with the full path.
Questions: - Are there cases where the library name from objdump
isn't actually the filename of the library?
- How do we decide wether a found file is really a usable
library? (parse /etc/ld.so.conf?)
Note that this method is used in the patch for #285857 which I'm
currently evaluating for inclusion. So it seems the answer to
question 1 is "no" (or else the patch in #285857 is useless)...
That patch fixes paths that dpkg -S will search.
That bug describes the following situation:
- objdump finds NEEDED libfoo.so.X
- ldd resolves that to /lib/libfoo.so.X.Y
- dpkg -S can't find /lib/libfoo.so.X.Y, because file existsing in dpkg's
database is /usr/lib/libfoo.so.X.Y; ldd finds it under /lib
because /usr/lib just a symlink to /lib under hurd (per bug reporter)

This has nothing to do about 'library name from objdump isn't actually the
filename of the library'.

As for that question, AFAIK library filename and soname are completely
unrelated. So there may be a soname 'ABCDE' that is provided by libxxx.so.
A binary linked against that library will get 'NEEDED ABCDE'. And dynamic
linker and ldd will find it, as long as there is 'ABCDE' symlink or file
in dynamic linker search path.
Post by Frank Lichtenheld
Post by Nikita V. Youshchenko
Yes. ldd will list all shared libraries pulled in by a binary,
regardless of whether they're NEEDED by the binary itself or just
And also LD_PRELOAD'd libraries, et cetera (which may include
fakeroot).
As far as I understand, dpkg-shlibdeps uses ldd only to get paths to
libraries. The list of the libraries is got using objdump.

Nikita
Steve Langasek
2006-01-20 11:36:36 UTC
Permalink
Post by Nikita V. Youshchenko
Post by Frank Lichtenheld
2) we could try to use the ldconfig cache to make to work of ldd for
ourself.
Questions: - Is this really an advantage? Or has the cache the same
problems ldd has?
Hmm. In theory, ldconfig shouldn't require the ability to execute 64-bit
binaries in order to build a cache of their paths. The only thing I don't
know is how 64-bit vs. 32-bit libs are cached? It would after all have to
be possible to distinguish between them in order to use the cache for this.
Post by Nikita V. Youshchenko
Post by Frank Lichtenheld
3) we could parse /etc/ld.so.conf and search for the libraries ourself.
this is essentially a more general version of the current way
dpkg-cross works AFAICS
I think the only question here is whether it's appropriate for dpkg to have
its own internal representation of the library search path, since that means
duplicating information between glibc and dpkg. IMHO, this is still the
least-bad option.
Post by Nikita V. Youshchenko
Post by Frank Lichtenheld
4) we could just parse _all_ shlib files, perhaps even generating a
cache from them.
Questions: - Seems a complex solution (if it uses caching). Is any of
the simpler solutions feasible?
- How slow is this?
I believe this is *the* correct way.
If should be *much* faster than calling dpkg -S - just because dpkg -S
actually loads all *.list files, which is many times larger than *.shlibs
files.
As I side effect, this will allow to remove LD_LIBRARY_PATH thing from
dh_shlibdeps, which gives some unwanted effects when cross-compiling
packages (like finding cross version of libz.so.1 in LD_LIBRARY_PATH and
refusing to load native binaries that link against libz.so.1).
The only question is that is will *require* packages to provide valid
shlibs.local files.
Hrm? It shouldn't; dpkg-shlibdeps is already capable of checking shlibs
files within the build directory.
Post by Nikita V. Youshchenko
One more question that came to mind right now: is it possible that more
than one installed package provides shlibs infomation for the same soname?
Yes, AFAIK *every* biarch package is going to provide shlibs for
libraries that differ only by library path, not by soname. So that's not
very useful for solving the problem at hand.
Post by Nikita V. Youshchenko
From the other hand, what is the wanted behaviour in this case?
If you aren't able to distinguish the libraries according to the path of the
library they provide, the *only* sensible thing to do in such a case is to
bail.
Post by Nikita V. Youshchenko
As for that question, AFAIK library filename and soname are completely
unrelated. So there may be a soname 'ABCDE' that is provided by libxxx.so.
A binary linked against that library will get 'NEEDED ABCDE'. And dynamic
linker and ldd will find it, as long as there is 'ABCDE' symlink or file
in dynamic linker search path.
Uh, consequently they are *not* unrelated: the soname specified in NEEDED
*must* be a filename that exists on the system, be it symlink or otherwise,
and per policy this file must be contained in the library package (as
opposed to just being created by ldconfig on package install).
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
***@debian.org http://www.debian.org/
Frank Lichtenheld
2006-01-20 12:12:57 UTC
Permalink
Post by Steve Langasek
Post by Frank Lichtenheld
2) we could try to use the ldconfig cache to make to work of ldd for
ourself.
Questions: - Is this really an advantage? Or has the cache the same
problems ldd has?
Hmm. In theory, ldconfig shouldn't require the ability to execute 64-bit
binaries in order to build a cache of their paths. The only thing I don't
know is how 64-bit vs. 32-bit libs are cached? It would after all have to
be possible to distinguish between them in order to use the cache for this.
It is possible:
***@feynman:~$ /sbin/ldconfig -p | grep libbz2.so.1.0
libbz2.so.1.0 (libc6,x86-64) => /usr/lib64/libbz2.so.1.0
libbz2.so.1.0 (libc6) => /lib/libbz2.so.1.0

Might still be a mess to parse all those architecture specific names,
though...

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Daniel Jacobowitz
2006-01-20 19:36:50 UTC
Permalink
Post by Nikita V. Youshchenko
I sometimes have to work with MontaVista toolchains. They to provide
cross-ldd tool that just implements the same library-searching logic for
non-native binaries as dynamic libker implements for native ones.
I don't know if this thing is free etc, but it could be easilly implemented
from scratch if we decide we need one.
It's not generally available; it was a fairly substantial patch to the
prelinker. There are other ways to do it.
--
Daniel Jacobowitz
CodeSourcery
Nikita V. Youshchenko
2006-01-24 16:06:47 UTC
Permalink
Post by Frank Lichtenheld
dpkg-shlibdeps calls ldd, which will just fail if LD_LIBRARY_PATH
won't point to directories with local libraries.
That's not true. ldd will just happily print "libfoo.so.1 => not found"
and exit with exit code 0. So this doesn't terminate dpkg-shlibdeps.
Hmm...
New information for me. I was sure that ldd, as any other [good-behaving]
unix utility, exits with non-zero status on errors. A bug in ldd?

But if ldd does not dislike unresolved libraries, I see no other problems
with dropping -l. Library files from non-standard paths won't be found by
dpkg anyway, so can't be processed in way other than shlibs.local
In this sence, new information that comes from ldd finding libraries in
LD_LIBRARY_PATH dirs, is useless.
Frank Lichtenheld
2006-01-24 16:29:14 UTC
Permalink
Post by Nikita V. Youshchenko
But if ldd does not dislike unresolved libraries, I see no other problems
with dropping -l. Library files from non-standard paths won't be found by
dpkg anyway, so can't be processed in way other than shlibs.local
shlibs.local isn't even normally needed in this case because dpkg-shlibdeps
happily searchs for _all_ shlibs files below debian/ before attempting
to search for "external" packages.

The problem with shlibs.local is that it easily produces packages
depending on themself since there is no check against that in the
code currently. For other "local" shlibs files dpkg-shlibdeps already
detects if they are from the same package as the executable and ignores
those dependencies.

I somehow doubt that many people know about these differences, though...
Post by Nikita V. Youshchenko
In this sence, new information that comes from ldd finding libraries in
LD_LIBRARY_PATH dirs, is useless.
Yeah, I'm still searching for a counter example myself.

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Continue reading on narkive:
Loading...