For discussion of Alpine Linux development and developer support

3 2

[alpine-devel] [PATCH 0/1] apk upgrade has issues when old deps still available

A. Wilcox
Details
Message ID
<20190601042516.56133-1-AWilcox@Wilcox-Tech.com>
Sender timestamp
1559363115
DKIM signature
missing
Download raw message
When running `apk upgrade -al` only packages listed in the world file are
forced to upgrade.  If a package in world has a dependency that is eligible
for upgrade, but the currently installed version is also still available in
the repository, it will not be considered for upgrade.

This causes considerable breakage for at least Adélie Linux; we keep built
packages around on the mirrors indefinitely.  This breakage was noticed
during upgrades of 1.0-BETA2 to 1.0-BETA3, and the solver log (using
DEBUG_PRINT) can be seen at [1].

The patch I am sending fixes this issue for us by adding an option to the
upgrade applet, '-d' or '--deep', which causes the solver to prefer newer
versions of all selected packages when available.  The difference in the
solver log can be seen at [2].

[1]: https://www.adelielinux.org/storage/apk-error.log
[2]: https://www.adelielinux.org/storage/deep.diff

A. Wilcox (1):
  upgrade: add --deep option to upgrade everything

 src/apk_solver.h |  1 +
 src/solver.c     | 10 ++++++++++
 src/upgrade.c    |  5 +++++
 3 files changed, 16 insertions(+)

-- 
2.21.0



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] [PATCH 1/1] upgrade: add --deep option to upgrade everything

A. Wilcox
Details
Message ID
<20190601042516.56133-2-AWilcox@Wilcox-Tech.com>
In-Reply-To
<20190601042516.56133-1-AWilcox@Wilcox-Tech.com> (view parent)
Sender timestamp
1559363116
DKIM signature
missing
Download raw message
Patch: +16 -0
---
 src/apk_solver.h |  1 +
 src/solver.c     | 10 ++++++++++
 src/upgrade.c    |  5 +++++
 3 files changed, 16 insertions(+)

diff --git a/src/apk_solver.h b/src/apk_solver.h
index b8b072d..908b3fd 100644
--- a/src/apk_solver.h
+++ b/src/apk_solver.h
@@ -35,6 +35,7 @@ struct apk_changeset {
 #define APK_SOLVERF_REINSTALL		0x0004
 #define APK_SOLVERF_LATEST		0x0008
 #define APK_SOLVERF_IGNORE_CONFLICT	0x0010
+#define APK_SOLVERF_DEEP		0x0020
 
 void apk_solver_set_name_flags(struct apk_name *name,
 			       unsigned short solver_flags,
diff --git a/src/solver.c b/src/solver.c
index e10cf8b..8437d61 100644
--- a/src/solver.c
+++ b/src/solver.c
@@ -40,6 +40,7 @@ struct apk_solver_state {
 	unsigned int pinning_inherit;
 	unsigned int default_repos;
 	unsigned ignore_conflict : 1;
+	unsigned deep_upgrade : 1;
 };
 
 static struct apk_provider provider_none = {
@@ -510,6 +511,14 @@ static int compare_providers(struct apk_solver_state *ss,
 	unsigned int solver_flags;
 	int r;
 
+	/* In deep upgrades, always return the greater version */
+	if (ss->deep_upgrade)
+		switch (apk_version_compare_blob(*pA->version, *pB->version)) {
+			case APK_VERSION_LESS:
+				return -1;
+			case APK_VERSION_GREATER:
+				return 1;
+		}
 
 	/* Prefer existing package */
 	if (pkgA == NULL || pkgB == NULL)
@@ -1006,6 +1015,7 @@ restart:
 	ss->changeset = changeset;
 	ss->default_repos = apk_db_get_pinning_mask_repos(db, APK_DEFAULT_PINNING_MASK);
 	ss->ignore_conflict = !!(solver_flags & APK_SOLVERF_IGNORE_CONFLICT);
+	ss->deep_upgrade = !!(solver_flags & APK_SOLVERF_DEEP);
 	list_init(&ss->dirty_head);
 	list_init(&ss->unresolved_head);
 
diff --git a/src/upgrade.c b/src/upgrade.c
index 14457b5..e48d8e3 100644
--- a/src/upgrade.c
+++ b/src/upgrade.c
@@ -38,6 +38,9 @@ static int option_parse_applet(void *ctx, struct apk_db_options *dbopts, int opt
 	case 'a':
 		uctx->solver_flags |= APK_SOLVERF_AVAILABLE;
 		break;
+	case 'd':
+		uctx->solver_flags |= APK_SOLVERF_DEEP;
+		break;
 	case 'l':
 		uctx->solver_flags |= APK_SOLVERF_LATEST;
 		break;
@@ -59,6 +62,8 @@ static const struct apk_option options_applet[] = {
 	{ 0x10000, "no-self-upgrade",
 	  "Do not do early upgrade of 'apk-tools' package" },
 	{ 0x10001, "self-upgrade-only", "Only do self-upgrade" },
+	{ 'd', "deep",
+	  "Include dependencies when upgrading world" },
 };
 
 static const struct apk_option_group optgroup_applet = {
-- 
2.21.0



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
A. Wilcox
Details
Message ID
<0c8c6097-92d5-c796-3e31-f5dfcbed3081@Wilcox-Tech.com>
In-Reply-To
<20190603094533.782ca468@vostro> (view parent)
Sender timestamp
1559556270
DKIM signature
missing
Download raw message
On 06/03/19 01:45, Timo Teras wrote:
> Hi,
> 
> On Fri, 31 May 2019 23:25:15 -0500
> "A. Wilcox" <AWilcox@Wilcox-Tech.com> wrote:
> 
>> When running `apk upgrade -al` only packages listed in the world file
>> are forced to upgrade.  If a package in world has a dependency that
>> is eligible for upgrade, but the currently installed version is also
>> still available in the repository, it will not be considered for
>> upgrade.
>>
>> This causes considerable breakage for at least Adélie Linux; we keep
>> built packages around on the mirrors indefinitely.  This breakage was
>> noticed during upgrades of 1.0-BETA2 to 1.0-BETA3, and the solver log
>> (using DEBUG_PRINT) can be seen at [1].
>>
>> The patch I am sending fixes this issue for us by adding an option to
>> the upgrade applet, '-d' or '--deep', which causes the solver to
>> prefer newer versions of all selected packages when available.  The
>> difference in the solver log can be seen at [2].
>>
>> [1]: https://www.adelielinux.org/storage/apk-error.log
>> [2]: https://www.adelielinux.org/storage/deep.diff
> 
> Thanks for the debug logs, and analysis so far. I have seen this also
> once or twice and been wondering what is causing it.
> 
> However, instead of adding yet another switch. I'd like to fix the root
> cause and make this default behaviour. Furthermore, I think the patch
> suggested breaks at least package pinning.
> 
> Was any package pinning used?
> 
> Is there any cycles, or errors in the repository? E.g. does "apk dot
> --errors --installed" see anything problematic?
> 
> Do you have an easy way for me to duplicate this setup, to further
> analyze the issue?
> 
> It is likely related to the solver doing things in wrong order which
> causes some of the dependency inherited flags to not be available at
> the time when the dependency gets processed.
> 
> Thanks,
> Timo


We rarely use package pinning over here, so it wasn't in use on any of
the computers that hit this bug.

The easiest way that I can think of for you to duplicate it would be to
install Adélie beta2 with qt5-qtbase-dev in a chroot, then attempt to
upgrade it to beta3:

# mkdir /test

# apk --root /test -X
https://distfiles.adelielinux.org/adelie/1.0-beta2/system -X
https://distfiles.adelielinux.org/adelie/1.0-beta2/user
--allow-untrusted --initdb add adelie-base qt5-qtbase-dev

# apk --root /test -X
https://distfiles.adelielinux.org/adelie/1.0-beta3/system -X
https://distfiles.adelielinux.org/adelie/1.0-beta3/user upgrade -al

On the x86_64 machine I tested this on, there was a slightly different
error, but the same cause/effect:

Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.10.3-r0 -> 2.10.3-r1)
Continuing the upgrade transaction with new apk-tools:
ERROR: unsatisfiable constraints:
  libpcrecpp-8.42-r1:
    breaks: pcre-dev-8.43-r0[libpcrecpp=8.43-r0]

(If you don't want to use --allow-untrusted for the initial install, you
can manually extract our adelie-keys apk.)


It looks like there are two cycles shown in dot, one that we should
probably fix (util-linux deps on libfdisk which deps on util-linux) and
one that isn't really fixable (bash deps on /bin/sh because it has an
install script, and /bin/sh is provided by bash-binsh which is part of
bash).  Neither of these seem to be related to the issue, but I may be
wrong.


I would really like to have this be the default behaviour but was unsure
if it would be desired; I'm glad to hear that it is.  Thank you for
looking in to this.  If you need more information, let me know.


Best to you and yours,
--arw


-- 
A. Wilcox (awilfox)
Open-source programmer (C, C++, Python)
https://code.foxkit.us/u/awilfox/
Timo Teras
Details
Message ID
<20190603094533.782ca468@vostro>
In-Reply-To
<20190601042516.56133-1-AWilcox@Wilcox-Tech.com> (view parent)
Sender timestamp
1559544333
DKIM signature
missing
Download raw message
Hi,

On Fri, 31 May 2019 23:25:15 -0500
"A. Wilcox" <AWilcox@Wilcox-Tech.com> wrote:

> When running `apk upgrade -al` only packages listed in the world file
> are forced to upgrade.  If a package in world has a dependency that
> is eligible for upgrade, but the currently installed version is also
> still available in the repository, it will not be considered for
> upgrade.
> 
> This causes considerable breakage for at least Adélie Linux; we keep
> built packages around on the mirrors indefinitely.  This breakage was
> noticed during upgrades of 1.0-BETA2 to 1.0-BETA3, and the solver log
> (using DEBUG_PRINT) can be seen at [1].
> 
> The patch I am sending fixes this issue for us by adding an option to
> the upgrade applet, '-d' or '--deep', which causes the solver to
> prefer newer versions of all selected packages when available.  The
> difference in the solver log can be seen at [2].
> 
> [1]: https://www.adelielinux.org/storage/apk-error.log
> [2]: https://www.adelielinux.org/storage/deep.diff

Thanks for the debug logs, and analysis so far. I have seen this also
once or twice and been wondering what is causing it.

However, instead of adding yet another switch. I'd like to fix the root
cause and make this default behaviour. Furthermore, I think the patch
suggested breaks at least package pinning.

Was any package pinning used?

Is there any cycles, or errors in the repository? E.g. does "apk dot
--errors --installed" see anything problematic?

Do you have an easy way for me to duplicate this setup, to further
analyze the issue?

It is likely related to the solver doing things in wrong order which
causes some of the dependency inherited flags to not be available at
the time when the dependency gets processed.

Thanks,
Timo



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---