Subject: [alpine-devel] [PATCH 0/1] apk upgrade has issues when old deps still available

From: "A. Wilcox" <[email-masked]>
Date: Fri, 31 May 2019 23:25:15 -0500

When running `apk upgrade -al` only packages listed in the world file are
forced to upgrade.  If a package in world has a dependency that is eligible
for upgrade, but the currently installed version is also still available in
the repository, it will not be considered for upgrade.

This causes considerable breakage for at least Adélie Linux; we keep built
packages around on the mirrors indefinitely.  This breakage was noticed
during upgrades of 1.0-BETA2 to 1.0-BETA3, and the solver log (using
DEBUG_PRINT) can be seen at [1].

The patch I am sending fixes this issue for us by adding an option to the
upgrade applet, '-d' or '--deep', which causes the solver to prefer newer
versions of all selected packages when available.  The difference in the
solver log can be seen at [2].

[1]: https://www.adelielinux.org/storage/apk-error.log
[2]: https://www.adelielinux.org/storage/deep.diff

A. Wilcox (1):
  upgrade: add --deep option to upgrade everything

 src/apk_solver.h |  1 +
 src/solver.c     | 10 ++++++++++
 src/upgrade.c    |  5 +++++
 3 files changed, 16 insertions(+)

-- 
2.21.0

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: "A. Wilcox" <[email-masked]>
Date: Fri, 31 May 2019 23:25:16 -0500

---
 src/apk_solver.h |  1 +

diff --git a/src/apk_solver.h b/src/apk_solver.h
index b8b072d..908b3fd 100644
--- a/src/apk_solver.h
+++ b/src/apk_solver.h
@@ -35,6 +35,7 @@ struct apk_changeset {
 #define APK_SOLVERF_REINSTALL		0x0004
 #define APK_SOLVERF_LATEST		0x0008
 #define APK_SOLVERF_IGNORE_CONFLICT	0x0010
+#define APK_SOLVERF_DEEP		0x0020
 
 void apk_solver_set_name_flags(struct apk_name *name,
 			       unsigned short solver_flags,
diff --git a/src/solver.c b/src/solver.c
index e10cf8b..8437d61 100644
--- a/src/solver.c
+++ b/src/solver.c
@@ -40,6 +40,7 @@ struct apk_solver_state {
 	unsigned int pinning_inherit;
 	unsigned int default_repos;
 	unsigned ignore_conflict : 1;
+	unsigned deep_upgrade : 1;
 };
 
 static struct apk_provider provider_none = {
@@ -510,6 +511,14 @@ static int compare_providers(struct apk_solver_state *ss,
 	unsigned int solver_flags;
 	int r;
 
+	/* In deep upgrades, always return the greater version */
+	if (ss->deep_upgrade)
+		switch (apk_version_compare_blob(*pA->version, *pB->version)) {
+			case APK_VERSION_LESS:
+				return -1;
+			case APK_VERSION_GREATER:
+				return 1;
+		}
 
 	/* Prefer existing package */
 	if (pkgA == NULL || pkgB == NULL)
@@ -1006,6 +1015,7 @@ restart:
 	ss->changeset = changeset;
 	ss->default_repos = apk_db_get_pinning_mask_repos(db, APK_DEFAULT_PINNING_MASK);
 	ss->ignore_conflict = !!(solver_flags & APK_SOLVERF_IGNORE_CONFLICT);
+	ss->deep_upgrade = !!(solver_flags & APK_SOLVERF_DEEP);
 	list_init(&ss->dirty_head);
 	list_init(&ss->unresolved_head);
 
diff --git a/src/upgrade.c b/src/upgrade.c
index 14457b5..e48d8e3 100644
--- a/src/upgrade.c
+++ b/src/upgrade.c
@@ -38,6 +38,9 @@ static int option_parse_applet(void *ctx, struct apk_db_options *dbopts, int opt
 	case 'a':
 		uctx->solver_flags |= APK_SOLVERF_AVAILABLE;
 		break;
+	case 'd':
+		uctx->solver_flags |= APK_SOLVERF_DEEP;
+		break;
 	case 'l':
 		uctx->solver_flags |= APK_SOLVERF_LATEST;
 		break;
@@ -59,6 +62,8 @@ static const struct apk_option options_applet[] = {
 	{ 0x10000, "no-self-upgrade",
 	  "Do not do early upgrade of 'apk-tools' package" },
 	{ 0x10001, "self-upgrade-only", "Only do self-upgrade" },
+	{ 'd', "deep",
+	  "Include dependencies when upgrading world" },
 };
 
 static const struct apk_option_group optgroup_applet = {
-- 
2.21.0

---

---

From: Timo Teras <[email-masked]>
Date: Mon, 3 Jun 2019 09:45:33 +0300

Hi,

On Fri, 31 May 2019 23:25:15 -0500
"A. Wilcox" <[email-masked]> wrote:

> When running `apk upgrade -al` only packages listed in the world file
> are forced to upgrade.  If a package in world has a dependency that
> is eligible for upgrade, but the currently installed version is also
> still available in the repository, it will not be considered for
> upgrade.
> 
> This causes considerable breakage for at least Adélie Linux; we keep
> built packages around on the mirrors indefinitely.  This breakage was
> noticed during upgrades of 1.0-BETA2 to 1.0-BETA3, and the solver log
> (using DEBUG_PRINT) can be seen at [1].
> 
> The patch I am sending fixes this issue for us by adding an option to
> the upgrade applet, '-d' or '--deep', which causes the solver to
> prefer newer versions of all selected packages when available.  The
> difference in the solver log can be seen at [2].
> 

Thanks for the debug logs, and analysis so far. I have seen this also
once or twice and been wondering what is causing it.

However, instead of adding yet another switch. I'd like to fix the root
cause and make this default behaviour. Furthermore, I think the patch
suggested breaks at least package pinning.

Was any package pinning used?

Is there any cycles, or errors in the repository? E.g. does "apk dot
--errors --installed" see anything problematic?

Do you have an easy way for me to duplicate this setup, to further
analyze the issue?

It is likely related to the solver doing things in wrong order which
causes some of the dependency inherited flags to not be available at
the time when the dependency gets processed.

Thanks,
Timo

---

---

From: "A. Wilcox" <[email-masked]>
Date: Mon, 3 Jun 2019 05:04:30 -0500

> the time when the dependency gets processed.

We rarely use package pinning over here, so it wasn't in use on any of
the computers that hit this bug.

The easiest way that I can think of for you to duplicate it would be to
install Adélie beta2 with qt5-qtbase-dev in a chroot, then attempt to
upgrade it to beta3:

# mkdir /test

# apk --root /test -X
https://distfiles.adelielinux.org/adelie/1.0-beta2/system -X
https://distfiles.adelielinux.org/adelie/1.0-beta2/user
--allow-untrusted --initdb add adelie-base qt5-qtbase-dev

# apk --root /test -X
https://distfiles.adelielinux.org/adelie/1.0-beta3/system -X
https://distfiles.adelielinux.org/adelie/1.0-beta3/user upgrade -al

On the x86_64 machine I tested this on, there was a slightly different
error, but the same cause/effect:

Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.10.3-r0 -> 2.10.3-r1)
Continuing the upgrade transaction with new apk-tools:
ERROR: unsatisfiable constraints:
  libpcrecpp-8.42-r1:
    breaks: pcre-dev-8.43-r0[libpcrecpp=8.43-r0]

(If you don't want to use --allow-untrusted for the initial install, you
can manually extract our adelie-keys apk.)

It looks like there are two cycles shown in dot, one that we should
probably fix (util-linux deps on libfdisk which deps on util-linux) and
one that isn't really fixable (bash deps on /bin/sh because it has an
install script, and /bin/sh is provided by bash-binsh which is part of
bash).  Neither of these seem to be related to the issue, but I may be
wrong.

I would really like to have this be the default behaviour but was unsure
if it would be desired; I'm glad to hear that it is.  Thank you for
looking in to this.  If you need more information, let me know.

Best to you and yours,
--arw

-- 
A. Wilcox (awilfox)
Open-source programmer (C, C++, Python)
https://code.foxkit.us/u/awilfox/