Subject: [alpine-user] Startup hangs (aports: sshd?)

From: Steffen Nurpmeso <[email-masked]>
Date: Thu, 17 Jan 2019 15:02:07 +0100

Hello.

After upgrading to [edge] i see a possibly endless hang upon
startup, which seems to be caused by PRNG init.  I have seen the
AlpineLinux bug report from two years ago.  I have seen hangs of
about one to two minutes on the VM since, hmm, not too long, maybe
two or three months?, and on the bare metal we do hang also about
a minute.  I presumed it was about PRNG seeding, and, whereas
i totally dislike that the precious entropy pool i save in between
boots does not count at all, geeh, what can i do about that...
Anyway.

So after upgrading [edge] today i get a possible endless hang, it
was about eleven minutes before my "homebrew watchdog" caused
starting of other services, which then overcame the seeding:

  [   10.386116] random: dnsmasq: uninitialized urandom read (128 bytes read)
  [  673.619579] random: crng init done

And this:

  Jan 17 14:04:14 init: starting pid 1997, tty '': '/sbin/openrc default'
  ...
  Jan 17 14:15:09 /etc/init.d/lighttpd[2355]: lighttpd: waiting for sshd (50 seconds)
  Jan 17 14:15:18 /etc/init.d/lighttpd[2355]: lighttpd: waiting for sshd (41 seconds)
  Jan 17 14:15:20 lighttpd[2406]: (server.c.1457) server started (lighttpd/1.4.52)

Eleven minutes!  Maybe longer if left alone!  So if it is like
that, (why does lighttpd wait for sshd via init.d, btw.??), and if
people are about to install haveged or whatever, wouldn't it make
sense to ensure that jitterentropy_rng is loaded during boot, and
rmmoded later on, unless it is also in /etc/modules?

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Natanael Copa <[email-masked]>
Date: Sat, 19 Jan 2019 00:47:24 +0000

On Thu, 17 Jan 2019 15:02:07 +0100
Steffen Nurpmeso <[email-masked]> wrote:

> startup, which seems to be caused by PRNG init. 

Try add random.trust_cpu=1 as boot option.

See:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.19.y&id=39a8883a2b989d1d21bd8dd99f5557f0c5e89694

and:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.19.y&id=9b25436662d5fb4c66eb527ead53cab15f596ee0

-nc

> I have seen the

---

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 19 Jan 2019 18:15:35 +0100

Natanael Copa wrote in <[email-masked]>:
 |On Thu, 17 Jan 2019 15:02:07 +0100
 |Steffen Nurpmeso <[email-masked]> wrote:
 |
 |> Hello.
 |> 
 |> After upgrading to [edge] i see a possibly endless hang upon
 |> startup, which seems to be caused by PRNG init. 
 |
 |Try add random.trust_cpu=1 as boot option.

Will do when i finally can upgrade to 4.19.y series.  Merci.

 |See:
 |https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/\
 |?h=linux-4.19.y&id=39a8883a2b989d1d21bd8dd99f5557f0c5e89694
 |
 |and:
 |https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/\
 |?h=linux-4.19.y&id=9b25436662d5fb4c66eb527ead53cab15f596ee0

I am back at 4.14.89 since the nf_conntrack crashes persist, and
even 4.19.16 does not seem to have anything related.  (4.19.15
had, but crashes, ehm, persist.)  That is: i have to wait.

But ok i mean all that seems to be problematic, for example with
the program i have posted:

  #?64|xhost:/etc/init.d# /sbin/entropy-saver save
  - 1588 bits of entropy available at /dev/random
  - 2422 bits of entropy remain at /dev/random
  Insufficient entropy to save from /dev/random (-834 bits)

Two adjacent ioctl() asking for entropy_avail with just one read
(!) of /dev/random in between causes the entropy to explode!
This _cannot_ be right, can it?
Thereafter it works for the second try:

  #?0|xhost:/etc/init.d# /sbin/entropy-saver save
  - 2435 bits of entropy available at /dev/random
  - 1411 bits of entropy remain at /dev/random
  - 1024 bits / 80 bytes of entropy saved to /var/lib/misc/random.dat

Very weakly rated, this stuff.
Say, isn't that flaky?  This VM really crawls along, with
jitterentropy, with haveged, and with the above.  While i am
writing, the entropy went down to 1343, with only postfix
(urandom), mailman (1000 x urandom), dnsmasq (using internal surf
rand with one-time seeding), lighttpd (uses getrandom/urandom)
and cgit running.  Whereas my laptop is stable at 38xy.

So sorry to bother you all with this!
I will turn to use linux-virt on the VM.  Sorry for the noise.

A nice weekend i wish.

--steffen
|

---

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 19 Jan 2019 00:00:12 +0100

Hello again.

Steffen Nurpmeso wrote in <[email-masked]>:
 |After upgrading to [edge] i see a possibly endless hang upon
 |startup, which seems to be caused by PRNG init.  I have seen the
 |AlpineLinux bug report from two years ago.  I have seen hangs of
 |about one to two minutes on the VM since, hmm, not too long, maybe
 |two or three months?, and on the bare metal we do hang also about
 ..

So what am i supposed to do, entirely switch to linux-virt on the
VM?  I mean, ok, i could do that, but for example on this laptop
i have multiple installations, and can either boot them from
within each other via qemu, as well as via refit.

Anyway, i have written something that saves entropy in a file and
fills it in via ioctl(2), the stuff goes

  #?0|essex:tmp$ ./zt save /tmp/t.dat
  - 3420 bits of entropy available at /dev/random
  - 2396 bits of entropy remain at /dev/random
  - 1024 bits / 115 bytes of entropy saved to /tmp/t.dat
...
  #?0|essex:/etc/init.d# /tmp/zt load /tmp/t.dat
  - 2631 bits of entropy available at /dev/random
  - 115 bytes / 1024 bits of entropy read from /tmp/t.dat
  - Entropy added
  - 2905 bits of entropy are at at /dev/random

So call this two hour hack stupid, but i will use it.
I will look out for linux-virt after the next kernel update.
A nice weekend i wish.

/*@ Save and load Linux (2.6.0+) entropy.
 *@ Synopsis: entropy-saver save [file]
 *@ Synopsis: entropy-saver load [file]
 *@ "file" defaults to a_RAND_FILE_STORE.
 *
 * Copyright (c) 2019 Steffen (Daode) Nurpmeso <[email-masked]>.
 * SPDX-License-Identifier: ISC
 *
 * Permission to use, copy, modify, and/or distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

/* Random device*/
#define a_RAND_DEV "/dev/random"

/* Maximum number of bytes we handle (must be LT INT_MAX/8!) */
#define a_RAND_NO_BYTES 512

/* Default storage */
#define a_RAND_FILE_STORE "/var/lib/misc/random.dat"

#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sysexits.h>
#include <unistd.h>

#include <linux/random.h>
#include <linux/version.h>

#if KERNEL_VERSION(2,6,0) >= LINUX_VERSION_CODE
# error Linux kernel version and IOCTL usage incompatible.
#endif

static char const a_rand_file_store[] = a_RAND_FILE_STORE;

int
main(int argc, char **argv){
   enum {a_LOAD, a_SAVE};

   struct{
      struct rand_pool_info rpi;
      char buf[a_RAND_NO_BYTES];
   } x;
   char *rpibuf; /* To make C++ happy: rpi.buf is char[0] */
   ssize_t len;
   char const *store;
   int rv, accu, randfd, storfd, iocarg;

   if(argc < 2 || argc > 3)
      goto jeuse;

   if(!strcmp(argv[1], "load"))
      accu = a_LOAD;
   else if(!strcmp(argv[1], "save"))
      accu = a_SAVE;
   else{
jeuse:
      fprintf(stderr,
         "Synopsis: entropy-saver save [storage-file]\n"
         "Synopsis: entropy-saver load [storage-file]\n"
         "\n"
         "storage-file defaults to " a_RAND_FILE_STORE "\n"
         "Exit: sysexits.h: EX_USAGE, EX_NOPERM, EX_IOERR, EX_TEMPFAIL\n");
      rv = EX_USAGE;
      goto jleave;
   }

   randfd = open(a_RAND_DEV, (O_RDONLY | O_NONBLOCK));
   if(randfd == -1){
      accu = errno;
      fprintf(stderr, "Failed to open " a_RAND_DEV ": %s\n",
         strerror(accu));
      rv = (accu == EACCES || accu == EPERM) ? EX_NOPERM : EX_IOERR;
      goto jleave;
   }

   store = (argv[2] != NULL) ? argv[2] : a_rand_file_store;
   storfd = open(store, (accu == a_LOAD ? O_RDONLY
         : O_WRONLY | O_CREAT | O_TRUNC), (S_IRUSR | S_IWUSR));
   if(storfd == -1){
      accu = errno;
      fprintf(stderr, "Failed to open %s: %s\n", store, strerror(accu));
      rv = (accu == EACCES || accu == EPERM) ? EX_NOPERM : EX_IOERR;
      goto jerr1;
   }

   /* For at least statistics */
   rv = ioctl(randfd, (int)RNDGETENTCNT, &iocarg);
   if(rv == -1){
      fprintf(stderr, "Failed to query available entropy of " a_RAND_DEV
         ": %s\n", strerror(errno));
      rv = EX_IOERR;
      goto jerr2;
   }
   x.rpi.entropy_count = iocarg;

   rpibuf = (char*)x.rpi.buf;

   if(accu == a_LOAD){
      printf("- %d bits of entropy available at " a_RAND_DEV "\n",
         x.rpi.entropy_count);

      /* INT: entropy bits */
      len = read(storfd, &x.rpi.entropy_count, sizeof x.rpi.entropy_count);
      if(len == -1){
         fprintf(stderr, "Failed to read from %s: %s\n",
            store, strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      if(len != sizeof x.rpi.entropy_count){
         fprintf(stderr, "Storage %s: seems corrupted (false format)\n",
            store);
         rv = EX_IOERR;
         goto jerr2;
      }
      if(x.rpi.entropy_count < 128 || x.rpi.entropy_count > a_RAND_NO_BYTES*8){
         fprintf(stderr, "Storage %s seems corrupted (%d entropy bits)\n",
            store, x.rpi.entropy_count);
         rv = EX_IOERR;
         goto jerr2;
      }

      /* REST: pool bytes */
      len = read(storfd, rpibuf, sizeof x.buf);
      if(len == -1){
         fprintf(stderr, "Failed to read from %s: %s\n",
            store, strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      if(len == 0){
         fprintf(stderr, "Storage %s: seems corrupted (no entropy)\n",
            store);
         rv = EX_TEMPFAIL;
         goto jerr2;
      }
      x.rpi.buf_size = (int)len;

      if(read(storfd, &accu, 1) != 0){
         fprintf(stderr, "Storage %s: seems corrupted (no EOF seen)\n",
            store);
         rv = EX_IOERR;
         goto jerr2;
      }

      printf("- %d bytes / %d bits of entropy read from %s\n",
         x.rpi.buf_size, x.rpi.entropy_count, store);

      accu = ioctl(randfd, RNDADDENTROPY, &x.rpi);
      if(accu == -1){
         fprintf(stderr, "Failed to add %d bits of entropy to " a_RAND_DEV
            ": %s\n", x.rpi.entropy_count, strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }

      printf("- Entropy added\n");

      /* For at least statistics */
      rv = ioctl(randfd, (int)RNDGETENTCNT, &iocarg);
      if(rv != -1)
         printf("- %d bits of entropy are at at " a_RAND_DEV "\n", iocarg);
   }else{
      printf("- %d bits of entropy available at " a_RAND_DEV "%s\n",
         x.rpi.entropy_count,
         (x.rpi.entropy_count <= 128 ? ": temporary failure" : ""));
      if(x.rpi.entropy_count <= 128){
         rv = EX_TEMPFAIL;
         goto jerr2;
      }

      len = read(randfd, rpibuf, sizeof x.buf);
      if(len == -1){
         fprintf(stderr, "Failed to read from " a_RAND_DEV ": %s\n",
            strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      x.rpi.buf_size = (int)len;

      rv = ioctl(randfd, (int)RNDGETENTCNT, &iocarg);
      if(rv == -1){
         fprintf(stderr, "Failed to query remaining entropy of " a_RAND_DEV
            ": %s\n", strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      printf("- %d bits of entropy remain at " a_RAND_DEV "\n", iocarg);
      x.rpi.entropy_count -= iocarg;

      if(x.rpi.entropy_count <= 128){
         fprintf(stderr, "Insufficient entropy to save from " a_RAND_DEV
            " (%d bits)\n", x.rpi.entropy_count);
         rv = EX_TEMPFAIL;
         goto jerr2;
      }

      if((ssize_t)sizeof(x.rpi.entropy_count) != write(storfd,
               &x.rpi.entropy_count, sizeof x.rpi.entropy_count) || 
            len != write(storfd, rpibuf, len)){
         fprintf(stderr, "Failed to write to %s: %s\n",
            store, strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }

      printf("- %d bits / %d bytes of entropy saved to %s\n",
         x.rpi.entropy_count, x.rpi.buf_size, store);
   }

   rv = 0;
jerr2:
   if(close(storfd) == -1)
      fprintf(stderr, "Error closing %s: %s\n", store, strerror(errno));
jerr1:
   if(close(randfd) == -1)
      fprintf(stderr, "Error closing " a_RAND_DEV ": %s\n",
         strerror(errno));
jleave:
   return rv;
}

/* s-it-mode */

--steffen
|

---

---

From: Steffen Nurpmeso <[email-masked]>
Date: Mon, 21 Jan 2019 22:00:48 +0100

Steffen Nurpmeso wrote in <[email-masked]>:
  ...
 |Anyway, i have written something that saves entropy in a file and
 |fills it in via ioctl(2), the stuff goes
 |
 |  #?0|essex:tmp$ ./zt save /tmp/t.dat
 |  - 3420 bits of entropy available at /dev/random
 |  - 2396 bits of entropy remain at /dev/random
 |  - 1024 bits / 115 bytes of entropy saved to /tmp/t.dat
 |...
 |  #?0|essex:/etc/init.d# /tmp/zt load /tmp/t.dat
 |  - 2631 bits of entropy available at /dev/random
 |  - 115 bytes / 1024 bits of entropy read from /tmp/t.dat
 |  - Entropy added
 |  - 2905 bits of entropy are at at /dev/random
 |
 |So call this two hour hack stupid, but i will use it.

Because i have posted it, please let me post an update which will
read more bytes than the latter, as many as possible above some
treshold etc. (since the program is meant as a pool server, say).
My notebook booted without any hang today!  And if people can
access the hardware / mount the device, well, in my case (no
crypto all-through) they could change the kernel or whatever.
Ciao!

/*@ Save and load Linux (2.6.0+) entropy.
 *@ Different to "cat XY > /dev/urandom" this increments "entropy_avail".
 *@ Synopsis: entropy-saver save [file]
 *@ Synopsis: entropy-saver load [file]
 *@ "file" defaults to a_RAND_FILE_STORE.
 *
 * 2019 Steffen (Daode) Nurpmeso <[email-masked]>.
 * Public Domain
 */

/* Random device*/

/* When saving, the minimum number of entropy_avail we keep in the pool.
 * _This_ is checked after we have read once (512 we test initially).
 * We will refuse to save a dump which offers less than 128 bits. */
#define a_RAND_ENTROPY_COUNT_MIN 1024

/* Default storage */

#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sysexits.h>
#include <unistd.h>

int

   struct{

   /* Command line handling */
   if(argc < 2 || argc > 3)
      goto jeuse;

   /* Open our two files according to chosen action */
   randfd = open(a_RAND_DEV, (O_RDONLY | O_NONBLOCK));
   if(randfd == -1){
      accu = errno;
      fprintf(stderr, "Failed to open " a_RAND_DEV ": %s\n",
         strerror(accu));
      rv = (accu == EACCES || accu == EPERM) ? EX_NOPERM : EX_IOERR;
      goto jleave;
   }

   /* For at least statistics query entropy count once */

   if(accu == a_LOAD){

      /* INT: entropy bits */
      len = read(storfd, &x.rpi.entropy_count, sizeof x.rpi.entropy_count);
      if(len == -1){
         fprintf(stderr, "Failed to read from %s: %s\n",
            store, strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      if(len != sizeof x.rpi.entropy_count){
         fprintf(stderr, "Storage %s: seems corrupted (false format)\n",
            store);
         rv = EX_IOERR;
         goto jerr2;
      }
      /* The former because we will refuse to save less than that, the latter
       * rather arbitrary (like the suff as such?) */
      if(x.rpi.entropy_count < 128 || x.rpi.entropy_count > 1000000){
         fprintf(stderr, "Storage %s seems corrupted (%d entropy bits)\n",
            store, x.rpi.entropy_count);
         rv = EX_IOERR;
         goto jerr2;
      }

      /* REST: pool bytes */

      /* For at least statistics */
      rv = ioctl(randfd, (int)RNDGETENTCNT, &iocarg);
      if(rv != -1)
         printf("- %d bits of entropy are at at " a_RAND_DEV "\n", iocarg);
   }else{
      /* Since we are reading in non-blocking mode, and since reading from
       * /dev/random returns not that much in this mode, read in a loop until
       * it no longer serves / the entropy count falls under a a_RAND_ */
      size_t rem_size;
      int entrop_cnt;

      entrop_cnt = x.rpi.entropy_count;
      printf("- %d bits of entropy available at " a_RAND_DEV "%s\n",
         entrop_cnt, (entrop_cnt <= 512 ? ": temporary failure" : ""));
      if(entrop_cnt <= 512){
         rv = EX_TEMPFAIL;
         goto jerr2;
      }

      x.rpi.buf_size = x.rpi.entropy_count = 0;
      rem_size = sizeof x.buf;
jread_more:
      len = read(randfd, rpibuf, rem_size);
      if(len == -1){
         /* Ignore the EAGAIN that /dev/random reports when it would block
          * (Bernd Petrovitsch (bernd at petrovitsch dot priv dot at)) */
         if(errno == EAGAIN)
            goto jread_done;

         fprintf(stderr, "Failed to read from " a_RAND_DEV ": %s\n",
            strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      x.rpi.buf_size += (int)len;
      rpibuf += len;
      rem_size -= len;

      rv = ioctl(randfd, (int)RNDGETENTCNT, &iocarg);
      if(rv == -1){
         fprintf(stderr, "Failed to query remaining entropy of " a_RAND_DEV
            ": %s\n", strerror(errno));
         rv = EX_IOERR;
         goto jerr2;
      }
      entrop_cnt -= iocarg;
      x.rpi.entropy_count += entrop_cnt;

      /* Try to read more? */
      if(len > 0 && (entrop_cnt = iocarg) >= a_RAND_ENTROPY_COUNT_MIN &&
            rem_size >= 64)
         goto jread_more;
      printf("- %d bits of entropy remain at " a_RAND_DEV "\n", iocarg);

jread_done:

      len = x.rpi.buf_size;

   rv = 0;
jerr2:
   if(close(storfd) == -1)

/* s-it-mode */

--steffen
|

---