X-Original-To: alpine-user@lists.alpinelinux.org Received: from mail92c50.megamailservers.eu (mail73c50.megamailservers.eu [91.136.10.83]) by lists.alpinelinux.org (Postfix) with ESMTP id C09AD5C55A0 for ; Tue, 14 Aug 2018 17:23:07 +0000 (GMT) X-Authenticated-User: alex@ewinkle.com Received: from Diaspar2 (host86-167-174-209.range86-167.btcentralplus.com [86.167.174.209]) (authenticated bits=0) by mail92c50.megamailservers.eu (8.14.9/8.13.1) with ESMTP id w7EHN43F011263 for ; Tue, 14 Aug 2018 17:23:05 +0000 From: "Alex Butler" To: Subject: [alpine-user] Alpine limit on file descriptors? Date: Tue, 14 Aug 2018 18:23:04 +0100 Message-ID: <055e01d433f3$76e95660$64bc0320$@ewinkle.com> X-Mailinglist: alpine-user Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_055F_01D433FB.D8AFBA30" X-Mailer: Microsoft Outlook 16.0 Content-Language: en-gb Thread-Index: AdQz60eO086cWmmIR0+pPiCoC0QVVA== X-CTCH-RefID: str=0001.0A0B020D.5B730FF9.0054,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0 X-CTCH-VOD: Unknown X-CTCH-Spam: Unknown X-CTCH-Score: 0.000 X-CTCH-Rules: X-CTCH-Flags: 0 X-CTCH-ScoreCust: 0.000 X-CSC: 0 X-CHA: v=2.3 cv=Y8HWTCWN c=1 sm=1 tr=0 a=yV+eBZsnerNgLmCiluTJ/w==:117 a=yV+eBZsnerNgLmCiluTJ/w==:17 a=DAwyPP_o2Byb1YXLmDAA:9 a=Q0fV-lfS_mZEWCGtoTUA:9 a=vBgh5oke-uClweai:21 a=8PSiHd5gzrldxXWN:21 a=CjuIK1q_8ugA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=KKM8aWBaAOlXMXBcoIcA:9 a=p2xpdk_ZxPcvYazh:21 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=BNvrlgU0_EZ7dGlRZGAA:9 a=7bk0SRsPWpQA:10 This is a multipart message in MIME format. ------=_NextPart_000_055F_01D433FB.D8AFBA30 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0560_01D433FB.D8AFBA30" ------=_NextPart_001_0560_01D433FB.D8AFBA30 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit We've been having some issues with what looks like some kind of limitation on the maximum number of file descriptors (or a /dev/shm semaphore limitation). Our application is in Python and uses the standard Python multiprocessing library to create processes and associated queues for communication (typically creating between 20 and 200 processes at start-up depending on hardware configuration). It runs fine on Raspbian/Debian with any number of processes we choose (within reason!) and runs fine under Alpine when we run with low numbers of processes. It however always barfs for larger numbers of processes under Alpine - suggesting (from the reported OSError) that it is running out of file descriptors. [Might be a red herring but might be related to the use of OS semaphore management in /dev/shm. Just not sure!] Anyway, after trying quite a few things we've narrowed it down to failing in every stock flavour of Alpine we've tried (x64, Raspberry Pi etc) but which just doesn't happen at all in the different flavours of Raspbian/Debian/Ubuntu etc. Is there some Alpine setting/limit which we haven't yet found which sets the maximum number of file descriptors (or some other subtle Alpine difference). We've tried all the "obvious" Linux file descriptor changes like ulimit, sysctl type changes etc. To help recreate this we've created a simple Python script (attached). Under Alpine (Raspberry Pi) it fails after the 85th process pair. If MAX_PAIRS is set to 85 it works fine. i.e. no exceptions. Put in anything bigger for MAX_PAIRS and we always get the following error message at the 86th: --- data for 83 was [83001, 83002, 83003, 83004, 83005, 83006, 83007, 83008, 83009] data for 84 was [84001, 84002, 84003, 84004, 84005, 84006, 84007, 84008, 84009] data for 85 was [85001, 85002, 85003, 85004, 85005, 85006, 85007, 85008, 85009] Traceback (most recent call last): File "queue_test.py", line 41, in q = Queue() File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue return Queue(maxsize) File "/usr/lib/python2.7/multiprocessing/queues.py", line 68, in __init__ self._wlock = Lock() File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 24] No file descriptors available --- As I said - on other Linux distro's this code runs fine. We'd _really_ like to use Alpine for a variety of obvious reasons. It's not obvious what is going on and not being able to run multiprocessing to the level of parallelism we need might be a deal-breaker. Incidentally, at MAX_PAIRS = 85 (when the test code runs fine), doing a "lsof | wc -l" reveals about 29991 file descriptors (~29k). I've attached a copy of the test python code for ease of replication. We just run it as root using "/usr/bin/python queue_test.py" Any help or suggestions as to what might be going on gratefully received! Cheers, Alex Butler UK ------=_NextPart_001_0560_01D433FB.D8AFBA30 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We’ve been having some issues with what looks = like some kind of limitation on the maximum number of file descriptors = (or a /dev/shm semaphore limitation).  Our application is in Python = and uses the standard Python multiprocessing library to create processes = and associated queues for communication (typically creating between 20 = and 200 processes at start-up depending on hardware = configuration).   It runs fine on Raspbian/Debian with any = number of processes we choose (within reason!) and runs fine under = Alpine when we run with low numbers of processes.  =

 

It however always barfs for larger numbers of = processes under Alpine – suggesting (from the reported OSError) = that it is running out of file descriptors.  [Might be a red = herring but might be related to the use of OS semaphore management in = /dev/shm.  Just not sure!]

 

Anyway, = after trying quite a few things we’ve narrowed it down to failing = in every stock flavour of Alpine we’ve tried (x64, Raspberry Pi = etc) but which just doesn’t happen at all in the different = flavours of Raspbian/Debian/Ubuntu etc.

 

Is there = some Alpine setting/limit which we haven’t yet found which sets = the maximum number of file descriptors (or some other subtle Alpine = difference).  We’ve tried all the “obvious” Linux = file descriptor changes like ulimit, sysctl type changes = etc.

 

To help recreate this we’ve created a simple = Python script (attached).

 

Under Alpine = (Raspberry Pi) it fails after the 85th process pair.  If = MAX_PAIRS is set to 85 it works fine.  i.e. no exceptions.  = Put in anything bigger for MAX_PAIRS and we always get the following = error message at the 86th:

 

---

data for 83 was = [83001, 83002, 83003, 83004, 83005, 83006, 83007, 83008, = 83009]

data for 84 was [84001, 84002, = 84003, 84004, 84005, 84006, 84007, 84008, 84009]

data for 85 was [85001, 85002, 85003, 85004, 85005, = 85006, 85007, 85008, 85009]

Traceback = (most recent call last):

  File = "queue_test.py", line 41, in <module>

    q =3D Queue()

  File = "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in = Queue

    return = Queue(maxsize)

  File = "/usr/lib/python2.7/multiprocessing/queues.py", line 68, in = __init__

    = self._wlock =3D Lock()

  File = "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, = in __init__

    = SemLock.__init__(self, SEMAPHORE, 1, 1)

  File = "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, = in __init__

    sl =3D = self._semlock =3D _multiprocessing.SemLock(kind, value, = maxvalue)

OSError: [Errno 24] No file = descriptors available

---

 

As I said = – on other Linux distro’s this code runs fine.  = We’d _really_ like to use Alpine for a variety of obvious = reasons.  It’s not obvious what is going on and not being = able to run multiprocessing to the level of parallelism we need might be = a deal-breaker.

 

Incidentally, at MAX_PAIRS =3D 85 (when the test code = runs fine), doing a “lsof | wc -l” reveals about 29991 file = descriptors (~29k).

 

I’ve = attached a copy of the test python code for ease of replication.  = We just run it as root using “/usr/bin/python = queue_test.py”

 

Any help or = suggestions as to what might be going on gratefully = received!

 

Cheers,

 

Alex = Butler

UK

------=_NextPart_001_0560_01D433FB.D8AFBA30-- ------=_NextPart_000_055F_01D433FB.D8AFBA30 Content-Type: text/plain; name="queue_test.py" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="queue_test.py" from multiprocessing import Process, Queue import time # Maximum pairs of communicating processes (all is good if 85 or lower) MAX_PAIRS = 100 def creator(data, q, x): """ Creates data to be consumed and waits for the consumer to finish processing """ print('Creator{0} putting data on the queue'.format(x)) while True: for item in data: time.sleep(1) q.put(item) def eater(q, x): """ Consumes some data and works on it """ while True: data = q.get() print "Eater{0} data was {1}".format(x, data) processed = data print(processed) process_c = [] process_eat = [] if __name__ == '__main__': all_q = [] for x in xrange(0,MAX_PAIRS): data = [] for y in xrange(1,10): data.append(x*1000 + y) print "data for {0} was {1}".format(x, data) q = Queue() all_q.append(q) process_c.append( Process(target=creator, args=(data, q, x)) ) process_eat.append( Process(target=eater, args=(q, x)) ) for p in process_eat: p.start() for p in process_c: p.start() #q.close() #q.join_thread() #for p in process_c: # p.join() #for p in process_eat: # p.join() ------=_NextPart_000_055F_01D433FB.D8AFBA30-- --- Unsubscribe: alpine-user+unsubscribe@lists.alpinelinux.org Help: alpine-user+help@lists.alpinelinux.org ---