X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id CCEFADC18A0 for ; Tue, 10 Jan 2012 21:33:55 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so73562obb.30 for ; Tue, 10 Jan 2012 13:33:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hBCHw3rgvy/mRjSSIThJv2wUBlWgXOHiyTbPyHljEEg=; b=W3q27c2HsXurVPVexV/S0BhhHtgNTGs2VtC7gUDt1vwjr5ipBdLHGdGUYpVaPT6bui Ty71Ka6re19fbYRupKHglc7/ZOs1J0k2fW1QnuyS7C+Oc2mUEeIvCuJ9c6jk/euXC9Lm QZtGOL9GdOB75WMMPUvPXVjup770JdZhGdU2s= MIME-Version: 1.0 Received: by 10.182.117.97 with SMTP id kd1mr19956228obb.50.1326231235545; Tue, 10 Jan 2012 13:33:55 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:33:55 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:33:55 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 22:33:55 +0100 Message-ID: Subject: Re: Bugs.a.o From: Carlo Landmeter To: Jeff Bilyk Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: multipart/alternative; boundary=f46d0447f0f8af4ae904b633459c --f46d0447f0f8af4ae904b633459c Content-Type: text/plain; charset=ISO-8859-1 The bad answer is, no dump available.... On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: > So that failed. We're hitting what seems to be the same bug I've hit > before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634) > that still hasn't been fixed. The workaround is to delete everything > in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump > backup. I'm really hoping that clandmeter has a backup... > > 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in > file btr0pcur.c line 425 > InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == > buf_block_get_page_no(btr_pcur_get_block(cursor)) > InnoDB: We intentionally generate a memory trap. > InnoDB: Submit a detailed bug report to http://bugs.mysql.com. > InnoDB: If you get repeated assertion failures or crashes, even > InnoDB: immediately after the mysqld startup, there may be > InnoDB: corruption in the InnoDB tablespace. Please refer to > InnoDB: > http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > InnoDB: about forcing recovery. > 120110 21:26:39 - mysqld got signal 6 ; > This could be because you hit a bug. It is also possible that this binary > or one of the libraries it was linked against is corrupt, improperly built, > or misconfigured. This error can also be caused by malfunctioning hardware. > We will try our best to scrape up some info that will hopefully help > diagnose > the problem, but since we have already crashed, something is definitely > wrong > and this may fail. > > key_buffer_size=16777216 > read_buffer_size=262144 > max_used_connections=0 > max_threads=151 > thread_count=0 > connection_count=0 > It is possible that mysqld could use up to > key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > 133434 K > bytes of memory > Hope that's ok; if not, decrease some variables in the equation. > > Thread pointer: 0x0 > Attempting backtrace. You can use the following information to find out > where mysqld died. If you see no messages after this, something went > terribly wrong... > stack_bottom = (nil) thread_stack 0x30000 > /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] > The manual page at http://dev.mysql.com/doc/mysql/en/crashing.htmlcontains > information that should help you find out what is causing the crash. > 120110 21:26:39 mysqld_safe mysqld from pid file > /var/run/mysqld/mysqld.pid ended > > On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk wrote: > > Only other thing I can think to try is to dump the binlogs since it's > during recovery that the crash occurs. Logs say there's only one > transaction that's trying to apply... > > > > Jeff > > > > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter > wrote: > > > >> I'm getting the below, not sure what to do now. Already updated mysql > >> to latest 2.2 version. > >> > >> > >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda. > >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 > >> 120110 19:53:23 InnoDB: Database was not shut down normally! > >> InnoDB: Starting crash recovery. > >> InnoDB: Reading tablespace information from the .ibd files... > >> InnoDB: Restoring possible half-written data pages from the doublewrite > >> InnoDB: buffer... > >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045 > >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up > >> InnoDB: in total 12 row operations to undo > >> InnoDB: Trx id counter is 6DE8E00 > >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to the > >> database... > >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal > 11 ; > >> This could be because you hit a bug. It is also possible that this > binary > >> or one of the libraries it was linked against is corrupt, improperly > built, > >> or misconfigured. This error can also be caused by malfunctioning > hardware. > >> We will try our best to scrape up some info that will hopefully help > diagnose > >> the problem, but since we have already crashed, something is definitely > wrong > >> and this may fail. > >> > >> key_buffer_size=16777216 > >> read_buffer_size=262144 > >> max_used_connections=0 > >> max_threads=151 > >> thread_count=0 > >> connection_count=0 > >> It is possible that mysqld could use up to > >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > 133434 K > >> bytes of memory > >> Hope that's ok; if not, decrease some variables in the equation. > >> > >> Thread pointer: 0x0 > >> Attempting backtrace. You can use the following information to find out > >> where mysqld died. If you see no messages after this, something went > >> terribly wrong... > >> stack_bottom = (nil) thread_stack 0x30000 > >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] > >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.htmlcontains > >> information that should help you find out what is causing the crash. > >> 120110 19:53:24 mysqld_safe mysqld from pid file > >> /var/run/mysqld/mysqld.pid ended > >> > >> > >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter > wrote: > >>> Ill be able to check in a minute if needed. > >>> > >>> Carlo > >>> > >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" > wrote: > >>>> > >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: > >>>>> > >>>>> Hi Everyone, > >>>>> > >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is > >>>>> crashing, by the looks of things because of multiple spiders hitting > >>>>> it at once. Banning the user agents in lightty config hasn't helped, > >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of > >>>>> time to keep trying to fix it. If anyone else has some time to take > a > >>>>> look, that'd be awesome, > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Jeff > >>>>> > >>>> > >>>> Mysql died, so far I'm unable to restart it > >>>> > >>> > > > > -- > Jeff > --f46d0447f0f8af4ae904b633459c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

The bad answer is, no dump available....

On Jan 10, 2012 10:30 PM, "Jeff Bilyk"= <jbilyk@gmail.com> wrote:
So that failed. =A0We're hitting what seems to be the same bug I've= hit
before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=3D62634)
that still hasn't been fixed. =A0The workaround is to delete everything=
in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump
backup. =A0I'm really hoping that clandmeter has a backup...

120110 21:26:39 =A0InnoDB: Assertion failure in thread 3071633072 in
file btr0pcur.c line 425
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) =3D=3D
buf_block_get_page_no(btr_pcur_get_block(cursor))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forci= ng-innodb-recovery.html
InnoDB: about forcing recovery.
120110 21:26:39 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built,=
or misconfigured. This error can also be caused by malfunctioning hardware.=
We will try our best to scrape up some info that will hopefully help diagno= se
the problem, but since we have already crashed, something is definitely wro= ng
and this may fail.

key_buffer_size=3D16777216
read_buffer_size=3D262144
max_used_connections=3D0
max_threads=3D151
thread_count=3D0
connection_count=3D0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =3D 133= 434 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom =3D (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html c= ontains
information that should help you find out what is causing the crash.
120110 21:26:39 mysqld_safe mysqld from pid file
/var/run/mysqld/mysqld.pid ended

On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:
> Only other thing I can think to try is to dump the binlogs since it= 9;s during recovery that the crash occurs. =A0Logs say there's only one= transaction that's trying to apply...
>
> Jeff
>
> On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
>
>> I'm getting the below, not sure what to do now. Already update= d mysql
>> to latest 2.2 version.
>>
>>
>> 120110 19:53:23 InnoDB: highest supported file format is Barracuda= .
>> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 >> 120110 19:53:23 =A0InnoDB: Database was not shut down normally! >> InnoDB: Starting crash recovery.
>> InnoDB: Reading tablespace information from the .ibd files...
>> InnoDB: Restoring possible half-written data pages from the double= write
>> InnoDB: buffer...
>> InnoDB: Doing recovery: scanned up to log sequence number 15163771= 045
>> InnoDB: 1 transaction(s) which must be rolled back or cleaned up >> InnoDB: in total 12 row operations to undo
>> InnoDB: Trx id counter is 6DE8E00
>> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log records = to the
>> database...
>> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal 11 ;
>> This could be because you hit a bug. It is also possible that this= binary
>> or one of the libraries it was linked against is corrupt, improper= ly built,
>> or misconfigured. This error can also be caused by malfunctioning = hardware.
>> We will try our best to scrape up some info that will hopefully he= lp diagnose
>> the problem, but since we have already crashed, something is defin= itely wrong
>> and this may fail.
>>
>> key_buffer_size=3D16777216
>> read_buffer_size=3D262144
>> max_used_connections=3D0
>> max_threads=3D151
>> thread_count=3D0
>> connection_count=3D0
>> It is possible that mysqld could use up to
>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_thread= s =3D 133434 K
>> bytes of memory
>> Hope that's ok; if not, decrease some variables in the equatio= n.
>>
>> Thread pointer: 0x0
>> Attempting backtrace. You can use the following information to fin= d out
>> where mysqld died. If you see no messages after this, something we= nt
>> terribly wrong...
>> stack_bottom =3D (nil) thread_stack 0x30000
>> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
>> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.h= tml contains
>> information that should help you find out what is causing the cras= h.
>> 120110 19:53:24 mysqld_safe mysqld from pid file
>> /var/run/mysqld/mysqld.pid ended
>>
>>
>> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
>>> Ill be able to check in a minute if needed.
>>>
>>> Carlo
>>>
>>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org> wrote:
>>>>
>>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
>>>>>
>>>>> Hi Everyone,
>>>>>
>>>>> Redmine on bugs.a.o is having issues this afternoon. = =A0Fastcgi is
>>>>> crashing, by the looks of things because of multiple s= piders hitting
>>>>> it at once. =A0Banning the user agents in lightty conf= ig hasn't helped,
>>>>> neither has upping the redmine sockets to 32 (from 4) = but I'm out of
>>>>> time to keep trying to fix it. =A0If anyone else has s= ome time to take a
>>>>> look, that'd be awesome,
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jeff
>>>>>
>>>>
>>>> Mysql died, so far I'm unable to restart it
>>>>
>>>



--
Jeff
--f46d0447f0f8af4ae904b633459c--