X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id EC0C0DC1843 for ; Tue, 10 Jan 2012 21:40:10 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so86388obb.30 for ; Tue, 10 Jan 2012 13:40:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+3TOraAn76p5UAh8mIQSqviURh7VteM+N8KIMCC91Es=; b=ZPWdAvFQ4maywGNbnsePCIRTHPS9FX9ACXclwPAofKkfSPguV0M+hkGtZ7BXtQbs/Q ffv5BRags5DDAszu7Lz09pTFElyYbcVExzc3w8aVo4RN8VzAbbniKfL4yZFvafB4ALeq StLYXK78ZX355hW82OSBzKbOv6mUrAAQDbV6I= MIME-Version: 1.0 Received: by 10.182.45.102 with SMTP id l6mr20174638obm.0.1326231610686; Tue, 10 Jan 2012 13:40:10 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:40:10 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:40:10 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 22:40:10 +0100 Message-ID: Subject: Re: Bugs.a.o From: Carlo Landmeter To: Jeff Bilyk Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: multipart/alternative; boundary=f46d0444ef0b0b7bdd04b6335c03 --f46d0444ef0b0b7bdd04b6335c03 Content-Type: text/plain; charset=ISO-8859-1 What about starting this var on Ubuntu or something and export? On Jan 10, 2012 10:37 PM, "Jeff Bilyk" wrote: > Urgh - I'm not sure how to recover that then... Carlo, I've got a copy > of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a > couple...invasive... recovery tricks on the copy in /varlib/mysql, but > they didn't work. In the testing we did for the bug below, the only > way I was ever able to get a copy running again was delete/restore > from backup. > > Anyone have any ideas where to go from here? > > On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter > wrote: > > The bad answer is, no dump available.... > > > > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: > >> > >> So that failed. We're hitting what seems to be the same bug I've hit > >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634) > >> that still hasn't been fixed. The workaround is to delete everything > >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump > >> backup. I'm really hoping that clandmeter has a backup... > >> > >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in > >> file btr0pcur.c line 425 > >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == > >> buf_block_get_page_no(btr_pcur_get_block(cursor)) > >> InnoDB: We intentionally generate a memory trap. > >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. > >> InnoDB: If you get repeated assertion failures or crashes, even > >> InnoDB: immediately after the mysqld startup, there may be > >> InnoDB: corruption in the InnoDB tablespace. Please refer to > >> InnoDB: > >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > >> InnoDB: about forcing recovery. > >> 120110 21:26:39 - mysqld got signal 6 ; > >> This could be because you hit a bug. It is also possible that this > binary > >> or one of the libraries it was linked against is corrupt, improperly > >> built, > >> or misconfigured. This error can also be caused by malfunctioning > >> hardware. > >> We will try our best to scrape up some info that will hopefully help > >> diagnose > >> the problem, but since we have already crashed, something is definitely > >> wrong > >> and this may fail. > >> > >> key_buffer_size=16777216 > >> read_buffer_size=262144 > >> max_used_connections=0 > >> max_threads=151 > >> thread_count=0 > >> connection_count=0 > >> It is possible that mysqld could use up to > >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > >> 133434 K > >> bytes of memory > >> Hope that's ok; if not, decrease some variables in the equation. > >> > >> Thread pointer: 0x0 > >> Attempting backtrace. You can use the following information to find out > >> where mysqld died. If you see no messages after this, something went > >> terribly wrong... > >> stack_bottom = (nil) thread_stack 0x30000 > >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] > >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html > >> contains > >> information that should help you find out what is causing the crash. > >> 120110 21:26:39 mysqld_safe mysqld from pid file > >> /var/run/mysqld/mysqld.pid ended > >> > >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk wrote: > >> > Only other thing I can think to try is to dump the binlogs since it's > >> > during recovery that the crash occurs. Logs say there's only one > >> > transaction that's trying to apply... > >> > > >> > Jeff > >> > > >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter > >> > wrote: > >> > > >> >> I'm getting the below, not sure what to do now. Already updated mysql > >> >> to latest 2.2 version. > >> >> > >> >> > >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda. > >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 > >> >> 120110 19:53:23 InnoDB: Database was not shut down normally! > >> >> InnoDB: Starting crash recovery. > >> >> InnoDB: Reading tablespace information from the .ibd files... > >> >> InnoDB: Restoring possible half-written data pages from the > doublewrite > >> >> InnoDB: buffer... > >> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045 > >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up > >> >> InnoDB: in total 12 row operations to undo > >> >> InnoDB: Trx id counter is 6DE8E00 > >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to > the > >> >> database... > >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got > signal > >> >> 11 ; > >> >> This could be because you hit a bug. It is also possible that this > >> >> binary > >> >> or one of the libraries it was linked against is corrupt, improperly > >> >> built, > >> >> or misconfigured. This error can also be caused by malfunctioning > >> >> hardware. > >> >> We will try our best to scrape up some info that will hopefully help > >> >> diagnose > >> >> the problem, but since we have already crashed, something is > definitely > >> >> wrong > >> >> and this may fail. > >> >> > >> >> key_buffer_size=16777216 > >> >> read_buffer_size=262144 > >> >> max_used_connections=0 > >> >> max_threads=151 > >> >> thread_count=0 > >> >> connection_count=0 > >> >> It is possible that mysqld could use up to > >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > >> >> 133434 K > >> >> bytes of memory > >> >> Hope that's ok; if not, decrease some variables in the equation. > >> >> > >> >> Thread pointer: 0x0 > >> >> Attempting backtrace. You can use the following information to find > out > >> >> where mysqld died. If you see no messages after this, something went > >> >> terribly wrong... > >> >> stack_bottom = (nil) thread_stack 0x30000 > >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] > >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html > >> >> contains > >> >> information that should help you find out what is causing the crash. > >> >> 120110 19:53:24 mysqld_safe mysqld from pid file > >> >> /var/run/mysqld/mysqld.pid ended > >> >> > >> >> > >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter < > clandmeter@gmail.com> > >> >> wrote: > >> >>> Ill be able to check in a minute if needed. > >> >>> > >> >>> Carlo > >> >>> > >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" > >> >>> wrote: > >> >>>> > >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: > >> >>>>> > >> >>>>> Hi Everyone, > >> >>>>> > >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is > >> >>>>> crashing, by the looks of things because of multiple spiders > hitting > >> >>>>> it at once. Banning the user agents in lightty config hasn't > >> >>>>> helped, > >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out > of > >> >>>>> time to keep trying to fix it. If anyone else has some time to > take > >> >>>>> a > >> >>>>> look, that'd be awesome, > >> >>>>> > >> >>>>> Thanks, > >> >>>>> > >> >>>>> Jeff > >> >>>>> > >> >>>> > >> >>>> Mysql died, so far I'm unable to restart it > >> >>>> > >> >>> > >> > >> > >> > >> -- > >> Jeff > > > > -- > Jeff > --f46d0444ef0b0b7bdd04b6335c03 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

What about starting this var on Ubuntu or something and export?

On Jan 10, 2012 10:37 PM, "Jeff Bilyk"= <jbilyk@gmail.com> wrote:
Urgh - I'm not sure how to recover that then... Carlo, I've got a c= opy
of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
couple...invasive... recovery tricks on the copy in /varlib/mysql, but
they didn't work. =A0In the testing we did for the bug below, the only<= br> way I was ever able to get a copy running again was delete/restore
from backup.

Anyone have any ideas where to go from here?

On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> The bad answer is, no dump available....
>
> On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
>>
>> So that failed. =A0We're hitting what seems to be the same bug= I've hit
>> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=3D62634)
>> that still hasn't been fixed. =A0The workaround is to delete e= verything
>> in /var/lib/mysql/, reinitialize the db, then restore from a mysql= dump
>> backup. =A0I'm really hoping that clandmeter has a backup... >>
>> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 3071633072 = in
>> file btr0pcur.c line 425
>> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) =3D= =3D
>> buf_block_get_page_no(btr_pcur_get_block(cursor))
>> InnoDB: We intentionally generate a memory trap.
>> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> InnoDB: If you get repeated assertion failures or crashes, even >> InnoDB: immediately after the mysqld startup, there may be
>> InnoDB: corruption in the InnoDB tablespace. Please refer to
>> InnoDB:
>> http://dev.mysql.com/doc/refman/5.5/en/forc= ing-innodb-recovery.html
>> InnoDB: about forcing recovery.
>> 120110 21:26:39 - mysqld got signal 6 ;
>> This could be because you hit a bug. It is also possible that this= binary
>> or one of the libraries it was linked against is corrupt, improper= ly
>> built,
>> or misconfigured. This error can also be caused by malfunctioning<= br> >> hardware.
>> We will try our best to scrape up some info that will hopefully he= lp
>> diagnose
>> the problem, but since we have already crashed, something is defin= itely
>> wrong
>> and this may fail.
>>
>> key_buffer_size=3D16777216
>> read_buffer_size=3D262144
>> max_used_connections=3D0
>> max_threads=3D151
>> thread_count=3D0
>> connection_count=3D0
>> It is possible that mysqld could use up to
>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_thread= s =3D
>> 133434 K
>> bytes of memory
>> Hope that's ok; if not, decrease some variables in the equatio= n.
>>
>> Thread pointer: 0x0
>> Attempting backtrace. You can use the following information to fin= d out
>> where mysqld died. If you see no messages after this, something we= nt
>> terribly wrong...
>> stack_bottom =3D (nil) thread_stack 0x30000
>> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
>> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.h= tml
>> contains
>> information that should help you find out what is causing the cras= h.
>> 120110 21:26:39 mysqld_safe mysqld from pid file
>> /var/run/mysqld/mysqld.pid ended
>>
>> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:
>> > Only other thing I can think to try is to dump the binlogs si= nce it's
>> > during recovery that the crash occurs. =A0Logs say there'= s only one
>> > transaction that's trying to apply...
>> >
>> > Jeff
>> >
>> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>
>> > wrote:
>> >
>> >> I'm getting the below, not sure what to do now. Alrea= dy updated mysql
>> >> to latest 2.2 version.
>> >>
>> >>
>> >> 120110 19:53:23 InnoDB: highest supported file format is = Barracuda.
>> >> InnoDB: Log scan progressed past the checkpoint lsn 15160= 684415
>> >> 120110 19:53:23 =A0InnoDB: Database was not shut down nor= mally!
>> >> InnoDB: Starting crash recovery.
>> >> InnoDB: Reading tablespace information from the .ibd file= s...
>> >> InnoDB: Restoring possible half-written data pages from t= he doublewrite
>> >> InnoDB: buffer...
>> >> InnoDB: Doing recovery: scanned up to log sequence number= 15163771045
>> >> InnoDB: 1 transaction(s) which must be rolled back or cle= aned up
>> >> InnoDB: in total 12 row operations to undo
>> >> InnoDB: Trx id counter is 6DE8E00
>> >> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log= records to the
>> >> database...
>> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal=
>> >> 11 ;
>> >> This could be because you hit a bug. It is also possible = that this
>> >> binary
>> >> or one of the libraries it was linked against is corrupt,= improperly
>> >> built,
>> >> or misconfigured. This error can also be caused by malfun= ctioning
>> >> hardware.
>> >> We will try our best to scrape up some info that will hop= efully help
>> >> diagnose
>> >> the problem, but since we have already crashed, something= is definitely
>> >> wrong
>> >> and this may fail.
>> >>
>> >> key_buffer_size=3D16777216
>> >> read_buffer_size=3D262144
>> >> max_used_connections=3D0
>> >> max_threads=3D151
>> >> thread_count=3D0
>> >> connection_count=3D0
>> >> It is possible that mysqld could use up to
>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*m= ax_threads =3D
>> >> 133434 K
>> >> bytes of memory
>> >> Hope that's ok; if not, decrease some variables in th= e equation.
>> >>
>> >> Thread pointer: 0x0
>> >> Attempting backtrace. You can use the following informati= on to find out
>> >> where mysqld died. If you see no messages after this, som= ething went
>> >> terribly wrong...
>> >> stack_bottom =3D (nil) thread_stack 0x30000
>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] >> >> The manual page at http://dev.mysql.com/doc/mysql/en/c= rashing.html
>> >> contains
>> >> information that should help you find out what is causing= the crash.
>> >> 120110 19:53:24 mysqld_safe mysqld from pid file
>> >> /var/run/mysqld/mysqld.pid ended
>> >>
>> >>
>> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com>
>> >> wrote:
>> >>> Ill be able to check in a minute if needed.
>> >>>
>> >>> Carlo
>> >>>
>> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos"= <nangel@nothome.org>
>> >>> wrote:
>> >>>>
>> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
>> >>>>>
>> >>>>> Hi Everyone,
>> >>>>>
>> >>>>> Redmine on bugs.a.o is having issues this aft= ernoon. =A0Fastcgi is
>> >>>>> crashing, by the looks of things because of m= ultiple spiders hitting
>> >>>>> it at once. =A0Banning the user agents in lig= htty config hasn't
>> >>>>> helped,
>> >>>>> neither has upping the redmine sockets to 32 = (from 4) but I'm out of
>> >>>>> time to keep trying to fix it. =A0If anyone e= lse has some time to take
>> >>>>> a
>> >>>>> look, that'd be awesome,
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Jeff
>> >>>>>
>> >>>>
>> >>>> Mysql died, so far I'm unable to restart it >> >>>>
>> >>>
>>
>>
>>
>> --
>> Jeff



--
Jeff
--f46d0444ef0b0b7bdd04b6335c03--