X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id 8A987DC1844 for ; Tue, 10 Jan 2012 21:46:59 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so102432obb.30 for ; Tue, 10 Jan 2012 13:46:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bd+MwYVXZDOZlfqqXUuMiUVdPzgK2Vfsr3G/54UZ3nQ=; b=U+o4su3bEiouA7pydepy9uOfH533Q54WCf1f9rjnV3/F6USHpUHkmhiolSwonV+24r 3fDLQ0nebxxOp3ENonadbm4A4Tv/IBJtDDxYOCDdnMuBYz1wpr5+pBhdB0lk0s9EPuEu umBR41Hs34L7hDJdycMNgAIVDX8LV2KKmkax0= MIME-Version: 1.0 Received: by 10.182.15.104 with SMTP id w8mr12290742obc.20.1326232019118; Tue, 10 Jan 2012 13:46:59 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:46:59 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 13:46:59 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 22:46:59 +0100 Message-ID: Subject: Re: Bugs.a.o From: Carlo Landmeter To: Jeff Bilyk Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: multipart/alternative; boundary=f46d04462f8663ab4c04b633746f --f46d04462f8663ab4c04b633746f Content-Type: text/plain; charset=ISO-8859-1 Should be no problem. It's an vserver. Ill try tonight. On Jan 10, 2012 10:44 PM, "Jeff Bilyk" wrote: > It's worth a shot, but I don't have the time/a box at work to try. > Could you give it a shot? > > On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter > wrote: > > What about starting this var on Ubuntu or something and export? > > > > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" wrote: > >> > >> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy > >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a > >> couple...invasive... recovery tricks on the copy in /varlib/mysql, but > >> they didn't work. In the testing we did for the bug below, the only > >> way I was ever able to get a copy running again was delete/restore > >> from backup. > >> > >> Anyone have any ideas where to go from here? > >> > >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter > >> wrote: > >> > The bad answer is, no dump available.... > >> > > >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: > >> >> > >> >> So that failed. We're hitting what seems to be the same bug I've hit > >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634) > >> >> that still hasn't been fixed. The workaround is to delete everything > >> >> in /var/lib/mysql/, reinitialize the db, then restore from a > mysqldump > >> >> backup. I'm really hoping that clandmeter has a backup... > >> >> > >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in > >> >> file btr0pcur.c line 425 > >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == > >> >> buf_block_get_page_no(btr_pcur_get_block(cursor)) > >> >> InnoDB: We intentionally generate a memory trap. > >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. > >> >> InnoDB: If you get repeated assertion failures or crashes, even > >> >> InnoDB: immediately after the mysqld startup, there may be > >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to > >> >> InnoDB: > >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html > >> >> InnoDB: about forcing recovery. > >> >> 120110 21:26:39 - mysqld got signal 6 ; > >> >> This could be because you hit a bug. It is also possible that this > >> >> binary > >> >> or one of the libraries it was linked against is corrupt, improperly > >> >> built, > >> >> or misconfigured. This error can also be caused by malfunctioning > >> >> hardware. > >> >> We will try our best to scrape up some info that will hopefully help > >> >> diagnose > >> >> the problem, but since we have already crashed, something is > definitely > >> >> wrong > >> >> and this may fail. > >> >> > >> >> key_buffer_size=16777216 > >> >> read_buffer_size=262144 > >> >> max_used_connections=0 > >> >> max_threads=151 > >> >> thread_count=0 > >> >> connection_count=0 > >> >> It is possible that mysqld could use up to > >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = > >> >> 133434 K > >> >> bytes of memory > >> >> Hope that's ok; if not, decrease some variables in the equation. > >> >> > >> >> Thread pointer: 0x0 > >> >> Attempting backtrace. You can use the following information to find > out > >> >> where mysqld died. If you see no messages after this, something went > >> >> terribly wrong... > >> >> stack_bottom = (nil) thread_stack 0x30000 > >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] > >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html > >> >> contains > >> >> information that should help you find out what is causing the crash. > >> >> 120110 21:26:39 mysqld_safe mysqld from pid file > >> >> /var/run/mysqld/mysqld.pid ended > >> >> > >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk > wrote: > >> >> > Only other thing I can think to try is to dump the binlogs since > it's > >> >> > during recovery that the crash occurs. Logs say there's only one > >> >> > transaction that's trying to apply... > >> >> > > >> >> > Jeff > >> >> > > >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter > > >> >> > wrote: > >> >> > > >> >> >> I'm getting the below, not sure what to do now. Already updated > >> >> >> mysql > >> >> >> to latest 2.2 version. > >> >> >> > >> >> >> > >> >> >> 120110 19:53:23 InnoDB: highest supported file format is > Barracuda. > >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 > >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally! > >> >> >> InnoDB: Starting crash recovery. > >> >> >> InnoDB: Reading tablespace information from the .ibd files... > >> >> >> InnoDB: Restoring possible half-written data pages from the > >> >> >> doublewrite > >> >> >> InnoDB: buffer... > >> >> >> InnoDB: Doing recovery: scanned up to log sequence number > >> >> >> 15163771045 > >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up > >> >> >> InnoDB: in total 12 row operations to undo > >> >> >> InnoDB: Trx id counter is 6DE8E00 > >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to > >> >> >> the > >> >> >> database... > >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got > >> >> >> signal > >> >> >> 11 ; > >> >> >> This could be because you hit a bug. It is also possible that this > >> >> >> binary > >> >> >> or one of the libraries it was linked against is corrupt, > improperly > >> >> >> built, > >> >> >> or misconfigured. This error can also be caused by malfunctioning > >> >> >> hardware. > >> >> >> We will try our best to scrape up some info that will hopefully > help > >> >> >> diagnose > >> >> >> the problem, but since we have already crashed, something is > >> >> >> definitely > >> >> >> wrong > >> >> >> and this may fail. > >> >> >> > >> >> >> key_buffer_size=16777216 > >> >> >> read_buffer_size=262144 > >> >> >> max_used_connections=0 > >> >> >> max_threads=151 > >> >> >> thread_count=0 > >> >> >> connection_count=0 > >> >> >> It is possible that mysqld could use up to > >> >> >> key_buffer_size + (read_buffer_size + > sort_buffer_size)*max_threads > >> >> >> = > >> >> >> 133434 K > >> >> >> bytes of memory > >> >> >> Hope that's ok; if not, decrease some variables in the equation. > >> >> >> > >> >> >> Thread pointer: 0x0 > >> >> >> Attempting backtrace. You can use the following information to > find > >> >> >> out > >> >> >> where mysqld died. If you see no messages after this, something > went > >> >> >> terribly wrong... > >> >> >> stack_bottom = (nil) thread_stack 0x30000 > >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] > >> >> >> The manual page at > http://dev.mysql.com/doc/mysql/en/crashing.html > >> >> >> contains > >> >> >> information that should help you find out what is causing the > crash. > >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file > >> >> >> /var/run/mysqld/mysqld.pid ended > >> >> >> > >> >> >> > >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter > >> >> >> > >> >> >> wrote: > >> >> >>> Ill be able to check in a minute if needed. > >> >> >>> > >> >> >>> Carlo > >> >> >>> > >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" > >> >> >>> wrote: > >> >> >>>> > >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: > >> >> >>>>> > >> >> >>>>> Hi Everyone, > >> >> >>>>> > >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi > is > >> >> >>>>> crashing, by the looks of things because of multiple spiders > >> >> >>>>> hitting > >> >> >>>>> it at once. Banning the user agents in lightty config hasn't > >> >> >>>>> helped, > >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm > out > >> >> >>>>> of > >> >> >>>>> time to keep trying to fix it. If anyone else has some time to > >> >> >>>>> take > >> >> >>>>> a > >> >> >>>>> look, that'd be awesome, > >> >> >>>>> > >> >> >>>>> Thanks, > >> >> >>>>> > >> >> >>>>> Jeff > >> >> >>>>> > >> >> >>>> > >> >> >>>> Mysql died, so far I'm unable to restart it > >> >> >>>> > >> >> >>> > >> >> > >> >> > >> >> > >> >> -- > >> >> Jeff > >> > >> > >> > >> -- > >> Jeff > > > > -- > Jeff > --f46d04462f8663ab4c04b633746f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Should be no problem. It's an vserver. Ill try tonight.

On Jan 10, 2012 10:44 PM, "Jeff Bilyk"= <jbilyk@gmail.com> wrote:
It's worth a shot, but I don't have the time/a box at work to try.<= br> Could you give it a shot?

On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> What about starting this var on Ubuntu or something and export?
>
> On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
>>
>> Urgh - I'm not sure how to recover that then... Carlo, I'v= e got a copy
>> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a<= br> >> couple...invasive... recovery tricks on the copy in /varlib/mysql,= but
>> they didn't work. =A0In the testing we did for the bug below, = the only
>> way I was ever able to get a copy running again was delete/restore=
>> from backup.
>>
>> Anyone have any ideas where to go from here?
>>
>> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com>
>> wrote:
>> > The bad answer is, no dump available....
>> >
>> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
>> >>
>> >> So that failed. =A0We're hitting what seems to be the= same bug I've hit
>> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id= =3D62634)
>> >> that still hasn't been fixed. =A0The workaround is to= delete everything
>> >> in /var/lib/mysql/, reinitialize the db, then restore fro= m a mysqldump
>> >> backup. =A0I'm really hoping that clandmeter has a ba= ckup...
>> >>
>> >> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 30= 71633072 in
>> >> file btr0pcur.c line 425
>> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, m= tr) =3D=3D
>> >> buf_block_get_page_no(btr_pcur_get_block(cursor))
>> >> InnoDB: We intentionally generate a memory trap.
>> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> >> InnoDB: If you get repeated assertion failures or crashes= , even
>> >> InnoDB: immediately after the mysqld startup, there may b= e
>> >> InnoDB: corruption in the InnoDB tablespace. Please refer= to
>> >> InnoDB:
>> >> http://dev.mysql.com/doc/refman/5.= 5/en/forcing-innodb-recovery.html
>> >> InnoDB: about forcing recovery.
>> >> 120110 21:26:39 - mysqld got signal 6 ;
>> >> This could be because you hit a bug. It is also possible = that this
>> >> binary
>> >> or one of the libraries it was linked against is corrupt,= improperly
>> >> built,
>> >> or misconfigured. This error can also be caused by malfun= ctioning
>> >> hardware.
>> >> We will try our best to scrape up some info that will hop= efully help
>> >> diagnose
>> >> the problem, but since we have already crashed, something= is definitely
>> >> wrong
>> >> and this may fail.
>> >>
>> >> key_buffer_size=3D16777216
>> >> read_buffer_size=3D262144
>> >> max_used_connections=3D0
>> >> max_threads=3D151
>> >> thread_count=3D0
>> >> connection_count=3D0
>> >> It is possible that mysqld could use up to
>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*m= ax_threads =3D
>> >> 133434 K
>> >> bytes of memory
>> >> Hope that's ok; if not, decrease some variables in th= e equation.
>> >>
>> >> Thread pointer: 0x0
>> >> Attempting backtrace. You can use the following informati= on to find out
>> >> where mysqld died. If you see no messages after this, som= ething went
>> >> terribly wrong...
>> >> stack_bottom =3D (nil) thread_stack 0x30000
>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] >> >> The manual page at http://dev.mysql.com/doc/mysql/en/c= rashing.html
>> >> contains
>> >> information that should help you find out what is causing= the crash.
>> >> 120110 21:26:39 mysqld_safe mysqld from pid file
>> >> /var/run/mysqld/mysqld.pid ended
>> >>
>> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:
>> >> > Only other thing I can think to try is to dump the b= inlogs since it's
>> >> > during recovery that the crash occurs. =A0Logs say t= here's only one
>> >> > transaction that's trying to apply...
>> >> >
>> >> > Jeff
>> >> >
>> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> I'm getting the below, not sure what to do n= ow. Already updated
>> >> >> mysql
>> >> >> to latest 2.2 version.
>> >> >>
>> >> >>
>> >> >> 120110 19:53:23 InnoDB: highest supported file f= ormat is Barracuda.
>> >> >> InnoDB: Log scan progressed past the checkpoint = lsn 15160684415
>> >> >> 120110 19:53:23 =A0InnoDB: Database was not shut= down normally!
>> >> >> InnoDB: Starting crash recovery.
>> >> >> InnoDB: Reading tablespace information from the = .ibd files...
>> >> >> InnoDB: Restoring possible half-written data pag= es from the
>> >> >> doublewrite
>> >> >> InnoDB: buffer...
>> >> >> InnoDB: Doing recovery: scanned up to log sequen= ce number
>> >> >> 15163771045
>> >> >> InnoDB: 1 transaction(s) which must be rolled ba= ck or cleaned up
>> >> >> InnoDB: in total 12 row operations to undo
>> >> >> InnoDB: Trx id counter is 6DE8E00
>> >> >> 120110 19:53:24 =A0InnoDB: Starting an apply bat= ch of log records to
>> >> >> the
>> >> >> database...
>> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld g= ot
>> >> >> signal
>> >> >> 11 ;
>> >> >> This could be because you hit a bug. It is also = possible that this
>> >> >> binary
>> >> >> or one of the libraries it was linked against is= corrupt, improperly
>> >> >> built,
>> >> >> or misconfigured. This error can also be caused = by malfunctioning
>> >> >> hardware.
>> >> >> We will try our best to scrape up some info that= will hopefully help
>> >> >> diagnose
>> >> >> the problem, but since we have already crashed, = something is
>> >> >> definitely
>> >> >> wrong
>> >> >> and this may fail.
>> >> >>
>> >> >> key_buffer_size=3D16777216
>> >> >> read_buffer_size=3D262144
>> >> >> max_used_connections=3D0
>> >> >> max_threads=3D151
>> >> >> thread_count=3D0
>> >> >> connection_count=3D0
>> >> >> It is possible that mysqld could use up to
>> >> >> key_buffer_size + (read_buffer_size + sort_buffe= r_size)*max_threads
>> >> >> =3D
>> >> >> 133434 K
>> >> >> bytes of memory
>> >> >> Hope that's ok; if not, decrease some variab= les in the equation.
>> >> >>
>> >> >> Thread pointer: 0x0
>> >> >> Attempting backtrace. You can use the following = information to find
>> >> >> out
>> >> >> where mysqld died. If you see no messages after = this, something went
>> >> >> terribly wrong...
>> >> >> stack_bottom =3D (nil) thread_stack 0x30000
>> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73= bb529]
>> >> >> The manual page at http://dev.mysql.com/doc/m= ysql/en/crashing.html
>> >> >> contains
>> >> >> information that should help you find out what i= s causing the crash.
>> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file=
>> >> >> /var/run/mysqld/mysqld.pid ended
>> >> >>
>> >> >>
>> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter=
>> >> >> <clan= dmeter@gmail.com>
>> >> >> wrote:
>> >> >>> Ill be able to check in a minute if needed.<= br> >> >> >>>
>> >> >>> Carlo
>> >> >>>
>> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angela= cos" <nangel@nothome.org&= gt;
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote= :
>> >> >>>>>
>> >> >>>>> Hi Everyone,
>> >> >>>>>
>> >> >>>>> Redmine on bugs.a.o is having issues= this afternoon. =A0Fastcgi is
>> >> >>>>> crashing, by the looks of things bec= ause of multiple spiders
>> >> >>>>> hitting
>> >> >>>>> it at once. =A0Banning the user agen= ts in lightty config hasn't
>> >> >>>>> helped,
>> >> >>>>> neither has upping the redmine socke= ts to 32 (from 4) but I'm out
>> >> >>>>> of
>> >> >>>>> time to keep trying to fix it. =A0If= anyone else has some time to
>> >> >>>>> take
>> >> >>>>> a
>> >> >>>>> look, that'd be awesome,
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>>
>> >> >>>>> Jeff
>> >> >>>>>
>> >> >>>>
>> >> >>>> Mysql died, so far I'm unable to res= tart it
>> >> >>>>
>> >> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Jeff
>>
>>
>>
>> --
>> Jeff



--
Jeff
--f46d04462f8663ab4c04b633746f--