Mail archive
alpine-infra

Re: Bugs.a.o

From: Carlo Landmeter <clandmeter_at_gmail.com>
Date: Wed, 11 Jan 2012 01:34:30 +0100

Im not able to bring it back online. I'm trying with ubuntu and start
the db (same version) but mysql crashes again.

If somebody else wants to take a look? im going to bed.

gnite.

On Tue, Jan 10, 2012 at 10:53 PM, Carlo Landmeter <clandmeter_at_gmail.com> wrote:
> Also had issues with pg on uc. It's the price we pay off trying to be
> differed...
>
> On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>>
>> Thanks.  I should be back home in a few hours, if you need a hand, I
>> can give it a try.  Fwiw, Ubuntu's MySQL package doesn't suffer the
>> same bug as far as we could tell.  Fabled figured it was something in
>> uclibc iirc that was causing this crash, but we weren't able to
>> isolate exactly what the bug was...
>>
>> On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> wrote:
>> > Should be no problem. It's an vserver. Ill try tonight.
>> >
>> > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>> >>
>> >> It's worth a shot, but I don't have the time/a box at work to try.
>> >> Could you give it a shot?
>> >>
>> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> >> wrote:
>> >> > What about starting this var on Ubuntu or something and export?
>> >> >
>> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>> >> >>
>> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a
>> >> >> copy
>> >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
>> >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql,
>> >> >> but
>> >> >> they didn't work.  In the testing we did for the bug below, the only
>> >> >> way I was ever able to get a copy running again was delete/restore
>> >> >> from backup.
>> >> >>
>> >> >> Anyone have any ideas where to go from here?
>> >> >>
>> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter
>> >> >> <clandmeter_at_gmail.com>
>> >> >> wrote:
>> >> >> > The bad answer is, no dump available....
>> >> >> >
>> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>> >> >> >>
>> >> >> >> So that failed.  We're hitting what seems to be the same bug I've
>> >> >> >> hit
>> >> >> >> before in MySQL on Alpine
>> >> >> >> (http://bugs.mysql.com/bug.php?id=62634)
>> >> >> >> that still hasn't been fixed.  The workaround is to delete
>> >> >> >> everything
>> >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a
>> >> >> >> mysqldump
>> >> >> >> backup.  I'm really hoping that clandmeter has a backup...
>> >> >> >>
>> >> >> >> 120110 21:26:39  InnoDB: Assertion failure in thread 3071633072
>> >> >> >> in
>> >> >> >> file btr0pcur.c line 425
>> >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
>> >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))
>> >> >> >> InnoDB: We intentionally generate a memory trap.
>> >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> >> >> >> InnoDB: If you get repeated assertion failures or crashes, even
>> >> >> >> InnoDB: immediately after the mysqld startup, there may be
>> >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to
>> >> >> >> InnoDB:
>> >> >> >>
>> >> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
>> >> >> >> InnoDB: about forcing recovery.
>> >> >> >> 120110 21:26:39 - mysqld got signal 6 ;
>> >> >> >> This could be because you hit a bug. It is also possible that
>> >> >> >> this
>> >> >> >> binary
>> >> >> >> or one of the libraries it was linked against is corrupt,
>> >> >> >> improperly
>> >> >> >> built,
>> >> >> >> or misconfigured. This error can also be caused by malfunctioning
>> >> >> >> hardware.
>> >> >> >> We will try our best to scrape up some info that will hopefully
>> >> >> >> help
>> >> >> >> diagnose
>> >> >> >> the problem, but since we have already crashed, something is
>> >> >> >> definitely
>> >> >> >> wrong
>> >> >> >> and this may fail.
>> >> >> >>
>> >> >> >> key_buffer_size=16777216
>> >> >> >> read_buffer_size=262144
>> >> >> >> max_used_connections=0
>> >> >> >> max_threads=151
>> >> >> >> thread_count=0
>> >> >> >> connection_count=0
>> >> >> >> It is possible that mysqld could use up to
>> >> >> >> key_buffer_size + (read_buffer_size +
>> >> >> >> sort_buffer_size)*max_threads
>> >> >> >> =
>> >> >> >> 133434 K
>> >> >> >> bytes of memory
>> >> >> >> Hope that's ok; if not, decrease some variables in the equation.
>> >> >> >>
>> >> >> >> Thread pointer: 0x0
>> >> >> >> Attempting backtrace. You can use the following information to
>> >> >> >> find
>> >> >> >> out
>> >> >> >> where mysqld died. If you see no messages after this, something
>> >> >> >> went
>> >> >> >> terribly wrong...
>> >> >> >> stack_bottom = (nil) thread_stack 0x30000
>> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
>> >> >> >> The manual page at
>> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html
>> >> >> >> contains
>> >> >> >> information that should help you find out what is causing the
>> >> >> >> crash.
>> >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file
>> >> >> >> /var/run/mysqld/mysqld.pid ended
>> >> >> >>
>> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk_at_gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Only other thing I can think to try is to dump the binlogs
>> >> >> >> > since
>> >> >> >> > it's
>> >> >> >> > during recovery that the crash occurs.  Logs say there's only
>> >> >> >> > one
>> >> >> >> > transaction that's trying to apply...
>> >> >> >> >
>> >> >> >> > Jeff
>> >> >> >> >
>> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter
>> >> >> >> > <clandmeter_at_gmail.com>
>> >> >> >> > wrote:
>> >> >> >> >
>> >> >> >> >> I'm getting the below, not sure what to do now. Already
>> >> >> >> >> updated
>> >> >> >> >> mysql
>> >> >> >> >> to latest 2.2 version.
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is
>> >> >> >> >> Barracuda.
>> >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn
>> >> >> >> >> 15160684415
>> >> >> >> >> 120110 19:53:23  InnoDB: Database was not shut down normally!
>> >> >> >> >> InnoDB: Starting crash recovery.
>> >> >> >> >> InnoDB: Reading tablespace information from the .ibd files...
>> >> >> >> >> InnoDB: Restoring possible half-written data pages from the
>> >> >> >> >> doublewrite
>> >> >> >> >> InnoDB: buffer...
>> >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number
>> >> >> >> >> 15163771045
>> >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned
>> >> >> >> >> up
>> >> >> >> >> InnoDB: in total 12 row operations to undo
>> >> >> >> >> InnoDB: Trx id counter is 6DE8E00
>> >> >> >> >> 120110 19:53:24  InnoDB: Starting an apply batch of log
>> >> >> >> >> records
>> >> >> >> >> to
>> >> >> >> >> the
>> >> >> >> >> database...
>> >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got
>> >> >> >> >> signal
>> >> >> >> >> 11 ;
>> >> >> >> >> This could be because you hit a bug. It is also possible that
>> >> >> >> >> this
>> >> >> >> >> binary
>> >> >> >> >> or one of the libraries it was linked against is corrupt,
>> >> >> >> >> improperly
>> >> >> >> >> built,
>> >> >> >> >> or misconfigured. This error can also be caused by
>> >> >> >> >> malfunctioning
>> >> >> >> >> hardware.
>> >> >> >> >> We will try our best to scrape up some info that will
>> >> >> >> >> hopefully
>> >> >> >> >> help
>> >> >> >> >> diagnose
>> >> >> >> >> the problem, but since we have already crashed, something is
>> >> >> >> >> definitely
>> >> >> >> >> wrong
>> >> >> >> >> and this may fail.
>> >> >> >> >>
>> >> >> >> >> key_buffer_size=16777216
>> >> >> >> >> read_buffer_size=262144
>> >> >> >> >> max_used_connections=0
>> >> >> >> >> max_threads=151
>> >> >> >> >> thread_count=0
>> >> >> >> >> connection_count=0
>> >> >> >> >> It is possible that mysqld could use up to
>> >> >> >> >> key_buffer_size + (read_buffer_size +
>> >> >> >> >> sort_buffer_size)*max_threads
>> >> >> >> >> =
>> >> >> >> >> 133434 K
>> >> >> >> >> bytes of memory
>> >> >> >> >> Hope that's ok; if not, decrease some variables in the
>> >> >> >> >> equation.
>> >> >> >> >>
>> >> >> >> >> Thread pointer: 0x0
>> >> >> >> >> Attempting backtrace. You can use the following information to
>> >> >> >> >> find
>> >> >> >> >> out
>> >> >> >> >> where mysqld died. If you see no messages after this,
>> >> >> >> >> something
>> >> >> >> >> went
>> >> >> >> >> terribly wrong...
>> >> >> >> >> stack_bottom = (nil) thread_stack 0x30000
>> >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
>> >> >> >> >> The manual page at
>> >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html
>> >> >> >> >> contains
>> >> >> >> >> information that should help you find out what is causing the
>> >> >> >> >> crash.
>> >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file
>> >> >> >> >> /var/run/mysqld/mysqld.pid ended
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter
>> >> >> >> >> <clandmeter_at_gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >>> Ill be able to check in a minute if needed.
>> >> >> >> >>>
>> >> >> >> >>> Carlo
>> >> >> >> >>>
>> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos"
>> >> >> >> >>> <nangel_at_nothome.org>
>> >> >> >> >>> wrote:
>> >> >> >> >>>>
>> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
>> >> >> >> >>>>>
>> >> >> >> >>>>> Hi Everyone,
>> >> >> >> >>>>>
>> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon.
>> >> >> >> >>>>>  Fastcgi
>> >> >> >> >>>>> is
>> >> >> >> >>>>> crashing, by the looks of things because of multiple
>> >> >> >> >>>>> spiders
>> >> >> >> >>>>> hitting
>> >> >> >> >>>>> it at once.  Banning the user agents in lightty config
>> >> >> >> >>>>> hasn't
>> >> >> >> >>>>> helped,
>> >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but
>> >> >> >> >>>>> I'm
>> >> >> >> >>>>> out
>> >> >> >> >>>>> of
>> >> >> >> >>>>> time to keep trying to fix it.  If anyone else has some
>> >> >> >> >>>>> time
>> >> >> >> >>>>> to
>> >> >> >> >>>>> take
>> >> >> >> >>>>> a
>> >> >> >> >>>>> look, that'd be awesome,
>> >> >> >> >>>>>
>> >> >> >> >>>>> Thanks,
>> >> >> >> >>>>>
>> >> >> >> >>>>> Jeff
>> >> >> >> >>>>>
>> >> >> >> >>>>
>> >> >> >> >>>> Mysql died, so far I'm unable to restart it
>> >> >> >> >>>>
>> >> >> >> >>>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Jeff
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Jeff
>> >>
>> >>
>> >>
>> >> --
>> >> Jeff
>>
>>
>>
>> --
>> Jeff
Received on Wed Jan 11 2012 - 01:34:30 GMT