Mail archive
alpine-infra

Re: Bugs.a.o

From: Jeff Bilyk <jbilyk_at_gmail.com>
Date: Tue, 10 Jan 2012 16:44:49 -0500

It's worth a shot, but I don't have the time/a box at work to try.
Could you give it a shot?

On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter_at_gmail.com> wrote:
> What about starting this var on Ubuntu or something and export?
>
> On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>>
>> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy
>> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
>> couple...invasive... recovery tricks on the copy in /varlib/mysql, but
>> they didn't work.  In the testing we did for the bug below, the only
>> way I was ever able to get a copy running again was delete/restore
>> from backup.
>>
>> Anyone have any ideas where to go from here?
>>
>> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> wrote:
>> > The bad answer is, no dump available....
>> >
>> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>> >>
>> >> So that failed.  We're hitting what seems to be the same bug I've hit
>> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)
>> >> that still hasn't been fixed.  The workaround is to delete everything
>> >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump
>> >> backup.  I'm really hoping that clandmeter has a backup...
>> >>
>> >> 120110 21:26:39  InnoDB: Assertion failure in thread 3071633072 in
>> >> file btr0pcur.c line 425
>> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
>> >> buf_block_get_page_no(btr_pcur_get_block(cursor))
>> >> InnoDB: We intentionally generate a memory trap.
>> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> >> InnoDB: If you get repeated assertion failures or crashes, even
>> >> InnoDB: immediately after the mysqld startup, there may be
>> >> InnoDB: corruption in the InnoDB tablespace. Please refer to
>> >> InnoDB:
>> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
>> >> InnoDB: about forcing recovery.
>> >> 120110 21:26:39 - mysqld got signal 6 ;
>> >> This could be because you hit a bug. It is also possible that this
>> >> binary
>> >> or one of the libraries it was linked against is corrupt, improperly
>> >> built,
>> >> or misconfigured. This error can also be caused by malfunctioning
>> >> hardware.
>> >> We will try our best to scrape up some info that will hopefully help
>> >> diagnose
>> >> the problem, but since we have already crashed, something is definitely
>> >> wrong
>> >> and this may fail.
>> >>
>> >> key_buffer_size=16777216
>> >> read_buffer_size=262144
>> >> max_used_connections=0
>> >> max_threads=151
>> >> thread_count=0
>> >> connection_count=0
>> >> It is possible that mysqld could use up to
>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
>> >> 133434 K
>> >> bytes of memory
>> >> Hope that's ok; if not, decrease some variables in the equation.
>> >>
>> >> Thread pointer: 0x0
>> >> Attempting backtrace. You can use the following information to find out
>> >> where mysqld died. If you see no messages after this, something went
>> >> terribly wrong...
>> >> stack_bottom = (nil) thread_stack 0x30000
>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
>> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
>> >> contains
>> >> information that should help you find out what is causing the crash.
>> >> 120110 21:26:39 mysqld_safe mysqld from pid file
>> >> /var/run/mysqld/mysqld.pid ended
>> >>
>> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk_at_gmail.com> wrote:
>> >> > Only other thing I can think to try is to dump the binlogs since it's
>> >> > during recovery that the crash occurs.  Logs say there's only one
>> >> > transaction that's trying to apply...
>> >> >
>> >> > Jeff
>> >> >
>> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> >> > wrote:
>> >> >
>> >> >> I'm getting the below, not sure what to do now. Already updated
>> >> >> mysql
>> >> >> to latest 2.2 version.
>> >> >>
>> >> >>
>> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.
>> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415
>> >> >> 120110 19:53:23  InnoDB: Database was not shut down normally!
>> >> >> InnoDB: Starting crash recovery.
>> >> >> InnoDB: Reading tablespace information from the .ibd files...
>> >> >> InnoDB: Restoring possible half-written data pages from the
>> >> >> doublewrite
>> >> >> InnoDB: buffer...
>> >> >> InnoDB: Doing recovery: scanned up to log sequence number
>> >> >> 15163771045
>> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up
>> >> >> InnoDB: in total 12 row operations to undo
>> >> >> InnoDB: Trx id counter is 6DE8E00
>> >> >> 120110 19:53:24  InnoDB: Starting an apply batch of log records to
>> >> >> the
>> >> >> database...
>> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got
>> >> >> signal
>> >> >> 11 ;
>> >> >> This could be because you hit a bug. It is also possible that this
>> >> >> binary
>> >> >> or one of the libraries it was linked against is corrupt, improperly
>> >> >> built,
>> >> >> or misconfigured. This error can also be caused by malfunctioning
>> >> >> hardware.
>> >> >> We will try our best to scrape up some info that will hopefully help
>> >> >> diagnose
>> >> >> the problem, but since we have already crashed, something is
>> >> >> definitely
>> >> >> wrong
>> >> >> and this may fail.
>> >> >>
>> >> >> key_buffer_size=16777216
>> >> >> read_buffer_size=262144
>> >> >> max_used_connections=0
>> >> >> max_threads=151
>> >> >> thread_count=0
>> >> >> connection_count=0
>> >> >> It is possible that mysqld could use up to
>> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads
>> >> >> =
>> >> >> 133434 K
>> >> >> bytes of memory
>> >> >> Hope that's ok; if not, decrease some variables in the equation.
>> >> >>
>> >> >> Thread pointer: 0x0
>> >> >> Attempting backtrace. You can use the following information to find
>> >> >> out
>> >> >> where mysqld died. If you see no messages after this, something went
>> >> >> terribly wrong...
>> >> >> stack_bottom = (nil) thread_stack 0x30000
>> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
>> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
>> >> >> contains
>> >> >> information that should help you find out what is causing the crash.
>> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file
>> >> >> /var/run/mysqld/mysqld.pid ended
>> >> >>
>> >> >>
>> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter
>> >> >> <clandmeter_at_gmail.com>
>> >> >> wrote:
>> >> >>> Ill be able to check in a minute if needed.
>> >> >>>
>> >> >>> Carlo
>> >> >>>
>> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel_at_nothome.org>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
>> >> >>>>>
>> >> >>>>> Hi Everyone,
>> >> >>>>>
>> >> >>>>> Redmine on bugs.a.o is having issues this afternoon.  Fastcgi is
>> >> >>>>> crashing, by the looks of things because of multiple spiders
>> >> >>>>> hitting
>> >> >>>>> it at once.  Banning the user agents in lightty config hasn't
>> >> >>>>> helped,
>> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out
>> >> >>>>> of
>> >> >>>>> time to keep trying to fix it.  If anyone else has some time to
>> >> >>>>> take
>> >> >>>>> a
>> >> >>>>> look, that'd be awesome,
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>>
>> >> >>>>> Jeff
>> >> >>>>>
>> >> >>>>
>> >> >>>> Mysql died, so far I'm unable to restart it
>> >> >>>>
>> >> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Jeff
>>
>>
>>
>> --
>> Jeff



-- 
Jeff
Received on Tue Jan 10 2012 - 16:44:49 GMT