Mail archive
alpine-infra

Re: Bugs.a.o

From: Jeff Bilyk <jbilyk_at_gmail.com>
Date: Tue, 10 Jan 2012 16:37:37 -0500

Urgh - I'm not sure how to recover that then... Carlo, I've got a copy
of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
couple...invasive... recovery tricks on the copy in /varlib/mysql, but
they didn't work. In the testing we did for the bug below, the only
way I was ever able to get a copy running again was delete/restore
from backup.

Anyone have any ideas where to go from here?

On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter_at_gmail.com> wrote:
> The bad answer is, no dump available....
>
> On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
>>
>> So that failed.  We're hitting what seems to be the same bug I've hit
>> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)
>> that still hasn't been fixed.  The workaround is to delete everything
>> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump
>> backup.  I'm really hoping that clandmeter has a backup...
>>
>> 120110 21:26:39  InnoDB: Assertion failure in thread 3071633072 in
>> file btr0pcur.c line 425
>> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
>> buf_block_get_page_no(btr_pcur_get_block(cursor))
>> InnoDB: We intentionally generate a memory trap.
>> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
>> InnoDB: If you get repeated assertion failures or crashes, even
>> InnoDB: immediately after the mysqld startup, there may be
>> InnoDB: corruption in the InnoDB tablespace. Please refer to
>> InnoDB:
>> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
>> InnoDB: about forcing recovery.
>> 120110 21:26:39 - mysqld got signal 6 ;
>> This could be because you hit a bug. It is also possible that this binary
>> or one of the libraries it was linked against is corrupt, improperly
>> built,
>> or misconfigured. This error can also be caused by malfunctioning
>> hardware.
>> We will try our best to scrape up some info that will hopefully help
>> diagnose
>> the problem, but since we have already crashed, something is definitely
>> wrong
>> and this may fail.
>>
>> key_buffer_size=16777216
>> read_buffer_size=262144
>> max_used_connections=0
>> max_threads=151
>> thread_count=0
>> connection_count=0
>> It is possible that mysqld could use up to
>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
>> 133434 K
>> bytes of memory
>> Hope that's ok; if not, decrease some variables in the equation.
>>
>> Thread pointer: 0x0
>> Attempting backtrace. You can use the following information to find out
>> where mysqld died. If you see no messages after this, something went
>> terribly wrong...
>> stack_bottom = (nil) thread_stack 0x30000
>> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
>> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
>> contains
>> information that should help you find out what is causing the crash.
>> 120110 21:26:39 mysqld_safe mysqld from pid file
>> /var/run/mysqld/mysqld.pid ended
>>
>> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk_at_gmail.com> wrote:
>> > Only other thing I can think to try is to dump the binlogs since it's
>> > during recovery that the crash occurs.  Logs say there's only one
>> > transaction that's trying to apply...
>> >
>> > Jeff
>> >
>> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> > wrote:
>> >
>> >> I'm getting the below, not sure what to do now. Already updated mysql
>> >> to latest 2.2 version.
>> >>
>> >>
>> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.
>> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415
>> >> 120110 19:53:23  InnoDB: Database was not shut down normally!
>> >> InnoDB: Starting crash recovery.
>> >> InnoDB: Reading tablespace information from the .ibd files...
>> >> InnoDB: Restoring possible half-written data pages from the doublewrite
>> >> InnoDB: buffer...
>> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045
>> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up
>> >> InnoDB: in total 12 row operations to undo
>> >> InnoDB: Trx id counter is 6DE8E00
>> >> 120110 19:53:24  InnoDB: Starting an apply batch of log records to the
>> >> database...
>> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal
>> >> 11 ;
>> >> This could be because you hit a bug. It is also possible that this
>> >> binary
>> >> or one of the libraries it was linked against is corrupt, improperly
>> >> built,
>> >> or misconfigured. This error can also be caused by malfunctioning
>> >> hardware.
>> >> We will try our best to scrape up some info that will hopefully help
>> >> diagnose
>> >> the problem, but since we have already crashed, something is definitely
>> >> wrong
>> >> and this may fail.
>> >>
>> >> key_buffer_size=16777216
>> >> read_buffer_size=262144
>> >> max_used_connections=0
>> >> max_threads=151
>> >> thread_count=0
>> >> connection_count=0
>> >> It is possible that mysqld could use up to
>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
>> >> 133434 K
>> >> bytes of memory
>> >> Hope that's ok; if not, decrease some variables in the equation.
>> >>
>> >> Thread pointer: 0x0
>> >> Attempting backtrace. You can use the following information to find out
>> >> where mysqld died. If you see no messages after this, something went
>> >> terribly wrong...
>> >> stack_bottom = (nil) thread_stack 0x30000
>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
>> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
>> >> contains
>> >> information that should help you find out what is causing the crash.
>> >> 120110 19:53:24 mysqld_safe mysqld from pid file
>> >> /var/run/mysqld/mysqld.pid ended
>> >>
>> >>
>> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter_at_gmail.com>
>> >> wrote:
>> >>> Ill be able to check in a minute if needed.
>> >>>
>> >>> Carlo
>> >>>
>> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel_at_nothome.org>
>> >>> wrote:
>> >>>>
>> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
>> >>>>>
>> >>>>> Hi Everyone,
>> >>>>>
>> >>>>> Redmine on bugs.a.o is having issues this afternoon.  Fastcgi is
>> >>>>> crashing, by the looks of things because of multiple spiders hitting
>> >>>>> it at once.  Banning the user agents in lightty config hasn't
>> >>>>> helped,
>> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of
>> >>>>> time to keep trying to fix it.  If anyone else has some time to take
>> >>>>> a
>> >>>>> look, that'd be awesome,
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Jeff
>> >>>>>
>> >>>>
>> >>>> Mysql died, so far I'm unable to restart it
>> >>>>
>> >>>
>>
>>
>>
>> --
>> Jeff



-- 
Jeff
Received on Tue Jan 10 2012 - 16:37:37 GMT