Mail archive
alpine-infra

Re: Bugs.a.o

From: Carlo Landmeter <clandmeter_at_gmail.com>
Date: Tue, 10 Jan 2012 22:40:10 +0100

What about starting this var on Ubuntu or something and export?
On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:

> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy
> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
> couple...invasive... recovery tricks on the copy in /varlib/mysql, but
> they didn't work. In the testing we did for the bug below, the only
> way I was ever able to get a copy running again was delete/restore
> from backup.
>
> Anyone have any ideas where to go from here?
>
> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter_at_gmail.com>
> wrote:
> > The bad answer is, no dump available....
> >
> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
> >>
> >> So that failed. We're hitting what seems to be the same bug I've hit
> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)
> >> that still hasn't been fixed. The workaround is to delete everything
> >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump
> >> backup. I'm really hoping that clandmeter has a backup...
> >>
> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in
> >> file btr0pcur.c line 425
> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
> >> buf_block_get_page_no(btr_pcur_get_block(cursor))
> >> InnoDB: We intentionally generate a memory trap.
> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
> >> InnoDB: If you get repeated assertion failures or crashes, even
> >> InnoDB: immediately after the mysqld startup, there may be
> >> InnoDB: corruption in the InnoDB tablespace. Please refer to
> >> InnoDB:
> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> >> InnoDB: about forcing recovery.
> >> 120110 21:26:39 - mysqld got signal 6 ;
> >> This could be because you hit a bug. It is also possible that this
> binary
> >> or one of the libraries it was linked against is corrupt, improperly
> >> built,
> >> or misconfigured. This error can also be caused by malfunctioning
> >> hardware.
> >> We will try our best to scrape up some info that will hopefully help
> >> diagnose
> >> the problem, but since we have already crashed, something is definitely
> >> wrong
> >> and this may fail.
> >>
> >> key_buffer_size=16777216
> >> read_buffer_size=262144
> >> max_used_connections=0
> >> max_threads=151
> >> thread_count=0
> >> connection_count=0
> >> It is possible that mysqld could use up to
> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
> >> 133434 K
> >> bytes of memory
> >> Hope that's ok; if not, decrease some variables in the equation.
> >>
> >> Thread pointer: 0x0
> >> Attempting backtrace. You can use the following information to find out
> >> where mysqld died. If you see no messages after this, something went
> >> terribly wrong...
> >> stack_bottom = (nil) thread_stack 0x30000
> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
> >> contains
> >> information that should help you find out what is causing the crash.
> >> 120110 21:26:39 mysqld_safe mysqld from pid file
> >> /var/run/mysqld/mysqld.pid ended
> >>
> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk_at_gmail.com> wrote:
> >> > Only other thing I can think to try is to dump the binlogs since it's
> >> > during recovery that the crash occurs. Logs say there's only one
> >> > transaction that's trying to apply...
> >> >
> >> > Jeff
> >> >
> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter_at_gmail.com>
> >> > wrote:
> >> >
> >> >> I'm getting the below, not sure what to do now. Already updated mysql
> >> >> to latest 2.2 version.
> >> >>
> >> >>
> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.
> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415
> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!
> >> >> InnoDB: Starting crash recovery.
> >> >> InnoDB: Reading tablespace information from the .ibd files...
> >> >> InnoDB: Restoring possible half-written data pages from the
> doublewrite
> >> >> InnoDB: buffer...
> >> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045
> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up
> >> >> InnoDB: in total 12 row operations to undo
> >> >> InnoDB: Trx id counter is 6DE8E00
> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to
> the
> >> >> database...
> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got
> signal
> >> >> 11 ;
> >> >> This could be because you hit a bug. It is also possible that this
> >> >> binary
> >> >> or one of the libraries it was linked against is corrupt, improperly
> >> >> built,
> >> >> or misconfigured. This error can also be caused by malfunctioning
> >> >> hardware.
> >> >> We will try our best to scrape up some info that will hopefully help
> >> >> diagnose
> >> >> the problem, but since we have already crashed, something is
> definitely
> >> >> wrong
> >> >> and this may fail.
> >> >>
> >> >> key_buffer_size=16777216
> >> >> read_buffer_size=262144
> >> >> max_used_connections=0
> >> >> max_threads=151
> >> >> thread_count=0
> >> >> connection_count=0
> >> >> It is possible that mysqld could use up to
> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
> >> >> 133434 K
> >> >> bytes of memory
> >> >> Hope that's ok; if not, decrease some variables in the equation.
> >> >>
> >> >> Thread pointer: 0x0
> >> >> Attempting backtrace. You can use the following information to find
> out
> >> >> where mysqld died. If you see no messages after this, something went
> >> >> terribly wrong...
> >> >> stack_bottom = (nil) thread_stack 0x30000
> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
> >> >> contains
> >> >> information that should help you find out what is causing the crash.
> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file
> >> >> /var/run/mysqld/mysqld.pid ended
> >> >>
> >> >>
> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <
> clandmeter_at_gmail.com>
> >> >> wrote:
> >> >>> Ill be able to check in a minute if needed.
> >> >>>
> >> >>> Carlo
> >> >>>
> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel_at_nothome.org>
> >> >>> wrote:
> >> >>>>
> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
> >> >>>>>
> >> >>>>> Hi Everyone,
> >> >>>>>
> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is
> >> >>>>> crashing, by the looks of things because of multiple spiders
> hitting
> >> >>>>> it at once. Banning the user agents in lightty config hasn't
> >> >>>>> helped,
> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out
> of
> >> >>>>> time to keep trying to fix it. If anyone else has some time to
> take
> >> >>>>> a
> >> >>>>> look, that'd be awesome,
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>>
> >> >>>>> Jeff
> >> >>>>>
> >> >>>>
> >> >>>> Mysql died, so far I'm unable to restart it
> >> >>>>
> >> >>>
> >>
> >>
> >>
> >> --
> >> Jeff
>
>
>
> --
> Jeff
>
Received on Tue Jan 10 2012 - 22:40:10 GMT