Mail archive
alpine-infra

Re: Bugs.a.o

From: Carlo Landmeter <clandmeter_at_gmail.com>
Date: Tue, 10 Jan 2012 22:53:47 +0100

Also had issues with pg on uc. It's the price we pay off trying to be
differed...
On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:

> Thanks. I should be back home in a few hours, if you need a hand, I
> can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the
> same bug as far as we could tell. Fabled figured it was something in
> uclibc iirc that was causing this crash, but we weren't able to
> isolate exactly what the bug was...
>
> On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter <clandmeter_at_gmail.com>
> wrote:
> > Should be no problem. It's an vserver. Ill try tonight.
> >
> > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
> >>
> >> It's worth a shot, but I don't have the time/a box at work to try.
> >> Could you give it a shot?
> >>
> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter_at_gmail.com>
> >> wrote:
> >> > What about starting this var on Ubuntu or something and export?
> >> >
> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
> >> >>
> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a
> copy
> >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
> >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql,
> but
> >> >> they didn't work. In the testing we did for the bug below, the only
> >> >> way I was ever able to get a copy running again was delete/restore
> >> >> from backup.
> >> >>
> >> >> Anyone have any ideas where to go from here?
> >> >>
> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <
> clandmeter_at_gmail.com>
> >> >> wrote:
> >> >> > The bad answer is, no dump available....
> >> >> >
> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk_at_gmail.com> wrote:
> >> >> >>
> >> >> >> So that failed. We're hitting what seems to be the same bug I've
> >> >> >> hit
> >> >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634
> )
> >> >> >> that still hasn't been fixed. The workaround is to delete
> >> >> >> everything
> >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a
> >> >> >> mysqldump
> >> >> >> backup. I'm really hoping that clandmeter has a backup...
> >> >> >>
> >> >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in
> >> >> >> file btr0pcur.c line 425
> >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
> >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))
> >> >> >> InnoDB: We intentionally generate a memory trap.
> >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
> >> >> >> InnoDB: If you get repeated assertion failures or crashes, even
> >> >> >> InnoDB: immediately after the mysqld startup, there may be
> >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to
> >> >> >> InnoDB:
> >> >> >>
> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
> >> >> >> InnoDB: about forcing recovery.
> >> >> >> 120110 21:26:39 - mysqld got signal 6 ;
> >> >> >> This could be because you hit a bug. It is also possible that this
> >> >> >> binary
> >> >> >> or one of the libraries it was linked against is corrupt,
> improperly
> >> >> >> built,
> >> >> >> or misconfigured. This error can also be caused by malfunctioning
> >> >> >> hardware.
> >> >> >> We will try our best to scrape up some info that will hopefully
> help
> >> >> >> diagnose
> >> >> >> the problem, but since we have already crashed, something is
> >> >> >> definitely
> >> >> >> wrong
> >> >> >> and this may fail.
> >> >> >>
> >> >> >> key_buffer_size=16777216
> >> >> >> read_buffer_size=262144
> >> >> >> max_used_connections=0
> >> >> >> max_threads=151
> >> >> >> thread_count=0
> >> >> >> connection_count=0
> >> >> >> It is possible that mysqld could use up to
> >> >> >> key_buffer_size + (read_buffer_size +
> sort_buffer_size)*max_threads
> >> >> >> =
> >> >> >> 133434 K
> >> >> >> bytes of memory
> >> >> >> Hope that's ok; if not, decrease some variables in the equation.
> >> >> >>
> >> >> >> Thread pointer: 0x0
> >> >> >> Attempting backtrace. You can use the following information to
> find
> >> >> >> out
> >> >> >> where mysqld died. If you see no messages after this, something
> went
> >> >> >> terribly wrong...
> >> >> >> stack_bottom = (nil) thread_stack 0x30000
> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
> >> >> >> The manual page at
> http://dev.mysql.com/doc/mysql/en/crashing.html
> >> >> >> contains
> >> >> >> information that should help you find out what is causing the
> crash.
> >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file
> >> >> >> /var/run/mysqld/mysqld.pid ended
> >> >> >>
> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk_at_gmail.com>
> >> >> >> wrote:
> >> >> >> > Only other thing I can think to try is to dump the binlogs since
> >> >> >> > it's
> >> >> >> > during recovery that the crash occurs. Logs say there's only
> one
> >> >> >> > transaction that's trying to apply...
> >> >> >> >
> >> >> >> > Jeff
> >> >> >> >
> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter
> >> >> >> > <clandmeter_at_gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> I'm getting the below, not sure what to do now. Already updated
> >> >> >> >> mysql
> >> >> >> >> to latest 2.2 version.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is
> >> >> >> >> Barracuda.
> >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415
> >> >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!
> >> >> >> >> InnoDB: Starting crash recovery.
> >> >> >> >> InnoDB: Reading tablespace information from the .ibd files...
> >> >> >> >> InnoDB: Restoring possible half-written data pages from the
> >> >> >> >> doublewrite
> >> >> >> >> InnoDB: buffer...
> >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number
> >> >> >> >> 15163771045
> >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned
> up
> >> >> >> >> InnoDB: in total 12 row operations to undo
> >> >> >> >> InnoDB: Trx id counter is 6DE8E00
> >> >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records
> >> >> >> >> to
> >> >> >> >> the
> >> >> >> >> database...
> >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got
> >> >> >> >> signal
> >> >> >> >> 11 ;
> >> >> >> >> This could be because you hit a bug. It is also possible that
> >> >> >> >> this
> >> >> >> >> binary
> >> >> >> >> or one of the libraries it was linked against is corrupt,
> >> >> >> >> improperly
> >> >> >> >> built,
> >> >> >> >> or misconfigured. This error can also be caused by
> malfunctioning
> >> >> >> >> hardware.
> >> >> >> >> We will try our best to scrape up some info that will hopefully
> >> >> >> >> help
> >> >> >> >> diagnose
> >> >> >> >> the problem, but since we have already crashed, something is
> >> >> >> >> definitely
> >> >> >> >> wrong
> >> >> >> >> and this may fail.
> >> >> >> >>
> >> >> >> >> key_buffer_size=16777216
> >> >> >> >> read_buffer_size=262144
> >> >> >> >> max_used_connections=0
> >> >> >> >> max_threads=151
> >> >> >> >> thread_count=0
> >> >> >> >> connection_count=0
> >> >> >> >> It is possible that mysqld could use up to
> >> >> >> >> key_buffer_size + (read_buffer_size +
> >> >> >> >> sort_buffer_size)*max_threads
> >> >> >> >> =
> >> >> >> >> 133434 K
> >> >> >> >> bytes of memory
> >> >> >> >> Hope that's ok; if not, decrease some variables in the
> equation.
> >> >> >> >>
> >> >> >> >> Thread pointer: 0x0
> >> >> >> >> Attempting backtrace. You can use the following information to
> >> >> >> >> find
> >> >> >> >> out
> >> >> >> >> where mysqld died. If you see no messages after this, something
> >> >> >> >> went
> >> >> >> >> terribly wrong...
> >> >> >> >> stack_bottom = (nil) thread_stack 0x30000
> >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
> >> >> >> >> The manual page at
> >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html
> >> >> >> >> contains
> >> >> >> >> information that should help you find out what is causing the
> >> >> >> >> crash.
> >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file
> >> >> >> >> /var/run/mysqld/mysqld.pid ended
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter
> >> >> >> >> <clandmeter_at_gmail.com>
> >> >> >> >> wrote:
> >> >> >> >>> Ill be able to check in a minute if needed.
> >> >> >> >>>
> >> >> >> >>> Carlo
> >> >> >> >>>
> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <
> nangel_at_nothome.org>
> >> >> >> >>> wrote:
> >> >> >> >>>>
> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
> >> >> >> >>>>>
> >> >> >> >>>>> Hi Everyone,
> >> >> >> >>>>>
> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon.
> Fastcgi
> >> >> >> >>>>> is
> >> >> >> >>>>> crashing, by the looks of things because of multiple spiders
> >> >> >> >>>>> hitting
> >> >> >> >>>>> it at once. Banning the user agents in lightty config
> hasn't
> >> >> >> >>>>> helped,
> >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but
> I'm
> >> >> >> >>>>> out
> >> >> >> >>>>> of
> >> >> >> >>>>> time to keep trying to fix it. If anyone else has some time
> >> >> >> >>>>> to
> >> >> >> >>>>> take
> >> >> >> >>>>> a
> >> >> >> >>>>> look, that'd be awesome,
> >> >> >> >>>>>
> >> >> >> >>>>> Thanks,
> >> >> >> >>>>>
> >> >> >> >>>>> Jeff
> >> >> >> >>>>>
> >> >> >> >>>>
> >> >> >> >>>> Mysql died, so far I'm unable to restart it
> >> >> >> >>>>
> >> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Jeff
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jeff
> >>
> >>
> >>
> >> --
> >> Jeff
>
>
>
> --
> Jeff
>
Received on Tue Jan 10 2012 - 22:53:47 GMT