Hi Everyone,
Redmine on bugs.a.o is having issues this afternoon. Fastcgi is
crashing, by the looks of things because of multiple spiders hitting
it at once. Banning the user agents in lightty config hasn't helped,
neither has upping the redmine sockets to 32 (from 4) but I'm out of
time to keep trying to fix it. If anyone else has some time to take a
look, that'd be awesome,
Thanks,
Jeff
On 01/10/2012 09:49 AM, Jeff Bilyk wrote:
> Hi Everyone,>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is> crashing, by the looks of things because of multiple spiders hitting> it at once. Banning the user agents in lightty config hasn't helped,> neither has upping the redmine sockets to 32 (from 4) but I'm out of> time to keep trying to fix it. If anyone else has some time to take a> look, that'd be awesome,>> Thanks,>> Jeff>
Mysql died, so far I'm unable to restart it
Only other thing I can think to try is to dump the binlogs since it's during recovery that the crash occurs. Logs say there's only one transaction that's trying to apply...
Jeff
On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> I'm getting the below, not sure what to do now. Already updated mysql> to latest 2.2 version.> > > 120110 19:53:23 InnoDB: highest supported file format is Barracuda.> InnoDB: Log scan progressed past the checkpoint lsn 15160684415> 120110 19:53:23 InnoDB: Database was not shut down normally!> InnoDB: Starting crash recovery.> InnoDB: Reading tablespace information from the .ibd files...> InnoDB: Restoring possible half-written data pages from the doublewrite> InnoDB: buffer...> InnoDB: Doing recovery: scanned up to log sequence number 15163771045> InnoDB: 1 transaction(s) which must be rolled back or cleaned up> InnoDB: in total 12 row operations to undo> InnoDB: Trx id counter is 6DE8E00> 120110 19:53:24 InnoDB: Starting an apply batch of log records to the> database...> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal 11 ;> This could be because you hit a bug. It is also possible that this binary> or one of the libraries it was linked against is corrupt, improperly built,> or misconfigured. This error can also be caused by malfunctioning hardware.> We will try our best to scrape up some info that will hopefully help diagnose> the problem, but since we have already crashed, something is definitely wrong> and this may fail.> > key_buffer_size=16777216> read_buffer_size=262144> max_used_connections=0> max_threads=151> thread_count=0> connection_count=0> It is possible that mysqld could use up to> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133434 K> bytes of memory> Hope that's ok; if not, decrease some variables in the equation.> > Thread pointer: 0x0> Attempting backtrace. You can use the following information to find out> where mysqld died. If you see no messages after this, something went> terribly wrong...> stack_bottom = (nil) thread_stack 0x30000> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains> information that should help you find out what is causing the crash.> 120110 19:53:24 mysqld_safe mysqld from pid file> /var/run/mysqld/mysqld.pid ended> > > On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:>> Ill be able to check in a minute if needed.>> >> Carlo>> >> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org> wrote:>>> >>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>>>> >>>> Hi Everyone,>>>> >>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>>>> crashing, by the looks of things because of multiple spiders hitting>>>> it at once. Banning the user agents in lightty config hasn't helped,>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of>>>> time to keep trying to fix it. If anyone else has some time to take a>>>> look, that'd be awesome,>>>> >>>> Thanks,>>>> >>>> Jeff>>>> >>> >>> Mysql died, so far I'm unable to restart it>>> >>
So that failed. We're hitting what seems to be the same bug I've hit
before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)
that still hasn't been fixed. The workaround is to delete everything
in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump
backup. I'm really hoping that clandmeter has a backup...
120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in
file btr0pcur.c line 425
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==
buf_block_get_page_no(btr_pcur_get_block(cursor))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
120110 21:26:39 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133434 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120110 21:26:39 mysqld_safe mysqld from pid file
/var/run/mysqld/mysqld.pid ended
On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:
> Only other thing I can think to try is to dump the binlogs since it's during recovery that the crash occurs. Logs say there's only one transaction that's trying to apply...>> Jeff>> On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:>>> I'm getting the below, not sure what to do now. Already updated mysql>> to latest 2.2 version.>>>>>> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.>> InnoDB: Log scan progressed past the checkpoint lsn 15160684415>> 120110 19:53:23 InnoDB: Database was not shut down normally!>> InnoDB: Starting crash recovery.>> InnoDB: Reading tablespace information from the .ibd files...>> InnoDB: Restoring possible half-written data pages from the doublewrite>> InnoDB: buffer...>> InnoDB: Doing recovery: scanned up to log sequence number 15163771045>> InnoDB: 1 transaction(s) which must be rolled back or cleaned up>> InnoDB: in total 12 row operations to undo>> InnoDB: Trx id counter is 6DE8E00>> 120110 19:53:24 InnoDB: Starting an apply batch of log records to the>> database...>> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal 11 ;>> This could be because you hit a bug. It is also possible that this binary>> or one of the libraries it was linked against is corrupt, improperly built,>> or misconfigured. This error can also be caused by malfunctioning hardware.>> We will try our best to scrape up some info that will hopefully help diagnose>> the problem, but since we have already crashed, something is definitely wrong>> and this may fail.>>>> key_buffer_size=16777216>> read_buffer_size=262144>> max_used_connections=0>> max_threads=151>> thread_count=0>> connection_count=0>> It is possible that mysqld could use up to>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133434 K>> bytes of memory>> Hope that's ok; if not, decrease some variables in the equation.>>>> Thread pointer: 0x0>> Attempting backtrace. You can use the following information to find out>> where mysqld died. If you see no messages after this, something went>> terribly wrong...>> stack_bottom = (nil) thread_stack 0x30000>> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]>> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains>> information that should help you find out what is causing the crash.>> 120110 19:53:24 mysqld_safe mysqld from pid file>> /var/run/mysqld/mysqld.pid ended>>>>>> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:>>> Ill be able to check in a minute if needed.>>>>>> Carlo>>>>>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org> wrote:>>>>>>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>>>>>>>>>> Hi Everyone,>>>>>>>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>>>>> crashing, by the looks of things because of multiple spiders hitting>>>>> it at once. Banning the user agents in lightty config hasn't helped,>>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of>>>>> time to keep trying to fix it. If anyone else has some time to take a>>>>> look, that'd be awesome,>>>>>>>>>> Thanks,>>>>>>>>>> Jeff>>>>>>>>>>>>> Mysql died, so far I'm unable to restart it>>>>>>>
--
Jeff
Urgh - I'm not sure how to recover that then... Carlo, I've got a copy
of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a
couple...invasive... recovery tricks on the copy in /varlib/mysql, but
they didn't work. In the testing we did for the bug below, the only
way I was ever able to get a copy running again was delete/restore
from backup.
Anyone have any ideas where to go from here?
On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> The bad answer is, no dump available....>> On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>>>> So that failed. We're hitting what seems to be the same bug I've hit>> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)>> that still hasn't been fixed. The workaround is to delete everything>> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump>> backup. I'm really hoping that clandmeter has a backup...>>>> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in>> file btr0pcur.c line 425>> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==>> buf_block_get_page_no(btr_pcur_get_block(cursor))>> InnoDB: We intentionally generate a memory trap.>> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.>> InnoDB: If you get repeated assertion failures or crashes, even>> InnoDB: immediately after the mysqld startup, there may be>> InnoDB: corruption in the InnoDB tablespace. Please refer to>> InnoDB:>> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html>> InnoDB: about forcing recovery.>> 120110 21:26:39 - mysqld got signal 6 ;>> This could be because you hit a bug. It is also possible that this binary>> or one of the libraries it was linked against is corrupt, improperly>> built,>> or misconfigured. This error can also be caused by malfunctioning>> hardware.>> We will try our best to scrape up some info that will hopefully help>> diagnose>> the problem, but since we have already crashed, something is definitely>> wrong>> and this may fail.>>>> key_buffer_size=16777216>> read_buffer_size=262144>> max_used_connections=0>> max_threads=151>> thread_count=0>> connection_count=0>> It is possible that mysqld could use up to>> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =>> 133434 K>> bytes of memory>> Hope that's ok; if not, decrease some variables in the equation.>>>> Thread pointer: 0x0>> Attempting backtrace. You can use the following information to find out>> where mysqld died. If you see no messages after this, something went>> terribly wrong...>> stack_bottom = (nil) thread_stack 0x30000>> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]>> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html>> contains>> information that should help you find out what is causing the crash.>> 120110 21:26:39 mysqld_safe mysqld from pid file>> /var/run/mysqld/mysqld.pid ended>>>> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:>> > Only other thing I can think to try is to dump the binlogs since it's>> > during recovery that the crash occurs. Logs say there's only one>> > transaction that's trying to apply...>> >>> > Jeff>> >>> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>>> > wrote:>> >>> >> I'm getting the below, not sure what to do now. Already updated mysql>> >> to latest 2.2 version.>> >>>> >>>> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.>> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415>> >> 120110 19:53:23 InnoDB: Database was not shut down normally!>> >> InnoDB: Starting crash recovery.>> >> InnoDB: Reading tablespace information from the .ibd files...>> >> InnoDB: Restoring possible half-written data pages from the doublewrite>> >> InnoDB: buffer...>> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045>> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up>> >> InnoDB: in total 12 row operations to undo>> >> InnoDB: Trx id counter is 6DE8E00>> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to the>> >> database...>> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal>> >> 11 ;>> >> This could be because you hit a bug. It is also possible that this>> >> binary>> >> or one of the libraries it was linked against is corrupt, improperly>> >> built,>> >> or misconfigured. This error can also be caused by malfunctioning>> >> hardware.>> >> We will try our best to scrape up some info that will hopefully help>> >> diagnose>> >> the problem, but since we have already crashed, something is definitely>> >> wrong>> >> and this may fail.>> >>>> >> key_buffer_size=16777216>> >> read_buffer_size=262144>> >> max_used_connections=0>> >> max_threads=151>> >> thread_count=0>> >> connection_count=0>> >> It is possible that mysqld could use up to>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =>> >> 133434 K>> >> bytes of memory>> >> Hope that's ok; if not, decrease some variables in the equation.>> >>>> >> Thread pointer: 0x0>> >> Attempting backtrace. You can use the following information to find out>> >> where mysqld died. If you see no messages after this, something went>> >> terribly wrong...>> >> stack_bottom = (nil) thread_stack 0x30000>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]>> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html>> >> contains>> >> information that should help you find out what is causing the crash.>> >> 120110 19:53:24 mysqld_safe mysqld from pid file>> >> /var/run/mysqld/mysqld.pid ended>> >>>> >>>> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com>>> >> wrote:>> >>> Ill be able to check in a minute if needed.>> >>>>> >>> Carlo>> >>>>> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>>> >>> wrote:>> >>>>>> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>> >>>>>>> >>>>> Hi Everyone,>> >>>>>>> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>> >>>>> crashing, by the looks of things because of multiple spiders hitting>> >>>>> it at once. Banning the user agents in lightty config hasn't>> >>>>> helped,>> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of>> >>>>> time to keep trying to fix it. If anyone else has some time to take>> >>>>> a>> >>>>> look, that'd be awesome,>> >>>>>>> >>>>> Thanks,>> >>>>>>> >>>>> Jeff>> >>>>>>> >>>>>> >>>> Mysql died, so far I'm unable to restart it>> >>>>>> >>>>>>>>>>> -->> Jeff
--
Jeff
It's worth a shot, but I don't have the time/a box at work to try.
Could you give it a shot?
On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> What about starting this var on Ubuntu or something and export?>> On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>>>> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy>> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a>> couple...invasive... recovery tricks on the copy in /varlib/mysql, but>> they didn't work. In the testing we did for the bug below, the only>> way I was ever able to get a copy running again was delete/restore>> from backup.>>>> Anyone have any ideas where to go from here?>>>> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com>>> wrote:>> > The bad answer is, no dump available....>> >>> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >>>> >> So that failed. We're hitting what seems to be the same bug I've hit>> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)>> >> that still hasn't been fixed. The workaround is to delete everything>> >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump>> >> backup. I'm really hoping that clandmeter has a backup...>> >>>> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in>> >> file btr0pcur.c line 425>> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==>> >> buf_block_get_page_no(btr_pcur_get_block(cursor))>> >> InnoDB: We intentionally generate a memory trap.>> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.>> >> InnoDB: If you get repeated assertion failures or crashes, even>> >> InnoDB: immediately after the mysqld startup, there may be>> >> InnoDB: corruption in the InnoDB tablespace. Please refer to>> >> InnoDB:>> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html>> >> InnoDB: about forcing recovery.>> >> 120110 21:26:39 - mysqld got signal 6 ;>> >> This could be because you hit a bug. It is also possible that this>> >> binary>> >> or one of the libraries it was linked against is corrupt, improperly>> >> built,>> >> or misconfigured. This error can also be caused by malfunctioning>> >> hardware.>> >> We will try our best to scrape up some info that will hopefully help>> >> diagnose>> >> the problem, but since we have already crashed, something is definitely>> >> wrong>> >> and this may fail.>> >>>> >> key_buffer_size=16777216>> >> read_buffer_size=262144>> >> max_used_connections=0>> >> max_threads=151>> >> thread_count=0>> >> connection_count=0>> >> It is possible that mysqld could use up to>> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =>> >> 133434 K>> >> bytes of memory>> >> Hope that's ok; if not, decrease some variables in the equation.>> >>>> >> Thread pointer: 0x0>> >> Attempting backtrace. You can use the following information to find out>> >> where mysqld died. If you see no messages after this, something went>> >> terribly wrong...>> >> stack_bottom = (nil) thread_stack 0x30000>> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]>> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html>> >> contains>> >> information that should help you find out what is causing the crash.>> >> 120110 21:26:39 mysqld_safe mysqld from pid file>> >> /var/run/mysqld/mysqld.pid ended>> >>>> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:>> >> > Only other thing I can think to try is to dump the binlogs since it's>> >> > during recovery that the crash occurs. Logs say there's only one>> >> > transaction that's trying to apply...>> >> >>> >> > Jeff>> >> >>> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>>> >> > wrote:>> >> >>> >> >> I'm getting the below, not sure what to do now. Already updated>> >> >> mysql>> >> >> to latest 2.2 version.>> >> >>>> >> >>>> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.>> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415>> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!>> >> >> InnoDB: Starting crash recovery.>> >> >> InnoDB: Reading tablespace information from the .ibd files...>> >> >> InnoDB: Restoring possible half-written data pages from the>> >> >> doublewrite>> >> >> InnoDB: buffer...>> >> >> InnoDB: Doing recovery: scanned up to log sequence number>> >> >> 15163771045>> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up>> >> >> InnoDB: in total 12 row operations to undo>> >> >> InnoDB: Trx id counter is 6DE8E00>> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to>> >> >> the>> >> >> database...>> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got>> >> >> signal>> >> >> 11 ;>> >> >> This could be because you hit a bug. It is also possible that this>> >> >> binary>> >> >> or one of the libraries it was linked against is corrupt, improperly>> >> >> built,>> >> >> or misconfigured. This error can also be caused by malfunctioning>> >> >> hardware.>> >> >> We will try our best to scrape up some info that will hopefully help>> >> >> diagnose>> >> >> the problem, but since we have already crashed, something is>> >> >> definitely>> >> >> wrong>> >> >> and this may fail.>> >> >>>> >> >> key_buffer_size=16777216>> >> >> read_buffer_size=262144>> >> >> max_used_connections=0>> >> >> max_threads=151>> >> >> thread_count=0>> >> >> connection_count=0>> >> >> It is possible that mysqld could use up to>> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads>> >> >> =>> >> >> 133434 K>> >> >> bytes of memory>> >> >> Hope that's ok; if not, decrease some variables in the equation.>> >> >>>> >> >> Thread pointer: 0x0>> >> >> Attempting backtrace. You can use the following information to find>> >> >> out>> >> >> where mysqld died. If you see no messages after this, something went>> >> >> terribly wrong...>> >> >> stack_bottom = (nil) thread_stack 0x30000>> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]>> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html>> >> >> contains>> >> >> information that should help you find out what is causing the crash.>> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file>> >> >> /var/run/mysqld/mysqld.pid ended>> >> >>>> >> >>>> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter>> >> >> <clandmeter@gmail.com>>> >> >> wrote:>> >> >>> Ill be able to check in a minute if needed.>> >> >>>>> >> >>> Carlo>> >> >>>>> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>>> >> >>> wrote:>> >> >>>>>> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>> >> >>>>>>> >> >>>>> Hi Everyone,>> >> >>>>>>> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>> >> >>>>> crashing, by the looks of things because of multiple spiders>> >> >>>>> hitting>> >> >>>>> it at once. Banning the user agents in lightty config hasn't>> >> >>>>> helped,>> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out>> >> >>>>> of>> >> >>>>> time to keep trying to fix it. If anyone else has some time to>> >> >>>>> take>> >> >>>>> a>> >> >>>>> look, that'd be awesome,>> >> >>>>>>> >> >>>>> Thanks,>> >> >>>>>>> >> >>>>> Jeff>> >> >>>>>>> >> >>>>>> >> >>>> Mysql died, so far I'm unable to restart it>> >> >>>>>> >> >>>>> >>>> >>>> >>>> >> -->> >> Jeff>>>>>>>> -->> Jeff
--
Jeff
On 01/10/2012 01:53 PM, Carlo Landmeter wrote:
> Also had issues with pg on uc. It's the price we pay off trying to be> differed...>> On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk@gmail.com> <jbilyk@gmail.com>> wrote:>> Thanks. I should be back home in a few hours, if you need a hand, I> can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the> same bug as far as we could tell. Fabled figured it was something in> uclibc iirc that was causing this crash, but we weren't able to> isolate exactly what the bug was...>
This is good for tracking down the bug then... we have a repeatable
scenario.
Thanks. I should be back home in a few hours, if you need a hand, I
can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the
same bug as far as we could tell. Fabled figured it was something in
uclibc iirc that was causing this crash, but we weren't able to
isolate exactly what the bug was...
On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> Should be no problem. It's an vserver. Ill try tonight.>> On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>>>> It's worth a shot, but I don't have the time/a box at work to try.>> Could you give it a shot?>>>> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com>>> wrote:>> > What about starting this var on Ubuntu or something and export?>> >>> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >>>> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy>> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a>> >> couple...invasive... recovery tricks on the copy in /varlib/mysql, but>> >> they didn't work. In the testing we did for the bug below, the only>> >> way I was ever able to get a copy running again was delete/restore>> >> from backup.>> >>>> >> Anyone have any ideas where to go from here?>> >>>> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com>>> >> wrote:>> >> > The bad answer is, no dump available....>> >> >>> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >> >>>> >> >> So that failed. We're hitting what seems to be the same bug I've>> >> >> hit>> >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)>> >> >> that still hasn't been fixed. The workaround is to delete>> >> >> everything>> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a>> >> >> mysqldump>> >> >> backup. I'm really hoping that clandmeter has a backup...>> >> >>>> >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in>> >> >> file btr0pcur.c line 425>> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==>> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))>> >> >> InnoDB: We intentionally generate a memory trap.>> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.>> >> >> InnoDB: If you get repeated assertion failures or crashes, even>> >> >> InnoDB: immediately after the mysqld startup, there may be>> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to>> >> >> InnoDB:>> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html>> >> >> InnoDB: about forcing recovery.>> >> >> 120110 21:26:39 - mysqld got signal 6 ;>> >> >> This could be because you hit a bug. It is also possible that this>> >> >> binary>> >> >> or one of the libraries it was linked against is corrupt, improperly>> >> >> built,>> >> >> or misconfigured. This error can also be caused by malfunctioning>> >> >> hardware.>> >> >> We will try our best to scrape up some info that will hopefully help>> >> >> diagnose>> >> >> the problem, but since we have already crashed, something is>> >> >> definitely>> >> >> wrong>> >> >> and this may fail.>> >> >>>> >> >> key_buffer_size=16777216>> >> >> read_buffer_size=262144>> >> >> max_used_connections=0>> >> >> max_threads=151>> >> >> thread_count=0>> >> >> connection_count=0>> >> >> It is possible that mysqld could use up to>> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads>> >> >> =>> >> >> 133434 K>> >> >> bytes of memory>> >> >> Hope that's ok; if not, decrease some variables in the equation.>> >> >>>> >> >> Thread pointer: 0x0>> >> >> Attempting backtrace. You can use the following information to find>> >> >> out>> >> >> where mysqld died. If you see no messages after this, something went>> >> >> terribly wrong...>> >> >> stack_bottom = (nil) thread_stack 0x30000>> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]>> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html>> >> >> contains>> >> >> information that should help you find out what is causing the crash.>> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file>> >> >> /var/run/mysqld/mysqld.pid ended>> >> >>>> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com>>> >> >> wrote:>> >> >> > Only other thing I can think to try is to dump the binlogs since>> >> >> > it's>> >> >> > during recovery that the crash occurs. Logs say there's only one>> >> >> > transaction that's trying to apply...>> >> >> >>> >> >> > Jeff>> >> >> >>> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter>> >> >> > <clandmeter@gmail.com>>> >> >> > wrote:>> >> >> >>> >> >> >> I'm getting the below, not sure what to do now. Already updated>> >> >> >> mysql>> >> >> >> to latest 2.2 version.>> >> >> >>>> >> >> >>>> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is>> >> >> >> Barracuda.>> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415>> >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!>> >> >> >> InnoDB: Starting crash recovery.>> >> >> >> InnoDB: Reading tablespace information from the .ibd files...>> >> >> >> InnoDB: Restoring possible half-written data pages from the>> >> >> >> doublewrite>> >> >> >> InnoDB: buffer...>> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number>> >> >> >> 15163771045>> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up>> >> >> >> InnoDB: in total 12 row operations to undo>> >> >> >> InnoDB: Trx id counter is 6DE8E00>> >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records>> >> >> >> to>> >> >> >> the>> >> >> >> database...>> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got>> >> >> >> signal>> >> >> >> 11 ;>> >> >> >> This could be because you hit a bug. It is also possible that>> >> >> >> this>> >> >> >> binary>> >> >> >> or one of the libraries it was linked against is corrupt,>> >> >> >> improperly>> >> >> >> built,>> >> >> >> or misconfigured. This error can also be caused by malfunctioning>> >> >> >> hardware.>> >> >> >> We will try our best to scrape up some info that will hopefully>> >> >> >> help>> >> >> >> diagnose>> >> >> >> the problem, but since we have already crashed, something is>> >> >> >> definitely>> >> >> >> wrong>> >> >> >> and this may fail.>> >> >> >>>> >> >> >> key_buffer_size=16777216>> >> >> >> read_buffer_size=262144>> >> >> >> max_used_connections=0>> >> >> >> max_threads=151>> >> >> >> thread_count=0>> >> >> >> connection_count=0>> >> >> >> It is possible that mysqld could use up to>> >> >> >> key_buffer_size + (read_buffer_size +>> >> >> >> sort_buffer_size)*max_threads>> >> >> >> =>> >> >> >> 133434 K>> >> >> >> bytes of memory>> >> >> >> Hope that's ok; if not, decrease some variables in the equation.>> >> >> >>>> >> >> >> Thread pointer: 0x0>> >> >> >> Attempting backtrace. You can use the following information to>> >> >> >> find>> >> >> >> out>> >> >> >> where mysqld died. If you see no messages after this, something>> >> >> >> went>> >> >> >> terribly wrong...>> >> >> >> stack_bottom = (nil) thread_stack 0x30000>> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]>> >> >> >> The manual page at>> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html>> >> >> >> contains>> >> >> >> information that should help you find out what is causing the>> >> >> >> crash.>> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file>> >> >> >> /var/run/mysqld/mysqld.pid ended>> >> >> >>>> >> >> >>>> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter>> >> >> >> <clandmeter@gmail.com>>> >> >> >> wrote:>> >> >> >>> Ill be able to check in a minute if needed.>> >> >> >>>>> >> >> >>> Carlo>> >> >> >>>>> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>>> >> >> >>> wrote:>> >> >> >>>>>> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>> >> >> >>>>>>> >> >> >>>>> Hi Everyone,>> >> >> >>>>>>> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi>> >> >> >>>>> is>> >> >> >>>>> crashing, by the looks of things because of multiple spiders>> >> >> >>>>> hitting>> >> >> >>>>> it at once. Banning the user agents in lightty config hasn't>> >> >> >>>>> helped,>> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm>> >> >> >>>>> out>> >> >> >>>>> of>> >> >> >>>>> time to keep trying to fix it. If anyone else has some time>> >> >> >>>>> to>> >> >> >>>>> take>> >> >> >>>>> a>> >> >> >>>>> look, that'd be awesome,>> >> >> >>>>>>> >> >> >>>>> Thanks,>> >> >> >>>>>>> >> >> >>>>> Jeff>> >> >> >>>>>>> >> >> >>>>>> >> >> >>>> Mysql died, so far I'm unable to restart it>> >> >> >>>>>> >> >> >>>>> >> >>>> >> >>>> >> >>>> >> >> -->> >> >> Jeff>> >>>> >>>> >>>> >> -->> >> Jeff>>>>>>>> -->> Jeff
--
Jeff
Ill be able to check in a minute if needed.
Carlo
On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org> wrote:
> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>>> Hi Everyone,>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>> crashing, by the looks of things because of multiple spiders hitting>> it at once. Banning the user agents in lightty config hasn't helped,>> neither has upping the redmine sockets to 32 (from 4) but I'm out of>> time to keep trying to fix it. If anyone else has some time to take a>> look, that'd be awesome,>>>> Thanks,>>>> Jeff>>>>> Mysql died, so far I'm unable to restart it>>
I'm getting the below, not sure what to do now. Already updated mysql
to latest 2.2 version.
120110 19:53:23 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 15160684415
120110 19:53:23 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 15163771045
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 12 row operations to undo
InnoDB: Trx id counter is 6DE8E00
120110 19:53:24 InnoDB: Starting an apply batch of log records to the
database...
InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133434 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120110 19:53:24 mysqld_safe mysqld from pid file
/var/run/mysqld/mysqld.pid ended
On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> Ill be able to check in a minute if needed.>> Carlo>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org> wrote:>>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>>>>>> Hi Everyone,>>>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is>>> crashing, by the looks of things because of multiple spiders hitting>>> it at once. Banning the user agents in lightty config hasn't helped,>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of>>> time to keep trying to fix it. If anyone else has some time to take a>>> look, that'd be awesome,>>>>>> Thanks,>>>>>> Jeff>>>>>>> Mysql died, so far I'm unable to restart it>>>
On Tue, Jan 10, 2012 at 7:46 PM, Nathan Angelacos <nangel@nothome.org> wrote:
> On 01/10/2012 01:53 PM, Carlo Landmeter wrote:>>>> Also had issues with pg on uc. It's the price we pay off trying to be>> differed...>>>> On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk@gmail.com>> <jbilyk@gmail.com>> wrote:>>>> Thanks. I should be back home in a few hours, if you need a hand, I>> can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the>> same bug as far as we could tell. Fabled figured it was something in>> uclibc iirc that was causing this crash, but we weren't able to>> isolate exactly what the bug was...>>>> This is good for tracking down the bug then... we have a repeatable> scenario.>>
Fabled, any chance we could get a bit of your time tomorrow to try to
track down this issue? It's the same bug you were helping me with
earlier in the year (http://bugs.mysql.com/bug.php?id=62634).
Unfortunately, upstream just keeps saying "can't replicate, try mysql
$LATESTVER"...
Jeff
The bad answer is, no dump available....
On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
> So that failed. We're hitting what seems to be the same bug I've hit> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)> that still hasn't been fixed. The workaround is to delete everything> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump> backup. I'm really hoping that clandmeter has a backup...>> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in> file btr0pcur.c line 425> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==> buf_block_get_page_no(btr_pcur_get_block(cursor))> InnoDB: We intentionally generate a memory trap.> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.> InnoDB: If you get repeated assertion failures or crashes, even> InnoDB: immediately after the mysqld startup, there may be> InnoDB: corruption in the InnoDB tablespace. Please refer to> InnoDB:> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html> InnoDB: about forcing recovery.> 120110 21:26:39 - mysqld got signal 6 ;> This could be because you hit a bug. It is also possible that this binary> or one of the libraries it was linked against is corrupt, improperly built,> or misconfigured. This error can also be caused by malfunctioning hardware.> We will try our best to scrape up some info that will hopefully help> diagnose> the problem, but since we have already crashed, something is definitely> wrong> and this may fail.>> key_buffer_size=16777216> read_buffer_size=262144> max_used_connections=0> max_threads=151> thread_count=0> connection_count=0> It is possible that mysqld could use up to> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads => 133434 K> bytes of memory> Hope that's ok; if not, decrease some variables in the equation.>> Thread pointer: 0x0> Attempting backtrace. You can use the following information to find out> where mysqld died. If you see no messages after this, something went> terribly wrong...> stack_bottom = (nil) thread_stack 0x30000> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.htmlcontains> information that should help you find out what is causing the crash.> 120110 21:26:39 mysqld_safe mysqld from pid file> /var/run/mysqld/mysqld.pid ended>> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:> > Only other thing I can think to try is to dump the binlogs since it's> during recovery that the crash occurs. Logs say there's only one> transaction that's trying to apply...> >> > Jeff> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>> wrote:> >> >> I'm getting the below, not sure what to do now. Already updated mysql> >> to latest 2.2 version.> >>> >>> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415> >> 120110 19:53:23 InnoDB: Database was not shut down normally!> >> InnoDB: Starting crash recovery.> >> InnoDB: Reading tablespace information from the .ibd files...> >> InnoDB: Restoring possible half-written data pages from the doublewrite> >> InnoDB: buffer...> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up> >> InnoDB: in total 12 row operations to undo> >> InnoDB: Trx id counter is 6DE8E00> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to the> >> database...> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal> 11 ;> >> This could be because you hit a bug. It is also possible that this> binary> >> or one of the libraries it was linked against is corrupt, improperly> built,> >> or misconfigured. This error can also be caused by malfunctioning> hardware.> >> We will try our best to scrape up some info that will hopefully help> diagnose> >> the problem, but since we have already crashed, something is definitely> wrong> >> and this may fail.> >>> >> key_buffer_size=16777216> >> read_buffer_size=262144> >> max_used_connections=0> >> max_threads=151> >> thread_count=0> >> connection_count=0> >> It is possible that mysqld could use up to> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads => 133434 K> >> bytes of memory> >> Hope that's ok; if not, decrease some variables in the equation.> >>> >> Thread pointer: 0x0> >> Attempting backtrace. You can use the following information to find out> >> where mysqld died. If you see no messages after this, something went> >> terribly wrong...> >> stack_bottom = (nil) thread_stack 0x30000> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.htmlcontains> >> information that should help you find out what is causing the crash.> >> 120110 19:53:24 mysqld_safe mysqld from pid file> >> /var/run/mysqld/mysqld.pid ended> >>> >>> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <clandmeter@gmail.com>> wrote:> >>> Ill be able to check in a minute if needed.> >>>> >>> Carlo> >>>> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>> wrote:> >>>>> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:> >>>>>> >>>>> Hi Everyone,> >>>>>> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is> >>>>> crashing, by the looks of things because of multiple spiders hitting> >>>>> it at once. Banning the user agents in lightty config hasn't helped,> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out of> >>>>> time to keep trying to fix it. If anyone else has some time to take> a> >>>>> look, that'd be awesome,> >>>>>> >>>>> Thanks,> >>>>>> >>>>> Jeff> >>>>>> >>>>> >>>> Mysql died, so far I'm unable to restart it> >>>>> >>>>>>> --> Jeff>
What about starting this var on Ubuntu or something and export?
On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a> couple...invasive... recovery tricks on the copy in /varlib/mysql, but> they didn't work. In the testing we did for the bug below, the only> way I was ever able to get a copy running again was delete/restore> from backup.>> Anyone have any ideas where to go from here?>> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com>> wrote:> > The bad answer is, no dump available....> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >>> >> So that failed. We're hitting what seems to be the same bug I've hit> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)> >> that still hasn't been fixed. The workaround is to delete everything> >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump> >> backup. I'm really hoping that clandmeter has a backup...> >>> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in> >> file btr0pcur.c line 425> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==> >> buf_block_get_page_no(btr_pcur_get_block(cursor))> >> InnoDB: We intentionally generate a memory trap.> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.> >> InnoDB: If you get repeated assertion failures or crashes, even> >> InnoDB: immediately after the mysqld startup, there may be> >> InnoDB: corruption in the InnoDB tablespace. Please refer to> >> InnoDB:> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html> >> InnoDB: about forcing recovery.> >> 120110 21:26:39 - mysqld got signal 6 ;> >> This could be because you hit a bug. It is also possible that this> binary> >> or one of the libraries it was linked against is corrupt, improperly> >> built,> >> or misconfigured. This error can also be caused by malfunctioning> >> hardware.> >> We will try our best to scrape up some info that will hopefully help> >> diagnose> >> the problem, but since we have already crashed, something is definitely> >> wrong> >> and this may fail.> >>> >> key_buffer_size=16777216> >> read_buffer_size=262144> >> max_used_connections=0> >> max_threads=151> >> thread_count=0> >> connection_count=0> >> It is possible that mysqld could use up to> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads => >> 133434 K> >> bytes of memory> >> Hope that's ok; if not, decrease some variables in the equation.> >>> >> Thread pointer: 0x0> >> Attempting backtrace. You can use the following information to find out> >> where mysqld died. If you see no messages after this, something went> >> terribly wrong...> >> stack_bottom = (nil) thread_stack 0x30000> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html> >> contains> >> information that should help you find out what is causing the crash.> >> 120110 21:26:39 mysqld_safe mysqld from pid file> >> /var/run/mysqld/mysqld.pid ended> >>> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com> wrote:> >> > Only other thing I can think to try is to dump the binlogs since it's> >> > during recovery that the crash occurs. Logs say there's only one> >> > transaction that's trying to apply...> >> >> >> > Jeff> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com>> >> > wrote:> >> >> >> >> I'm getting the below, not sure what to do now. Already updated mysql> >> >> to latest 2.2 version.> >> >>> >> >>> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda.> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!> >> >> InnoDB: Starting crash recovery.> >> >> InnoDB: Reading tablespace information from the .ibd files...> >> >> InnoDB: Restoring possible half-written data pages from the> doublewrite> >> >> InnoDB: buffer...> >> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up> >> >> InnoDB: in total 12 row operations to undo> >> >> InnoDB: Trx id counter is 6DE8E00> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to> the> >> >> database...> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got> signal> >> >> 11 ;> >> >> This could be because you hit a bug. It is also possible that this> >> >> binary> >> >> or one of the libraries it was linked against is corrupt, improperly> >> >> built,> >> >> or misconfigured. This error can also be caused by malfunctioning> >> >> hardware.> >> >> We will try our best to scrape up some info that will hopefully help> >> >> diagnose> >> >> the problem, but since we have already crashed, something is> definitely> >> >> wrong> >> >> and this may fail.> >> >>> >> >> key_buffer_size=16777216> >> >> read_buffer_size=262144> >> >> max_used_connections=0> >> >> max_threads=151> >> >> thread_count=0> >> >> connection_count=0> >> >> It is possible that mysqld could use up to> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads => >> >> 133434 K> >> >> bytes of memory> >> >> Hope that's ok; if not, decrease some variables in the equation.> >> >>> >> >> Thread pointer: 0x0> >> >> Attempting backtrace. You can use the following information to find> out> >> >> where mysqld died. If you see no messages after this, something went> >> >> terribly wrong...> >> >> stack_bottom = (nil) thread_stack 0x30000> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html> >> >> contains> >> >> information that should help you find out what is causing the crash.> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file> >> >> /var/run/mysqld/mysqld.pid ended> >> >>> >> >>> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter <> clandmeter@gmail.com>> >> >> wrote:> >> >>> Ill be able to check in a minute if needed.> >> >>>> >> >>> Carlo> >> >>>> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>> >> >>> wrote:> >> >>>>> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:> >> >>>>>> >> >>>>> Hi Everyone,> >> >>>>>> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi is> >> >>>>> crashing, by the looks of things because of multiple spiders> hitting> >> >>>>> it at once. Banning the user agents in lightty config hasn't> >> >>>>> helped,> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out> of> >> >>>>> time to keep trying to fix it. If anyone else has some time to> take> >> >>>>> a> >> >>>>> look, that'd be awesome,> >> >>>>>> >> >>>>> Thanks,> >> >>>>>> >> >>>>> Jeff> >> >>>>>> >> >>>>> >> >>>> Mysql died, so far I'm unable to restart it> >> >>>>> >> >>>> >>> >>> >>> >> --> >> Jeff>>>> --> Jeff>
Should be no problem. It's an vserver. Ill try tonight.
On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
> It's worth a shot, but I don't have the time/a box at work to try.> Could you give it a shot?>> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com>> wrote:> > What about starting this var on Ubuntu or something and export?> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >>> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a> >> couple...invasive... recovery tricks on the copy in /varlib/mysql, but> >> they didn't work. In the testing we did for the bug below, the only> >> way I was ever able to get a copy running again was delete/restore> >> from backup.> >>> >> Anyone have any ideas where to go from here?> >>> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <clandmeter@gmail.com>> >> wrote:> >> > The bad answer is, no dump available....> >> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >> >>> >> >> So that failed. We're hitting what seems to be the same bug I've hit> >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634)> >> >> that still hasn't been fixed. The workaround is to delete everything> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a> mysqldump> >> >> backup. I'm really hoping that clandmeter has a backup...> >> >>> >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in> >> >> file btr0pcur.c line 425> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))> >> >> InnoDB: We intentionally generate a memory trap.> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.> >> >> InnoDB: If you get repeated assertion failures or crashes, even> >> >> InnoDB: immediately after the mysqld startup, there may be> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to> >> >> InnoDB:> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html> >> >> InnoDB: about forcing recovery.> >> >> 120110 21:26:39 - mysqld got signal 6 ;> >> >> This could be because you hit a bug. It is also possible that this> >> >> binary> >> >> or one of the libraries it was linked against is corrupt, improperly> >> >> built,> >> >> or misconfigured. This error can also be caused by malfunctioning> >> >> hardware.> >> >> We will try our best to scrape up some info that will hopefully help> >> >> diagnose> >> >> the problem, but since we have already crashed, something is> definitely> >> >> wrong> >> >> and this may fail.> >> >>> >> >> key_buffer_size=16777216> >> >> read_buffer_size=262144> >> >> max_used_connections=0> >> >> max_threads=151> >> >> thread_count=0> >> >> connection_count=0> >> >> It is possible that mysqld could use up to> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads => >> >> 133434 K> >> >> bytes of memory> >> >> Hope that's ok; if not, decrease some variables in the equation.> >> >>> >> >> Thread pointer: 0x0> >> >> Attempting backtrace. You can use the following information to find> out> >> >> where mysqld died. If you see no messages after this, something went> >> >> terribly wrong...> >> >> stack_bottom = (nil) thread_stack 0x30000> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html> >> >> contains> >> >> information that should help you find out what is causing the crash.> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file> >> >> /var/run/mysqld/mysqld.pid ended> >> >>> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com>> wrote:> >> >> > Only other thing I can think to try is to dump the binlogs since> it's> >> >> > during recovery that the crash occurs. Logs say there's only one> >> >> > transaction that's trying to apply...> >> >> >> >> >> > Jeff> >> >> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter <clandmeter@gmail.com> >> >> >> > wrote:> >> >> >> >> >> >> I'm getting the below, not sure what to do now. Already updated> >> >> >> mysql> >> >> >> to latest 2.2 version.> >> >> >>> >> >> >>> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is> Barracuda.> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415> >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!> >> >> >> InnoDB: Starting crash recovery.> >> >> >> InnoDB: Reading tablespace information from the .ibd files...> >> >> >> InnoDB: Restoring possible half-written data pages from the> >> >> >> doublewrite> >> >> >> InnoDB: buffer...> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number> >> >> >> 15163771045> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up> >> >> >> InnoDB: in total 12 row operations to undo> >> >> >> InnoDB: Trx id counter is 6DE8E00> >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records to> >> >> >> the> >> >> >> database...> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got> >> >> >> signal> >> >> >> 11 ;> >> >> >> This could be because you hit a bug. It is also possible that this> >> >> >> binary> >> >> >> or one of the libraries it was linked against is corrupt,> improperly> >> >> >> built,> >> >> >> or misconfigured. This error can also be caused by malfunctioning> >> >> >> hardware.> >> >> >> We will try our best to scrape up some info that will hopefully> help> >> >> >> diagnose> >> >> >> the problem, but since we have already crashed, something is> >> >> >> definitely> >> >> >> wrong> >> >> >> and this may fail.> >> >> >>> >> >> >> key_buffer_size=16777216> >> >> >> read_buffer_size=262144> >> >> >> max_used_connections=0> >> >> >> max_threads=151> >> >> >> thread_count=0> >> >> >> connection_count=0> >> >> >> It is possible that mysqld could use up to> >> >> >> key_buffer_size + (read_buffer_size +> sort_buffer_size)*max_threads> >> >> >> => >> >> >> 133434 K> >> >> >> bytes of memory> >> >> >> Hope that's ok; if not, decrease some variables in the equation.> >> >> >>> >> >> >> Thread pointer: 0x0> >> >> >> Attempting backtrace. You can use the following information to> find> >> >> >> out> >> >> >> where mysqld died. If you see no messages after this, something> went> >> >> >> terribly wrong...> >> >> >> stack_bottom = (nil) thread_stack 0x30000> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]> >> >> >> The manual page at> http://dev.mysql.com/doc/mysql/en/crashing.html> >> >> >> contains> >> >> >> information that should help you find out what is causing the> crash.> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file> >> >> >> /var/run/mysqld/mysqld.pid ended> >> >> >>> >> >> >>> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter> >> >> >> <clandmeter@gmail.com>> >> >> >> wrote:> >> >> >>> Ill be able to check in a minute if needed.> >> >> >>>> >> >> >>> Carlo> >> >> >>>> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <nangel@nothome.org>> >> >> >>> wrote:> >> >> >>>>> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:> >> >> >>>>>> >> >> >>>>> Hi Everyone,> >> >> >>>>>> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. Fastcgi> is> >> >> >>>>> crashing, by the looks of things because of multiple spiders> >> >> >>>>> hitting> >> >> >>>>> it at once. Banning the user agents in lightty config hasn't> >> >> >>>>> helped,> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm> out> >> >> >>>>> of> >> >> >>>>> time to keep trying to fix it. If anyone else has some time to> >> >> >>>>> take> >> >> >>>>> a> >> >> >>>>> look, that'd be awesome,> >> >> >>>>>> >> >> >>>>> Thanks,> >> >> >>>>>> >> >> >>>>> Jeff> >> >> >>>>>> >> >> >>>>> >> >> >>>> Mysql died, so far I'm unable to restart it> >> >> >>>>> >> >> >>>> >> >>> >> >>> >> >>> >> >> --> >> >> Jeff> >>> >>> >>> >> --> >> Jeff>>>> --> Jeff>
Also had issues with pg on uc. It's the price we pay off trying to be
differed...
On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:
> Thanks. I should be back home in a few hours, if you need a hand, I> can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the> same bug as far as we could tell. Fabled figured it was something in> uclibc iirc that was causing this crash, but we weren't able to> isolate exactly what the bug was...>> On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter <clandmeter@gmail.com>> wrote:> > Should be no problem. It's an vserver. Ill try tonight.> >> > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >>> >> It's worth a shot, but I don't have the time/a box at work to try.> >> Could you give it a shot?> >>> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com>> >> wrote:> >> > What about starting this var on Ubuntu or something and export?> >> >> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >> >>> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a> copy> >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a> >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql,> but> >> >> they didn't work. In the testing we did for the bug below, the only> >> >> way I was ever able to get a copy running again was delete/restore> >> >> from backup.> >> >>> >> >> Anyone have any ideas where to go from here?> >> >>> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter <> clandmeter@gmail.com>> >> >> wrote:> >> >> > The bad answer is, no dump available....> >> >> >> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:> >> >> >>> >> >> >> So that failed. We're hitting what seems to be the same bug I've> >> >> >> hit> >> >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=62634> )> >> >> >> that still hasn't been fixed. The workaround is to delete> >> >> >> everything> >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a> >> >> >> mysqldump> >> >> >> backup. I'm really hoping that clandmeter has a backup...> >> >> >>> >> >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072 in> >> >> >> file btr0pcur.c line 425> >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==> >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))> >> >> >> InnoDB: We intentionally generate a memory trap.> >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.> >> >> >> InnoDB: If you get repeated assertion failures or crashes, even> >> >> >> InnoDB: immediately after the mysqld startup, there may be> >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to> >> >> >> InnoDB:> >> >> >>> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html> >> >> >> InnoDB: about forcing recovery.> >> >> >> 120110 21:26:39 - mysqld got signal 6 ;> >> >> >> This could be because you hit a bug. It is also possible that this> >> >> >> binary> >> >> >> or one of the libraries it was linked against is corrupt,> improperly> >> >> >> built,> >> >> >> or misconfigured. This error can also be caused by malfunctioning> >> >> >> hardware.> >> >> >> We will try our best to scrape up some info that will hopefully> help> >> >> >> diagnose> >> >> >> the problem, but since we have already crashed, something is> >> >> >> definitely> >> >> >> wrong> >> >> >> and this may fail.> >> >> >>> >> >> >> key_buffer_size=16777216> >> >> >> read_buffer_size=262144> >> >> >> max_used_connections=0> >> >> >> max_threads=151> >> >> >> thread_count=0> >> >> >> connection_count=0> >> >> >> It is possible that mysqld could use up to> >> >> >> key_buffer_size + (read_buffer_size +> sort_buffer_size)*max_threads> >> >> >> => >> >> >> 133434 K> >> >> >> bytes of memory> >> >> >> Hope that's ok; if not, decrease some variables in the equation.> >> >> >>> >> >> >> Thread pointer: 0x0> >> >> >> Attempting backtrace. You can use the following information to> find> >> >> >> out> >> >> >> where mysqld died. If you see no messages after this, something> went> >> >> >> terribly wrong...> >> >> >> stack_bottom = (nil) thread_stack 0x30000> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]> >> >> >> The manual page at> http://dev.mysql.com/doc/mysql/en/crashing.html> >> >> >> contains> >> >> >> information that should help you find out what is causing the> crash.> >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file> >> >> >> /var/run/mysqld/mysqld.pid ended> >> >> >>> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com>> >> >> >> wrote:> >> >> >> > Only other thing I can think to try is to dump the binlogs since> >> >> >> > it's> >> >> >> > during recovery that the crash occurs. Logs say there's only> one> >> >> >> > transaction that's trying to apply...> >> >> >> >> >> >> >> > Jeff> >> >> >> >> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter> >> >> >> > <clandmeter@gmail.com>> >> >> >> > wrote:> >> >> >> >> >> >> >> >> I'm getting the below, not sure what to do now. Already updated> >> >> >> >> mysql> >> >> >> >> to latest 2.2 version.> >> >> >> >>> >> >> >> >>> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is> >> >> >> >> Barracuda.> >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415> >> >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!> >> >> >> >> InnoDB: Starting crash recovery.> >> >> >> >> InnoDB: Reading tablespace information from the .ibd files...> >> >> >> >> InnoDB: Restoring possible half-written data pages from the> >> >> >> >> doublewrite> >> >> >> >> InnoDB: buffer...> >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number> >> >> >> >> 15163771045> >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned> up> >> >> >> >> InnoDB: in total 12 row operations to undo> >> >> >> >> InnoDB: Trx id counter is 6DE8E00> >> >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log records> >> >> >> >> to> >> >> >> >> the> >> >> >> >> database...> >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got> >> >> >> >> signal> >> >> >> >> 11 ;> >> >> >> >> This could be because you hit a bug. It is also possible that> >> >> >> >> this> >> >> >> >> binary> >> >> >> >> or one of the libraries it was linked against is corrupt,> >> >> >> >> improperly> >> >> >> >> built,> >> >> >> >> or misconfigured. This error can also be caused by> malfunctioning> >> >> >> >> hardware.> >> >> >> >> We will try our best to scrape up some info that will hopefully> >> >> >> >> help> >> >> >> >> diagnose> >> >> >> >> the problem, but since we have already crashed, something is> >> >> >> >> definitely> >> >> >> >> wrong> >> >> >> >> and this may fail.> >> >> >> >>> >> >> >> >> key_buffer_size=16777216> >> >> >> >> read_buffer_size=262144> >> >> >> >> max_used_connections=0> >> >> >> >> max_threads=151> >> >> >> >> thread_count=0> >> >> >> >> connection_count=0> >> >> >> >> It is possible that mysqld could use up to> >> >> >> >> key_buffer_size + (read_buffer_size +> >> >> >> >> sort_buffer_size)*max_threads> >> >> >> >> => >> >> >> >> 133434 K> >> >> >> >> bytes of memory> >> >> >> >> Hope that's ok; if not, decrease some variables in the> equation.> >> >> >> >>> >> >> >> >> Thread pointer: 0x0> >> >> >> >> Attempting backtrace. You can use the following information to> >> >> >> >> find> >> >> >> >> out> >> >> >> >> where mysqld died. If you see no messages after this, something> >> >> >> >> went> >> >> >> >> terribly wrong...> >> >> >> >> stack_bottom = (nil) thread_stack 0x30000> >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]> >> >> >> >> The manual page at> >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html> >> >> >> >> contains> >> >> >> >> information that should help you find out what is causing the> >> >> >> >> crash.> >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file> >> >> >> >> /var/run/mysqld/mysqld.pid ended> >> >> >> >>> >> >> >> >>> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter> >> >> >> >> <clandmeter@gmail.com>> >> >> >> >> wrote:> >> >> >> >>> Ill be able to check in a minute if needed.> >> >> >> >>>> >> >> >> >>> Carlo> >> >> >> >>>> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" <> nangel@nothome.org>> >> >> >> >>> wrote:> >> >> >> >>>>> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:> >> >> >> >>>>>> >> >> >> >>>>> Hi Everyone,> >> >> >> >>>>>> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon.> Fastcgi> >> >> >> >>>>> is> >> >> >> >>>>> crashing, by the looks of things because of multiple spiders> >> >> >> >>>>> hitting> >> >> >> >>>>> it at once. Banning the user agents in lightty config> hasn't> >> >> >> >>>>> helped,> >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but> I'm> >> >> >> >>>>> out> >> >> >> >>>>> of> >> >> >> >>>>> time to keep trying to fix it. If anyone else has some time> >> >> >> >>>>> to> >> >> >> >>>>> take> >> >> >> >>>>> a> >> >> >> >>>>> look, that'd be awesome,> >> >> >> >>>>>> >> >> >> >>>>> Thanks,> >> >> >> >>>>>> >> >> >> >>>>> Jeff> >> >> >> >>>>>> >> >> >> >>>>> >> >> >> >>>> Mysql died, so far I'm unable to restart it> >> >> >> >>>>> >> >> >> >>>> >> >> >>> >> >> >>> >> >> >>> >> >> >> --> >> >> >> Jeff> >> >>> >> >>> >> >>> >> >> --> >> >> Jeff> >>> >>> >>> >> --> >> Jeff>>>> --> Jeff>
Im not able to bring it back online. I'm trying with ubuntu and start
the db (same version) but mysql crashes again.
If somebody else wants to take a look? im going to bed.
gnite.
On Tue, Jan 10, 2012 at 10:53 PM, Carlo Landmeter <clandmeter@gmail.com> wrote:
> Also had issues with pg on uc. It's the price we pay off trying to be> differed...>> On Jan 10, 2012 10:49 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>>>> Thanks. I should be back home in a few hours, if you need a hand, I>> can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the>> same bug as far as we could tell. Fabled figured it was something in>> uclibc iirc that was causing this crash, but we weren't able to>> isolate exactly what the bug was...>>>> On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter <clandmeter@gmail.com>>> wrote:>> > Should be no problem. It's an vserver. Ill try tonight.>> >>> > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >>>> >> It's worth a shot, but I don't have the time/a box at work to try.>> >> Could you give it a shot?>> >>>> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter <clandmeter@gmail.com>>> >> wrote:>> >> > What about starting this var on Ubuntu or something and export?>> >> >>> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >> >>>> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a>> >> >> copy>> >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a>> >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql,>> >> >> but>> >> >> they didn't work. In the testing we did for the bug below, the only>> >> >> way I was ever able to get a copy running again was delete/restore>> >> >> from backup.>> >> >>>> >> >> Anyone have any ideas where to go from here?>> >> >>>> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter>> >> >> <clandmeter@gmail.com>>> >> >> wrote:>> >> >> > The bad answer is, no dump available....>> >> >> >>> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" <jbilyk@gmail.com> wrote:>> >> >> >>>> >> >> >> So that failed. We're hitting what seems to be the same bug I've>> >> >> >> hit>> >> >> >> before in MySQL on Alpine>> >> >> >> (http://bugs.mysql.com/bug.php?id=62634)>> >> >> >> that still hasn't been fixed. The workaround is to delete>> >> >> >> everything>> >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a>> >> >> >> mysqldump>> >> >> >> backup. I'm really hoping that clandmeter has a backup...>> >> >> >>>> >> >> >> 120110 21:26:39 InnoDB: Assertion failure in thread 3071633072>> >> >> >> in>> >> >> >> file btr0pcur.c line 425>> >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) ==>> >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor))>> >> >> >> InnoDB: We intentionally generate a memory trap.>> >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com.>> >> >> >> InnoDB: If you get repeated assertion failures or crashes, even>> >> >> >> InnoDB: immediately after the mysqld startup, there may be>> >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to>> >> >> >> InnoDB:>> >> >> >>>> >> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html>> >> >> >> InnoDB: about forcing recovery.>> >> >> >> 120110 21:26:39 - mysqld got signal 6 ;>> >> >> >> This could be because you hit a bug. It is also possible that>> >> >> >> this>> >> >> >> binary>> >> >> >> or one of the libraries it was linked against is corrupt,>> >> >> >> improperly>> >> >> >> built,>> >> >> >> or misconfigured. This error can also be caused by malfunctioning>> >> >> >> hardware.>> >> >> >> We will try our best to scrape up some info that will hopefully>> >> >> >> help>> >> >> >> diagnose>> >> >> >> the problem, but since we have already crashed, something is>> >> >> >> definitely>> >> >> >> wrong>> >> >> >> and this may fail.>> >> >> >>>> >> >> >> key_buffer_size=16777216>> >> >> >> read_buffer_size=262144>> >> >> >> max_used_connections=0>> >> >> >> max_threads=151>> >> >> >> thread_count=0>> >> >> >> connection_count=0>> >> >> >> It is possible that mysqld could use up to>> >> >> >> key_buffer_size + (read_buffer_size +>> >> >> >> sort_buffer_size)*max_threads>> >> >> >> =>> >> >> >> 133434 K>> >> >> >> bytes of memory>> >> >> >> Hope that's ok; if not, decrease some variables in the equation.>> >> >> >>>> >> >> >> Thread pointer: 0x0>> >> >> >> Attempting backtrace. You can use the following information to>> >> >> >> find>> >> >> >> out>> >> >> >> where mysqld died. If you see no messages after this, something>> >> >> >> went>> >> >> >> terribly wrong...>> >> >> >> stack_bottom = (nil) thread_stack 0x30000>> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529]>> >> >> >> The manual page at>> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html>> >> >> >> contains>> >> >> >> information that should help you find out what is causing the>> >> >> >> crash.>> >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file>> >> >> >> /var/run/mysqld/mysqld.pid ended>> >> >> >>>> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk <jbilyk@gmail.com>>> >> >> >> wrote:>> >> >> >> > Only other thing I can think to try is to dump the binlogs>> >> >> >> > since>> >> >> >> > it's>> >> >> >> > during recovery that the crash occurs. Logs say there's only>> >> >> >> > one>> >> >> >> > transaction that's trying to apply...>> >> >> >> >>> >> >> >> > Jeff>> >> >> >> >>> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter>> >> >> >> > <clandmeter@gmail.com>>> >> >> >> > wrote:>> >> >> >> >>> >> >> >> >> I'm getting the below, not sure what to do now. Already>> >> >> >> >> updated>> >> >> >> >> mysql>> >> >> >> >> to latest 2.2 version.>> >> >> >> >>>> >> >> >> >>>> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is>> >> >> >> >> Barracuda.>> >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn>> >> >> >> >> 15160684415>> >> >> >> >> 120110 19:53:23 InnoDB: Database was not shut down normally!>> >> >> >> >> InnoDB: Starting crash recovery.>> >> >> >> >> InnoDB: Reading tablespace information from the .ibd files...>> >> >> >> >> InnoDB: Restoring possible half-written data pages from the>> >> >> >> >> doublewrite>> >> >> >> >> InnoDB: buffer...>> >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number>> >> >> >> >> 15163771045>> >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned>> >> >> >> >> up>> >> >> >> >> InnoDB: in total 12 row operations to undo>> >> >> >> >> InnoDB: Trx id counter is 6DE8E00>> >> >> >> >> 120110 19:53:24 InnoDB: Starting an apply batch of log>> >> >> >> >> records>> >> >> >> >> to>> >> >> >> >> the>> >> >> >> >> database...>> >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got>> >> >> >> >> signal>> >> >> >> >> 11 ;>> >> >> >> >> This could be because you hit a bug. It is also possible that>> >> >> >> >> this>> >> >> >> >> binary>> >> >> >> >> or one of the libraries it was linked against is corrupt,>> >> >> >> >> improperly>> >> >> >> >> built,>> >> >> >> >> or misconfigured. This error can also be caused by>> >> >> >> >> malfunctioning>> >> >> >> >> hardware.>> >> >> >> >> We will try our best to scrape up some info that will>> >> >> >> >> hopefully>> >> >> >> >> help>> >> >> >> >> diagnose>> >> >> >> >> the problem, but since we have already crashed, something is>> >> >> >> >> definitely>> >> >> >> >> wrong>> >> >> >> >> and this may fail.>> >> >> >> >>>> >> >> >> >> key_buffer_size=16777216>> >> >> >> >> read_buffer_size=262144>> >> >> >> >> max_used_connections=0>> >> >> >> >> max_threads=151>> >> >> >> >> thread_count=0>> >> >> >> >> connection_count=0>> >> >> >> >> It is possible that mysqld could use up to>> >> >> >> >> key_buffer_size + (read_buffer_size +>> >> >> >> >> sort_buffer_size)*max_threads>> >> >> >> >> =>> >> >> >> >> 133434 K>> >> >> >> >> bytes of memory>> >> >> >> >> Hope that's ok; if not, decrease some variables in the>> >> >> >> >> equation.>> >> >> >> >>>> >> >> >> >> Thread pointer: 0x0>> >> >> >> >> Attempting backtrace. You can use the following information to>> >> >> >> >> find>> >> >> >> >> out>> >> >> >> >> where mysqld died. If you see no messages after this,>> >> >> >> >> something>> >> >> >> >> went>> >> >> >> >> terribly wrong...>> >> >> >> >> stack_bottom = (nil) thread_stack 0x30000>> >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529]>> >> >> >> >> The manual page at>> >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html>> >> >> >> >> contains>> >> >> >> >> information that should help you find out what is causing the>> >> >> >> >> crash.>> >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file>> >> >> >> >> /var/run/mysqld/mysqld.pid ended>> >> >> >> >>>> >> >> >> >>>> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter>> >> >> >> >> <clandmeter@gmail.com>>> >> >> >> >> wrote:>> >> >> >> >>> Ill be able to check in a minute if needed.>> >> >> >> >>>>> >> >> >> >>> Carlo>> >> >> >> >>>>> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos">> >> >> >> >>> <nangel@nothome.org>>> >> >> >> >>> wrote:>> >> >> >> >>>>>> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote:>> >> >> >> >>>>>>> >> >> >> >>>>> Hi Everyone,>> >> >> >> >>>>>>> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon.>> >> >> >> >>>>> Fastcgi>> >> >> >> >>>>> is>> >> >> >> >>>>> crashing, by the looks of things because of multiple>> >> >> >> >>>>> spiders>> >> >> >> >>>>> hitting>> >> >> >> >>>>> it at once. Banning the user agents in lightty config>> >> >> >> >>>>> hasn't>> >> >> >> >>>>> helped,>> >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but>> >> >> >> >>>>> I'm>> >> >> >> >>>>> out>> >> >> >> >>>>> of>> >> >> >> >>>>> time to keep trying to fix it. If anyone else has some>> >> >> >> >>>>> time>> >> >> >> >>>>> to>> >> >> >> >>>>> take>> >> >> >> >>>>> a>> >> >> >> >>>>> look, that'd be awesome,>> >> >> >> >>>>>>> >> >> >> >>>>> Thanks,>> >> >> >> >>>>>>> >> >> >> >>>>> Jeff>> >> >> >> >>>>>>> >> >> >> >>>>>> >> >> >> >>>> Mysql died, so far I'm unable to restart it>> >> >> >> >>>>>> >> >> >> >>>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >> -->> >> >> >> Jeff>> >> >>>> >> >>>> >> >>>> >> >> -->> >> >> Jeff>> >>>> >>>> >>>> >> -->> >> Jeff>>>>>>>> -->> Jeff
On Wed, 11 Jan 2012 01:34:30 +0100
Carlo Landmeter <clandmeter@gmail.com> wrote:
> Im not able to bring it back online. I'm trying with ubuntu and start> the db (same version) but mysql crashes again.> > If somebody else wants to take a look? im going to bed.
Me and clandmeter got bugs.a.o back online.
Here is what i did:
* copied /var/lib/mysql to /var/lib/mysql.try1 in case someone had made
parital progress.
* remove /var/lib/mysql and copied over /var/lib/mysql.bak
to /var/lib/mysql (this i did various times)
* tried to start with different innodb_force_recovery settings
in /etc/mysql/my.cnf
* tried to remove parts of /var/lib/mysql/* dirs to try to disable
single databases (in case it was only one database was broke)
Nothing worked.
Then i saw that disk usage was 97%. I increased the /vservers partition
from 60G to 70G (thanks to who ever set up the LVM!) and resizefs'ed it.
I removed my mysql.try1 and the ubuntu vserver /var/lib/mysql to free
up some space.
After that the database started up with innodb_force_recovery=6 and
clandmeter was able to mysqldump out the alpine-bugs database.
From there clandmeter took over. I think he restored the alpine-bugs
some other place and pointed redmine to there.
-nc