X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id E13CFDC1843 for ; Tue, 10 Jan 2012 21:37:37 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so80715obb.30 for ; Tue, 10 Jan 2012 13:37:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=mD/5Gz2kP+DRqfDan3heljjF7vwyzTxW2dCM9RjHCwo=; b=LZJFp4Obqu3cYyZk0j9xNCes7i2uLbBrh7cd3MkIk3sthdD8HQ4dr4z0lvSQk7ajfO nSSfPsMx9c0gnlEk0dAvqJLs2D+0X5mb9hPYfo3weoEmy4TWpgGVPbnpVK0jP5eYC7WP 3ssEg6tWNMU/MdfK4F/OMp7zTHShEXDMdEPrg= MIME-Version: 1.0 Received: by 10.182.86.229 with SMTP id s5mr19888864obz.37.1326231457439; Tue, 10 Jan 2012 13:37:37 -0800 (PST) Received: by 10.182.14.3 with HTTP; Tue, 10 Jan 2012 13:37:37 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 16:37:37 -0500 Message-ID: Subject: Re: Bugs.a.o From: Jeff Bilyk To: Carlo Landmeter Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Urgh - I'm not sure how to recover that then... Carlo, I've got a copy of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a couple...invasive... recovery tricks on the copy in /varlib/mysql, but they didn't work. In the testing we did for the bug below, the only way I was ever able to get a copy running again was delete/restore from backup. Anyone have any ideas where to go from here? On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter wro= te: > The bad answer is, no dump available.... > > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: >> >> So that failed. =A0We're hitting what seems to be the same bug I've hit >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=3D62634) >> that still hasn't been fixed. =A0The workaround is to delete everything >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldump >> backup. =A0I'm really hoping that clandmeter has a backup... >> >> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 3071633072 in >> file btr0pcur.c line 425 >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) =3D=3D >> buf_block_get_page_no(btr_pcur_get_block(cursor)) >> InnoDB: We intentionally generate a memory trap. >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. >> InnoDB: If you get repeated assertion failures or crashes, even >> InnoDB: immediately after the mysqld startup, there may be >> InnoDB: corruption in the InnoDB tablespace. Please refer to >> InnoDB: >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html >> InnoDB: about forcing recovery. >> 120110 21:26:39 - mysqld got signal 6 ; >> This could be because you hit a bug. It is also possible that this binar= y >> or one of the libraries it was linked against is corrupt, improperly >> built, >> or misconfigured. This error can also be caused by malfunctioning >> hardware. >> We will try our best to scrape up some info that will hopefully help >> diagnose >> the problem, but since we have already crashed, something is definitely >> wrong >> and this may fail. >> >> key_buffer_size=3D16777216 >> read_buffer_size=3D262144 >> max_used_connections=3D0 >> max_threads=3D151 >> thread_count=3D0 >> connection_count=3D0 >> It is possible that mysqld could use up to >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =3D >> 133434 K >> bytes of memory >> Hope that's ok; if not, decrease some variables in the equation. >> >> Thread pointer: 0x0 >> Attempting backtrace. You can use the following information to find out >> where mysqld died. If you see no messages after this, something went >> terribly wrong... >> stack_bottom =3D (nil) thread_stack 0x30000 >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> contains >> information that should help you find out what is causing the crash. >> 120110 21:26:39 mysqld_safe mysqld from pid file >> /var/run/mysqld/mysqld.pid ended >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk wrote: >> > Only other thing I can think to try is to dump the binlogs since it's >> > during recovery that the crash occurs. =A0Logs say there's only one >> > transaction that's trying to apply... >> > >> > Jeff >> > >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter >> > wrote: >> > >> >> I'm getting the below, not sure what to do now. Already updated mysql >> >> to latest 2.2 version. >> >> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda. >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 >> >> 120110 19:53:23 =A0InnoDB: Database was not shut down normally! >> >> InnoDB: Starting crash recovery. >> >> InnoDB: Reading tablespace information from the .ibd files... >> >> InnoDB: Restoring possible half-written data pages from the doublewri= te >> >> InnoDB: buffer... >> >> InnoDB: Doing recovery: scanned up to log sequence number 15163771045 >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up >> >> InnoDB: in total 12 row operations to undo >> >> InnoDB: Trx id counter is 6DE8E00 >> >> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log records to = the >> >> database... >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got signal >> >> 11 ; >> >> This could be because you hit a bug. It is also possible that this >> >> binary >> >> or one of the libraries it was linked against is corrupt, improperly >> >> built, >> >> or misconfigured. This error can also be caused by malfunctioning >> >> hardware. >> >> We will try our best to scrape up some info that will hopefully help >> >> diagnose >> >> the problem, but since we have already crashed, something is definite= ly >> >> wrong >> >> and this may fail. >> >> >> >> key_buffer_size=3D16777216 >> >> read_buffer_size=3D262144 >> >> max_used_connections=3D0 >> >> max_threads=3D151 >> >> thread_count=3D0 >> >> connection_count=3D0 >> >> It is possible that mysqld could use up to >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = =3D >> >> 133434 K >> >> bytes of memory >> >> Hope that's ok; if not, decrease some variables in the equation. >> >> >> >> Thread pointer: 0x0 >> >> Attempting backtrace. You can use the following information to find o= ut >> >> where mysqld died. If you see no messages after this, something went >> >> terribly wrong... >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> >> contains >> >> information that should help you find out what is causing the crash. >> >> 120110 19:53:24 mysqld_safe mysqld from pid file >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter >> >> wrote: >> >>> Ill be able to check in a minute if needed. >> >>> >> >>> Carlo >> >>> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" >> >>> wrote: >> >>>> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: >> >>>>> >> >>>>> Hi Everyone, >> >>>>> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. =A0Fastcgi is >> >>>>> crashing, by the looks of things because of multiple spiders hitti= ng >> >>>>> it at once. =A0Banning the user agents in lightty config hasn't >> >>>>> helped, >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm out = of >> >>>>> time to keep trying to fix it. =A0If anyone else has some time to = take >> >>>>> a >> >>>>> look, that'd be awesome, >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> Jeff >> >>>>> >> >>>> >> >>>> Mysql died, so far I'm unable to restart it >> >>>> >> >>> >> >> >> >> -- >> Jeff --=20 Jeff