X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id 65142DC1843 for ; Tue, 10 Jan 2012 21:44:49 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so97491obb.30 for ; Tue, 10 Jan 2012 13:44:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=4+ksMriHpDrrt+q2WwrmmAiv8HmarrwgQkYjRJ8RRcA=; b=CgObFZGtuF1etPgL8Rrr1gdPsE0EOkWs10Xs5JSbOTV1xDGkVJWePBaBYE74soXhI/ iYtX9oWJs3ETiuzv8Ogdta6wkKdrBPfomihuDUen0GnsPvcOh5u73lhnATjI7pCZcAq9 KZyfhsTKFPZtlH1vHD+TXlfSpjNdlPzuR5ev8= MIME-Version: 1.0 Received: by 10.182.86.229 with SMTP id s5mr19907862obz.37.1326231889173; Tue, 10 Jan 2012 13:44:49 -0800 (PST) Received: by 10.182.14.3 with HTTP; Tue, 10 Jan 2012 13:44:49 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 16:44:49 -0500 Message-ID: Subject: Re: Bugs.a.o From: Jeff Bilyk To: Carlo Landmeter Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable It's worth a shot, but I don't have the time/a box at work to try. Could you give it a shot? On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter wro= te: > What about starting this var on Ubuntu or something and export? > > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" wrote: >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a copy >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a >> couple...invasive... recovery tricks on the copy in /varlib/mysql, but >> they didn't work. =A0In the testing we did for the bug below, the only >> way I was ever able to get a copy running again was delete/restore >> from backup. >> >> Anyone have any ideas where to go from here? >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter >> wrote: >> > The bad answer is, no dump available.... >> > >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: >> >> >> >> So that failed. =A0We're hitting what seems to be the same bug I've h= it >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=3D62634) >> >> that still hasn't been fixed. =A0The workaround is to delete everythi= ng >> >> in /var/lib/mysql/, reinitialize the db, then restore from a mysqldum= p >> >> backup. =A0I'm really hoping that clandmeter has a backup... >> >> >> >> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 3071633072 in >> >> file btr0pcur.c line 425 >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) =3D=3D >> >> buf_block_get_page_no(btr_pcur_get_block(cursor)) >> >> InnoDB: We intentionally generate a memory trap. >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. >> >> InnoDB: If you get repeated assertion failures or crashes, even >> >> InnoDB: immediately after the mysqld startup, there may be >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to >> >> InnoDB: >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html >> >> InnoDB: about forcing recovery. >> >> 120110 21:26:39 - mysqld got signal 6 ; >> >> This could be because you hit a bug. It is also possible that this >> >> binary >> >> or one of the libraries it was linked against is corrupt, improperly >> >> built, >> >> or misconfigured. This error can also be caused by malfunctioning >> >> hardware. >> >> We will try our best to scrape up some info that will hopefully help >> >> diagnose >> >> the problem, but since we have already crashed, something is definite= ly >> >> wrong >> >> and this may fail. >> >> >> >> key_buffer_size=3D16777216 >> >> read_buffer_size=3D262144 >> >> max_used_connections=3D0 >> >> max_threads=3D151 >> >> thread_count=3D0 >> >> connection_count=3D0 >> >> It is possible that mysqld could use up to >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = =3D >> >> 133434 K >> >> bytes of memory >> >> Hope that's ok; if not, decrease some variables in the equation. >> >> >> >> Thread pointer: 0x0 >> >> Attempting backtrace. You can use the following information to find o= ut >> >> where mysqld died. If you see no messages after this, something went >> >> terribly wrong... >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> >> contains >> >> information that should help you find out what is causing the crash. >> >> 120110 21:26:39 mysqld_safe mysqld from pid file >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk wrote: >> >> > Only other thing I can think to try is to dump the binlogs since it= 's >> >> > during recovery that the crash occurs. =A0Logs say there's only one >> >> > transaction that's trying to apply... >> >> > >> >> > Jeff >> >> > >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter >> >> > wrote: >> >> > >> >> >> I'm getting the below, not sure what to do now. Already updated >> >> >> mysql >> >> >> to latest 2.2 version. >> >> >> >> >> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is Barracuda= . >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 >> >> >> 120110 19:53:23 =A0InnoDB: Database was not shut down normally! >> >> >> InnoDB: Starting crash recovery. >> >> >> InnoDB: Reading tablespace information from the .ibd files... >> >> >> InnoDB: Restoring possible half-written data pages from the >> >> >> doublewrite >> >> >> InnoDB: buffer... >> >> >> InnoDB: Doing recovery: scanned up to log sequence number >> >> >> 15163771045 >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned up >> >> >> InnoDB: in total 12 row operations to undo >> >> >> InnoDB: Trx id counter is 6DE8E00 >> >> >> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log records = to >> >> >> the >> >> >> database... >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got >> >> >> signal >> >> >> 11 ; >> >> >> This could be because you hit a bug. It is also possible that this >> >> >> binary >> >> >> or one of the libraries it was linked against is corrupt, improper= ly >> >> >> built, >> >> >> or misconfigured. This error can also be caused by malfunctioning >> >> >> hardware. >> >> >> We will try our best to scrape up some info that will hopefully he= lp >> >> >> diagnose >> >> >> the problem, but since we have already crashed, something is >> >> >> definitely >> >> >> wrong >> >> >> and this may fail. >> >> >> >> >> >> key_buffer_size=3D16777216 >> >> >> read_buffer_size=3D262144 >> >> >> max_used_connections=3D0 >> >> >> max_threads=3D151 >> >> >> thread_count=3D0 >> >> >> connection_count=3D0 >> >> >> It is possible that mysqld could use up to >> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_thread= s >> >> >> =3D >> >> >> 133434 K >> >> >> bytes of memory >> >> >> Hope that's ok; if not, decrease some variables in the equation. >> >> >> >> >> >> Thread pointer: 0x0 >> >> >> Attempting backtrace. You can use the following information to fin= d >> >> >> out >> >> >> where mysqld died. If you see no messages after this, something we= nt >> >> >> terribly wrong... >> >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] >> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> >> >> contains >> >> >> information that should help you find out what is causing the cras= h. >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file >> >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter >> >> >> >> >> >> wrote: >> >> >>> Ill be able to check in a minute if needed. >> >> >>> >> >> >>> Carlo >> >> >>> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" >> >> >>> wrote: >> >> >>>> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: >> >> >>>>> >> >> >>>>> Hi Everyone, >> >> >>>>> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. =A0Fastcgi= is >> >> >>>>> crashing, by the looks of things because of multiple spiders >> >> >>>>> hitting >> >> >>>>> it at once. =A0Banning the user agents in lightty config hasn't >> >> >>>>> helped, >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'm o= ut >> >> >>>>> of >> >> >>>>> time to keep trying to fix it. =A0If anyone else has some time = to >> >> >>>>> take >> >> >>>>> a >> >> >>>>> look, that'd be awesome, >> >> >>>>> >> >> >>>>> Thanks, >> >> >>>>> >> >> >>>>> Jeff >> >> >>>>> >> >> >>>> >> >> >>>> Mysql died, so far I'm unable to restart it >> >> >>>> >> >> >>> >> >> >> >> >> >> >> >> -- >> >> Jeff >> >> >> >> -- >> Jeff --=20 Jeff