X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id 42402DC1843 for ; Wed, 11 Jan 2012 00:34:31 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so627008obb.30 for ; Tue, 10 Jan 2012 16:34:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=LggL2ou+/RnN+Wi4o+fLIVizDxvd3GDHsR14d4SqyHQ=; b=ld+sSsCQqTYTG5MjSNGMousyAWBxihn9dA3fponvvmX1+W/ypc16uUPP1lVcJkmg8c IP+TUNxZjiPHy3Le+Ki+KIhHD9uEkuIHvGJe0d19RKtkGtJ6kdA7bhFF1i0GbNQOKTKE k1/vexeQmPapdVoIS6UnYv4Dw1+PknbL7jBnk= MIME-Version: 1.0 Received: by 10.182.42.37 with SMTP id k5mr20374950obl.40.1326242070686; Tue, 10 Jan 2012 16:34:30 -0800 (PST) Received: by 10.182.113.73 with HTTP; Tue, 10 Jan 2012 16:34:30 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Wed, 11 Jan 2012 01:34:30 +0100 Message-ID: Subject: Re: Bugs.a.o From: Carlo Landmeter To: Jeff Bilyk Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Im not able to bring it back online. I'm trying with ubuntu and start the db (same version) but mysql crashes again. If somebody else wants to take a look? im going to bed. gnite. On Tue, Jan 10, 2012 at 10:53 PM, Carlo Landmeter wr= ote: > Also had issues with pg on uc. It's the price we pay off trying to be > differed... > > On Jan 10, 2012 10:49 PM, "Jeff Bilyk" wrote: >> >> Thanks. =A0I should be back home in a few hours, if you need a hand, I >> can give it a try. =A0Fwiw, Ubuntu's MySQL package doesn't suffer the >> same bug as far as we could tell. =A0Fabled figured it was something in >> uclibc iirc that was causing this crash, but we weren't able to >> isolate exactly what the bug was... >> >> On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter >> wrote: >> > Should be no problem. It's an vserver. Ill try tonight. >> > >> > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" wrote: >> >> >> >> It's worth a shot, but I don't have the time/a box at work to try. >> >> Could you give it a shot? >> >> >> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter >> >> wrote: >> >> > What about starting this var on Ubuntu or something and export? >> >> > >> >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" wrote: >> >> >> >> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a >> >> >> copy >> >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a >> >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql, >> >> >> but >> >> >> they didn't work. =A0In the testing we did for the bug below, the = only >> >> >> way I was ever able to get a copy running again was delete/restore >> >> >> from backup. >> >> >> >> >> >> Anyone have any ideas where to go from here? >> >> >> >> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter >> >> >> >> >> >> wrote: >> >> >> > The bad answer is, no dump available.... >> >> >> > >> >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: >> >> >> >> >> >> >> >> So that failed. =A0We're hitting what seems to be the same bug = I've >> >> >> >> hit >> >> >> >> before in MySQL on Alpine >> >> >> >> (http://bugs.mysql.com/bug.php?id=3D62634) >> >> >> >> that still hasn't been fixed. =A0The workaround is to delete >> >> >> >> everything >> >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a >> >> >> >> mysqldump >> >> >> >> backup. =A0I'm really hoping that clandmeter has a backup... >> >> >> >> >> >> >> >> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 30716330= 72 >> >> >> >> in >> >> >> >> file btr0pcur.c line 425 >> >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) = =3D=3D >> >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor)) >> >> >> >> InnoDB: We intentionally generate a memory trap. >> >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. >> >> >> >> InnoDB: If you get repeated assertion failures or crashes, even >> >> >> >> InnoDB: immediately after the mysqld startup, there may be >> >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to >> >> >> >> InnoDB: >> >> >> >> >> >> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.= html >> >> >> >> InnoDB: about forcing recovery. >> >> >> >> 120110 21:26:39 - mysqld got signal 6 ; >> >> >> >> This could be because you hit a bug. It is also possible that >> >> >> >> this >> >> >> >> binary >> >> >> >> or one of the libraries it was linked against is corrupt, >> >> >> >> improperly >> >> >> >> built, >> >> >> >> or misconfigured. This error can also be caused by malfunctioni= ng >> >> >> >> hardware. >> >> >> >> We will try our best to scrape up some info that will hopefully >> >> >> >> help >> >> >> >> diagnose >> >> >> >> the problem, but since we have already crashed, something is >> >> >> >> definitely >> >> >> >> wrong >> >> >> >> and this may fail. >> >> >> >> >> >> >> >> key_buffer_size=3D16777216 >> >> >> >> read_buffer_size=3D262144 >> >> >> >> max_used_connections=3D0 >> >> >> >> max_threads=3D151 >> >> >> >> thread_count=3D0 >> >> >> >> connection_count=3D0 >> >> >> >> It is possible that mysqld could use up to >> >> >> >> key_buffer_size + (read_buffer_size + >> >> >> >> sort_buffer_size)*max_threads >> >> >> >> =3D >> >> >> >> 133434 K >> >> >> >> bytes of memory >> >> >> >> Hope that's ok; if not, decrease some variables in the equation= . >> >> >> >> >> >> >> >> Thread pointer: 0x0 >> >> >> >> Attempting backtrace. You can use the following information to >> >> >> >> find >> >> >> >> out >> >> >> >> where mysqld died. If you see no messages after this, something >> >> >> >> went >> >> >> >> terribly wrong... >> >> >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] >> >> >> >> The manual page at >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html >> >> >> >> contains >> >> >> >> information that should help you find out what is causing the >> >> >> >> crash. >> >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file >> >> >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk >> >> >> >> wrote: >> >> >> >> > Only other thing I can think to try is to dump the binlogs >> >> >> >> > since >> >> >> >> > it's >> >> >> >> > during recovery that the crash occurs. =A0Logs say there's on= ly >> >> >> >> > one >> >> >> >> > transaction that's trying to apply... >> >> >> >> > >> >> >> >> > Jeff >> >> >> >> > >> >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter >> >> >> >> > >> >> >> >> > wrote: >> >> >> >> > >> >> >> >> >> I'm getting the below, not sure what to do now. Already >> >> >> >> >> updated >> >> >> >> >> mysql >> >> >> >> >> to latest 2.2 version. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is >> >> >> >> >> Barracuda. >> >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn >> >> >> >> >> 15160684415 >> >> >> >> >> 120110 19:53:23 =A0InnoDB: Database was not shut down normal= ly! >> >> >> >> >> InnoDB: Starting crash recovery. >> >> >> >> >> InnoDB: Reading tablespace information from the .ibd files..= . >> >> >> >> >> InnoDB: Restoring possible half-written data pages from the >> >> >> >> >> doublewrite >> >> >> >> >> InnoDB: buffer... >> >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number >> >> >> >> >> 15163771045 >> >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleane= d >> >> >> >> >> up >> >> >> >> >> InnoDB: in total 12 row operations to undo >> >> >> >> >> InnoDB: Trx id counter is 6DE8E00 >> >> >> >> >> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log >> >> >> >> >> records >> >> >> >> >> to >> >> >> >> >> the >> >> >> >> >> database... >> >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld g= ot >> >> >> >> >> signal >> >> >> >> >> 11 ; >> >> >> >> >> This could be because you hit a bug. It is also possible tha= t >> >> >> >> >> this >> >> >> >> >> binary >> >> >> >> >> or one of the libraries it was linked against is corrupt, >> >> >> >> >> improperly >> >> >> >> >> built, >> >> >> >> >> or misconfigured. This error can also be caused by >> >> >> >> >> malfunctioning >> >> >> >> >> hardware. >> >> >> >> >> We will try our best to scrape up some info that will >> >> >> >> >> hopefully >> >> >> >> >> help >> >> >> >> >> diagnose >> >> >> >> >> the problem, but since we have already crashed, something is >> >> >> >> >> definitely >> >> >> >> >> wrong >> >> >> >> >> and this may fail. >> >> >> >> >> >> >> >> >> >> key_buffer_size=3D16777216 >> >> >> >> >> read_buffer_size=3D262144 >> >> >> >> >> max_used_connections=3D0 >> >> >> >> >> max_threads=3D151 >> >> >> >> >> thread_count=3D0 >> >> >> >> >> connection_count=3D0 >> >> >> >> >> It is possible that mysqld could use up to >> >> >> >> >> key_buffer_size + (read_buffer_size + >> >> >> >> >> sort_buffer_size)*max_threads >> >> >> >> >> =3D >> >> >> >> >> 133434 K >> >> >> >> >> bytes of memory >> >> >> >> >> Hope that's ok; if not, decrease some variables in the >> >> >> >> >> equation. >> >> >> >> >> >> >> >> >> >> Thread pointer: 0x0 >> >> >> >> >> Attempting backtrace. You can use the following information = to >> >> >> >> >> find >> >> >> >> >> out >> >> >> >> >> where mysqld died. If you see no messages after this, >> >> >> >> >> something >> >> >> >> >> went >> >> >> >> >> terribly wrong... >> >> >> >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] >> >> >> >> >> The manual page at >> >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html >> >> >> >> >> contains >> >> >> >> >> information that should help you find out what is causing th= e >> >> >> >> >> crash. >> >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file >> >> >> >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter >> >> >> >> >> >> >> >> >> >> wrote: >> >> >> >> >>> Ill be able to check in a minute if needed. >> >> >> >> >>> >> >> >> >> >>> Carlo >> >> >> >> >>> >> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" >> >> >> >> >>> >> >> >> >> >>> wrote: >> >> >> >> >>>> >> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: >> >> >> >> >>>>> >> >> >> >> >>>>> Hi Everyone, >> >> >> >> >>>>> >> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. >> >> >> >> >>>>> =A0Fastcgi >> >> >> >> >>>>> is >> >> >> >> >>>>> crashing, by the looks of things because of multiple >> >> >> >> >>>>> spiders >> >> >> >> >>>>> hitting >> >> >> >> >>>>> it at once. =A0Banning the user agents in lightty config >> >> >> >> >>>>> hasn't >> >> >> >> >>>>> helped, >> >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but >> >> >> >> >>>>> I'm >> >> >> >> >>>>> out >> >> >> >> >>>>> of >> >> >> >> >>>>> time to keep trying to fix it. =A0If anyone else has some >> >> >> >> >>>>> time >> >> >> >> >>>>> to >> >> >> >> >>>>> take >> >> >> >> >>>>> a >> >> >> >> >>>>> look, that'd be awesome, >> >> >> >> >>>>> >> >> >> >> >>>>> Thanks, >> >> >> >> >>>>> >> >> >> >> >>>>> Jeff >> >> >> >> >>>>> >> >> >> >> >>>> >> >> >> >> >>>> Mysql died, so far I'm unable to restart it >> >> >> >> >>>> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Jeff >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Jeff >> >> >> >> >> >> >> >> -- >> >> Jeff >> >> >> >> -- >> Jeff