X-Original-To: alpine-infra@alpinelinux.org Delivered-To: alpine-infra@mail.alpinelinux.org Received: from mail-tul01m020-f171.google.com (mail-tul01m020-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.alpinelinux.org (Postfix) with ESMTPS id 366FEDC1844 for ; Tue, 10 Jan 2012 21:49:16 +0000 (UTC) Received: by obbwd20 with SMTP id wd20so108449obb.30 for ; Tue, 10 Jan 2012 13:49:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=9S7iAGFVR2vFHsL5EGbvRTboZhu0hQfOQNoDnyMyQRs=; b=EUGoQfxfXUAQP7mTlqYu1paS5DdMXkuAykr58tZWFj7w1WuB8HPX+GsHkNqf1qKTbi iHvY7MY2Wr8iN63khLPtviI90Hy8MhKM8TnEyw8v4Ar1blfro8LI4Lp8iPmKkWiZ/ajp hzMzX/nqgXqTWBn1+vwXODkhuRVEgV59NtN08= MIME-Version: 1.0 Received: by 10.182.7.42 with SMTP id g10mr20005128oba.7.1326232156618; Tue, 10 Jan 2012 13:49:16 -0800 (PST) Received: by 10.182.14.3 with HTTP; Tue, 10 Jan 2012 13:49:16 -0800 (PST) In-Reply-To: References: <4F0C8B9E.40808@nothome.org> <2B149571-CAE6-463D-BDD7-EE2CA5B74977@gmail.com> Date: Tue, 10 Jan 2012 16:49:16 -0500 Message-ID: Subject: Re: Bugs.a.o From: Jeff Bilyk To: Carlo Landmeter Cc: Nathan Angelacos , alpine-infra@alpinelinux.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks. I should be back home in a few hours, if you need a hand, I can give it a try. Fwiw, Ubuntu's MySQL package doesn't suffer the same bug as far as we could tell. Fabled figured it was something in uclibc iirc that was causing this crash, but we weren't able to isolate exactly what the bug was... On Tue, Jan 10, 2012 at 4:46 PM, Carlo Landmeter wro= te: > Should be no problem. It's an vserver. Ill try tonight. > > On Jan 10, 2012 10:44 PM, "Jeff Bilyk" wrote: >> >> It's worth a shot, but I don't have the time/a box at work to try. >> Could you give it a shot? >> >> On Tue, Jan 10, 2012 at 4:40 PM, Carlo Landmeter >> wrote: >> > What about starting this var on Ubuntu or something and export? >> > >> > On Jan 10, 2012 10:37 PM, "Jeff Bilyk" wrote: >> >> >> >> Urgh - I'm not sure how to recover that then... Carlo, I've got a cop= y >> >> of /var/lib/mysql copied over to /var/lib/mysql.bak/, and tried a >> >> couple...invasive... recovery tricks on the copy in /varlib/mysql, bu= t >> >> they didn't work. =A0In the testing we did for the bug below, the onl= y >> >> way I was ever able to get a copy running again was delete/restore >> >> from backup. >> >> >> >> Anyone have any ideas where to go from here? >> >> >> >> On Tue, Jan 10, 2012 at 4:33 PM, Carlo Landmeter >> >> wrote: >> >> > The bad answer is, no dump available.... >> >> > >> >> > On Jan 10, 2012 10:30 PM, "Jeff Bilyk" wrote: >> >> >> >> >> >> So that failed. =A0We're hitting what seems to be the same bug I'v= e >> >> >> hit >> >> >> before in MySQL on Alpine (http://bugs.mysql.com/bug.php?id=3D6263= 4) >> >> >> that still hasn't been fixed. =A0The workaround is to delete >> >> >> everything >> >> >> in /var/lib/mysql/, reinitialize the db, then restore from a >> >> >> mysqldump >> >> >> backup. =A0I'm really hoping that clandmeter has a backup... >> >> >> >> >> >> 120110 21:26:39 =A0InnoDB: Assertion failure in thread 3071633072 = in >> >> >> file btr0pcur.c line 425 >> >> >> InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) =3D= =3D >> >> >> buf_block_get_page_no(btr_pcur_get_block(cursor)) >> >> >> InnoDB: We intentionally generate a memory trap. >> >> >> InnoDB: Submit a detailed bug report to http://bugs.mysql.com. >> >> >> InnoDB: If you get repeated assertion failures or crashes, even >> >> >> InnoDB: immediately after the mysqld startup, there may be >> >> >> InnoDB: corruption in the InnoDB tablespace. Please refer to >> >> >> InnoDB: >> >> >> http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.htm= l >> >> >> InnoDB: about forcing recovery. >> >> >> 120110 21:26:39 - mysqld got signal 6 ; >> >> >> This could be because you hit a bug. It is also possible that this >> >> >> binary >> >> >> or one of the libraries it was linked against is corrupt, improper= ly >> >> >> built, >> >> >> or misconfigured. This error can also be caused by malfunctioning >> >> >> hardware. >> >> >> We will try our best to scrape up some info that will hopefully he= lp >> >> >> diagnose >> >> >> the problem, but since we have already crashed, something is >> >> >> definitely >> >> >> wrong >> >> >> and this may fail. >> >> >> >> >> >> key_buffer_size=3D16777216 >> >> >> read_buffer_size=3D262144 >> >> >> max_used_connections=3D0 >> >> >> max_threads=3D151 >> >> >> thread_count=3D0 >> >> >> connection_count=3D0 >> >> >> It is possible that mysqld could use up to >> >> >> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_thread= s >> >> >> =3D >> >> >> 133434 K >> >> >> bytes of memory >> >> >> Hope that's ok; if not, decrease some variables in the equation. >> >> >> >> >> >> Thread pointer: 0x0 >> >> >> Attempting backtrace. You can use the following information to fin= d >> >> >> out >> >> >> where mysqld died. If you see no messages after this, something we= nt >> >> >> terribly wrong... >> >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb74e6529] >> >> >> The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html >> >> >> contains >> >> >> information that should help you find out what is causing the cras= h. >> >> >> 120110 21:26:39 mysqld_safe mysqld from pid file >> >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> On Tue, Jan 10, 2012 at 4:14 PM, Jeff Bilyk >> >> >> wrote: >> >> >> > Only other thing I can think to try is to dump the binlogs since >> >> >> > it's >> >> >> > during recovery that the crash occurs. =A0Logs say there's only = one >> >> >> > transaction that's trying to apply... >> >> >> > >> >> >> > Jeff >> >> >> > >> >> >> > On Jan 10, 2012, at 3:02 PM, Carlo Landmeter >> >> >> > >> >> >> > wrote: >> >> >> > >> >> >> >> I'm getting the below, not sure what to do now. Already updated >> >> >> >> mysql >> >> >> >> to latest 2.2 version. >> >> >> >> >> >> >> >> >> >> >> >> 120110 19:53:23 InnoDB: highest supported file format is >> >> >> >> Barracuda. >> >> >> >> InnoDB: Log scan progressed past the checkpoint lsn 15160684415 >> >> >> >> 120110 19:53:23 =A0InnoDB: Database was not shut down normally! >> >> >> >> InnoDB: Starting crash recovery. >> >> >> >> InnoDB: Reading tablespace information from the .ibd files... >> >> >> >> InnoDB: Restoring possible half-written data pages from the >> >> >> >> doublewrite >> >> >> >> InnoDB: buffer... >> >> >> >> InnoDB: Doing recovery: scanned up to log sequence number >> >> >> >> 15163771045 >> >> >> >> InnoDB: 1 transaction(s) which must be rolled back or cleaned u= p >> >> >> >> InnoDB: in total 12 row operations to undo >> >> >> >> InnoDB: Trx id counter is 6DE8E00 >> >> >> >> 120110 19:53:24 =A0InnoDB: Starting an apply batch of log recor= ds >> >> >> >> to >> >> >> >> the >> >> >> >> database... >> >> >> >> InnoDB: Progress in percents: 0 1 120110 19:53:24 - mysqld got >> >> >> >> signal >> >> >> >> 11 ; >> >> >> >> This could be because you hit a bug. It is also possible that >> >> >> >> this >> >> >> >> binary >> >> >> >> or one of the libraries it was linked against is corrupt, >> >> >> >> improperly >> >> >> >> built, >> >> >> >> or misconfigured. This error can also be caused by malfunctioni= ng >> >> >> >> hardware. >> >> >> >> We will try our best to scrape up some info that will hopefully >> >> >> >> help >> >> >> >> diagnose >> >> >> >> the problem, but since we have already crashed, something is >> >> >> >> definitely >> >> >> >> wrong >> >> >> >> and this may fail. >> >> >> >> >> >> >> >> key_buffer_size=3D16777216 >> >> >> >> read_buffer_size=3D262144 >> >> >> >> max_used_connections=3D0 >> >> >> >> max_threads=3D151 >> >> >> >> thread_count=3D0 >> >> >> >> connection_count=3D0 >> >> >> >> It is possible that mysqld could use up to >> >> >> >> key_buffer_size + (read_buffer_size + >> >> >> >> sort_buffer_size)*max_threads >> >> >> >> =3D >> >> >> >> 133434 K >> >> >> >> bytes of memory >> >> >> >> Hope that's ok; if not, decrease some variables in the equation= . >> >> >> >> >> >> >> >> Thread pointer: 0x0 >> >> >> >> Attempting backtrace. You can use the following information to >> >> >> >> find >> >> >> >> out >> >> >> >> where mysqld died. If you see no messages after this, something >> >> >> >> went >> >> >> >> terribly wrong... >> >> >> >> stack_bottom =3D (nil) thread_stack 0x30000 >> >> >> >> /usr/sbin/mysqld(my_print_stacktrace+0x46)[0xb73bb529] >> >> >> >> The manual page at >> >> >> >> http://dev.mysql.com/doc/mysql/en/crashing.html >> >> >> >> contains >> >> >> >> information that should help you find out what is causing the >> >> >> >> crash. >> >> >> >> 120110 19:53:24 mysqld_safe mysqld from pid file >> >> >> >> /var/run/mysqld/mysqld.pid ended >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Jan 10, 2012 at 8:08 PM, Carlo Landmeter >> >> >> >> >> >> >> >> wrote: >> >> >> >>> Ill be able to check in a minute if needed. >> >> >> >>> >> >> >> >>> Carlo >> >> >> >>> >> >> >> >>> On Jan 10, 2012 8:03 PM, "Nathan Angelacos" >> >> >> >>> wrote: >> >> >> >>>> >> >> >> >>>> On 01/10/2012 09:49 AM, Jeff Bilyk wrote: >> >> >> >>>>> >> >> >> >>>>> Hi Everyone, >> >> >> >>>>> >> >> >> >>>>> Redmine on bugs.a.o is having issues this afternoon. =A0Fast= cgi >> >> >> >>>>> is >> >> >> >>>>> crashing, by the looks of things because of multiple spiders >> >> >> >>>>> hitting >> >> >> >>>>> it at once. =A0Banning the user agents in lightty config has= n't >> >> >> >>>>> helped, >> >> >> >>>>> neither has upping the redmine sockets to 32 (from 4) but I'= m >> >> >> >>>>> out >> >> >> >>>>> of >> >> >> >>>>> time to keep trying to fix it. =A0If anyone else has some ti= me >> >> >> >>>>> to >> >> >> >>>>> take >> >> >> >>>>> a >> >> >> >>>>> look, that'd be awesome, >> >> >> >>>>> >> >> >> >>>>> Thanks, >> >> >> >>>>> >> >> >> >>>>> Jeff >> >> >> >>>>> >> >> >> >>>> >> >> >> >>>> Mysql died, so far I'm unable to restart it >> >> >> >>>> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Jeff >> >> >> >> >> >> >> >> -- >> >> Jeff >> >> >> >> -- >> Jeff --=20 Jeff