Bug #2816

Storeconfigs - MySQL server has gone away

Added by Dan Carley about 2 years ago. Updated almost 2 years ago.

Status:Closed Start date:11/13/2009
Priority:Normal Due date:
Assignee:- % Done:

0%

Category:Rails
Target version:0.25.2
Affected Puppet version:0.25.1 Branch:http://github.com/jes5199/puppet/tree/ticket/0.25.x/2816
Keywords:
Votes: 1

Description

I am receiving the following error from storeconfigs with a MySQL backend.

Mysql::Error: MySQL server has gone away

Respawning one client in a loop doesn’t produce the error. Spawning all clients (17 on this particular network) at once from cron reproduces the error every time. Several of the clients then fail to complete. Attached is a puppetmasterd log of such a run. Observing MySQL’s processlist at the time shows that it holds up to 18 idle connections open until the point at which the errors are logged and then connections are closed.

The puppetmaster is setup like so.

  • Gentoo Linux
  • Puppet 0.25.0
  • Mongrel 1.1.5 (5 puppetmasterd processes)
  • Nginx 0.8.17
  • Rails 2.3.4
  • ActiveRecord 2.3.4
  • MySQL/Ruby 2.7.5
  • MySQL 5.0.70 (unix socket, default settings)

I have tried combinations of the following with no success.

  • Downgrading to Rails 2.2 and ActiveRecord 2.2.3 (with patch from #2508).
  • Upgrading to MySQL/Ruby 2.8.1

I have noticed that amending @rails.rb@ to set the ActiveRecord connection option @:reconnect = true@ seems to resolve the issue. However I’m not clear whether such an approach is transaction safe for Puppet to use.

storeconfigs_mysql_error-20091113.txt (2.4 kB) Dan Carley, 11/13/2009 05:45 pm


Related issues

related to Puppet - Bug #2764: Active Record 2.1 no longer compatible with 0.25.1 Closed 10/29/2009
related to Puppet - Bug #2880: Fix for mysql 'server has gone away' under passenger Duplicate 11/30/2009
related to Puppet - Bug #4694: Storeconfig issue with AR/MySQL Duplicate 09/02/2010

History

Updated by Markus Roberts about 2 years ago

  • Status changed from Unreviewed to Investigating
  • Target version set to 0.25.2

Dan —

I suspect this may be resolved by the patch for #2764; could you test that branch and, if it resolves it, mark this ticket as a duplicate of #2764? If it doesn’t do it, add a note, assign the ticket to me, and I’ll look into it further.

Thanks,

— Markus

Updated by Dan Carley about 2 years ago

I did take a look at #2764 previously. It doesn’t apply because @allow_concurrency@ was deprecated in ActiveRecord 2.2 and later.

It might also be worth noting that the same behaviour occurs with a single Webrick instance as it does multiple Mongrel instances.

Updated by Adam Gibbins about 2 years ago

  • Affected Puppet version changed from 0.25.0 to 0.25.1

I’m receiving this exact same problem. All machines received this error:

Nov 17 17:47:34 puppet.example.com puppetmasterd[19059]: Mysql::Error: MySQL server has gone away: SELECT * FROM `hosts`     WHERE (`hosts`.`name` = 'slave1.example.com')  LIMIT 1

Restarting puppetmaster/mongrel resolved this instantly however it reoccurs again 5 minutes later.

Puppet Version (Master and Client): 0.25.1 MySQL Version: MySQL 5.0.75-0ubuntu10.2 Operating System: Ubuntu Jaunty 9.04 Mongrel: 1.1.5 (10 puppetmasterd processes) Apache: 2.2.11 (Prefork 64bit) Ruby: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

Please let me know if you require any additional details (My ruby knowledge is practically non existent).

Updated by Brice Figureau about 2 years ago

Adam Gibbins wrote:

I’m receiving this exact same problem. All machines received this error: […] Restarting puppetmaster/mongrel resolved this instantly however it reoccurs again 5 minutes later.

Puppet Version (Master and Client): 0.25.1 MySQL Version: MySQL 5.0.75-0ubuntu10.2 Operating System: Ubuntu Jaunty 9.04 Mongrel: 1.1.5 (10 puppetmasterd processes) Apache: 2.2.11 (Prefork 64bit) Ruby: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

Please let me know if you require any additional details (My ruby knowledge is practically non existent).

You forgot to print the most important information: rails version. I highly suspect you run 2.1 and thus you’re seeing #2764.

Updated by Luke Kanies about 2 years ago

  • Assignee set to Jesse Wolfe

Updated by Jesse Wolfe about 2 years ago

  • Status changed from Investigating to In Topic Branch Pending Review
  • Branch set to http://github.com/jes5199/puppet/tree/ticket/0.25.x/2816

This patch sets :reconnect to true, as suggested by the ticket.

Updated by Markus Roberts about 2 years ago

  • Assignee deleted (Jesse Wolfe)

Updated by James Turnbull about 2 years ago

Added this to the commit.

diff --git a/spec/unit/rails.rb b/spec/unit/rails.rb
index 0302873..f61288f 100755
--- a/spec/unit/rails.rb
+++ b/spec/unit/rails.rb
@@ -90,7 +90,7 @@ end
 describe Puppet::Rails, "when initializing a mysql or postgresql connection" do
     confine "Cannot test without ActiveRecord" => Puppet.features.rails?
 
-    it "should provide the adapter, log_level, and host, username, password, and database arguments" do
+    it "should provide the adapter, log_level, and host, username, password, database, and reconnect arguments" do
         Puppet.settings.stubs(:value).with(:dbadapter).returns("mysql")
         Puppet.settings.stubs(:value).with(:rails_loglevel).returns("testlevel")
         Puppet.settings.stubs(:value).with(:dbserver).returns("testserver")
@@ -105,11 +105,12 @@ describe Puppet::Rails, "when initializing a mysql or postgresql connection" do
             :host => "testserver",
             :username => "testuser",
             :password => "testpassword",
-            :database => "testname"
+            :database => "testname",
+            :reconnect => true
         }
     end
 
-    it "should provide the adapter, log_level, and host, username, password, database, and socket arguments" do
+    it "should provide the adapter, log_level, and host, username, password, database, socket, and reconnect arguments" do
         Puppet.settings.stubs(:value).with(:dbadapter).returns("mysql")
         Puppet.settings.stubs(:value).with(:rails_loglevel).returns("testlevel")
         Puppet.settings.stubs(:value).with(:dbserver).returns("testserver")
@@ -125,7 +126,8 @@ describe Puppet::Rails, "when initializing a mysql or postgresql connection" do
             :username => "testuser",
             :password => "testpassword",
             :database => "testname",
-            :socket => "testsocket"
+            :socket => "testsocket",
+            :reconnect => true
         }
     end
 end

Updated by James Turnbull about 2 years ago

  • Status changed from In Topic Branch Pending Review to Closed

Pushed in commit:b86decc0ea274eb6d9ffa3170fd4ec81d735519f in branch 0.25.x

Also available in: Atom PDF