These servers have been running for 98 days. I think considering all the water that’s gone under the bridge, they are really close. To be honest, they are a lot closer than I personally thought they’d be.

Can I tell you why they are off - No. I do know of two separate occasions where the slave SQL thread stopped due to a bad query. In both cases there was some manual intervention. This could possible have had an affect on the binlogs.

I guess I’ll be doing a lot more investigation into this in the next couple of days. :-/

Even then, I don’t have a system for automatic failover of the intermediate nodes, and I’m not sure I would trust the binlog positions to be absolutely the same.

My current procedure in the event of a failure is to assign the standard replicated servers with one of the other intermediate servers as master. Before manually issuing a change master on each slave, I will compare the relay-log on the slave to the bin-log on the new master.

First, do a SHOW SLAVE STATUS on the slave you want to change master on. Make a note of RELAY_MASTER_LOG_FILE, EXEC_MASTER_LOG_POS, RELAY_LOG_FILE and RELAY_LOG_SPACE. On the slave, do mysqlbinlog RELAY_LOG_FILE and look at the timestamp and the command identified by RELAY_LOG_SPACE. On the new master, do mysqlbinlog RELAY_MASTER_LOG_FILE and look at the timestamp and the command identified by EXEC_MASTER_LOG_POS. If these commands are the same, it’s a safe bet I can just point the slave to it’s new master and use it’s current coordinates. If the commands are different, then we need to find the correct coordinate on the new master that reflect the next command that the slave needs to use before we can do a change master.

There was lots of talk in our shop about using a VIP in our intermediate layer, so that if one of those servers failed, we could just point the VIP to one of the still functioning intermediate servers. I put that conversation on hold for two reasons - I don’t trust the binlog positions to be in sync, and I only have n+1 intermediate servers, and if I assign all of my orphaned slaves to only one of those servers, it’s going to have double the load of all the others. There’s probably a way to carefully automate it all, but we have bigger fish to fry right now.

http://bugs.mysql.com/bug.php?id=36541

I also changed MASTER_LOG_FILE to RELAY_MASTER_LOG_FILE reflecting Baron’s comment.

RELAY_MASTER_LOG_FILE is the name of the master binary log file containing the most recent event executed by the SQL thread. It’s companion EXEC_MASTER_LOG_POSITION is the position of the last event executed by the SQL thread from the master’s binary log. See Documentation.

I would note that in the case of one of my intermediate slaves going down, the SQL thread on the end slaves is not going to stop. It’s going to process all the way through the relay logs. The I/O thread would keep trying to connect to the master every 60 seconds. If the slaves were guaranteed in sync, one could theoretically map the IP Address of the failed server to one of your working servers and never miss a beat.

I’m still working on automating this with mk-slave-move from Maatkit. You can already do it *before* the server crashes, with the current code. But after it crashes — that’s harder. Even while the servers are all functioning normally there are a lot of tricky cases to cover.

You want to look in to STOP SLAVE ; START SLAVE UNTIL instead, this guarantees that no statements run after the point that you want to be at.

This was discussed a little before on Baron’s blog (and in the comments) here:

http://www.xaprb.com/blog/2007/01/20/how-to-make-mysql-replication-reliable/