As your cluster starts growing, you have to worry about the communication between cluster data nodes. If you have setup you cluster optimally, all of the data nodes should have their own private communications network. This gives all the data nodes plenty of bandwidth to trade data back and forth and stay synchronized.

While talking to a MySQL consultant, he gave us a few guidelines for the data node private network. He said a gigabit network is OK for clusters of 8 data nodes or smaller. But when you start going above 8 data nodes the network traffic will start to overload the gigabit and you will hit bandwidth limitations. He recommended we start looking at Dolphin interconnect cards as a viable upgrade option. Dolphin interconnects are extremely fast at sending small packets of data and allow for much faster throughput than standard gigabit networks. Check out the benchmarks on the Dolphin cards. It's actually pretty amazing.