How to syncing data between application instances? Easy!

Syncing data with 12k requests/sec messages between instances

Replication. What is it?

Data replication is the process of syncing data between you services (databases, applications, etc). It can help you not to lose data in critical incidents and easily scale the increasing load on your services.

Replication description from the Wiki:
Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
Me after a week recover lost data from the broken server

Ok. Suppose I convinced you and you want to make your own replication in your application

But how?

First steps. Making the transaction model.

The basic transaction must have some data and an identifier for the transfer.

// In my opinion, protobufs is the best solution for the replication.
// Our identifier will be unix nano timestamp
syntax = "proto3";

message Transaction {
int64 timestamp = 1;
map<string, bytes> data = 2;
}
var transactionPool = sync.Pool{
New: func() interface{} {
return new(Transaction)
},
}

func GetTransaction() *Transaction {
return transactionPool.Get().(*Transaction)
}

func PutTransaction(transaction *Transaction) {
transaction.Reset()
transactionPool.Put(transaction)
}
func NewTransaction(data map[string][]byte) *Transaction {
transaction := GetTransaction()
transaction.Timestamp = time.Now().UnixNano()
transaction.Data = data
return transaction
}

Making the replication.

The simple replication must have the transmitter and receiver (for send and receive transactions), list of received transactions and list of any application hosts for sending transactions.

type Replication struct {
Transmitter chan *Transaction
Receiver chan *Transaction

Received map[string]int64

Replicas map[string]string
}
func (r *Replication) Transmit() {
for {
transaction := <-r.Transmitter

go func() {
body, err := proto.Marshal(transaction)

if err == nil {
for addr, host := range r.Replicas {
//That's url what you services will listen for transactions
_, err := http.Post(host + "/replication/receiver", "application/protobuf", bytes.NewReader(body))

if err != nil {
log.Println("Master", addr, "is down:", err)
}
}
} else {
log.Println("Replication error:", err)
}

PutTransaction(transaction)
}()
}
}
func (r *Replication) Receive() {
for {
transaction := <-r.Receiver

go func() {
for key, data := range transaction.Data {
if timestamp, ok := r.Received[key]; ok && timestamp <= transaction.Timestamp {
continue
}
// Put transaction data to you data store
err := database.Client.Put(key, data)

if err == nil {
r.Received[key] = transaction.Timestamp
} else {
log.Println("Replication error:", err)
}
}

PutTransaction(transaction)
}()
}
}
func (r *Replication) SendMessage(data map[string][]byte) {
if len(r.Replicas) > 0 {
r.Transmitter <- NewTransaction(data)
}
}
replication := &Replication{
Transmitter: make(chan *Transaction, 2048),
Receiver: make(chan *Transaction, 2048),
Received: make(map[string]int64),
Replicas: hosts,
}

Replication usage

Just run receiver and transmitter in the new thread

go replication.Receive()
go replication.Transmit()
http.HandleFunc("/replication/receiver", func(writer http.ResponseWriter, request *http.Request) {
transaction := GetTransaction()
body, _ := ioutil.ReadAll(request.Body)
err := proto.Unmarshal(body, transaction)
if err == nil {
replication.Receiver <- transaction
fmt.Fprintln(writer, "Received transaction")
} else {
fmt.Fprintln(writer, err)
}
})
replication.SendMessage()

Conclusion

Replication can really save the life of your product. But it is not a panacea.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store