What a Weekend!
posted in Frenzied Daddy, Rantings |
It’s been a hell weekend.
Saturday was Ms B’s work picnic at the zoo. How cool is that? We got free admission to the zoo, late at night after the zoo closed. Most of the animals, were, sadly, sleeping, but we had food, played bingo, looked at elephants, and even rode the zoo train. Miss K had way too much fun and threw a total tantrum when we left. I mean a serious tantrum. We’ll have to go back; she loved it. I think she’d move there if we’d let her.
Sunday was a different story. I’m nominally in charge for my “regular” job while Chris gets married and has his honeymoon. Saturday morning before I came on, the main database server failed. The main replicator from this server failed. When I came on, it took about an hour to an hour and a half to get it back online. However, I didn’t check for the other servers; and shortly afterwards, they failed too. That wasn’t so great, because when that happened I was dealing with another issue and then mowing the lawn. If I had dealt with it right away, it would have been (relatively painless). But since I waited, the log files were cleaned up, and I had to suffer.
The other issue I was dealing with? One of my main clients has a webserver and does web hosting. They were having phishing sites uploaded to their servers and spam messages being sent out. It took me a while to straighten this out, and then since I don’t have as much time for them. I scripted some protection. And updated their firewall. And … well, you get the idea. It was a heavy job.
Then it was time to check back with my “real” job; and start the laborous process of redoing mysql replication from square one. I got those started and it was time to go get some food for dinner. Hurray for me! I went shopping. And made dinner, and came back into the office to find out that the drives on the slave database servers were full and replication still wasn’t going.
I turned the task over to the two other guys on call this weekend so I could get some sleep. I’m logging in now to find … yup, mysql replication is still broken. So, it’s being a rough week.
