The return to the mess... - Blog - Open Source - schlitt.info

schlitt.info - php, photography and private stuff

The return to the mess...

When I returned from vacation, which already wasn't as relaxing as expected due to priate issues, I directly returned to a huge mess, which you might already have been noticing by the downtime of phpunit.de website.

The first experience I had after booting my notebook was, that there did not seem to be any new emails in my private inbox. While this would usually be quite convenient after vacation, it made me kind of affraid, where the expected huge email load would have gone? SSHing into my private little server showed some interessting facts: First of, the load-indicator was at about 2500 (as Sebastian showed in his article already), a cool value in a way, if it is not your own server. The second fact was, that the /var partition had run full 4 days previously.

The latter issue was easily fixable: MySQL wrote binary logs of any change happening to a database. Don't ask me why, I did not (at least I'm not aware I did) switch this feature on, so it's possible a standard setting on Gentoo. Deleting these logs freed 6 GB of space on /var and made Lighttpd start up again to serve at least some of the websites. Sadly it did not serve phpunit.de, but more on that later.

Digging into my Qmail installation I noted, that about 5000 mails got stuck in the local queue, not being delivered (qmHandle helped a lot here). A closer look told me, that these were at least some of the messages I had expected in my inbox, which calmed me down a bit. Restarting Qmail and forcing it to flush its queues let my Thunderbird start downloading mails, so I left it for a while that way. When returning ah hour later, my inbox showed over 24000 emails. Wow! I had expected a huge lot, but that was a bit too much.

Guess what, each email was there at least duplicated, while most mails existed 7-10 times. I guess this was the effect of me trying out Qmailadmins auto-reply feature right before vacation. This test did not work as expected, so I switched the flag of again, being sure it would not do any damage. Still, even with having delivered a 24000 mails mess to me, Qmail had 5000 mails stuck... Those turned out to me mostly spam and bounces, so I decided to delete them from the queue directly and about 200 mails were left over from the 4 days with a full partition. Sounds Windozish, but a final reboot helped to flush those, too. I guess there was some Qmail process stuck, which kept the MTA to deliver further on, like a lock.

Sorting out all the bulsh``**`` from my inbox took about the rest of Saturday, so I did not find time to take care of the webserver. On Sunday I spontaniously decided to join the geeks at FrOSCon, which was the right decision. Right now I notice, that I did not take the chance of shaking the hand of Henry Bergius, who was attending to give a talk at the PHP room we organized, because I was too busy discussion stuff with other people. Sorry for that, Henry, I hope we will meet again soonish!

Anyway, during some session Sebastian and me took the chance to check through Lighttpd, which still did not serve Sebastians site on PHPUnit. I have to admit, this was my fault actually: Short before vacation I updated the world of the server, because of some security issues I heard of. Since times were heavily busy, I forgot to take further care of the update, like restarting processes and running post-update tasks indicated by emerge. That way, Trac for phpunit.de was updated, but the necessary Python updates were incorrect, since I did not run python-updater on the new Python versions installed. Doing so finally fixed phpunit.de.

So, what did I learn from this mess? I guess 3 things: 1. If you change a running system, do it right and keep thinking of the update. 2. Monitor your server more closely, so that you see issues like full partitions before they get serious. 3. Vacation auto-reply is in general a bad idea! ;)

Anyway, the server is running smooth again and Sebastian was smiling all over the face, that phpunit.de is up and running again, while he looked over my shoulder and corrected this article.

So, after that messy weekend and the borked vacation before, it's time for me to get into eZ Components development again. I did not do much in the past 2 months in that field, but the following weeks are still university free, so you can expect some progress. Today I already caught up with about 400 mails in my eZ mail account so development can start right away tomorrow.

Sorry to everyone who missed phpunit.de last week! Since there is the still the slight possibility, that some mails were bounced our eaten, please email me again, if I did not reply to you, yet! Thanks for your appreciation!

If you liked this blog post or learned something, please consider using flattr to contribute back: .

Trackbacks

Comments

Add new comment

Fields with bold names are mandatory.