Comment me (out) if you want to live!

What a day to launch our tech blog (on a Friday, no less).

For the last couple of weeks, we’ve been kicking around the idea of starting a tech blog where we could talk about the things that go on internally behind the scenes, and inside the servers, at HiretheWorld. Today, we actually launched it.

The first article to be posted was Importing contacts with Cloudsponge in CodeIgniter, because that happened to be latest feature that we launched – or so we thought.

It turns out that at the same time, we also decided to start using Markdown on our blog. The combination wasn’t good because the next thing we knew, the server appeared to die. For us, server death is normally a consequence of too much memory being used, causing excessive swapping to disk. After the panic and subsequent restart of the server, we did as most good server admins do: look into the server logs to figure out what happened.

Yes, it was excessive memory usage, as we thought, and as top showed, but since everything was working before the post, we thought that there was no reason for the server to die. Tracing into apache’s error log, we had line after line of (e.g.):

[Fri May  4 16:40:18 2012] [apc-warning] Unable to allocate memory for pool. in /hiretheworld.com/blog/wp-includes/comment-template.php on line 913.
[Fri May  4 16:40:18 2012] [apc-warning] Unable to allocate memory for pool. in /hiretheworld.com/blog/wp-includes/script-loader.php on line 25.
[Fri May  4 16:40:18 2012] [apc-warning] Unable to allocate memory for pool. in /hiretheworld.com/blog/wp-content/plugins/facebook-comments-for-wordpress/facebook-comments.php on line 30.
[Fri May  4 16:40:18 2012] [apc-warning] Unable to allocate memory for pool. in /hiretheworld.com/blog/wp-content/plugins/shortcode-exec-php/shortcode-exec-php.php on line 45.
[Fri May  4 16:40:18 2012] [apc-warning] Unable to allocate memory for pool. in /hiretheworld.com/blog/wp-content/plugins/shortcodes-ultimate/shortcodes-ultimate.php on line 15.

etc...

which brought us to the conclusion that we were probably loading too many plugins inside wordpress.

And if we were just good programmers, we would have left it there: increase the memory, restart apache, and hope everything would continue working. But we’re better than that: we’re semi-paranoid server admins, so we also looked into apache’s access log to see what the system was doing at the time it died.

And here’s what we found:

37.59.16.126 - - [04/May/2012:16:32:38 -0700] "HEAD /blog/tech-blog/importing-contacts-with-cloudsponge-in-codeigniter?utm_source=rss&utm_medium=dlvr.it&utm_campaign=dlvr.it&https%3A%2F%2Fwww.hiretheworld.com%2F%3Futm_source=dlvr.it HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
37.59.16.129 - - [04/May/2012:16:33:30 -0700] "HEAD /blog/company/site-updates/invite-your-friends-make-money-helping-your-friends-find-work-or-workers?utm_source=rss&utm_medium=dlvr.it&utm_campaign=dlvr.it&https%3A%2F%2Fwww.hiretheworld.com%2F%3Futm_source=dlvr.it HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
37.59.16.129 - - [04/May/2012:16:33:39 -0700] "HEAD /blog/company/site-updates/invite-your-friends-make-money-helping-your-friends-find-work-or-workers?utm_source=rss&utm_medium=dlvr.it&utm_campaign=dlvr.it&https%3A%2F%2Fwww.hiretheworld.com%2F%3Futm_source=dlvr.it HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
37.59.16.137 - - [04/May/2012:16:33:44 -0700] "HEAD /blog/company/site-updates/invite-your-friends-make-money-helping-your-friends-find-work-or-workers?utm_source=rss&utm_medium=dlvr.it&utm_campaign=dlvr.it&https%3A%2F%2Fwww.hiretheworld.com%2F%3Futm_source=dlvr.it HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
50.57.110.189 - - [04/May/2012:16:38:16 -0700] "GET /blog_integration/getIntegrationData/?share_url=&share_ref=&og_title= HTTP/1.1" 301 313 "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
50.57.110.189 - - [04/May/2012:16:38:23 -0700] "GET /blog_integration/getIntegrationData/?share_url=&share_ref=&og_title= HTTP/1.1" 301 313 "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)"
50.18.135.144 - - [04/May/2012:16:17:47 -0700] "GET /blog/tech-blog/importing-contacts-with-cloudsponge-in-codeigniter?utm_source=rss&utm_medium=dlvr.it&utm_campaign=dlvr.it&https%3A%2F%2Fwww.hiretheworld.com%2F%3Futm_source=dlvr.it HTTP/1.1" 200 36634 "-" "Python-urllib/2.7"

16:38:16… 16:38:23… 16:17:47… 16:17?!? — the blog post that went back in time and killed the server! The terminator blog post ;)

It was actually apache taking forever to retire an old request (because of the swapping), but it certainly looked like the server went back in time.

The moral of the story? Don’t write blog posts about inviting your friends, because they just might invite crawlers to your site. But if you insist on doing that, at least make sure that there’s enough memory on the server to give it a fighting chance.

If you don’t, you might end up with an apparent causality loop, forcing the system to send in the Terminators:

[Fri May 04 16:41:03 2012] [warn] child process 27889 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27870 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27890 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27872 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27379 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27873 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27891 still did not exit, sending a SIGTERM
[Fri May 04 16:41:03 2012] [warn] child process 27874 still did not exit, sending a SIGTERM

Won’t someone think of the child processes?

# I want to live!

This entry was posted in Tech Blog. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>