We have been using DotCloud as our hosting platform for months now, and overall I have been extremely pleased with their service.  There were some bumps in the road early on while they were still in their beta phase, but things have been running very smoothly for a few months now.  Everything except an uncomfortable number of seemingly random nginX “upstream timed out (110: Connection timed out) while reading response header from upstream” errors.

If you want to get what you need and not read the rest of this post, I will make it super simple for you:

If you are using php-fpm, gzip, xdebug and CodeIgniter, disable xdebug!!!

For anyone who feels like listening to my story, please read on.  Not only were these errors unsettling, but they were causing DotCloud’s load balancer to bounce POST requests  back to our servers multiple times because it was getting error responses and assuming nothing was happening on our end.  This in turn was causing user input (comments, image uploads, password requests, etc) to be saved (or emails sent) on our application multiple times.  Super embarrassing.

After weeks of research and floundering, a DotCloud tech and I finally discovered the issue.  There is a known bug with xdebug and ob_gzhandler which was causing our php processes to seg fault.  The bug is documented here:

http://grokbase.com/p/php/php-bugs/0365rtcdgx/23985-bgs-ob-gzhandler-make-segmentation-fault

What was happening was the request was sent to our server, the php process was doing everything it was supposed to, then when CodeIgniter’s output buffer was being gzipped by ob_gziphandler(), the php process was segmentation faulting and causing nginx to time out waiting for the response from php-fpm.  So, while everything was successfully happening in the php script, the output back to the client was failing.

By disabling the xdebug extension in the php configuration, the php processes stopped seg faulting, and everything is happy again!  No more Upstream Timeouts! It took a really long time to track this issue down, so I hope this post helps someone =)