James F. Roberts IV

Upstream timeout issues with nginX + php-fpm + CodeIgniter + gzip + xdebug on DotCloud – [resolved]

james — Thu, 10 May 2012 23:14:59 +0000

We have been using DotCloud as our hosting platform for months now, and overall I have been extremely pleased with their service. There were some bumps in the road early on while they were still in their beta phase, but things have been running very smoothly for a few months now. Everything except an uncomfortable number of seemingly random nginX “upstream timed out (110: Connection timed out) while reading response header from upstream” errors.

If you want to get what you need and not read the rest of this post, I will make it super simple for you:

If you are using php-fpm, gzip, xdebug and CodeIgniter, disable xdebug!!!

For anyone who feels like listening to my story, please read on. Not only were these errors unsettling, but they were causing DotCloud’s load balancer to bounce POST requests back to our servers multiple times because it was getting error responses and assuming nothing was happening on our end. This in turn was causing user input (comments, image uploads, password requests, etc) to be saved (or emails sent) on our application multiple times. Super embarrassing.

After weeks of research and floundering, a DotCloud tech and I finally discovered the issue. There is a known bug with xdebug and ob_gzhandler which was causing our php processes to seg fault. The bug is documented here:

http://grokbase.com/p/php/php-bugs/0365rtcdgx/23985-bgs-ob-gzhandler-make-segmentation-fault

What was happening was the request was sent to our server, the php process was doing everything it was supposed to, then when CodeIgniter’s output buffer was being gzipped by ob_gziphandler(), the php process was segmentation faulting and causing nginx to time out waiting for the response from php-fpm. So, while everything was successfully happening in the php script, the output back to the client was failing.

By disabling the xdebug extension in the php configuration, the php processes stopped seg faulting, and everything is happy again! No more Upstream Timeouts! It took a really long time to track this issue down, so I hope this post helps someone =)

AddThis Pintrest button hack to look better in 32×32 format

james — Mon, 30 Apr 2012 22:17:37 +0000

Today I had to add a Pinterest Share button on our website, and run it through our AddThis account so we can track the analytics. Unfortunately, there are only two options for the Pinterest button via AddThis, neither of which look good along side our 32×32 style share buttons. So, I decided to hack it up =)

If you apply a fixed height to the a tag which holds the Pinterest button’s iframe, then set overflow to hidden, then apply a negative margin to the nested iframe, you can essentially get rid of that pesky count bubble that will not go away.

Resulting css is something like:

 
	.addthis_toolbox > a.addthis_button_pinterest_pinit{
		margin:8px 0px 4px;
		height:24px;
		overflow:hidden;
		vertical-align:baseline;
	}
	.addthis_toolbox > a.addthis_button_pinterest_pinit >; iframe{
		margin-top:-34px;
	}

mongodb geospatial query example in php with distance radius in miles and km

james — Mon, 12 Mar 2012 06:44:07 +0000

Recently for a project involving 311 reports in New York city, I had to find a speedy way to query over 3 million records within a certain radius of a latitude longitude point. Mysql is slow for this sort of thing, so I decided to try the data set out with Solr. Solr worked great and was quite responsive, but required a massive amount of RAM on our server to hold the entire dataset. This was “ok” for our current dataset, but what would happen when we started to expand beyond NYC? or if we kept adding on new 311 reports from NYC as they come in over time? Solr was not scaleable. Mongodb to the rescue!

I wrote a simple import script in PHP to pull in the data we had already imported into mysql into mongodb. The only special part of the data mirgation was the creation of a “lnglat” field in each imported document array which is a “2d index” in mongo. You have to set a multivalued field with the lat/lng values cast as floats in the document array, like this:

$doc['lnglat'] = array('lng'=>(float)$row['lng'],'lat'=>(float)$row['lat']);

After the data has been imported to mongo, you create the 2d index on that multivalued field using the mongo command line, with this command:

my_mongodb.my_collection.ensureIndex( { lnglat : "2d" } )

Once I had the data in mongo, I had a hard time finding good examples of Geospatial queries for mongodb in php. I finally found this page:
http://comments.gmane.org/gmane.comp.db.mongodb.user/48789
which solved all my problems and pointed out that when querying mongo with a lat/lng point as an array, mongodb expects the lng to be first, not the lat. Durrrr…

Anyway, here is the code snippit that I pulled from the url above which was the first comprehensible example I found of php code for geospatial queries in with a distance radius specified:

public function getRadiusMiles($lng, $lat, $radius = 3) {
	$radiusOfEarth = 3956; //avg radius of earth in miles
	$cursor = $this->collection->find(
		array('lnglat' =>
			array('$within' =>
				array('$centerSphere' =>
					array(
						array(floatval($lng), floatval($lat)), $radius/$radiusOfEarth
					)
				)
			)
		)
	);
	$resultset = $this->jsonCursor($cursor);
	return json_encode($resultset);
}
 
public function getRadiusKm($lng, $lat, $radius = 5) {
	$radiusOfEarth = 6378.1; //avg radius of earth in km
	$cursor = $this->collection->find(
		array('lnglat' =>
			array('$within' =>
				array('$centerSphere' =>
					array(
						array(floatval($lng), floatval($lat)), $radius/$radiusOfEarth
					)
				)
			)
		)
	);
 
	$resultset = $this->jsonCursor($cursor);
	return json_encode($resultset);
}

Hopefully this saves someone some time!

mysqldump 5.1.36 unable to successfully dump a MySQL database in UTF-8 character encoding issues

james — Sat, 25 Feb 2012 01:53:44 +0000

Today I had a character encoding problem when importing some data from a mysqldump from a contractor we had hired to do some work. Everything about the dump was fine, the database, tables, and fields were all using utf8_unicode_ci utf8 collation, the dump file was encoded in utf8, and the database I was importing to was all utf8 as well. The contractor said he was able to re-import the dump into his server and local environment fine, but I was still unable to get the dump imported without all the utf8 characters getting garbled. As it turns out, his mysql and mysqldump versions were older than mine and it was messing everything up. Apparently, according to this mysql bug post:

http://bugs.mysql.com/bug.php?id=28969

“As of today (Feb 6, 2012) mysqldump 5.1.36 still exhibits the faulty behavior (unable to successfully dump a MySQL database in UTF-8)”. We were able to fix this issue by following these steps:

add latin1 default character set to the mysql dump command from the older versioned database, like so:
```
mysqldump -u dbuser -p --default-character-set=latin1 my_database_name > db_dump.sql
```
You can also try adding a “-r” flag to this command which should allow non-latin characters to dump properly as well, but this did not work in our case.
before importing into the newer version database, open the in the newly created dump file and change the line that reads:
```
/*!40101 SET NAMES latin1 */;
```
to:
```
/*!40101 SET NAMES utf8 */;
```
then save the file

import to the newer version database using:

mysql -u dbuser -p --default-character-set=utf8 my_database_name < db_dump.sql

Took a lot of emails back and forth to finally come to this solution. Hopefully this helps some one and thank god for the original bug post linked above!

disable firefox native inspect element context menu

james — Tue, 07 Feb 2012 18:26:19 +0000

With the latest update of Firefox to version 10.0, they included some new developer tools which are similar to what is available via Firebug or WebKit’s developer tools. About time! Unfortunately, I still am so used to firebug that will continue using it instead of these new native development tools. One thing that really grinds my gears was the native “Inspect Element” context menu item because it lives directly above the Firebug one. This was extremely frustrating for me because I kept clicking on that instead of the “Inspect with Firebug” menu item. This would bring up the native inspector console, which I would then have to close, then go back and right click again and make sure I click on the inspect with firebug menu item. Sigh…..how annoying.

Anyway, there is a simple way to disable the native inspect element context menu item. In the Firefox address bar, put “about:config” and press Enter. This will bring up Firefox’s preference configurations. In the “filter” search box enter “inspector”. This will bring up three configuration preferences. Simply double click on the “devtools.inspector.enabled” preference to change it to “false” and viola! No more native inspect element in the context menu! Horray.

Don’t muck around in here too much or you will mess up your firefox configuration. I hope someone finds this useful =)

php mysql collation change script

james — Wed, 04 Jan 2012 06:56:38 +0000

There are times when we get stuck with bad collations on our mysql tables and we are overwhelmed by having to change them by hand. Well, I have written a php script which will go through all your tables and fields in a database and update them to the collation which works for you. In my case for this example, I have a database which contains a mix of collations and changes them all to utf8_unicode_ci. This example uses the mysqli object in php as the database connection.

Your will need to update the database connection credentials as they are all bogus in this example. You will also need to change the “$execute_sql” to equal true if you want the script to actually perform the collation change via this script.


 
	$execute_sql = false;
 
	$host = 'mydbhost';
	$username = 'username';
	$password = 'passsword';
	$dbname = 'databasename';
 
	$db = new mysqli($host, $username, $password, $dbname);
 
	$mysqldatabase = 'databasename';
	$collation = 'CHARACTER SET utf8 COLLATE utf8_unicode_ci';
 
	echo '';
	$db->query("ALTER DATABASE $mysqldatabase $collation");
 
	$result = $db->query("SHOW TABLES");
 
	$count = 0;
	while($row = $result->fetch_assoc()) {
		$table = $row['Tables_in_'.$mysqldatabase];
		if($execute_sql) $db->query("ALTER TABLE $table DEFAULT $collation");
		$result1 = $db->query("SHOW COLUMNS FROM $table");
		$alter = '';
		while($row1 = $result1->fetch_assoc()) {
			if (preg_match('~char|text|enum|set~', $row1["Type"])) {
				if(strpos($row1["Field"], 'uuid')){
					// why does this not work
				}else{
					$alter .= (strlen($alter)?", \n":" ") . "MODIFY `$row1[Field]` $row1[Type] $collation" . ($row1["Null"] ? "" : " NOT NULL") . ($row1["Default"] && $row1["Default"] != "NULL" ? " DEFAULT '$row1[Default]'" : "");
				}
			}
		}
		if(strlen($alter)){
			$sql = "ALTER TABLE $table".$alter.";";
			echo "$sql\n\n
";
			if($execute_sql) $db->query($sql);
		}
		$count++;
	}
	echo '
';
 
?>

The only gotcha that I have run into with my script that I have yet to solve is that when you change the collation of an existing uuid field which has a foreign key, the copy table fails in mysql. Not that the collation of a uuid field really matters, but you can see a comment in the routine which questions why this does not work.

Anyway, I hope someone finds this useful =) Enjoy…

MySQL alter table to re-order columns

james — Sun, 02 Oct 2011 18:50:36 +0000

Sometimes because of my completely anal nature, I want the fields that I’ve already added to a table in MySQL to be in a different order. To do this you can simply alter that table column, without changing any of its properties, then adding “AFTER column_3″.

So for instance you had a table like this:

mysql> describe test;
+-----------+---------+------+-----+---------+----------------+
| Field     | Type    | Null | Key | Default | Extra          |
+-----------+---------+------+-----+---------+----------------+
| column_id | int(11) | NO   | PRI | NULL    | auto_increment |
| column_1  | int(11) | NO   |     | NULL    |                |
| column_2  | int(11) | NO   |     | NULL    |                |
| column_3  | int(11) | NO   |     | NULL    |                |
+-----------+---------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

But you really want column_1 to be ordered after column_3 because you are as ridiculous as I am. You can run this query:

ALTER TABLE `test` CHANGE `column_1` `column_1` INT( 11 ) NULL AFTER column_3;

Now your table looks like this:

mysql> describe test;
+-----------+---------+------+-----+---------+----------------+
| Field     | Type    | Null | Key | Default | Extra          |
+-----------+---------+------+-----+---------+----------------+
| column_id | int(11) | NO   | PRI | NULL    | auto_increment |
| column_2  | int(11) | NO   |     | NULL    |                |
| column_3  | int(11) | NO   |     | NULL    |                |
| column_1  | int(11) | YES  |     | NULL    |                |
+-----------+---------+------+-----+---------+----------------+
4 rows in set (0.00 sec)

I find this to be useful for me on larger tables that have grown over time and I want some relevant columns grouped together for when I am viewing data in phpMyAdmin or whatever.

Hope this helps someone =)

CSS3 dual background glow highlight button

james — Thu, 13 Jan 2011 00:27:47 +0000

I have been working on a new UI at work and decided to go full on with it in HTML5 and CSS3 because its 98% for internal company users who all use a “modern browser” and I don’t have to worry about those pesky IE users. The following code is for a cool looking glow highlight button which involves multiple backgrounds with rgba color. I have included three states: normal, hover, and active.

Enjoy!

php imagick complex gradient composite image

james — Sun, 20 Jun 2010 20:57:27 +0000

Manipulating images with imagemagick for php can be difficult, especially when you are trying to create complex images for a web template. Here is a more complex example of how to create sweet dynamic web graphics. This is a version of a header image that I created for a template design. It involves 4 colors, and adds a nice gradient sheen over the top of the base gradients.

The script first creates an new base image using the background color. Next, the script uses imagickdraw to create some lines and solid rectangles. Then the script creates another image which will fill in a gradient between the top two lines. Now comes the more complicated part, adding the sheen over the image that was just created.

The hard part of this is the fact we need an almost horizontal gradient. This involves creating two vertical gradients, reversing the color directions, then compositing them together. Once they have been combined, we rotate the image just beyond vertical, then set the transparency to 50%.

Now, the script puts the gradient image and sheen image together, and you have a sweet looking header image….


 
$background = "#FFFFFF";
$color1 = '#999999';
$color2 = '#333333';
$color3 = '#CCCCCC';
 
//$cachefile = 'path/to/cachefile.jpg';
//if (!file_exists($cachefile)) {
 
try{
 
	$im = new Imagick();
	$im->newImage( 400, 86, $background, "jpg" );
 
	$draw1 = new ImagickDraw();
	$draw1->setFillColor($color1);
	$draw1->rectangle(0, 0, 400, 3);
	$draw1->setFillColor($color1);
	$draw1->rectangle(0, 73, 400, 86);
 
	$draw1->setFillColor($color1);
	$draw1->line(0, 2, 400, 2);
	$draw1->setFillColor($color3);
	$draw1->line(0, 73, 400, 73);
 
	$gradient1 = new Imagick();
	$gradient1->newPseudoImage(400, 70, "gradient:".$color1."-".$color2);
	$im->compositeImage( $gradient1, imagick::COMPOSITE_OVER, 0, 3 );
 
	$im->drawImage($draw1);
 
 
	$gradient_over1 = new Imagick();
	$gradient_over1->newPseudoImage(400, 200, "gradient:transparent-".$background);
	$gradient_over2 = new Imagick();
	$gradient_over2->newPseudoImage(400, 200, "gradient:".$background."-transparent");
 
	$gradient_over = new Imagick();
	$gradient_over->newImage(400,400,$color2,'png');
	$gradient_over->compositeImage( $gradient_over1, imagick::COMPOSITE_OVER, 0, 0 );
	$gradient_over->compositeImage( $gradient_over2, imagick::COMPOSITE_OVER, 0, 200 );
	$gradient_over->rotateImage('transparent',120);
	$gradient_over->setImageOpacity(.5);
 
	$im->compositeImage($gradient_over, imagick::COMPOSITE_OVER, -100, -200 );	
 
	header( "Content-Type: image/jpg" );
	echo $im;
 
	$im->writeImage($cacheFilepath);
 
	$im->clear();
	$im->destroy();
 
}catch(Exception $e){
	echo 'Error: ',  $e->getMessage(), "";
}
 
/*
}else{
 
	$canvas = new Imagick($cachefile);
	header("Content-Type: image/png");
	echo $canvas;
 
}
*/
 
?>

As usual, I have commented out file caching code. Simply uncomment those lines and put a correct cachefile path (directory must be writable by the server).

simple but useful php base object for mysql table interaction

james — Sun, 20 Jun 2010 20:05:11 +0000

I started a project creating an online game which involved lots of objects and tables. Rather than rewrite the same code over and over, I decided to create a base object in which i could simple extend the base object to work with any mySQL table by simply listing all the fields.

Here is the code for the base object:


abstract class _base
{
 
	protected $db = null;
	protected $table = '';
	protected $primary_field = '';
	protected $primary_id = -1;
	protected $fields = array();
	protected $values = array();
 
	private $db_info = array(
		'host'=>'localhost',
		'username'=>'',
		'password'=>'',
		'database'=>''
	);
 
	public function __construct($primary_id=-1)
	{
 
		$this->db = $this->db_connect();
 
		if($this->primary_id>0){
			$this->primary_id = $primary_id;
			$this->load($this->primary_id);
		}
 
		return true;
	}
 
	public function load($primary_id=-1)
	{
		if($primary_id>0) $this->primary_id = $primary_id;
 
		if($this->primary_id<0) return false;
 
		$sql = "SELECT * FROM ".$this->table." WHERE ".$this->primary_field."='".intval($this->primary_id)."' LIMIT 1";
		$result = $this->db->query($sql);
		if($this->db->errno) echo 'Query Fail ('.$this->db->error.')
sql: '.$sql.'
';
 
		if($result && $result->num_rows>0){
			$tmp_vals = $result->fetch_assoc();
			$this->values = $tmp_vals;
			foreach($tmp_vals as $k=>$v){
				$this->$k = $v;
			}
			return true;
		}else{
			return false;
		}
	}
 
	public function load_from_array($load_array = array())
	{
		foreach($load_array as $field=>$value){
			$this->$field = $value;
		}
		return true;
	}
 
	public function save()
	{
		$primary_field = $this->primary_field;
 
		$save_new = true;
 
		if($this->$primary_field>0) $save_new = false;
 
		$this->prepare_save();
 
		if($save_new){
 
			$sql = "INSERT INTO ".$this->table;
			$fields = "(".implode(',',array_keys($this->values)).")";
			$values = "('".implode("','",array_values($this->values))."')";
			$sql .= ' '.$fields. ' VALUES '.$values; 
 
		}else{
 
			$sql = "UPDATE ".$this->table." SET ";
			$set = '';
			foreach ($this->values as $k => $v){
				$set.=(strlen($set)?', ':'').$k."='".$v."'";
			}
			$sql .= $set." WHERE ".$primary_field."=".intval($this->primary_id);
		}
 
		$result = $this->db->query($sql);
		if($this->db->errno) echo 'Query Fail ('.$this->db->error.')
sql: '.$sql.'
';
 
		if($result){
 
			if($save_new) $this->$primary_field = $this->db->insert_id;
 
			return true;
		}else{
			return false;
		}
	}
 
	protected function prepare_save()
	{
 
		$this->values = array();
		foreach($this->fields as $field){
			if(isset($this->$field)) $this->values[$field] = $this->db->real_escape_string($this->$field);
		}
		unset($this->values[$this->primary_field]);
	}
 
	protected function set_or_get($key='',$value=null)
	{
		if(!strlen($key)) return false;
 
		if(is_null($value) || (is_string($value) && !strlen($value)) || (is_int($value) && $value < 0)){
			return $this->$key;
		}else{
			$this->$key = $value;
			return true;
		}
	}
 
	public function set($key='',$value=null)
	{
		if(!strlen($key) || is_null($value)) return false;
		if(!in_array($key,$this->fields)) return false;
 
		$this->$key = $value;
 
	}
 
	public function get($key='')
	{
		if(!strlen($key)) return false;
		if(!in_array($key,$this->fields)) return false;
		return $this->$key;
	}
 
	public function object_info()
	{
		$object_info = array();
		$object_info['primary_field'] = $this->primary_field;
		$object_info['table'] = $this->table;
		$object_info['fields'] = $this->fields;
 
		return $object_info;
	}
 
	public function db_connect(){
		$this->db = new mysqli($this->db_info['host'],$this->db_info['username'],$this->db_info['password'],$this->db_info['database']);
		return $this->db;
	}
 
}
?>

To extend the object to use a certain mysql table, you simply set the table name, list the name of the fields, and the primary key. Then you ca access all the values of the mysql table with the variable name of the field names…. here is an example:

class user extends _base
{
	protected $primary_field = 'user_id';
	protected $table = 'users';
	protected $fields = array(
		'user_id ',
		'username',
		'password'
	);
 
	protected $user_id  = -1;
	protected $username  = '';
	protected $password  = '';
 
	public function __construct($user_id=-1)
	{
 
		parent::__construct();
 
		if($user_id>0){
			$this->user_id = $user_id;
			$this->load($user_id);
		}
 
		return true;
	}
 
	public function login($username='', $password='')
	{
		if(!strlen($username) || !strlen($password)) return false;
		$sql = "SELECT user_id FROM users WHERE username='".$this->db->real_escape_string($username)."' AND password='".$this->db->real_escape_string($password)."'";
		$result = $this->db->query($sql);
		if($this->db->errno) echo 'Query Fail ('.$this->db->error.')
sql: '.$sql.'
';
		if($result && $result->num_rows > 0){
			$user_data = $result->fetch_assoc();
			$this->load($user_data['user_id']);
 
			// do user login stuff
 
			return true;
		}else{
			return false;
		}
	}
 
}