-- the statement below is almost the same as
-- select lpad(mycolumn,'0',10) from mytable

select substr('0000000000' || mycolumn, -10, 10) from mytable

-- the statement below is almost the same as
-- select rpad(mycolumn,'0',10) from mytable

select substr(mycolumn || '0000000000', 1, 10) from mytable

The statement is fairly self-explanatory, but in case it doesn’t make sense we’re simply adding a big long string of characters to the original value and then truncating it down to the desired length.  The string used for concatenation has to be at least the same length that you are padding – 10 characters is used in this example (’0000000000′).

In most cases this workaround produces the same results as lpad/rpad except in the case where the length of your original value is greater than the length that you are padding.  In which case the original value would get truncated.  Usually when you are padding you know what the maximum length of the column anyway.

If you know of a more efficient technique, please post a comment.

Josh’s instructions include everything that I need except mcrypt.  Luckily Michael Gracie has provided a walk-through for installing mcrypt on Snow Leopard which involves re-compiling some things, but isn’t as tough as it first appears.

After getting PHP and mcrypt going, the last step for me is setting up MySQL which has some caveats on OSX.  The main problem is that the MySQL installer places the mysql.sock file in a non-standard place and so you have to either create a link or alter your php.ini file before PHP will be able to talk to MySQL.  (If you get “Can’t connect to local MySQL server through socket ‘/var/mysql/mysql.sock’” or  “No such file or directory” when calling mysql_connect, then this is the problem)

After completing the steps on Josh’s & Michael’s pages and running the MySQL installation package, here are the additional steps that I like to take to get my local environment configured:

Edit /private/etc/php.ini

display_errors = On
mysql.default_socket = /tmp/mysql.sock
pdo_mysql.default_socket=/tmp/mysql.sock
date.timezone = ‘America/Chicago’

Edit /private/etc/apache2/httpd.conf

DocumentRoot “/Users/jason/Sites”

Edit /private/etc/apache2/users/jason.conf

Append FollowSymLinks to the Options

I also like to change the permissions on php.ini and httpd.conf to allow my user account edit permissions and create an alias to those two files somewhere easy to get to.

Installing MAMP is probably easier, but for some reason I prefer using all of the services that are pre-bundled with OSX.  Also Snow Leopard includes PHP 5.3 which led me to find a couple of deprecated functions in some of our libraries.

Solution 1: Create a symbolic link

Open terminal and do the following:

sudo su
mkdir /var/mysql
ln -s /tmp/mysql.sock /var/mysql/mysql.sock

You just created a symbolic link in the place where PHP expects the socket file to be located so it should be happy.

Solution 2: Edit php.ini

If you don’t like the idea of creating a symbolic link, you can also simply alter your php.ini file to point PHP to the real location of mysql.sock.

Locate /etc/php.ini. (If php.ini doesn’t exist on your system, copy /etc/php.ini.default to /etc/php.ini).  You will likely have to do this from the terminal unless you have Finder configured to show hidden files.  Open the file and update the setting mysql.default_socket so it looks like this:

mysql.default_socket = /tmp/mysql.sock

To commit the change you need to restart Apache.  You can do that in System Settings -> Sharing, then  uncheck, then recheck Web Sharing.

Working with SQLite and Flex/AIR Date values can be tricky and various caveats are not particularly well documented. The confusion (for me) is that ActionScript is loaded with UTC functions, and SQLite will happily insert them into DATETIME columns. Everything appears fine, however SQLite does not actually recognize this format as a Date and treats it as plain text. You have no way to see this happened until you try to apply some date formatting functions and notice SQLite returning NULL. SQLite is so lax about data integrity that you can insert anything into any column type and will never receive any warnings. AIR, though, will attempt to cast values behind the scenes based on column types and so you will run into ‘Invalid Date’ errors and weird glitches when attempting to update data.

The magic solution is the Julian Date Format which both SQLite and AIR recognize as a date value. This is somewhat surprising as ActionScript has no built-in support for outputting Julian dates. If you’re like me, you may have already hacked up workarounds using int fields with timestamps, however your matching ActionScript class properties have to be hacked to match, and the hacking can trickle down throughout your code. This also prevents you from using the SQLStatement.itemClass functionality, which is nice when using Cairngorm, DAOs, value objects, etc.

To avoid the pain follow these rules when working with dates:

1. If you want a strongly typed Date field in AIR, the relevant SQLite column must be defined as DATETIME.  The interesting thing about this is that DATETIME is not technically a recognized SQLite column type and according to the SQLite docs it will be considered numeric.  But AIR is obviously looking at the column definition somewhere in the framework because it will refuse to automatically cast any value where the column type is not DATETIME.

2. Whenever inserting or updating DATETIME fields, you must store it in Julian format (or NULL).  SQLite will happily accept many common date formats. However AIR will behave inconsistently. Here is how to insert Julian dates in a variety of ways:

Inserting a Julian date manually via SQL:

To do this, simply have SQLite format your date value to Julian format using ‘%J’

UPDATE my_table SET my_column = STRFTIME('%J','2008-01-02 03:04:05')

Inserting a Julian date via AIR (with parameters):

Parameters are the best way to build SQL statements as you can use strongly typed Date variables and AIR will deal with the formatting for you.

statement.text = "UPDATE my_table SET my_column = :my_value";
statement.parameters[":my_value"] = new Date(2008,0,2,3,4,5); // Jan 02, 2008 03:04:05

Inserting a Julian date via AIR (without parameters):

If you are not using parameters, you have to pre-format the date into something that SQLite can parse. This is surprisingly obnoxious and requires you to write a couple of helper functions. (Note – if you know of an easier way to do this, please post a comment.)

public function lpad(original:Object, length:int, pad:String):String
{
var padded:String = original == null ? "" : original.toString();
while (padded.length < length) padded = pad + padded;
return padded;
}

public function toSqlDate(dateVal:Date):String
{
return dateVal == null ? null : dateVal.fullYear
+ "-" + lpad(dateVal.month + 1,2,'0')  // month is zero-based
+ "-" + lpad(dateVal.date,2,'0')
+ " " + lpad(dateVal.hours,2,'0')
+ ":" + lpad(dateVal.minutes,2,'0')
+ ":" + lpad(dateVal.seconds,2,'0')
;
}

var myDate:Date = new Date(2008,0,2,3,4,5); // Jan 02, 2008 03:04:05
statement.text = "UPDATE my_table SET my_column = strftime('%J','" + toSqlDate(myDate) + "')";

Fudging data to work around AIRs validation

If you absolutely refuse to change your schema (for example you insist on using timestamps, or you have to maintain compatibility with other clients) you can get AIR to play along during READ operations by altering your select statement like so:

SELECT STRFTIME('%J',my_column) as my_column from my_table

This does assume that the data is in a format that SQLite recognizes as a date. If SQLite can’t parse the date value, then it will just return NULL. For hilarity sake, you can also use this ridiculous date format which surprisingly works with AIR. A word of warning about this workaround is that, even though you will be able to read data, you may not be able to update data via SQLCommand parameters if your column types are DATETIME because AIR will complain about an invalid date (see errors below). You will have either have to write your own SQL statements without parameters or else change your column types to int or varchar.

Formatting a Julian date manually in SQL so you can read it:

Julian values are great and all that, but it’s pretty much impossible to eyeball them when you’re working at the command line. SQLite recognizes Julian formatting as a valid date, so you can use the STRFTIME function to format and output it any way you like. Below is a simple example that is easier to read:

SELECT STRFTIME('%Y-%m-%d %H:%M:%S',my_column) as my_column_formatted FROM my_table

Common errors that occur while working with dates:

Invalid Date

You may see this in a DataGrid instead of the expected date value. This is because you have a DATETIME column in SQLite, however the value is not in Julian format. Even though SQLite may recognize it as a date value, AIR does not. The solution is to clean your data so that all dates are Julian format, or alternatively change the column type to VARCHAR.

‘Error #3115: SQL Error.’, details:’could not convert string value to date’

This error occurs when you try to update a record that has one or more DATETIME columns that do not have the date stored in Julian format. The weird part is that even if you are not touching that specific column in your insert/update statement – AIR will still validate the Date and throw this error. The solution is to clean your data so that all dates are Julian format, or alternatively change the column type to VARCHAR.

If you have any tips or corrections please post a comment and I’ll incorporate it into the article.

SELECT
   CASE WHEN (  strftime('%m',my_column)   = '01') THEN ('Jan') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '02') THEN ('Feb') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '03') THEN ('Mar') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '04') THEN ('Apr') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '05') THEN ('May') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '06') THEN ('Jun') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '07') THEN ('Jul') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '08') THEN ('Aug') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '09') THEN ('Sep') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '10') THEN ('Oct') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '11') THEN ('Nov') ELSE '' END
|| CASE WHEN (  strftime('%m',my_column)   = '12') THEN ('Dec') ELSE '' END
|| STRFTIME(' %d %H:%M:%S GMT-0600 %Y', my_column)
FROM my_table

There is a catch with unsigned ints however that can create problems for you if you’re not careful. That is, when you subtract unsigned ints, you will always get a positive value back no matter what. What does that mean?

Say you have a table for products with two unsigned fields: qty_in_stock and qty_sold. Lets assume for whatever reason you oversold your inventory and qty_in_stock now equals 100, but qty_sold = 101. So take the following statement:

SELECT qty_in_stock - qty_sold as qty_remaining FROM products

You would expect qty_remaining to be -1, right? Wrong! In fact MySQL returns 18446744073709551615 !

Why would this happen? Well, because MySQL is built so that when you perform a math operation on two unsigned ints, it will always return an unsigned int. And -1 is technically an out-of-bounds value for an unsigned field type. Instead of complaining, MySQL returns the value that would have represented -1 if it were unsigned which is 18446744073709551615.

Mathematically, I suppose this is correct, however it doesn’t make any logical sense to me. If anything, I think MySQL should throw an out-of-bounds error when this happens because the default behavior seems very unintuitive to me. It is as though they are trying to say that one positive value subtracted from another can never result in a negative value. Of course this is false, but that is how MySQL works.

There are two solutions to this. The first is to simply use signed ints if you ever expect them to be used in a subtraction formula. You waste some space, but it’s an easy fix.

The second solution is, if you are able to edit your SQL statement that have the subtraction, you can use the MySQL cast function. The example from above could be converted to the following:

SELECT cast(qty_in_stock - qty_sold as signed) as qty_remaining FROM products

So there is a solution to this “problem” if you will. You can read more about this subject on the MySQL site at http://dev.mysql.com/doc/refman/4.1/en/cast-functions.html

For a recent assignment we needed to populate a table that reflects seating in a stadium. The seating was typical of most venues consisting of Aisles and Seats. For example, A-1, A-2, A-3, B-1, B-2, B-3, and so on. When the table was initially populated, the seat_number field was all set at it’s default value of ’1′. So we needed sequential numbers for the seats, but the count had to start over with each new Aisle. This could have been done easily in a scripting language, but what fun is that! We wanted to accomplish the numbering in one regular SQL statement.

This problem would be easily solved using Oracle’s rownum functionwhere we can get a sequential id for each row within a resultset. But, alas, MySQL doesn’t have a rownum function. What we did find, however, is a wonderful hack on Mark Malakonov’s blog that duplicates this functionality for MySQL.

The final query, which only updates Aisle ‘A’ is here:

update venue_seat set seat_number =
(select @rownum:=@rownum+1 rownum FROM (SELECT @rownum:=0) r)
where aisle = 'A'

These fields obviously relate to a specific schema, however the @rownum trick can be used to generate sequential numbers for your tables as well. If you need to update or populate a table with sequential numbers but need some additional logic, this is a handy trick to have in your toolbox.

If you get this error while trying to create a foreign key, it can be pretty frustrating. The error about not being able to create a .frm file seems like it would be some kind of OS file permission error or something but this is not the case. This error has been reported as a bug on the MySQL developer list for ages, but it is actually just a misleading error message.

In every case this is due to something about the relationship that MySQL doesn’t like. Unfortunately it doesn’t specify what the exact issue is. Here is a running list of causes that people have reported for the dreaded errno 150. I’ve tried to put them in order based on the frequency that I hear about a particular cause.

You may want to start by running the MySQL command “SHOW INNODB STATUS” immediately after receiving the error. This command displays log info and error details. (Thanks Jonathan for the tip)

Note: If your script runs fine on one server, but gives an error when you try to run it on a different server, then there is a good chance that #6 is the problem.  Different versions of MySQL have different default charset setting.

Known Causes:

1. The two key fields type and/or size doesn’t match exactly. For example, if one is INT(10) the key field needs to be INT(10) as well and not INT(11) or TINYINT. You may want to confirm the field size using SHOW CREATE TABLE because Query Browser will sometimes visually show just INTEGER for both INT(10) and INT(11). You should also check that one is not SIGNED and the other is UNSIGNED. They both need to be exactly the same. (More about signed vs unsigned here).
2. One of the key field that you are trying to reference does not have an index and/or is not a primary key. If one of the fields in the relationship is not a primary key, you must create an index for that field. (thanks to Venkatesh and Erichero and Terminally Incoherent for this tip)
3. The foreign key name is a duplicate of an already existing key. Check that the name of your foreign key is unique within your database. Just add a few random characters to the end of your key name to test for this. (Thanks to Niels for this tip)
4. One or both of your tables is a MyISAM table. In order to use foreign keys, the tables must both be InnoDB. (Actually, if both tables are MyISAM then you won’t get an error message – it just won’t create the key.) In Query Browser, you can specify the table type.
5. You have specified a cascade ON DELETE SET NULL, but the relevant key field is set to NOT NULL.  You can fix this by either changing your cascade or setting the field to allow NULL values. (Thanks to Sammy and J Jammin)
6. Make sure that the Charset and Collate options are the same both at the table level as well as individual field level for the key columns. (Thanks to FRR for this tip)
7. You have a default value (ie default=0) on your foreign key column (Thanks to Omar for the tip)
8. One of the fields in the relationship is part of a combination (composite) key and does not have it’s own individual index. Even though the field has an index as part of the composite key, you must create a separate index for only that key field in order to use it in a constraint. (Thanks to Alex for this tip)
9. You have a syntax error in your ALTER statement or you have mistyped one of the field names in the relationship (Thanks to Christian & Mateo for the tip)
10. The name of your foreign key exceeds the max length of 64 chars.  (Thanks to Nyleta for the tip)

The MySQL documentation includes a page explaining requirements for foreign keys. Though they don’t specifically indicate it, these are all potential causes of errno 150. If you still haven’t solved your problem you may want to check there for deeper technical explainations.If you run into this error and find that it’s caused by something else, please leave a comment and I’ll add it to the list.

These tips are not necessarily MySQL specific, but I’m using MySQL syntax as an example for the sake of simplicity.

SIGNED vs UNSIGNED

This is probably the easiest habit to practice because you have nothing to lose and everything to gain by using SIGNED /UNSIGNED data types in the right place. There’s a simple question you can ask – will this field ever contain a negative value? If the answer is no, then you want an UNSIGNED data type.

A common mistake that I see is a primary key that is an auto-increment INT starting at zero, yet the type is SIGNED. An UNSIGNED INT can go from 0 to 4294967295 (aprox 4 billion). A SIGNED INT , on the other hand, starts at -2147483648 and goes to 2147483648 (aprox 2 billion). If you use an auto-increment starting at 0, you’ll never touch any of those negative numbers and your Id column will top out at around 2 billion.

Use a Smaller ID Data Type for Small Tables

You don’t usually know how many records your application will eventually store. You don’t want to use too small a data type in the event that your app gets popular. You might tend to use INT for your primary key column just out of habit.

In some situations, though, you do have a pretty good idea of how many records you’ll have and so you don’t need to use an INT as the primary key. For tables that contain things like permissions, system settings, account types, etc you may know that you will never need a huge number of records. You may not even create a user interface to access these tables or insert new records.

In cases like this, I find an UNSIGNED TINYINT is perfect. This will hold values from 0 to 255. Sometimes 255 is more than enough, but still gives you room to grow. You don’t want to do this if you expect to get close to this limit of course. If the data is strictly internal and there’s no user interface, the developer is usually in control of these records.

A lot of people use UNSIGNED INT as their primary key which as I mentioned earlier goes up to over 4 billion records. That is a lot of records! An UNSIGNED MEDIUMINT goes up to 16 million records which is still more than most tables need. An UNSIGNED SMALLINT goes up to 65 thousand records. You don’t want to run out of space, but you can try to optimize a bit for the best performance. Keep in mind that your primary key is the field that will be theoretically used the most for record lookups, so the more efficient you can make it the faster your app will run.

Read more about numerical datatypes on the MySQL site…

Know the Difference Between CHAR and VARCHAR

CHAR and VARCHAR both hold string/text data. However they work a little differently because CHAR is like a fixed-size array that is always the same size, where as VARCHAR will expand and contract to fit the data.

The CHAR type is useful if you have a good idea of the size that the field will be. A product code that is always 10 digits, for example, is a great candidate for a CHAR data type. The negative side of the CHAR type is that it will use up the disk space whether or not you need it. For example, if you have a CHAR(25) but your record only uses up 5 characters, MySQL will still use up all 25 on disk.

The VARCHAR is convenient because you can set it to 25, but if you only store 5 characters, the VARCHAR will only use the space it needs. Two tradeoffs for this convenience are performance and fragmentation. In particular, when you update a VARCHAR field, you might outgrow the space that was originally allocated. Your data becomes fragmented so that one record may be spread out across the DB file instead of in one nice, continuous chunk. So as time goes by, your access to the VARCHAR data will require more processing power.

MySQL provides a solution in the way of the OPTIMIZE TABLE statement. This basically will de-fragment your table VARCHAR fields. If you are using VARCHARS and doing a lot of updating, you should make sure to optimize the tables periodically. This is something that can be automated through cron or Scheduled Tasks so that you don’t forget.

In case you think it’s a simple choice, keep in mind that table size and performance are closely related. So, it follows that the VARCHAR would deliver performance benefits due to it being the smallest possible size. That would offset the performance benefit that the consistency of CHAR delivers. My rule of thumb is to use CHAR if you have a pretty good idea of the field size.

More reading is available on the MySQL site, however I find the user comments on the end of this page more interesting than the official documentation.

Don’t mix “SELECT * FROM” with TEXT/BLOB fields

TEXT and BLOB fields are extremently useful for storing large datatypes. File uploads, XML data, long text content, etc usually require one of these types of fields. Unfortunately, these fields can make a table grow large quickly. Storing binary file data for example can make a DB table enormous.

Depending on your application, this may just be the best way to store this type of data. However, if you don’t need this data on every query, then don’t select it. Here’s an example:

select * from file_upload where fu_owner_id = ’1′

What we’re doing here is grabbing the full row contents for the file uploads, though we may not need all of that data. You can imagine that this query might select all of the files that have been uploaded by a specific user for us to show in a datagrid, dropdown list or whatever. The issue here is that you rarely need to have the *contents* of the files when you are displaying them in a list. If possible, this might be better:

select fu_id, fu_name from file_upload where fu_owner_id = ’1′

In this case we have the primary key (fu_id) and the file name (fu_name). We can show this in the list Now, if the user wants to download or see the contents of the file, you can grab it using the primary key:

select fu_name, fu_data from file_upload where fu_id = ’55′

An alternative approach is to use a dedicated table with only a primary key and the BLOB/TEXT field. Any other tables that need to store large values can simply have a reference to that table’s primary key instead. If you only do lookups from the dedicated table by it’s primary key, then you’ll avoid any full table scans.

Read The Fine Manual

The MySQL site has plenty of information describing the various data types. The little bit of time and effort that you put into tuning your database will pay off as your application grows.

More reading on MySQL 5 data types can be found here: http://dev.mysql.com/doc/refman/5.0/en/data-types.html

To transfer these, you have to use the “Transfer Objects” and change a few of the defaults. If you just accept all the defaults, you won’t get any of your keys on the copied database.

1. Right click on the source database and from All Tasks click “Export”

2. Choose the source DB and click next

3. Choose the destination DB and click next

NEXT STEPS ARE IMPORTANT SO KEYS AND IDENTITIES IMPORT CORRECTLY – PAY ATTENTION!!

4. Choose the option to “Copy Objects and Data between SQL Server Databases”

5. Check the box that says “Include Extended Properties” (I haven’t confirmed, but I think this is what copies over the Identity property)

6. If you don’t want all the tables, de-select “Copy all Objects” and click the “Select Objects…” button to choose the tables to copy.

7. De-select “Use Default Options” and click the “Options…” button. On the options screen, de-select “Copy database users and database roles” unless you have admin access to both servers and you want the same usernames to be used on both servers. If you get errors that the user does not exist, or you get duplicate tables with different username (schemas) then this is the cause.

8. Click next and run immediately or schedule. The data migration wizard should run. Check your new tables to make sure they have keys and identities correct.

Another thing to pay attention is that you can have duplicate tables like so:

[dbo].[Table1]
[username].[Table1]

This can cause all kinds of grief because they are treated as totally different tables and you will see only the one for which you are logged in. So, if you login to to SQL server as sa, you’ll see both tables. Your web app might only see the user-specific one. Pay attention to step 7 above to prevent this situation.