First, type about:config in the address bar, search for gecko.handlerService.allowRegisterFromDifferentHost, and set the value to true.

Then, log into GMail or your Google Apps mail URL, paste this line of JavaScript into your address bar, and hit enter:
For Gmail:
javascript:window.navigator.registerProtocolHandler("mailto","https://mail.google.com/mail/?extsrc=mailto&url=%s","GMail")

For Google Apps:
javascript:window.navigator.registerProtocolHandler("mailto","https://mail.google.com/a/example.com/mail/?extsrc=mailto&url=%s","GMail")
Replace “example.com” with your domain.

After you hit enter, Firefox should ask permission to add Gmail as the default mailto handler.

To undo this, click Edit Preferences > Applications. Search for mailto. Choose Application Details from dropdown under Actions. Highlight Gmail and click Remove.

I used these instructions on Firefox 3.0.1 on Ubuntu 8.10. This post was compiled from info found in this article and its comments at lifehacker.com.