SQLitePersistentObjects Presentation

I just noticed this thread over at iPhoneDevSDK, where one of the organizers of the 360 iDev Conference states that they don't mind if presenters post their conference presentations.

Therefore, once the conference is over, I will be posting my Keynote presentation and the speaker notes that go along with my SQLitePersistentObjects presentation I'm giving. Hopefully that will help fill in the gap until we get the documentation all squared away.

Another SQLite Library

I just learned of another library for using SQLite 3 from the iPhone called iPhoneLite3. It's still a pretty young project (classified as "alpha"), and takes a different approach than I did in sqlitepersistentobjects, but looks to be a project worth keeping an eye on.

Updated SQLitePersistence

As I think I mentioned earlier, Marcel Molina is doing some great work adding Ruby ActiveRecord style associations and also refactoring some of my code to make it much cleaner and neater.

Today, I did a little surgery of my own, albeit minor outpatient surgery. I added a bunch of methods to the database instance manager to allow greater control over database configuration, allowing you to specify such things as the autovacuum level, cache size, and the like.

Even if you don't have an interest in using SQLitePersistentObjects, if you use SQLite for either iPhone or Mac/Cocoa applications, it's probably worth your time to take a look at the SQLiteInstanceManager class in the project. It stands on its own and can be used without the rest of the package and it makes managing your database a heck of a lot easier. Just drop it into your project, grab a reference to the sharedManager, and start working with it - no need to create or open the database, or specify a file location or anything.

And, when you realize how nice it is not to have to write code to do any of that, maybe you'll become interested in checking out the rest of SQLitePersistentObjects.

Note: Anybody out there who has successfully used SQLitePersistentObjects for a delivered application, please let me know, and also let me know of any problems you encountered along the way.

SQLite Persistent Object Update

Last night, I fixed a bunch of bugs in the SQLite Persistent Objects code. I also wrote an iPhone application that uses it to save its data, and checked the project source code into Subversion as well. I plan to write a basic tutorial around this application, but wanted to make it available for anyone who wanted to see it in action. If you check out the project from Subversion, you should get a new folder called Sample Code. The Xcode project is in there; the links from the project back to the SQLite Persistent Objct code is project-relative, so if you put the Sample Code project anywhere else but in your sqlitepersistentobjects folder, you're going to have to relink those files.

I would appreciate it if someone could try checking it out and compiling it to make sure it works. The program uses the camera, but it can be run on the simulator,the "Take Picture" just won'tt work. If you want to run the sample code on the device, you'll have to change the bundle identifier, developer identity, and developer cert, of course.

Wh00t!

Our tech reviewer just finished the last chapter, so now the chapters are off to the formatter and then the copy editor. It's amazing how involved the commercial publishing process is. Having only written for magazines and online publishers before, this is a whole new ball o' wax for me. That's not a bad thing, but it does give me a new appreciation for just how many people contribute to those books you see on the shelves of the book store. There may be one or two names on the cover, but likely at least eight people (and possibly many more) were actively involved in shaping the content of the book, not to mention the layout, production, printing and all the other tasks, which likely drives the number up into the dozens.

If anyone's interested in becoming a committer on the SQLite Persistent Objects project, drop me a line at jeff underscore lamarche at mac dot com.

SQLitePersistentObjects now in Google Code Repository

Check it out - you can always get the latest and greatest from here:

http://code.google.com/p/sqlitepersistentobjects/

There aren't any substantive changes since the initial release - I'm starting to get chapters back now fast and furious and have to get as much done as I can before I leave on vacation. Anyone who is interested in contributing to the project, just drop me a line.

SQLite Persistent Objects

Wouldn't it be nice if your Objective-C data objects just knew how to save and load themselves? It'd be nice if you could just call "save" and trust that your object would save itself properly somewhere, and that when you wanted to load it back in, you could just call a class method to retrieve the object or objects you wanted?

Well, now you can. I've been talking about this project for a while, but it got put on hold for the book. I have now finished the first draft of the code. This is very much a beta release, but I am interested in feedback from people, so am releasing it. In the next release, I will provide more sample code and unit tests, and other niceties, but for this release, you just get the source code files and a little bit of information about how to use them.

What does it do?

It lets you create Objective-C classes that know how to persist themselves into a SQLite database. Not only that, but it completely hides the implementation details from you - you do not need to create the database, create the tables, or do anything else except work with your ojbects. you simply subclass SQLitePersistentObject and create Objective-C 2.0 properties for every data element you want persisted. When you create an instance of this object and send it the save message, it will get saved into the database.

How does it work

Every subclass of SQLitePersistentObject gets its own table in the database. Every property that's not a collection class (NSDictionary, NSArray, NSSet or mutable variants) will get persisted into a column in the database. Properties that are pointers to other objects that are also subclasses of SQLitePersistentObject will get stored as a reference to the right row in that object's corresponding table. Collection classes gets stored as child tables, and are capable of storing either a foreign key reference (when the object they hold is a subclass of SQLitePersistentObject) or in a field on the child table.

Can all properties be stored?

No. But most can. This currently does not support properties that are c-strings, void pointers, structs, or unions. All scalars (ints, floats, etc) get saved into appropriate fields. When it comes to Cocoa objects, any class that conforms to NSCoding can be stored in a column. It is also possible to provide support for specific classes by adding a category on the class you wish to support that tells the system how to store and retrieve that object from a column's data. There are provided categories for NSDate, NSString, NSData, NSMutableData, NSNumber, and (of course) NSObject. Creating new ones to let other objects be persisted is relatively easy - just look at one of the included categories and implement the same methods. The methods are documented in NSObject-SQLitePersistence.h.

Classes that don't have direct support (the ones listed above or any that you add), will use NSObject's persistence mechanism, which archives the object into a BLOB using an NSKeyedArchiver. This is inefficient for some objects because you can't search or compare on these fields, but at least most object can be persisted. Some classes like NSImage, this method actually works quite well and there's probably no reason to add a specific category.

How do I use it?

Add all the files from the zip file to your project, link in libsqlite3.dylib. Then, declare data objects like this:

#import <foundation/foundation.h>
#import "SQLitePersistentObject.h"

@interface PersistablePerson : SQLitePersistentObject {
 NSString *lastName;
 NSString *firstName;
}
@property (nonatomic, retain) NSString * lastName;
@property (nonatomic, retain) NSString * firstName;
@end

Once you've done that, you can just create your objects as usual:

PersistablePerson *person = [[PersistablePerson alloc] init];
person.firstName = @"Joe";
person.lastName = @"Smith";

Now, you can save it to the database (which will be created if necessary) like so:

[person save];

Loading it back in is almost as easy. Any persistable object gets dynamic class methods added to it to allow you to search. So, we could retrieve all the PersistablePerson objects that had a last name of "Smith" like so:

NSArray *people = [PersistablePerson findByLastName:@"Smith"]

Or, you could specify the exact criteria like this:

PeristablePerson *joeSmith = [PersistablePerson findFirstByCriteria:@"WHERE last_name = 'Smith' AND first_name = 'Joe'];

Notice that the camel case word divisions (marked by capital letters) got changed into an underscore, so if you want to use findByCriteria: or findFirstByCriteria: you have to make sure you get the column names correct. I plan on adding additional dynamic class methods to allow searching based on multiple criteria, but for now, if you want to search on more than one field you have to manually specify the criteria. Don't worry, it's not hard.

Can I create Indexes?

Yep. Just need to override a class method

+(NSArray *)indices;

You should return an NSArray of NSArrays. Each array contained in the returned array represents one index, and should contain the name of the properties to build the index on. Use the property names - although, in some cases, the names are changed, this method should return the actual property names, not the database column names. Here is an example implementation of indices

+(NSArray *)indices
{
 NSArray *index1 = [NSArray arrayWithObject:@"lastName"];
 NSArray *index2 = [NSArray arrayWithObjects:@"lastName", @"firstName", nil];
 NSArray *index3 = [NSArray arrayWithObjects:@"age", @"lastName", @"firstName", nil];
 return [NSArray arrayWithObjects:index1, index2, index3, nil];
}

Assuming the strings passed back all describe valid properties, the table that holds this class' data will have three indices. Easy enough, right?

What happens if I load the same object in multiple times?

These classes maintain a map of all persistable objects in memory. If you load one that's already in memory, it returns a reference to that instead of going back to the database. If you need to be able to make a copy of the object, you'll have to implement NSCopying, as SQLitePersistableObject does not currently conform to NSCopying, though it probably will before I consider this "done".

How is this Licensed

It's a very liberal license. You can use it however you want, without attribution, in any software commercial or otherwise. You are not required to contribute back your changes (although they are certainly welcome), nor are you required to distribute your changed version. The only restriction that I place on this is that if you distribute the source code, modified or unmodified, that you leave my original comments, copyright notice, and contact information, and ask that you comment your changes.

Is it fully functional?

Close to it. As I said earlier, I want to add more robust methods for finding objects. I also haven't implemented rollback. Because this maintains a memory map, it's actually difficult right now to go back to the way an object was when it was last saved because you have to make sure that every object that has a reference to that object releases it, make sure that it is gone, then re-load it from the database. Rollback is high on my priority list. Because this is an early release, there are bound to be bugs and I'm sure you can come up with enhancements that could make it better.

Does it use private or undocumented libraries?

No. It does not. It does make extensive use of the Objective-C runtime, but that is fully documented and open to developers. There is also nothing in here covered by the NDA. It was developed under Cocoa for OS X, but since it only uses foundation objects, it should, in theory, work just as well on the iPhone. Because of the NDA, I'm not actually going to say it works on the iPhone... I can neither confirm or deny that it works on said platform. But it should.

So, Where is it Already?

You can get it right here.

You should check out the file SQLitePersistentObject.h, which contains fairly extensive documentation in headerdoc format.

Dynamically adding class objects

Okay, folks, the first draft of the book is done, and I have some breathing room until the chapters start coming back from the tech reviewer and copy editor. Mostly, I'm using that time to catch up on sleep and house jobs and spend some time with the kids, but I've also been working on an idea I mentioned earlier about creating a way for NSObjects to automatically persist themselves into a SQLite database. I just finished my first draft of the code to do that, and once I've done some more thorough testing, I'll do a blog post about it.

Today, I'm going to post one of the little gotchas that bit me while writing that code, and which is really, really hard to figure out from Apple's documentation: how to dynamically add a class method at runtime. What I did in my persistence classes was to create class methods to help you retrieve objects. So, if you wanted to retrieve all the objects in the database where the name property equaled "John", you could do this

NSArray *johns = [Person findByName:@"John"];

Anybody who has worked with Ruby's excellent ActiveRecord implementation will recognize this pattern as the one used there. The way to implement this is to override resolveClassMethod: (you would use resolveInstanceMethod: to do the same thing for instance methods, but that is documented well and easy to implement so I won't discuss it here).

The resolveClassMethod: documentation refers you to resolveInstanceMethod: for an example implementation, which looks like this:

+ (BOOL) resolveInstanceMethod:(SEL)aSEL
{
  if (aSEL == @selector(resolveThisMethodDynamically))
  {
        class_addMethod([self class], aSEL, (IMP) dynamicMethodIMP, "v@:");
        return YES;
  }
  return [super resolveInstanceMethod:aSel];
}

If you use code like that in your resolveClassMethod:, however, you'll get an unrecognized selector error at runtime. Why? Because that code adds a method to the class object, which means you added an instance method to that class. That's not what you want. What you want is to add a method to your class' metaclass object instance, which means going to the runtime to get the Class object that represents the class' metaclass object (say that five times fast!). So, the example above, when turned into a class method override, would look like this:

+ (BOOL) resolveClassMethod:(SEL)aSEL
{
  if (aSEL == @selector(resolveThisMethodDynamically))
  {
        Class selfMetaClass = objc_getMetaClass([[self className] UTF8String]);
        class_addMethod([selfMetaClass, aSEL, (IMP) dynamicMethodIMP, "v@:");
        return YES;
  }
  return [super resolveClassMethod aSEL];
}

And, in the immortal words of Bugs Bunny: Viola! You've added a class method to your class at runtime. Of course, you have to create the function dynamicMethodIMP in order for this to work and do all the other assorted steps necessary to make dynamic methods work, but this isn't a tutorial on dynamic method creation, just a solution to one gotcha you may encounter while doing it.

I am alive...

Sorry for the delay... a bunch of stuff that I had put on the back burner all came to a boil at the same time, and I haven't had a lot of time for anything other than work of late. I still plan to get the SQLite persistence class a little more stable and then release it into the wild, but I wouldn't feel right putting it out in its current state.

19 days until WWDC, hope to see you there!

SQLite Adventures

With the release of OS X 10.4 (Tiger), Apple gave us developers something pretty cool called "Core Data". Core Data is an ORM tool based (at least in concept, if not in actual code) on Enterprise Objects Framework (or EOF) which was NeXT's revolutionary tool for persisting object-oriented data to a database: revolutionary, that is, back in 1993.

Now, I don't mean to slight EOF or Core Data in that statement. Both are cool tools, and are vastly superior to embedding SQL calls to a procedural API in your object-oriented code. In 1993, strongly-typed languages ruled the desktop development roost and "web development" was yet to even really exist as a defined form of programming since, for all practical purposes, 1993 was the year the web was born¹.

 C and C++ were the indisputed champions of desktop application development and the arrival of Java two years later would cement the dominance of strongly-typed languages for quite some time. In 1993, weakly-typed languages had been around for a while but, for a number of reasons, were largely considered ill-suited to writing "serious" applications. NeXTStep's use of Objective-C was definitely an odd-man out in terms of desktop application development.

Things have changed quite a lot since 1993, though. Computers have gotten faster, dynamic languages like PERL, Ruby, and Python have become increasingly popular, and even most of the statically typed languages — including C++ — have gained some level of runtime introspection capability.

Probably one of the coolest approaches to object-relational mapping is the ActiveRecord design pattern. A good ActiveRecord implementation requires a dynamic language with robust type introspection which allows the object to dynamically add variables and methods to itself based on the contents of a database table.

But ActiveRecord, as cool as it is, is not ideally suited for desktop development with an embedded database because it is database table-driven. What would be ideal for embedded work is a sort-of "reverse ActiveRecord" where tables are created based on the existing attributes of objects. So, if you added a string variable to your object, it would know to add a varchar column to the table and would also know to save the contents of the variable in that column.  If there is a name for this approach, please let me know it; I've never heard one, but I suspect one exists.

Anyway, I do have a point here, and it has to do with the iPhone and SQLite. I realized yesterday while writing an article proposal on SQLite that Objective-C's dynamic nature would allow this theoretical "reverse ActiveRecord" idea to be implemented and, in fact, that it could probably be encapsulated almost entirely in a single abstract super class. Well, not technically abstract, because Objective-C doesn't support abstract classes, but that's neither here nor there.

Yesterday, I wrote a proof-of-concept version of just such a class. It has methods to create tables, insert, update, and delete data from the database as well as rollback to the last values stored in the database. It's capable of using foreign key relationships for one-to-one associations between objects and cross-reference tables for one-to-many associations represented in Objective-C by the use of arrays and dictionaries.

My code is still very rough around the edges and is definitely not production ready, but I think the concept is sound and with Core Data not being available on the iPhone, I think it's a great approach to avoid having to write tons of custom embedded SQL for each object. In fact, all you do is create a subclass, add instance variables and property declarations and you're done. You don't actually have to write any SQL to handle the persistence and there's even a mechanism to specify indices for your object's table.

I will clean up my code a little bit and put it out there for comments and feedback in the next day or so.

1 - Yes, I know that, in theory the web dates back to 1989 and in implementation dates back to 1990, but CERN announced on April 30, 1993 that the World Wide Web would be free for everyone to use, and that date (in my mind) marks the real beginning of the web as we know it.

Problems Reading and Writing Files on the iPhone?

If you're having problems reading or writing data using the beta iPhone SDK, you should head over to the iPhone Dev Site at Apple, and download the SQLiteBooks sample code they just posted. There's a workaround in the App Delegate of that project for the problem. 

I can't give details because of the NDA, but if you're trying to do any file reads or writes, or to use SQLite, you should check out the new sample code to find out how to get those working.

Brain Surgery?

Craig Hockenberry has an interesting post on his blog today about the iPhone background processing issue. Craig speaks from personal experience on this matter and I'd classify his post as a must-read  for anyone who has an opinion on the background processing issue in the official iPhone SDK.

Unfortunately, his posting comes off as a bit smug and superior for my personal tastes. Though I have tremendous respect for Craig's ability, this posting comes across as a "shut up you naughty children, you're not as smart as me and Apple so you can't have an opinion on this". It is definitely true that he has first-hand experience that most of us don't, but since the decision he's justifying is one that prevents us from gaining the very experience he's claiming we need in order to have an informed opinion (without jailbreaking our phones, at least), I'm finding the argument a bit circular for my taste.

To work with Craig's own metaphor: Don't cut off our hands to keep us from shooting ourselves in the foot. Some of us idiot developers who are apparently not smart enough to have an opinion on this, might just surprise you by finding a way to work around the problems you encountered. Or, just maybe, our applications have different background processing needs than the one application you've tried this with. Since Apple has set themselves up as gatekeeper to the App Store, and since no self-respecting developer is going to ship a product without testing it extensively on a real iPhone first, I don't see the need for such drastic exclusions from the SDK itself. They could control this by subjecting apps that use background processing to greater scrutiny, or providing a sanctioned mechanism that can enforce limitations, not by making those parts of the iPhone completely verboten.

The iPhone and the Enterprise

With Apple's simultaneous announcement of the iPhone SDK and the licensing of the protocol needed to work with Microsoft Exchange servers, it does seem like Apple is finally getting serious about pushing into the Enterprise. I've worked a lot in this realm, and have long felt that Apple's development tools - especially Cocoa — which started life as NextSTEP — was especially well suited to doing the type of custom development that I've seen such a desperate need for in the Enterprise. Though most people aren't aware of it, the tools underlying the iPhone SDK as well as those commonly used for doing software development for Mac OS X have long-since been proven in the Enterprise environment.

I've spent spent close to a decade working in Enterprise software, first as a developer at PeopleSoft, and afterwards as an implementation consultant. During most of this time, I have wished there was some way to convince all these big, predominately Windows-based shops that there was something better available. But "Apple" is a dirty word in most large corporate IT departments, and Apple has never really seemed interested in challenging that. Old ideas persist long after they are no longer supported by facts. The fact that OS X is a solid Posix-compliant Unix which runs Microsoft Office, supports Windows printing and file sharing, and has virtually no impact from viruses or malware doesn't seem to hold much sway even when it is known. It is sheep mentality at its worse, but there it is. 

Large corporations are not known for encouraging individuality, and no decision is ever made by a single person at most large companies. I think Heinlein's description of decision by committee is spot on: He described a committee as a beast with multiple heads and no brain. Any committee is less likely to reach the right conclusion than any individual member is. The whole is considerably less than the sum of its parts in this context.

Most of places I've done work for also balk at even hearing the term "custom software"; bizarrely enough, they are frequently willing to spend literally millions of their corporation's dollars to implement so called "Enterprise Software" - software to handle some set of tasks less efficiently than what they could have had by hiring a couple good Cocoa developers for a month. These projects often fail, and rarely do they come in on-time or under-budget, even when the schedule and budget are luxurious. Despite that, suggesting to a large American corporation that they develop in-house solutions in Cocoa would be openly laughed at. "It's not Windows" has been a sufficient argument against that idea for well over a decade now.

In the short term, I do not see the situation changing, but if the iPhone sees any serious penetration into the Enterprise market (and it certainly seems poised to), it could expose a lot of people to just how awesome Apple's development tools are. Shops that want to develop custom iPhone apps will have to buy some Macs, and will have to hire or train Mac-savvy developers. Like a benevolent virus, a love for Cocoa tends to be infectious, and some hands-on experience with modern Macs might just disabuse some of the Corporate IT drones of their outdates ideas about just what a Mac is. 

I've worked on projects where handheld clients to ERP applications were being implemented. Though I've never been directly involved with the mobile implementation work, I have been involved enough to see that the offerings tend to be expensive, not particularly intuitive or easy to use, and often require you to hire the vendor's consultants if you've customized your system at all (and everybody customizes their system). In my experience, I can't recall having seen a mobile client to an ERP application whose functionality couldn't be replicated in a couple of days for the iPhone.

I would conservatively guess that a company willing to standardize on the iPhone OS and do custom development for their mobile needs could reduce the cost of their mobile implementation in half. Savings of 90% or more are absolutely not out of the question.

We do live in interesting times.

Still Playing

I started working with the audio toolbox on the iPhone today. This morning, I added sounds to the little dice rolling application I've been writing. I recorded a total of fifteen sounds, five of a single die being rolled, five of two dice being rolled, and five of four dice being rolled. I let my kids roll the dice while I recorded the sounds, which they thought was cool, although they can't understand why I can't put the program on my iPhone. 

Do you hear that, Steve Jobs? You are making my kids sad. Do you really want that on your conscience?  ;) 

Unfortunately, the "Coding How-Tos" on the iPhone web site don't give a full code sample on how to play audio files. They show the command to play a sound, but not how to find and load the file from your application bundle (it may be in the how-tos somewhere, but it wasn't under the section on playing audio). Figuring that out took a little poking around in the documentation and header files (and, of course, a couple of application crashes), but I got it working without too much trouble. 

Boy, it's been a while since I've worked with pointers... I spent way too long working almost exclusively with Java. I don't want to bash Java - I know a lot of people like it, and it's been the basis for a lot of my income over the years - but it just feels so much less elegant than objective-C. 

Since I wrote my first AppleScript Basic program in 1980, I have used literally dozens of different programming and scripting languages, many extensively and as part of my professional life, and I can say that Objective-C and Cocoa (and now Cocoa Touch) are, by far and without a doubt my favorite. They take a little while to understand when you first start using them, but once you get over the "grokking curve" and really start to understand how it all pieces together, coding in objective-C really becomes a pleasurable experience. It's just plain fun. 

In a way, I envy all those people coming to the iPhone from other languages and toolset. They are about to discover something very, very cool if they have the patience to get past the initial obstacles.