Cocoa Programming in OpenMCL


Table of Contents

Overview.
General.
Some steps in the right direction.
Objective-C and CLOS
Defining Objective-C classes
Instantiating ObjC classes
Using DEFMETHOD to define Objective-C methods
"first class" instances.
Saved images and the new scheme
The application kit and native threads.
Writing (and reading) Cocoa code
@class [Macro]
@selector [Macro]
define-objc-method [Macro]
define-objc-class-method [Macro]
Method-definition syntax examples.
Coercion of method arguments and result types.
Method redefinition constraints.

Overview.

For the last year or so, OpenMCL source distributions have been released with examples that showed that it was possible to access the Cocoa application framework via OpenMCL's foreign function interface. On the one hand, these examples seemed to show off the power of Cocoa: it was apparently possible to create a working (if rudimentary) lisp development environment with a relatively small amount of lisp code. On the other hand, the examples seemed to demonstrate that a lot of infrastructure was missing:

  • Cocoa's runtime system assumes that all threads are preemptively scheduled (“native threads”); subtle and bizarre bugs can arise when this assumption is violated.

  • Cocoa's exception-handling system and lisp's condition system provide similar functionality but were not integrated with (or even aware of) each other. A typo in an Objective-C message name was enough to terminate the application (with no opportunity for a lisp handler to intervene); a lisp error during Cocoa event handling generally meant that Cocoa event handling was suspended until the resulting break loop was dismissed.

  • The examples provided a thin layer (in the form of reader macros and other constructs) to make programming in terms of ObjC message sends a little easier than it might otherwise be, but it was a pretty thin layer on top of the FFI and not very “lispy”; some people may have mistakenly thought that the bulk of the example code was written in ObjC and not lisp. Some people were able to build on this layer and do interesting things, but I think that it's fair to say that programming at this level was a tedious and error-prone process.

General.

This document assumes some general familarity with Cocoa and with the Objective C programming language and its runtime conventions.

OpenMCL is ordinarily a command-line application (it doesn't have a connection to the OSX Window server, doesn't have its own menubar or dock icon, etc.) By opening some libraries and jumping through some hoops, it's able to to sort of transform itself into a full-fledged GUI application (while retaining its original TTY-based listener.) The general idea is that this hybrid environment can be used to test and protoype UI ideas and the resulting application can eventually be fully transformed into a bundled, double-clickable application. This is to some degree possible, but there needs to be a bit more infrastructure in place before many people would find it easy.

As it was originally built, the demo Cocoa IDE used .nib files created by Apple's Interface Builder to describe the layout and structure of its windows and menus. Integration with IB is certainly a good thing (IB is very good at what it does), but it's certainly possible to create and modify windows and menus dynamically. There should ideally be some (more) examples that show how to do this in OpenMCL.

Cocoa application use the NSLog function to write informational/warning/error messages to the application's standard output stream. A GUI application's standard output is diverted to a logging facility that can be monitored with the Console application (found in /Applications/Utilities/Console.app); in the hybrid environment, the application's standard output stream is usually the initial listener's standard output stream. With two different buffered stream mechanisms trying to write to the same underlying Unix file descriptor, it's not uncommon to see NSLog output mixed with lisp output on the initial listener.

Some steps in the right direction.

OpenMCL 0.14's native threads make it practical to address some aspects of the first two of these concerns; Randall Beer has developed and contributed a “Cocoa Bridge”, which provides many features to make Cocoa programming in OpenMCL more robust and convenient. The bridge is documented in greater detail in “ccl:examples;CocoaBridgeDoc.txt”; the OpenMCL Cocoa examples have been rewritten to use the features offered by the bridge.

Objective-C and CLOS

Randall Beer and I have been working on the integration of Objective-C and CLOS; if you're also interested in working on this, please let me know.

As of early February, 2004, OpenMCL's CLOS implementation has been extended to recognize that certain MACPTRs can also be "standard objects" (objects to which SLOT-VALUE can meaningfully be applied) and that certain of these foreign objects are classes (that can be subclassed and instantiated.)

The class of most "standard" CLOS classes is the class named STANDARD-CLASS. In the Objective-C object model, each class is an instance of a (usually unique) metaclass, which is itself an instance of a "base" metaclass (often the metaclass of the class named "NSObject".) So, the Objective-C class named "NSWindow" and the ObjC class "NSArray" are (sole) instances of their distinct metaclasses whose names are also "NSWindow" and "NSArray", respectively. (In the Objective-C world, it's much more common and useful to specialize class behavior such as instance allocation.)

When foreign libraries containing Objective-C classes are first loaded, the classes they contain are identified. The foreign class name (e.g., "NSWindow") is mapped to an external symbol in the "NS" package via the bridge's translation rules (yielding a result such as NS:NS-WINDOW); a similar transformation happens to the metaclass name with a "+" prepended, yielding something like NS:+NS-WINDOW. These classes are integrated into CLOS such that the metaclass is an instance of the class OBJC:OBJC-METACLASS and the class is an instance of the metaclass. SLOT-DESCRIPTION metaobjects are created for each instance variable, and the class and metaclass go through something very similar to the "standard" CLOS class initialization protocol (with a difference being that these classes have already been allocated.) This process currently takes several seconds; it could conceivably be sped up some, but it's never likely to be fast.

When the process is complete, CLOS is aware of several hundred new ObjC classes and their metaclasses. OpenMCL's runtime system can reliably recognize MACPTRs to ObjC classes as being CLASS objects, and can (fairly reliably but heuristically) recognize instances of those classes (though there are complicating factors here; see below.) SLOT-VALUE can be used to access (and, with care, set) instance variables in ObjC instances. To see this, do:

? (require "COCOA")

and, after waiting a bit longer for a Cocoa listener window to appear, activate that Cocoa listener and do:

? (describe (ccl::send ccl::*NSApp* 'key-window))  ; the "keyWindow" is the window that has input focus

As we can see, NSWindows (ahem. I meant NS:NS-WINDOWs) have lots of interesting slots.

Defining Objective-C classes

We can use the MOP to define new Objective-C classes (or subclasses of existing ones), but we have to do something a little funny: the :METACLASS that we'd want to use in a DEFCLASS option generally doesn't exist until we've created the class (recall that ObjC classes have, for the sake of argument, unique and private metaclasses.) We can sort of sleaze our way around this by specifying a known ObjC metaclass object name as the value of the DEFCLASS :METACLASS object; the metaclass of the root class NS:NS-OBJECT (NS:+NS-OBJECT) makes a good choice. To make a subclass of NS:NS-WINDOW (that, for simplicity's sake, doesn't define any new slots), we could do:

(defclass example-window (ns:ns-window)
  ()
  (:metaclass ns:+ns-object))

That'll create a new ObjC class named EXAMPLE-WINDOW whose metaclass is the class named +EXAMPLE-WINDOW. The class will be an object of type OBJC:OBJC-CLASS, and the metaclass will be of type OBJC:OBJC-METACLASS.

Defining classes with foreign slots

If a slot specifcation in an Objective-C class definition contains the keyword :FOREIGN-TYPE, the slot will be a "foreign slot" (i.e. an ObjC instance variable).[1]

The value of the :FOREIGN-TYPE initarg should be a foreign type specifier. For example, if we wanted (for some reason) to define a subclass of NS:NS-WINDOW that kept track of the number of key events it had received (and needed an instance variable to keep that information in), we could say:

(defclass key-event-counting-window (ns:ns-window)
  ((key-event-count :foreign-type :init :initform 0 :accessor window-key-event-count))
  (:metaclass ns:+ns-object))

Foreign slots are always SLOT-BOUNDP, and AFAIK the initform above is redundant: foreign slots are initialized to binary 0.

Defining classes with Lisp slots

A slot specifcation in an ObjC class definition that doesn't contain the :FOREIGN-TYPE initarg defines a pretty-much normal lisp slot that'll happen to be associated with "an instance of a foreign class". For instance:

(defclass hemlock-buffer-string (ns:ns-string)
  ((hemlock-buffer :type hi::hemlock-buffer :initform hi::%make-hemlock-buffer :accessor string-hemlock-buffer))
  (:metaclass ns:+ns-object))

As one might expect, this has memory-management implications: we have to maintain an association between a MACPTR and a set of lisp objects (its slots) as long as the ObjC instance exists, and we have to ensure that the ObjC instance exists (does not have its -dealloc method called) while lisp is trying to think of it as a first-class object that can't be "deallocated" while it's still possible to reference it. Associating one or more lisp objects with a foreign instance is something that's often very useful; if you were to do this "by hand", you'd have to face many of the same memory-management issues.

Instantiating ObjC classes

Making an instance of an ObjC class (whether the class in question is predefined or defined by the application) involves calling MAKE-INSTANCE with the class and a set of initargs as arguments. As with STANDARD-CLASS, making an instance involves initializing (with INITIALIZE-INSTANCE) an object allocated with ALLOCATE-INSTANCE.

Allocating an instance of an ObjC class involves sending the class an "alloc" message, and then using those initargs that -don't- correspond to slot initags to identify and send an "init" message to the newly-allocated instance.[2]

An ObjC initialization method may return a NULL pointer or may return something other than what was allocated. ALLOCATE-INSTANCE returns NIL if the "init" message returned a null pointer and returns what the "init" message returned otherwise. If ALLOCATE-INSTANCE returns a non-NIL pointer, INITIALIZE-INSTANCE is called on that object and that object is returned, otherwise NIL is returned. (Note that this means that it's possible for MAKE-INSTANCE to return NIL.)

Using DEFMETHOD to define Objective-C methods

[At this point, it's not yet clear whether DEFMETHOD can be used to define ObjC methods or what extensions to its syntax might be necessary to support this. DEFMETHOD can't yet be used to define "foreign" methods (things that can take the place of methods implemented in ObjC), and the older DEFINE-OBJC-METHOD macro (described below) must be used for that purpose.]

"first class" instances.

In most cases, pointers to instances of Objective-C classes are recognized as such; the recognition is (and probably always will be) slightly heuristic. Basically, any pointer that passes basic sanity checks and whose first word is a pointer to a known ObjC class is considered to be an instance of that class; the Objective-C runtime system would reach the same conclusion.

It's certainly possible that a random pointer to an arbitrary memory address could look enough like an ObjC instance to fool the lisp runtime system, and it's possible that pointers could have their contents change so that something that had either been a true ObjC instance (or had looked a lot like one) is changed (possibly by virtue of having been deallocated.)

In the first case, we can improve the heuristics substantially: we can make stronger assertions that a particular pointer is really "of type :ID" when it's a parameter to a function declared to take such a pointer as an argument or a similarly declare function result; we can be more confident of something we obtained via SLOT-VALUE of a slot defined to be of type :ID than if we just dug a pointer out of memory somewhere.

The second case is a little more subtle: ObjC memory management is based on a reference-counting scheme, and it's possible for an object to ... cease to be an object while lisp is still referencing it. If we don't want to deal with this possibility (and we don't), we'll basically have to ensure that the object is not deallocating while lisp is still thinking of it as a first-class object. There's some support for this in the case of objects created with MAKE-INSTANCE, but we may need to give similar treatment to foreign objects that are introduced to the lisp runtime in other ways (as function arguments, return values, SLOT-VALUE results, etc. as well as those instances that're created under lisp control.)

This doesn't all work yet (in fact, not much of it works yet); in practice, this has not yet been as much of a problem as anticipated, but that may be because existing Cocoa code deals primarily with relatively long-lived objects such as windows, views, menus, etc.

Saved images and the new scheme

When an image which had contained ObjC classes (which are also CLOS classes) is re-launched, those classes are "revived": all preexisting classes have their addresses updated destructively, so that existing subclass/superclass/metaclass relationships are maintained. It is not yet possible to run an image that contains a different set of foreign classes than the Cocoa libraries contain (in practice, this means that the image can only run in the exact same OS version that it was saved from.) Handling the more general case is certainly desirable, but it's also more complicated.

It's not possible (and may never be) to preserve foreign instances across SAVE-APPLICATION. (It may be the case that NSArchiver and NSCoder and related classes offer some approximation of that.)

The application kit and native threads.

AppKit (the parts of Cocoa that deal with events, window management, drawing, and other aspects) really wants all event handling, GUI object creation, and drawing to take place on a distinguished thread. Apple has published some guidelines that discuss these issues in some detail (see here, for instance) but there can sometimes be unexpected behavior when objects are created in threads other than the distinguished event thread (e.g., the event thread sometimes starts performing operations on objects that aren't fully initialized.) It's certainly more convenient to do certain types of exploratory programming by typing things into a listener or evaluating a “defun” in an Emacs buffer; it may sometimes be necessary to be aware of this issue while doing so.

Each thread in the Cocoa runtime system is expected to maintain a current “autorelease pool” (an instance of the NSAutoreleasePool class); newly created objects are often added to the current autorelease pool (via the -autorelease method), and periodically the current autorelease pool is sent a “-release” message, which causes it to send “-release” messages to all of the objects that've been added to it.

If the current thread doesn't have a current autorelease pool, the attempt to autorelease any object will result in a severe-looking warning being written via NSLog. The event thread maintains an autorelease pool (it releases the current pool after each event is processed and creates a new one for the next event), so code that only runs in that thread should never provoke any of these severe-looking NSLog messages.

To try to suppress these message (and still participate in the Cocoa memory management scheme), each listener thread (the initial listener and any created via the “New Listener” command in the IDE) is given a default autorelease pool; there are REPL colon-commands for manipulating the current listener's “toplevel auturelease pool”.

In the current scheme, every time that Cocoa calls lisp code, a lisp error handler is established which maps any lisp conditions to ObjC exceptions and arranges that this exception is raised when the callback to lisp returns. Whenever lisp code invokes a Cocoa method, it does so with an ObjC exception handler in place; this handler maps ObjC exceptions to lisp conditions and signals those conditions.

Any unhandled lisp error or ObjC exception that occurs during the execution of the distinguished event thread's event loop causes a message to be NSLog'ed and the event loop to (try to) continue execution. Any error that occurs in other threads is handled at the point of the outermost Cocoa method invocation. (Note that the error is not necessarily “handled” in the dynamic context in which it occurs.)

Both of these behaviors could possibly be improved; both of them seem to be substantial improvements over previous behaviors (where, for instance, a misspelled message name typically terminated the application.)

Writing (and reading) Cocoa code

The syntax of the constructs used to define Cocoa classes and methods has changed a bit (it was never documented outside of the source code and never too well documented at all), largely as the result of functionality offered by Randall Beer's bridge; the “standard name-mapping conventions” referenced below are described in his CocoaBridgeDoc.txt file, as are the constructs used to invoke (“send messages to”) Cocoa methods.

All of the symbols described below are currently internal to the CCL package.

@class [Macro]

Syntax

@class class-name

Description

Used to refer to a known ObjC class by name. (Via the use LOAD-TIME-VALUE, the results of a class-name -> class lookup are cached.)

Arguments

 

class-name

a string which denotes an existing class name, or a symbol which can be mapped to such a string via the standard name-mapping conventions for class names

@selector [Macro]

Syntax

@selector string

Description

Used to refer to an ObjC method selector (method name). Uses LOAD-TIME-VALUE to cache the result of a string -> selector lookup.

Arguments

 

string

a string constant, used to canonically refer to an ObjC method selector

define-objc-method [Macro]

Syntax

define-objc-method (selector class-name) &body body

Description

Defines an ObjC-callable method which implements the specified message selector for instances of the existing ObjC class class-name.

Arguments

 

selector

either a string which represents the name of the selector or a list which describ+es the method's return type, selector components, and argument types (see below.) If the first form is used, then the first form in the body must be a list which describes the selector's argument types and return value type, as per DEFCALLBACK.

class-name

either a string which names an existing ObjC class name or a list symbol which can map to such a string via the standard name-mapping conventions for class names. (Note that the "canonical" lisp class name is such a symbol)

define-objc-class-method [Macro]

Syntax

define-objc-class-method (selector class-name) &body body

Description

Like DEFINE-OBJC-METHOD, only used to define methods on the class named by class-name and on its subclasses.

Arguments

As per DEFINE-OBJC-METHOD

For both DEFINE-OBJC-METHOD and DEFINE-OBJC-CLASS-METHOD, the “selector” argument can be a list whose first element is a foreign type specifier for the method's return value type and whose subsequent elements are either:

  • a non-keyword symbol, which can be mapped to a selector string for a parameterless method according to the standard name-mapping conventions for method selectors.

  • a list of alternating keywords and variable/type specifiers, where the set of keywords can be mapped to a selector string for a parameteriezed method according to the standard name-mapping conventions for method selectors and each variable/type-specifier is either a variable name (denoting a value of type :ID) or a list whose CAR is a variable name and whose CADR is the corresponding argument's foreign type specifier.

Method-definition syntax examples.

The following method definitions are equivalent.

(define-objc-method ("applicationShouldTerminate:"
                     "LispApplicationDelegate")
   (:id sender :<BOOL>)
   (declare (ignore sender))
   nil)
;;;
(define-objc-method ((:<BOOL> 
                      :application-should-terminate sender)
                     lisp-application-delegate)
   (declare (ignore sender))
   nil)
  

Coercion of method arguments and result types.

The method definition macros coerce method arguments of foreign type :<BOOL> to T/NIL (this is done via an assignment at the beginning of the body.) If the method's declared result type is :<BOOL>, a non-nil, non-0 value returned by the body is coerced to 1 (#$YES) and NIL or 0 is returned as 0 (#$NO).

Method redefinition constraints.

Objective C methods can be redefined at runtime, but their signatures (the foreign result type and the types of foreign arguments) shouldn't change. If a method shadows a method defined in a superclass, it should have the same type signature as the superclass's method (though it's legal for methods defined in disjoint classes to have different signatures.)

Note that adding or removing arguments has the effect of changing the selector used to name and invoke the method.



[1] It is an error to redefine an ObjC class so that its foreign slots change in any way. I'm not sure if or how well we check for this yet,

[2] For example, the non-slot initargs in (:with-foo foo :and-bar bar) would cause the message "initWithFoo:andBar:" to be sent to the instance)