v

Wrapping C++ classes in Lua

Lua is a script language frequently used in game development. Typically the game engine is written in C/C++ and the game itself is defined in Lua. Meaning setting up behaviour, placement of objects like powerups, doors, enemies etc is done in Lua. That way the game can be quickly tweaked without having to go through a time consuming re-compile.



But I am not going to go in depth about the different usage of Lua, nor explain the language itself. An online book at the Lua web page does that very well already. I will here assume you already have some basic knowledge about Lua.



However finding any documentation on how to wrap a C++ class so that it can be used in Lua is difficult. One could of course use one of the ready made bridges. But here we are going to look at how to do it yourself.



The tricky part is deciding on how to do it. Because Lua is such a flexible language there is really a lot of ways you could achieve it.



The naive approach


First lets look at several options, what works and doesn't. My first approach was to light user data to store a pointer to my C++ class. I will use a Sprite class and related classes as examples here as that was the type of classes I was wrapping.



static int newSprite(lua_State *L)

{

    int n = lua_gettop(L);  // Number of arguments

    if (n != 4)

        return luaL_error(L, "Got %d arguments expected 4", n);

  

    double x = luaL_checknumber (L, 1);      

    double y = luaL_checknumber (L, 2);

    double dir = luaL_checknumber (L, 3);      

    double speed = luaL_checknumber (L, 4);



    Sprite *s = new Sprite(Point2(x, y), dir, speed);

    

    lua_pushlightuserdata(L, s);



  return 1;

}


The code snippet above shows a naive implementation of this approach. Unfortunately it doesn't work. The problem is that a light user data is just a simple pointer. Lua does not store any information with it. For instance a metatable which we could use to define the methods the class supports.



An approach with limited functionality


The next approach would be to use user data. Unlike light user data it can store a reference to a metatable.



static int newSprite(lua_State *L)

{

    int n = lua_gettop(L);  // Number of arguments

    if (n != 4)

        return luaL_error(L, "Got %d arguments expected 4", n);

  

    // Allocate memory for a pointer to to object

    Sprite **s = (Sprite **)lua_newuserdata(L, sizeof(Sprite *));  



    double x = luaL_checknumber (L, 1);      

    double y = luaL_checknumber (L, 2);

    double dir = luaL_checknumber (L, 3);      

    double speed = luaL_checknumber (L, 4);



    *s = new Sprite(Point2(x, y), dir, speed);

    

    lua_getglobal(L, "Sprite"); // Use global table 'Sprite' as metatable

    lua_setmetatable(L, -2);       

    

  return 1;

}


For us to be able to use sprite like this we need to register class first. Basically we need to create a table Sprite which contains all the methods that our user data should support.


// Show registration of class

static const luaL_Reg gSpriteFuncs[] = {

  // Creation

  {"new", newSprite},

  {"position", position},

  {"nextPosition", nextPosition},    

  {"setPosition", setPosition},  

  {"render", render},      

  {"update", update},          

  {"collision", collision},   

  {"move", move},    

  {"accelerate", accelerate},      

  {"rotate", rotate},  

  {NULL, NULL}

};



void registerSprite(lua_State *L)

{

  luaL_register(L, "Sprite", gSpriteFuncs);  

  lua_pushvalue(L,-1);

  lua_setfield(L, -2, "__index");    

}


This will allow us to create instances of Sprite and call methods on it in Lua like this:


-- Create an instance of sprite an call some methods

local sprite = Sprite.new(x, y, dir, speed)

sprite:setPosition(x,y)

sprite:render()



The final approach


In most cases this approach is sufficient but it has one major limitation. It does not support inheritance. You can change the methods of Sprite in Lua but that will change the behavior of all instances of Sprite. What you would want to do is to be able to change method on just the instance and then use that instance as a prototype for new Sprite instances, effectively creating a class inheritance system.


To do this we need to change the instance into being a table. How do we access our C++ object then? Simple, we just store the pointer to it as user data in one of the field of the table. You might think that this time light user data will be sufficient. However the problem is that only user data is informed of garbage collection, not tables or light user data. So if you want to delete the C++ object when corresponding lua table is garbage collected you need to use user data.


So we then arrive at our final solution. We will store a pointer to our C++ object as user data on the key __self in the table that represents our instance. __self is an arbitrary selected name. It could be anything. We will not register Sprite as as the metatable for our instance but instead register the first argument to the new function as it. This will allow us to support inheritance. Further the garbage collection function will be register on a separate table which will be used as metatable only for the user data. This is to allow it to be garbage collected.


static int newSprite(lua_State *L)

{

    int n = lua_gettop(L);  // Number of arguments

    if (n != 5)

        return luaL_error(L, "Got %d arguments expected 5 (class, x, y, dir, speed)", n);

    // First argument is now a table that represent the class to instantiate        

    luaL_checktype(L, 1, LUA_TTABLE);   

    

    lua_newtable(L);      // Create table to represent instance



    // Set first argument of new to metatable of instance

    lua_pushvalue(L,1);       

    lua_setmetatable(L, -2);



    // Do function lookups in metatable

    lua_pushvalue(L,1);

    lua_setfield(L, 1, "__index");  



    // Allocate memory for a pointer to to object

    Sprite **s = (Sprite **)lua_newuserdata(L, sizeof(Sprite *));  



    double x = luaL_checknumber (L, 2);      

    double y = luaL_checknumber (L, 3);

    double dir = luaL_checknumber (L, 4);      

    double speed = luaL_checknumber (L, 5);



    *s = new Sprite(Point2(x, y), dir, speed);

    

    // Get metatable 'Lusion.Sprite' store in the registry

    luaL_getmetatable(L, "Lusion.Sprite");



    // Set user data for Sprite to use this metatable

    lua_setmetatable(L, -2);       

    

    // Set field '__self' of instance table to the sprite user data

    lua_setfield(L, -2, "__self");  

    

  return 1;

}


We can now work with sprite instances in Lua like this:


-- Instantiate from Sprite class

local sprite = Sprite:new(x, y, dir, speed)

sprite:setPosition(x,y)

sprite:render()



-- Add method to instance as use it as class

function sprite:doSomething()

  print("do something")

end



local derived = sprite:new(x, y, dir, speed)

derived:render()

derived:doSomething() -- This is now a legal operation

There are still a couple of loose ends. We haven't showed how the methods are registered with this new solution nor how we access C++ object pointer in methods. But this is fairly straight forward as I will show.


void registerSprite(lua_State *L)

{  

  // Register metatable for user data in registry

  luaL_newmetatable(L, "Lusion.Sprite");

  luaL_register(L, 0, gDestroySpriteFuncs);      

  luaL_register(L, 0, gSpriteFuncs);      

  lua_pushvalue(L,-1);

  lua_setfield(L,-2, "__index");  

  

  // Register the base class for instances of Sprite

  luaL_register(L, "Sprite", gSpriteFuncs);  

}


We can then implement a method of Sprite like this


static int setSpeed(lua_State *L)

{

  int n = lua_gettop(L);  // Number of arguments

  

  if (n == 2) {

    Sprite* sprite = checkSprite(L);

    assert(sprite != 0);

    real speed = luaL_checknumber (L, 2);

    sprite->setSpeed(speed);

  }

  else

    luaL_error(L, "Got %d arguments expected 2 (self, speed)", n);

  

  return 0;

}


To extract the pointer to the C++ object and make sure it is of the correct type we use the following code:


Sprite* checkSprite(lua_State* L, int index)

{

  void* ud = 0;

  luaL_checktype(L, index, LUA_TTABLE);

  lua_getfield(L, index, "__self");

  ud = luaL_checkudata(L, index, "Lusion.Sprite");

  luaL_argcheck(L, ud != 0, "`Lusion.Sprite' expected");  

  

  return *((Sprite**)ud);      

}


The only thing left is dealing with garbage collection but I leave that as an exercise. You should already have the basic idea of how to deal with it. Please note that I have not tested the exact same code as written here so why the principles are correct there might be minor errors in the code. In my own code I have separated more of the code into separate functions since the code for creating an instance is almost identical for any class, as well as the code for extracting the __self pointer.



Conclusion


While using a bridge might be better for bigger projects I think it is good to know for yourself exactly what goes on under the hood and when you do it yourself you can more easily fine tune what you export and not and in which way. Typically you would want the Lua interface to be simpler and more limited than the C++ interface to your classes.

Intersection test assertion failure in CGAL

I have just started using the CGAL library. It is a library for doing computational geometry. It contains classes and functions for different geometric data structures like triangles, line segments, polygons and functions for testing for intersection, calculating convex hull and a lot more.

This week I fell into the first trap of computational geometry. I stored polygons in Polygon_2 classes and then performed an intersection test with the do_intersect function. Unfortunately that too often lead to the program crashing with the assertion failure:


CGAL error: precondition violation!
Expr: Segment_assertions::_assert_is_point_on (p, cv, Has_exact_division()) && compare_xy(cv.left(), p) == SMALLER && compare_xy(cv.right(), p) == LARGER
File: /Users/Shared/CGAL-3.2.1/include/CGAL/Arr_segment_traits_2.h
Line: 730


or


CGAL error: assertion violation!
Expr: *slIter == curve
File: /Users/Shared/CGAL-3.2.1/include/CGAL/Basic_sweep_line_2.h
Line: 591


Eventually I narrowed it down to discover that it happened in specific degenerate cases. For instance when a corner of a polygon intersected an edge of another polygon. Which means two line segments are intersecting another line segment in the exact same spot. Now I know this would normally be a problem for line sweep algorithms but I also know that any computational geometry library with respect for itself should handle such degeneracies. So I could not understand why CGAL should fail.

Well it turns out that I didn't read this part of the CGAL manual:

If it is crucial for you that the computation is reliable, the right choice is probably a number type that guarantees
exact computation. The Filtered kernel provides a way to apply filtering techniques [BBP01] to achieve a
kernel with exact and efficient predicates. Still other people will prefer the built-in type double, because they
need speed and can live with approximate results, or even algorithms that, from time to time, crash or compute
incorrect results due to accumulated rounding errors.


I had expected this kind of problem. So I thought it was because I used float. I changed to double and got the same problem. It turns out I should have used CGAL::Exact_predicates_exact_constructions_kernel.

String manipulation: Qt vs Cocoa

Background


I am coding a raytracer program using C++ and Qt. It is program that supports many different algorithms and file formats. One of the algorithms had some bugs that I had problems finding, so I made a OpenGL visualizer of the scene with the hope to be able to visualize different steps in the algorithm to find the error.



Unfortunately the Qt OpenGL implementation proved extremely slow. Instead of looking all over the place to find a solution I decided to create the OpenGL viewer in Cocoa. It seems like Qt4 has several performance problems compared to Qt3. Qt Designer is quite slow when editing properties. Before the last update, it was so slow to type in a text field that I had to type in another window and then cut and paste the text in. Likewise Assistant was so slow I ended up using Safari instead to browse the help documentation.



The scene I read was written for the RenderBitch raytracer. The format is basically like this:


lookat 0 .15 -4.0 ''0 0 0 ''0 1 0

persp 40 1 4.3 .0



material cyan

r_diff .7 1 1

r_refl 0 0 0



# this adds vertices to the scene

vertex 1.25 1 1.3

vertex 1.25 1 -3.3

That is it is a text based file format with some command at each line. So to read the file I had to do a bunch of string manipulation.


I had already done code to read this text in using Qt, so it proved an interesting comparison doing the same task in Cocoa


String classes in Qt and Cocoa


The approach used by Qt and Cocoa is quite different. When looking through Cocoa documentation it reminded me more of how I did string manipulation in script languages like Python and Ruby. Except the methods names were extremely long as is usual in Cocoa (One of the things I dislike the most about Cocoa).



In Qt you deal with strings using the QString, QStringList, QTextStream and QRegExp classes. In Cocoa the equivalents would more or less be: NSString, NSArray and NSScanner. Cocoa does not have an class for regular expressions.



The difference caused by Cocoa being dynamically typed shows itself in the use of NSArray instead of a special string list container. It is general purpose and can hold any kind of objects. Since objects in general can create string representations of itself NSArray also contains methods to joint elements into a string with a separator like QStringList.



One of the creates strength of Qt comes is visible when dealing with primitive types like int and float. By using templates and operator overloading these can be handled a lot easier than Cocoa. In Cocoa you must explicitly box primitive types.


So with Qt I could easily read in data from the text file using QTextStream and have operator overloading automatically take care of type conversion


The example below shows how I can read in e.g. a vertex line in the scene file. Placing the vertex keyword in the keyword variable and the coordinates in x, y and z:


// Reading in a line of text using Qt

QString keyword;

float x, y, z;

QTextStream in(line);

in >> keyword >> x >> y >> z;


In Cocoa I would have to do it explicitly:


// Reading a line of text using Cocoa

NSString* keyword;

float x, y, z;

NSScanner* scanner = [NSScanner scannerWithString:line];

[scanner scanUpToString:@" " intoString:&keyword];

[scanner scanFloat:&x];

[scanner scanFloat:&y];

[scanner scanFloat:&z];


However I didn't actually end up using the operator overloading in Qt. There was some benefits to doing in another way and perhaps it was influenced by my experience dealing with strings in script languages that don't have operator overloading and templates.



So instead I chopped the lines into pieces and converted each string separate like this:


// Split element on line separated by whitespace

QString keyword;

float x, y, z;

QRegExp whitespace("\\s");

QStringList list = line.split(whitespace, QString::SkipEmptyParts);

keyword = line[0];

x = line[1].toFloat();

y = line[2].toFloat();

z = line[3].toFloat();


This looks more complicated but it didn't turn out much different when considering that I read parameters in bulk and they were usually all floats or ints.



In Cocoa I could to similar with this code (although ironically I ended up using the stream approach there):


// Split element on line separated by whitespace

NSString* keyword;

float x, y, z;

NSArray* list = [line componentsSeparatedByString:@" "];

keyword = [list objectAtIndex:0];

x = [[list objectAtIndex:1] floatValue];

y = [[list objectAtIndex:2] floatValue];

z = [[list objectAtIndex:3] floatValue];


It is not entirely equal because it separates on single space and not on any whitespace. This is why I ended up with NSScanner because it has more options on separators.


Unicode


Both Qt and Cocoa uses unicode strings. However in Qt this is not something you have to think much about due to operator overloading. You can write line[0] == '#' and it works, despite line[0] actually representing a unicode character and '#' is not. But overloading of == takes care of the translation trasparently.


With Cocoa it is a different story, because of operator overloading.So to compare a unicode character to a range of other characters one would have to use the NSCharacterSet class.


// line[0] == '#'

NSCharacterSet* hash = [NSCharacterSet characterSetWithCharactersInString:@"#"];

if ([hash characterIsMember: [line characterAtIndex:0]]) {

''''// Do somehting...

}


Of course in this particular instance when we are just checking for a character at the start of the line there is a simpler solution:


// line[0] == '#'

if ([line hasPrefix:@"#"]) {

''''// Do somehting...

}


Experiences with the whole development cycle


It is of course hard to compare the development effort for both environments for this task given that I have much more experience with Qt and did the program in Qt first and then later in Cocoa. So it might even out in the sense that I didn't have to think that much about program design in Cocoa but I had to look up more documentation on how to deal with strings.


With debugging I think Cocoa overall was easier to deal with than Qt. Although I have noticed earlier that the weak typing used in C can cause from problems in Cocoa programs. A number of type mistakes are more easily caught using Qt because it uses C++ which has stronger typing than C.


Some problems with C was also enhanced by how Apple has chosen to let xCode build by default. It uses a Zero linking technique which I thought was great at first because it severely cuts down the compile and link process. However it is quite dangerous because it lets you compile and run programs with calls to functions that don't exist, or with the wrong number of parameters. Now one might argue that this is the case with most dynamic languages in general like script languages. However the combination of late binding and C is a very bad one. There are no proper error handlers when something goes wrong so you remain quite clueless to what went wrong.


However apart from that it is much easier to trace through and debug Objective-C programs in gdb than C++ programs. Almost all Objective-C classes give descriptions of themselves so they can easily be inspected. C functions and Objective-C methods are easier to call because they don't have problems with name mangling like C++ methods and functions.


For instance to debug Qt program I frequently had to make standalone methods that printed out specific objects I had to inspect.


Not an ideal solution because you have to add that code to every Qt program you debug and you have to make code for every class you want to inspect.


Since C++ methods are tricky to call, in debug mode it is hard to change their state through method calls in the debugger.


Conclusion


Although Qt and C++ have some distinct benefits like strong typing, templates and less verbose code, I found it quicker to develop my Cocoa solution because it was much faster to hunt down bugs in the program because debugging was easier.

Graphics: Qt vs Cocoa

The logic of how things work in Cocoa is quite different from most other OO toolkits, especially when it comes to graphics. Here I will give a comparison to how one does some basic graphics operations in Cocoa as opposed to Qt.



I might have chosen to compare Cocoa to Java or .NET frameworks but I am most familiar with Qt. Qt graphics model is quite similar to that use in Java and .NET so the differences explained here should be of interest to Java and C# programers as well.



Introduction


Cocoa and Qt have similar image models for graphics. Both are able to describe images independent of output device (device on which image is displayed) and information about points in drawings (vertices) are retained in the model so that different transformations can be applied to it (e.g. affine transformations.)



Programmin models


However their programming models are different. A programming model can be statefull or stateless and procedure oriented or object oriented. E.g. OpenGL has a statefull and procedure oriented programming model while DirectX has a stateless and object oriented programming model.



In a statefull programming model all drawing operations are affected by the state of the graphics system. E.g. a command to draw a box might be drawBox(x, y, width, height). This would draw a box with current color. The color is then a state in the graphics system. In a stateless system the command would be drawBox(x, y, width, height, color).



The reason why many people find Cocoa confusing at first is that it uses a statefull and object oriented programming model. This is uncommon as Qt, Java and .NET all use a stateless object oriented programming model. In this respect Cocoa has a lot of similarities with OpenGL.



Setting drawing target


In Qt drawing commands change the image or graphics of a QPaintDevice. Typically one of the subclasses like QWidget is used. Below is an example on how one can draw a line of thickness 2 and blue color through three points p1, p2 and p3 on some arbitrary widget returned by getCanvas().


QWidget* w = getCanvas(); ''// Returns our drawing surface

QPainter paint(w);



QPen pen(Qt::blue);

pen.setWidthF(2.0);

paint.setPen(pen);



QPoint p1(x1, y1);

QPoint p2(x2, y2);

QPoint p3(x3, y3);



paint.drawLine(p1, p2);

paint.drawLine(p2, p3);

As one can see the surface to draw on is provided specifically to the QPainter object as well as the pen and color to use. The graphics system is not in any state. The result of the drawing operations is determined exclusively by the objects involved an their attributes.


Below is an example on how to do the same with Cocoa:


NSView* view = getCanvas(); // Returns our drawing surface

NSBezierPath* path = [[NSBezierPath alloc] init];



[[NSColor] blueColor] set];

[path setLineWidth:2.0];



NSPoint p1 = NSMakePoint(x1, y1);

NSPoint p2 = NSMakePoint(x2, y2);

NSPoint p2 = NSMakePoint(x3, y3);



[path moveToPoint:p1];

[path lineToPoint:p2];

[path lineToPoint:p3];



[view lockFocus];

[path stroke];

[view unlockFocus];

As one can see there are some noticeable differences. There is no one object that represents the state of the graphics system like QPainter. NSBezierPath might look like it but it merely keeps track of points on a path and how the lines that connect the points are. No color information is passed to NSBezierPath in the form of a NSColor object nor is the surface on which to draw specified as in the Qt example.


Instead the state of the graphics system itself is changed. [color set] is used to change the color state of the graphics system. Likewise [view lockFocus] is used change the state for current drawing area in the graphics system. Usually when a views method to update its area is called, the state of the drawing area has already been set to that view. So most of the time the user does not have to call lockFocus himself.


Drawing bitmaps


One are that is difficult to understand once one gets started with Cocoa graphics is how to deal with bitmaps. It is confusing precisely because when one is not used statefull programming models for graphics it is not obvious how one draws into a bitmap.

Both Qt and Cocoa have two different image formats. However comparing them is difficult because there is no one to one correspondence in functionality. Qt has QPixmap and QImage. QPixmap is used for drawing images that will go on screen. Because graphics can be created in many different ways on screen individual pixel access is not possible. QImage on the other hand exists off screen in memory an allows individual pixel manipulation. However to put a QImage on screen it has to be converted to a QPixmap


In Cocoa there is a similar situation. NSImage vaguely coresponds to QPixmap. You can not access its individual pixels but you can draw into it and display it on screen. NSBitmapImageRep corresponds roughly to QImage. You can access its pixels individually and set up exactly how pixels are stored, how many color components are used etc. However until recently you could not draw directly into it. Instead you would draw to a NSImage. The reason for this is that NSImage can represent an offscreen window, while NSBitmapImageRep is just a bunch of bytes in memory. Windows area can be drawn on by graphics hardware so they can be represented in any number of ways. They could exist in special graphics memory or on a remote machine. Thus accessing individual pixels on an NSImage makes no sense. However giving drawing commands does because the graphics hardware have drawing commands implemented to do it fast. Doing drawing on NSBitmapImageRep does not make sense because it is not accessible to graphics hardware and drawing can be done by graphics hardware.


Below is an example on how to manipulate the pixels in a QPixmap


Pixmap pixmap = getImage(); // Get some image we want to manipulate

QImage img = pixmap.toImage();



for (int y=0; y < img.height(); ++y) {

''''for (int x=0; x < img.width(); ++x) {

''''''''QRgb pix = img.pixel(x,y);

''''''''doSomethingWithPixel(&pix);

''''''''img.setPixel(x,y,pix);

''''}

}


The code below shows how to do the same thing in Cocoa. That is, to manipulate the pixels in a NSImage. Notice how you must lockFocus on the NSImage to for the pixel grabbing to occur on that particular image.


NSImage* pixmap = getImage(); // Get some image we want to manipulate

NSRect rect = NSMakeRect(NSZeroPoint, [pixmap size]);



[pixmap lockFocus]; // Make pixmap target for drawing commands

NSBitmapImageRep* img = [[NSBitmapImageRep alloc] initWithFocusedViewRect:rect];

[pixmap unlockFocus];



for (int y=0; y < [img pixelsHigh]; ++y) {

''''for (int x=0; x < [img pixelsWide]; ++x) {

''''''''NSColor* pix = [img colorAtX:x y:y];

''''''''doSomethingWithPixel(pix);

''''''''[img setColor:pix atX:x y:y];

''''}

}



[pixmap addRepresentation:img];

Drawing images on screen


To round I need to show how you draw images on screen, or more specifically in a window. A QWidget in Qt's case and a NSView in Cocoa's case. Below we can see how to draw rectangular area within an image given by srcRect inside a rectangular area inside a window (QWidget) given by dstRect

QWidget* w = getCanvas(); ''// Returns our drawing surface

Pixmap pixmap = getImage(); // Return some image we want to draw



QPainter paint(w);



QRectF dstRect(x1, y1, width1, height1) // Area of window to draw into

QRectF srcRect(x1, y1, width2, height2); // Area of image to draw



paint.drawPixmap(dstRect, pixmap, srcRect);



Below we have the corresponding code for Cocoa. Notice that the drawing method is called on the NSImage itself. This does not however mean that drawing is performed inside the image as Qt, Java and C# programmers would easily assume. The target for drawing commands is always the surface/canvas which has focus.


NSView* w = getCanvas(); ''// Returns our drawing surface

NSImage* pixmap = getImage(); // Return some image we want to draw



NSRect dstRect = NSMakeRect(x1, y1, width1, heigth1); // Area of window to draw into

NSRect srcRect = NSMakeRect(x2, y2, width2, heigth2); // Area of image to draw



[w lockFocus];

[pixmap drawInRect:dstRect fromRect:srcRect operation:NSCompositeCopy fraction:1.0];

[w unlockFocus];

Final thoughts


It should be clear that drawing in Cocoa takes some time getting used to for Qt, Java or C# programmers. I have only scratched the surface in this post. From my own experience using both Java and Qt, it is a lot easier to get up to speed on graphics in Qt and Java at first.

However as is typical with everything else with Cocoa it might not be fast to get started but when you dwell into more complicate things, that is when Cocoa start to shine. Likewise with graphics. My own impression from using it (although I am no wizard in Cocoa graphics) is that when it comes to more complicated graphics Cocoa is much more capable than Qt.


It is also my impression that for short examples as given here Qt and Java would usually require less code, but when the graphics is more complicated less code is required in Cocoa.


However I must say that dealing with bitmap graphics seems overly complicated at times in Cocoa. Probably owing a lot to the fact that NEXTStep had totally vector based graphics. The graphics system would output post script drawing commands to the window server and not individual pixels. Basically vector graphics was made easy at the expense of pixel manipulation and raster graphics.

Thoughts on Objective-C 2.0

One of the most exciting changes coming in Mac OS X Leopard, is the new xCode 3.0 with a major update of the Objective-C language. While I think Objective-C is a great programming language and Cocoa is a great framework, it is getting a little bit long in the tooth in a few places.


This has attracted considerable bashing from different people. Nitesh Dhanjani of O'Reilly for instance started a big debate about whether Apple should offer more choices to developers.


The basic argument is that Microsoft is way ahead of the curve with C# and the .NET framework. .NET allows developers to choose any language in theory, thus giving them more choice. This is not something apple offers at the moment.


However with Objective-C 2.0 it seems like Apple has narrowed the gap to more modern languages like C# and Java. It is not official yet but it seems like Objective-C will offer:



  • Optional garbage collection. This was actually an option in Tiger, but toggling it didn't do anything.

  • for each statement. Which makes iterating through objects in containers easier. This is similar to the C# for each.

  • Properties. You don't need to explicitly use setter and getter methods. If you access or set a variable, access methods will be used if they are defined.

  • Method attributes. Not so sure about this but apparently it is similar to what is found in C#.



Here are some examples of the new syntax. The for each syntax:



NSArray *arr = randomListOfStrings();

for (NSString *s in arr) {

  NSLog("%@\n", s);

}


And here is an example of properties:


@interface MyClass : NSObject {

  int   foo;

  float baz;

}



@property int foo;

@property(getter=_getBaz,setter=_setBaz) float baz;



@end



@implementation MyClass



@synthesize foo;



- (int)_getBaz { return baz; }

- (void)_setBaz: (float)newBaz { baz = abs(newBaz); }



@end



int main(void) {

  MyClass *m = [MyClass new];

  m.baz = -5.0;

  printf("%f\n", m.baz); /* -&gt; 5.0000 */

}


I think this means that Mac developers don't have to consider themselves second class citizens. Although I think it never was that bad. A lot of the advantages of .NET and C# are highly theoretical.


Even though you can make any language that language is still limited in functionality to that of the IL code and the .NET framework. The Cocoa framework exploits a lot of the particular strengths of the Objective-C language. It would be difficult to make a framework similar to Cocoa within the .NET world.


So far it seems like most .NET developers are using C#. The other major .NET languages offer little extra functionality or benefits.


Even if Apple does not have an IL type language on which to build other languages they are not really that trapped in language offerings. Because of the highly dynamic nature of Objective-C which is based on message passing instead of method invocation. Other dynamic languages can easily be made to interface with Objective-C classes. Some examples are Ruby Cocoa and PyObjC, which is a Python bridge to Cocoa. One only needs to create a bridge or in other words code that translates the message passing in Objective-C to that of the other language. E.g. in Java there needs to be defined a type for each Objective-C class to maintain the bridge. Unknown Objective-C classes can not be used. For more in-depth look at the problems with bridging Objective-C and Java look at cocoadev.

Performance


Another important point is that I think Objective-C is simply a much more versatile language than C#. Although high performance can be squeezed out of C# it is not nearly as easy. "Easy" is a keyword here, because no language can really do anything another language can't. All popular programming languages today are Turing complete. The difference is then about with what flexibility, ease and performance one can do a certain task.


In a lot of language you can create modules written in C for critical sections and call them from the "mother" language. You can do this in Java through Java native calls. In Ruby you can add your own classes written in C. In C# you can mix managed and unmanaged code. E.g. you can have certain sections written in plain C++.


However the ease which one does this differs. Objective-C is a superset of C and interacts with it easily. Writing performance intensive parts of the program is thus easy and straight forward. One simply tries to use as much plain C as possible and as little object oriented features.


The same can not be said about .NET. I have learned the hard way that mixing managed and unmanaged code is no trivial task. It looks easy in theory and doesn't seem that hard when one does it with small example code. But as soon as you have to deal with a large real world application, problems start piling up.


This means that for performance critical tasks it will be much easier to fine tune a Objective-C application than a .NET application.


Missing features from Objective-C


So if the rumors are right, what will be missing from Objective-C to make it competitive with C# and other modern languages? I could actually only thing of one thing and that is support for namespaces. At the moment this is solved by prefixing class and function names with two letters, to distinguish them from classes and functions with the same name.


Some might argue templates or generics is missing but that is only something that is needed on a static language like C++, Java and C#. In Objective-C data types (except primitives) can be easily inserted and accessed from a container class. No up-casting is needed.


Conclusion


Mac development is going to be better than ever. The only thing that I miss now is a better IDE. I think the debugger in xCode has a lot of problems. It should be improved and so should code completion. Code navigation and re-factoring support could also be improved. At the moment I use TextMate a lot for my coding because it is a lot faster to jump around in code. Apple+T brings up a window that does name completion of files so I can jump to a new file with some quick typing. it also does tabs.


Perhaps Apple should simply improve the integration of the code editor with third party tools. For instance allows some hooks so that after a failed build one can jump to errors in an external code editor.


Also they should do something to reduce the amount of windows that are open at any given time. I do like multiple windows because it allows you to organize your work more when you have a lot of screen real-estate. But at the moment it just seem to become too many windows to keep track of: debug window, code window, run window, console window, project window etc.


If not reducing the number of windows at least provide a better way of managing them (although I guess Spaces might solve a lot of the problem).


My last wish is that Cocoa method names was a bit shorter. Do we really need:


[array insertObject:obj atIndex:2];



Isn't it just as readable to simply write:



[array insert:obj at:2];


Actually I find it almost hard to read Objective-C code at times because the Cocoa method names are so long that you can't actually glaze at them to see that they say. E.g. "thisIsAVeryLongMethodNameIndeed" takes some time to read. Of course this is all wishful thinking because it is too late to change the names not. But one can always dream.

Cocoa Help System

Today I wasted almost the whole day figuring out how to get the Cocoa help system working. I read guides and tutorials and every time I did I got more and more angry at the articles going on about how simple it was to get working, "I just worked like magic".

Well, I found other users didn't quite agree, and I am definitely one of them. One thing that can take some time getting used to with Cocoa is that naming convention is so important. If you don't give things the right name and are consistent, things easily get screwed up.

So to help others to avoid trouble here are the pitfalls I ran into:


  1. The name of the menu item for help HAS to be the same as the name used in the AppleTitle metatags content attribute in your main html help file.

  2. CFBundleName key in the info.plist is not set automatically, so you need to do this. Set it to the same name as your executable.

  3. And finally what took me longest time to discover. When you add your folder with help files, you need to select the "Create Folder References for any added folders." radio button when adding the files to project (use add existing files... under resource folder). If you have done correctly the folder should show up as blue. If it shows up as yellow then you did wrong.



Anyway what really bugs me with this thing, is that there is no feedback on what went wrong when you miss one of the steps.

I will add a step by step guide on how to to it later. In the meantime feel free to post questions.

Oh and this link explains what the difference between copying added folders and using references is.

The Dangers of Static initialization

As I have mentioned before C++ has a rich set of ways for you too shoot yourself in the foot. And that is what I did today. This time it was because of static initialization.

For those familiar with C++, you should know that one can declear variables like this:



static MyClass myVariable;



I am writing a raytracer and this raytracer has a color class. For convenience I added some static variables for commonly used colors.


Color Color::Red(1.0f, 0.0f, 0.0f);
Color Color::Green(0.0f, 1.0f, 0.0f);
Color Color::Blue(0.0f, 0.0f, 1.0f);



In my code I have a singleton class, AppSettings used to access and store and access settings in the class. I used to create a default scene in my application, so a scene could be rendered when starting the app instead of having to read from file. This scene I was creating in constructor for my main window. For various reasons I decided to move the creation of the default scene to AppSettings.

Now suddenly nothing was rendered anymore. I spent a lot of time figuring out what was wrong. Until it hit me that the AppSettings singleton was allocated statically. Using the debugger I discovered the problem: The AppSettings object was constructed before the static color objects. Since my static color objects were used to create the scene they were set to 0.0f, 0.0f, 0.0f when being used by AppSettings.

It also occurred to me that I have had this problem before. But I had forgotten about the dangers of static initialization.

So my own advice on this is. Avoid using static initialization for complex objects because you never know when the initialization code will actually be executed and in which sequence.

Cracking Software on OS X

You might be curious on how people actually do it, or you work with securing software and want to know what methods the other side use so you can better figure out how to protect you software. If so, read on.



I am by now means an expert in the area, but I recently did crack a piece of software to learn about it.



To learn about this I suggest you find a small shareware application or similar to practice on. They usually have copy protection that is weak enough to be cracked by a novice.



Tools

To be able to crack you need a number of different tools. Most of them will already be installed if you have installed Apple’s development tools.



  • Interface Builder. You will use this application to poke around in the user interface of the program you wish to crack.


  • otool. With this command line tool you can disassemble a file, or print

    out other useful information about the executable. For instance you can use it to find out where in memory your program will be loaded.


  • class-dump. Command line utility you have to download. It has some of the same functionality as otool, but allows you to see Objective-C class definitions as they normally appear in the header file.


  • gdb. This is our most important tool. The GNU Debugger. You will use it to trace through the program you wish to crack to find where int he code the copy protection resides


  • HexEditor. To be able to patch your changes into an executable you need an editor that can edit the raw data on a file.




Knowledge

Before you can learn and do cracking you have to know some basics first.




  • Objective-C. You do not need to be a wiz, but some basic knowledge of Objective-C is needed to crack a OS X program since most are written using it. At least most of the little shareware programs you find

  • Interface Builder. You should have basic knowledge about how to work with interface builder.

  • Assembly programming. Again no need to be great at it, or know PowerPC assembly. But you will need to get a reference over the instruction set. IBM also has a nice intro about PPC assembly.

  • Know about the Mac OS X binary application interface (ABI). This you can read about in the Apple developer documentation. Info about the format of OS X executable can be found here and here.

  • Most important though is to make yourself familiar with calling conventions for functions on OS X. This will tell you what the different registers are typically used for, which helps reading and understanding assembly code.



GDB crash course

To run a program through the gdb debugger. write:

gdb nameOfProgram.app/Contents/MacOS/nameOfProgram

The program has now been loaded into memory.



Commands in GDB

gdb has a lot of commands and they all have long descriptive names. You can get help about a command by writing help commandname. However to save you typing, gdb allows you to write abbrivations for the names. You can write the first letter and gdb will choose the first command that match these letters. E.g. instead of writing print you can write p.



A lot of the commands will take a line number, function name, or address in memory as argument. Usually you just write the function name as it is. gdb supports command completion with tab if you don’t remember exact name. Line number can also be written directly. If you write an address you have to prefix it with *. Example *0x00fbc.



Useful commands:



  • run Starts running the loaded program


  • info fun Gets information about a given function. Address it is located at etc.


  • info sym Dumps information about which segment and function the address or symbol is in. Example:

    | (gdb) info sym 0x50fd0
    | [StateController validState] in section LC_SEGMENT.__TEXT.__text


    Says that address 0×50fd0 is in segment __TEXT and section __text.



  • break Set a breakpoint at given function, line number or address.Short hand b



  • rbreak Set breakpoint at all functions with matching regular expression


  • delete Delete all breakpoints or breakpoint given by an index number


  • disassemble Disassemble code in current function


  • backtrace Show a stack backtrace of all functions that were called up to current function.


  • stepi Step one instruction at a time in code. Short hand si



  • where Print out current location in code. Where the program counter is.


  • print Print the value of a memory location, expression or register. Short hand p



  • po Interpret expression as the address of a Objective-C object and print out result one would get from asString. This is used to e.g. print Objective-C strings. Since print does not allow you to do that.


  • x Examine memory by giving and address or register.


  • disp Set expression to display each time program is stopped. For instance after each break or step instruction. Use it to display registers or memory you want to follow the changes in.




You can look up more detailed explanation of each command in the gdb online help. One thing that is useful to know though is that most command that give information about memory or registers let you choose format. E.g. x/x *0x00fb dumps content of address 0x00fb as hexadecimal number. x/s *0x00fb dumps contents as a c string.



Supported format types:



  • o octal

  • x hex

  • d decimal

  • u unsigned

  • t binary

  • a address

  • i instruction. Lets you interpret location in memory as PowerPC assembly instructions


  • o octal

  • c char

  • s string

  • T OSType



Showing contenst of registers:

The PowerPC CPU has a lot of register. The most used general purpose registers are numbered from 0 to 31. To show contents of register you specify a register by writing $ and then its name. E.g. $r1 denotes the second general purpose register. $pc is the program counter.



info register $r3 Dumps contents of register 3 and contents of memory it points to.

p $r3 Dump contents of memory pointed to by register 3.

x/x $r3 Dump contents of memory pointed to by register 3. as hexadecimal format.



Setting value of register:

You can set the value of a register using set command. E.g. set $r3=0x44



otool

As with gdb you have to specify the path to the executable. E.g. otool -l nameOfProgram.app/Contents/MacOS/nameOfProgram. otool has several options, you can read about them all by writing man otool in Termina.app. However there are a few options that are of most use to us.





  • -l Display load commands. This will give you information about where the different segments in the code get loaded into memory. And at which offset in your executable this segment can be found. E.g. if you look at the output you will see e.g.:


    Section
    sectname __text
    segname __TEXT
    addr 0x00002884
    size 0x0009149c
    offset 6276
    align 2^2 (4)
    reloff 0
    nreloc 0
    flags 0x80000400
    reserved1 0
    reserved2 0


    This tells us that this section can be found at offset (decimal) 6276 in executable. When this program is loaded into memory that section is placed at address (in hex) 0×00002884.




  • -tVv Disassembles the give executable and dumps to standard output.



class-dump

This is easy to use just type class-dump nameOfProgram.app/Contents/MacOS/nameOfProgram to dump out definition of Objective-C classes in executable. When you download the file though it doesn’t come with an installer so you need to put it in your search path. E.g. I copied it directly to /usr/bin but you have to be root user to do that. So do a sudo cp class-dump /usr/bin e.g.



Lets get started

First thing to do is to go into the application bundle by right clicking and select Show Package Contents. Then you navigate to e.g Contens/Resources/English.lproj. Here there will be .nib files for the user interface. I opened MainMenu.nib. Here I found the registration sheet. I selected the register… button. Pressing shift+apple+i brings up the inspector panel. Here I looked at the target attribute. It shows which method the button click is connected to. Say it is called registerAction: and exist in the AppController. Try looking for something similar. Then you will have a starting point to start looking for the registration code.



An alternative is to use class-dump on the executable and search for method names called something with register, serial number etc or a class nammed RegHandler, Reg or something similar. E.g. class-dump executable | grep register.



Now that we found a method to inspect, load the executable in gdb. Type

break registerA and hit tab to complete our method. Then we have set a break point. Type run and out application should start running and then stop executing when it reaches our breakpoint.



When you trace through the code using stepi you will often see lines like this:

00010070 bl 0x12e10 ; symbol stub for: _objc_msgSend



What this is a dispatching of a message to an Objective-C object. This is how methods are called in Objective-C. r3 contains pointer to object that will receive message. You can find out what kind of object it is by typing po $r3, this will give a description of object. If object is a NSString it will display the string. r4 contains the message selector. This basically says which method to call. The good thing about this is that the message selector also happens to be a pointer to the c string name of the method. So by typing x/s $r4 you can find out what the name is of the objective-c method being called. r5, r6.. contains the first, second… arguments of the method call. The result of the method call is returned in r3.



r1 and r30 typically points to the start of the stack. So stw and lwz using these registers usually access local variables in the method.



How copy protection works

Copy protection can work in many ways. But the way it worked in my case is that I register the product with a name. I then get back a serial number for that registration name from the application maker. So say I register the application on “Erik” then there will be a unique serial number associated with that name.



When the program loads it reads my registered name from a file and then performs a calculation on it. This will produce a serial number or seed that is compared with the serial number I have put into my application.



What I managed to do was to eventually locate the method that did generation of serial number or seed. I then put a break at that method and rerun the application. Then I did a backtrace to find out which method called the serial number generator at startup.



From there on I was able to find the place where the serial number method was called and the code that compared the results afterwards.



What the code did was doing a comparison with a fixed number against the returned seed (the registered code and provided serial number are used together to produce some unique seed numbers).



In the debugger I used set to set the registered that were compared to the exact same number they were compared against. Then I typed continue to continue execution of code. The program now ran as registered. So it worked.



Patching

The last step to do when you have found out how to circumvent the copy protection is to change the executable (patch it). This you can do with HexEdit. The problem is to find out where in the file to make changes.



There are a number of things that are handy to know when doing this:



  • Since PowerPC is a RISC processor every instruction has the same length, which is 32 bits (4 bytes). Thus you can easily calculate the number of bytes a chunk of code takes. Just multiply number of instructions with 4. This also means you can easily jump over an instruction when tracing by writing jump *($pc+4)


  • As described before you can use otool -l to get information about code or data segments. Using this information you can find the location of an address in memory in the file as follows: location_in_file = location_in_memory - segment_address - segment_offset



So to find the location in the executable to change, you use info sym as mentioned before to figure out in which segment and section it is in. Then use otool -l to find the coresponding segment and section. Then read of the offset and address to calculate location of code in the executable.



To find out what hex codes corresponds to instructions you can write a simple assembly program as below. You can just fill in the instructions you need the hex codes for.


.text
.globl _main
_main:
li r0,3241
stw r0,84(r30)
lwz r0,84(r30)



Name the file with the filename extension .s. The file can then be compiled with e.g.: gcc newfunc.s -o temp if your file is named newfunc.s.



You can now load your app into gdb and look at the hex code for your application by writing x/24xb main. Which will list the 24 first bytes starting from main as hex values. 24 bytes corresponds to 24/4 = 6 instructions.



If you have any questions or comments please let me know. I will update this post later.



References

Here you can find more information:

www.phrack.org

PowerPC cracking with GDB