Cracking Eggs

This post has been lurking for a while, but with the release of OJMoq 2, I decided I should probably just put this out there.  If you don't want to read about the motivation, skip to the sixth paragraph.

I've been on a testing kick for the past few months, motivated by the maturity of the OJTest library (thanks largely to the hard work of Derek Hammer).  OJTest has support for stubs, mocks and spies, which turns out to be very helpful when testing Cappuccino applications.  But one of the issues that I quickly ran into was writing tests around classes that make calls to Cappuccino classes that use default singletons.  For example, many parts of the Cappuccino code will use [CPNotificationCenter defaultCenter] for notifications or [CPBundle mainBundle] to bring in resources/read properties from an Info.plist file.  

So say you have a CPDocument subclass, called SCDocument, that you are writing tests for.  In a unit test, you may just instantiate SCDocument manually and exercise methods for it, but let's say you are being a bit more integrative in your tests and you want to instantiate your document the way Cappuccino normally would - by calling [[CPDocumentController sharedDocumentController] newDocument:self].  You'll realize that CPDocumentController looks to [CPBundle mainBundle] to find the document type to instantiate.  But if you don't have an Info.plist in your Tests directory, this fails.  So how can you tell CPDocumentController the class to instantiate?

One common way people work around problems like this is by using dependency injection.  To do this with the above example, you would first change CPDocumentController to accept a bundle and change all the calls to [[CPBundle mainBundle] call] to be [injectedBundle call].  Then, in your test, you would create a stub called stubBundle that returns the proper document type when you call [[stubBundle infoDictionary] objectForType:@"CPBundleDocumentTypes"].  Then you would inject this stub into CPDocumentController by calling [[CPDocumentController sharedController] setInjectedBundle:stubBundle].  

Simple, right? I didn't think so either.  When you add to this the fact that you would need to refactor a significant portion of Cappuccino to make all the classes accept dependencies, and that you would be making the design of Cappuccino's classes be based heavily on your testing needs, which is arguably bad, and that you would probably be making the API diverge farther away from Cocoa, you would end up with a mess quite a bit more messy than this sentence.

So, I decided to this another way.  Turns out in Cappuccino, you can treat class names like any other declared variable.  Since Cappuccino classes are also objects that understand selectors, you can really just stub out method calls to the class itself and simply reassign the global variable referring to the class to point to your stub.  So, in the above example, instead of making a stub CPBundle instance, I do the following (note that this is code written for OJMoq 1 - in the newer version, I would probably need to make a stub of CPBundle and put in fake implementations of a few more methods):

Now, any classes within Cappuccino that refer to CPBundle will get my stub instead without me having to change anything about these classes.  And once my test is done, the tearDown makes sure everything is back to normal.  

I've been trying to figure out how to do the collaboration part of real-time collaboration, and I've found a lot over the past week.  I've been meaning to do an actual blog entry that tries to explain all the algorithms succinctly (but with examples), but in the mean-time, I've made a post on Quora with a short summary of what I've found: http://www.quora.com/What-are-good-frameworks-for-real-time-collaboration-in-a-web-application?__snid__=688223#answer_30021. If you don't know about Quora, it's an awesome new real-time Q&A site that has a lot of smart people answering questions. If you don't have an account on Quora, this is the paste of what I wrote:

Question: What are good frameworks for real-time collaboration in a web application?

I am probably going to go into more detail about this later, but in case people are curious, this is what I found.  I've added links to articles that don't require an ACM subscription where I could. First, here are the possible algorithms you can use:

1. Operational transformations.  This is a term for a set of different algorithms that let you do real-time collaboration in a generic way.  It's what Google Wave, Etherpad, and Subethaedit use.  There are a few ways to implement this algorithm:
   1. A fully decentralized approach.  This algorithm is described in the first real paper on real-time collaboration here:  http://www.itu.dk/stud/speciale/....  The algorithm is a peer-to-peer way of doing things.  The paper is actually very readable (coming from someone who gets easily intimidated by academia), but as far as I know, no real-time collaboration system actually implements this algorithm other than GROVE (though it is provably correct).
   2. Client-server model.  The algorithm used here is much simpler than the one used for peer-to-peer collaboration, but requires you to have a central server that all the clients can talk to.  This algorithm is described here: ftp://ftp.lambda.moo.mud.org/pub/MOO/p... .  This paper is also very readable, though they do kind of skip over explaining how their model actually applies when you have multiple clients.  I explain a bit more about how that actually works in my stack overflow answer here: http://stackoverflow.com/questio... .
   3. Google Wave approach.  Wave actually uses the algorithm from b), with a slight change that allows them to reduce the amount of state the server needs to keep around.  The Wave algorithm is described here: http://www.waveprotocol.org/whit....  It also has the advantage of being open source.  If people aren't sure why their changes actually allow them to reduce the memory usage on a server, this is something I might expand upon later as it isn't the most obvious.  
   4. Modification to Google Wave approach that doesn't introduce any latency into the network.  The idea is to treat each client-server pair in a client-server model as a p2p network, and apply the algorithm from a).  I made a post about it here: http://groups.google.com/group/wave-protocol/browse_thread/thread/8ffe6e7ab70d0ab.  This is my own algorithm, and I haven't proven it, but it seems like it should work.  It's more complicated than the Wave approach though and I probably wouldn't recommend actually using it.
2. Differential synchronization.  This is the algorithm used by Google Mobwrite and is described in more detail here: http://neil.fraser.name/writing/... .  The idea behind this is to basically have "synchronization cycles" where diffs get sent to a server, merged there, and then the changes on the server get sent back to the client to get merged into the client's base.  It's pretty simple to implement for plain-text, though thinking through how to do it for structured data, it seemed less easy.
3. WOOT.  It stands for "without operational transformations" and you can read more about it here: http://www.inria.fr/rrrt/rr-5580....  Instead of using diffs, WOOT sends a context for every action and each action is applied based on this context.  It never deletes context, just marks things as hidden - so you can be sure that the context you use when sending an edit will exist.
4. Total ordering of commands.  This is the approach used by the Synchro library that builds on top of Google Wave: http://www.processwave.org/2010/....  With this approach, each client keeps around an array of actions it has taken that haven't yet been acknowledged by the server.  When the client gets a new action, it reverts all of its unacknowledged actions, applies the new action, then re-applies its unacknowledged actions.  For some subclass of applications, this can turn out to be simpler than a full operational transformation approach.

By the way, if you run across this paper while trying to find operational transformations (titled "An Updated Client-Server Real-time Collaborative Editor"), don't follow it: http://jcit.org/files/csc562-pro... .  That algorithm does NOT work in all cases.  I can provide details if people are interested.

If you don't want to implement these algorithms yourself, there are also a number of platforms that let you build real-time collaborative apps on top of them (as far as I know, they all use operational transformations to handle things generically).  These include:

1) Google Wave
2) http://www.beweevee.com/
3) http://www.xtwip.com/
4) Probably others, but it seemed like they were mostly flash/silverlight/.NET/paid services, so I didn't look too much into it.

Hope that helps!

If you follow our UserVoice forums, you might know that my latest directive, by popular vote, is to add some kind of collaboration to Mockingbird.  Though I haven't really decided on how this is going to look, I couldn't help but check out the variety of real-time-push-comet-right-here-right-now-web frameworks. Since I knew nothing (and now know next to nothing) about what's out there, the first step was, well, to see what's out there.  I ended up deciding to play around with Node.js + WebSockets (specifically, using Socket.IO - a nifty little framework on top of Node.js that will use WebSockets when possible and fall back to other Comet methods otherwise).

Instead of making the usual sample chat client, I decided to try to make a very simple collaborative drawing application by integrating Socket.IO with Cappuccino so that I can also try to play around with various ways to synchronize user data (see my Hacker News thread about it).  I haven't actually put in any kind of sophisticated data synchronization yet, but the code for what I do have so far is here: http://github.com/saikat/drawtogether.

And here it is in action:

Check out the README at the Github repo to get it up and running.  The code itself is fairly simple.  All the backend work is in server.js, and in there, the actual interesting parts that are doing anything other than serving static files start with the line "var listener = io.listen...".  On the client-side, I made a very simple Objective-J class that wraps the Socket.IO client called SCSocket (located in client/SCSocket.j).  It's up to you to simply set a delegate on this class and implement any of four methods (see setDelegate: on SCSocket) to handle notifications from the backend.  To send notifications to the Node server, simple call [[SCSocket sharedSocket] sendMessage:].

We just deployed a new version of Mockingbird that is officially running SCAuth for all of its authentication needs.

Which leads to the obvious question - what exactly is SCAuth and why would you care about it?  

If, like me, you have an application running on Cappuccino that requires users to login, you probably have a bunch of code that deals with keeping track of the users' session data, keeping it in sync with the backend, and showing users a login dialog at the right times.  There are a couple of not-so-well-known classes/hooks in Cappuccino that make this easier for you (I'm looking at you, CPUserSessionManager and setClassDelegate on CPURLConnection), but it wasn't quite enough for me, so I made SCAuth on top of it to do a bit more. SCAuth comes with SCUserSessionManager which subclasses CPUserSessionManager to make the class a bit more robust (letting you do things like logout and sync your session data).  Eventually, I'd like to make SCAuth handle all kinds of session data depending on what login provider is being used (for example, if your app uses Facebook Connect to login, there are probably facebook connect-y things you want to access about the user from your Cappuccino application).  This should be pretty easily possible by just switching out the login provider that SCAuth uses.

SCAuth also uses setClassDelegate on CPURLConnection to deal with making your users login whenever they do something that requires them to log in.  It uses the login provider that is set on the SCUserSessionManager to deal with logging the user in.  By default, this login provider is a login dialog that comes with SCAuth, but it's pretty easy to make this login provider whatever you want.  

After refactoring Mockingbird to abstract out all this authentication nonsense, I cut down quite a few lines of code, so hopefully someone else will also find this useful (or, if not, make suggestions for how it can be more useful).  You can read the README at the github page for more details.  Oh, and there is an example.

This isn't going to be a real post, but in case anyone else is having issues with performance on canvas-backed views, I learned something fairly important this past week.  Canvas, while it does perform fairly well when drawing lots of objects, is absolutely horrendous at drawing on a large space.  In Mockingbird, this meant that people with pages that were bigger than about 2000x2000 were suffering from choppiness and slowdowns, and as soon as I just removed drawing on a canvas element this big, things sped up tremendously.  This is a blog post that has some numbers and graphs.

For those that drink the Cappuccino, Cappuccino will call lockFocus on a view when the view needs display, even if the view has nothing in drawRect.  This means that if you have a giant CPView, even if you don't implement drawRect on it, a giant canvas element gets created in lockFocus.  It's not a huge performance hit, but getting rid of the lockFocus call did speed things up an a lower powered Macbook running Safari.  I made the following view subclass to do this:

Really simple - by default, a BigView will not draw anything.  But you can still set the isDrawable flag to make it call your drawRect method.  

EDIT: Just talked with the guys at 280 North and they explained to me that Cappuccino is, in fact, smart about not calling lockFocus unnecessarily.  The reason for the performance differences on the Macbook Pro vs the Macbook was that the code on the Macbook Pro did not have a drawRect method at all, whereas the code on the Macbook had an empty drawRect method on the view.  Cappuccino, apparently, figures out if your CPView subclass has an overridden drawRect method before calling lockFocus and all the other drawing code.  So, the default CPView already optimizes for this (just don't have an empty drawRect method in your CPView subclass).

Hope this helps anyone else running into performance problems with the Canvas tag.

I posted an update to Mockingbird earlier today that should make loading large projects much faster.  Most of what I did is probably fairly obvious, but I thought I should share what I did anyway for any Cappuccino newbies looking to help the performance of their app.

1. Use the Safari profiler

The first step to optimization is to always, ALWAYS profile your app.  The things slowing you down will often surprise you, and the guys at 280North have done an amazing job to make Safari's profile very usable with Cappuccino.  You can read more about what they've done here.  After using the profiler, I found a few easy ways to speed things up, but no matter what I did, some of the larger projects were still giving me slow script errors on Firefox.  It was at this point that I realized that if I could somehow break up the work that's being done up front when a new project is loaded, things would work out much better.  And then I remembered that I could...

2. Use Javascript's setTimeout method to break up a large chunk of work

The main issue with loading large projects up front is that each page was being recreated serially in one go.  Instead of doing this, I used setTimeout to load each page, and then give the browser a break between each page load.  My function was basically something like:

- (void)loadPageFunction { window.setTimeout(function() { [self loadPageFunction]; }, 0); }

This way, the browser doesn't think that loading all the pages is one giant chunk of work and, so, for large projects, unless the project has a really crazed page, the browser shouldn't complain.

But then, I noticed one more odd thing - whenever I loaded a project after already having a large project, things ran much more slowly.  After using the profiler to see what was going on, I noticed that my app was spending a lot of time cleaning up notifications from the old project.  After some investigation, I found out that the implementation for - [CPNotificationCenter removeObserver:] is way slower than explicitly removing your observer for each notification using - [CPNotificationtCenter removeObserver:name:object:], and I was using the former everywhere.  So, I decided to change these to...

3. Use - [CPNotificationtCenter removeObserver:name:object:] instead of - [CPNotificationCenter removeObserver:]

One of the problems with Javascript is its lack of weak references, which becomes an issue with singletons like CPNotificationCenter's defaultCenter.  I've tried fairly hard to make sure I always remove my observers wherever I set them up, but I'm pretty sure there are still leaks in there somewhere.  These leaks don't really affect performance until you use a - [CPNotificationCenter removeObserver:] call since this will end up iterating through all your notifications (so if you have many, that becomes a problem).  This is, however, a hack around a more fundamental problem - either we need to figure out a way to implement some kind of weak referencing system in Objective-J or, (the easier route), make it easier to not use the default notification center to handle notifications.  A lot of classes internal to the framework use the default notification center for messaging, but it would definitely be a lot easier for me to just trash the notification center whenever I open a new project (still not perfect, I know, but I think better than what currently happens).  

On the way to implementing PDF export, I needed to be able to transform Mockingbird pages into static HTML.  So, I added a simple category to CPView.  This category creates a cloned DOMElement of the CPView's DOMElement that has the canvas elements converted to PNG images.  I then use XMLSerializer to serialize and send this data to my server.  You can get the gist of it here - http://gist.github.com/253272 .

This uses Nihilogic's Canvas2Image library, which requires base64.j .  The one problem with this code is it won't work in IE since it requires a canvas-enabled browser.

In the meantime, hope this can help someone else trying to do something similar =).

Though Cappuccino has treated me very well so far, text support in Cappuccino is still an area that needs much improvement.  Many newcomers still ask about multiline text support, and so far, the solution has been the minimal implementation of CPTextView located here.  I've been using this for a while, but recently ran into a few problems, and decided to make it just a little less hacked together =).

First, the code:  http://gist.github.com/239696

This code is a bit of an amalgam of code from CPTextField and the old CPTextView.  I've tried to make it a bit more cross-browser compatible and have added support for tabbing.  I've also changed the behavior to be based roughly on the behavior Ross Boucher implemented for CPTextField as described here.  I implement the same delegate methods, except that for CPTextView, the action fires on escape instead of return.  Note that this is not actually the behavior of CPTextView in Cocoa, so if you want to override this behavior, I've made sure to let CPTextView be easily overridable.  There are no file-scoped functions - you can just override the private methods _keyUpFunction, _keyDownFunction, _keyPressFunction and _blurFunction. 

There's still obviously a lot lacking in the code.  For one, I didn't really adhere strictly to the Cocoa API for CPTextView when writing this.  Additionally, autoscrolling doesn't work, entering text is fairly slow after scrollbars appear, and I would like to implement mouse tracking so that clicking on the CPTextView's scroll bar doesn't fire the blur method for the CPTextView.  But this is doing the job for now on Mockingbird.

You may see me posting sometimes with divia at http://lispservice.posterous.com/ .

These are the steps I followed to get Weblocks working on OS X 10.5 (Leopard). Note that you should have Mercurial, CVS, and Darcs installed.

1. Install SBCL with multithread support. The multi-threaded part is important and not enabled by default, and not doing it will result in a page that gives you an obscure error message that reads:

   Socket error in "bind": EADDRINUSE (Address already in use)

   To install SBCL with multithread support do the following:
   1. Download the source for SBCL from http://www.sbcl.org/platform-table.html
   2. After unarchiving the source, in the root directory of the source, create a file called customize-target-features.lisp
   3. In that file add the following to enable threads when building SBCL:

      (lambda (features)
             (flet ((enable (x)
                      (pushnew x features))
                    (disable (x)
                      (setf features (remove x features))))
              ;;; Threading support, available only on x86/x86-64 Linux, x86 Solaris
              ;;; and x86 Mac OS X (experimental).
              (enable :sb-thread))) 
      

   4. Compile SBCL with "sh make.sh" from your shell (this takes a while)
   5. Install SBCL with sh install.sh
2. Get the Weblocks source with

   hg clone http://www.bitbucket.org/skypher/weblocks-stable/.
   

   You will need Mercurial.
3. Add commands in your .sbclrc file to require asdf, asdf-install, and push your Weblocks source into the ASDF central registry. This file should be located in your home directory. If it does not exist you will need to create it. For me, the file looks like this:

   (require 'asdf)
   (require 'asdf-install)
   (push #p"/Users/saikat/Development/Source/weblocks/" asdf:*central-registry*)
   

4. Install ASDF-System-Connections. This wasn't listed anywhere but a lot of the libraries that Weblocks depends on complained if I didn't have this installed. This step may not be necessary - I haven't tried without it. I couldn't get ASDF-System-Connections to install by just doing (asdf-install:install :asdf-system-connections). I downloaded the source and installed it using asdf-install. To do this, get the source as a gzipped tar file from here. Then use asdf-install to install it from your local directory as described here
5. Install all the dependencies listed here. Some notes about this before you begin:
   1. Before installing hunchentoot, I had to install cl-json, metatilities-base, and cl+ssl by hand (asdf-install by name failed on these). For cl-json, I got the source from here. You will need darcs to get it. For metatalities-base, I did this in my shell to get the source:

      $ wget http://common-lisp.net/project/metatilities-base/metatilities-base.tar.gz
      

      For cl+ssl, I got the source from here. You will need CVS to get it. Then I installed all of these using ASDF as is described in step 4.
   2. If you get the following message at any point during the install process you should just CONTINUE (0):

      debugger invoked on a ASDF-INSTALL::KEY-NOT-TRUSTED:
      GPG warns that the key id 0xNIL () is not fully trusted
      
      Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.
      
      restarts (invokable by number or by possibly-abbreviated name):
        0: [CONTINUE     ] Install the package anyway
        1: [SKIP-GPG-CHECK] Don't check GPG signature for this package
        2: [RETRY        ] Retry installation
        3: [ABORT        ] Exit debugger, returning to top level.
      
      (ASDF-INSTALL::VERIFY-GPG-SIGNATURE/STRING
       "-----BEGIN PGP SIGNATURE-----
      

   3. SBCL-port fails to compile. When this happens, simply ACCEPT it (the option that tells you to accept it as if it worked).

You should now be able to follow the instructions in the user's manual to start up weblocks.