ThoughtSpark.org

I've been running Trac for years now but usually on my Linux box. Since I don't have a web host that has a good way for the full Trac stack (Trac+Subversion), I needed access to a Trac instance from my laptop at all times. That being said, I decided to install Trac on my MacBook running OS X Leopard. After resolving a few issues:

• Trac doesn't work with Subversion 1.5: To work around this, instead of pointing easy_install to a Trac branch/tag/trunk, I just exported the branch/tag/trunk and pointed easy_install to that local directory.
• mod_python on Leopard has to be built with x86_64 support: To work around this, I followed: http://matterkkila.com/2007/12/20/building-mod_python-for-leopard/

I had Trac installed but when I tried to create a new Trac project, I kept getting the following error: SubversionException: ("Expected FS format '2'; found format '3'", 160043). I had PYTHONPATH set properly but for some reason, my Subversion 1.5 bindings weren't being used. As usual, I go to Google. I find many suggestions that "work" but they all seemed to suggest overwriting the Leopard-supplied Subversion Python bindings. (I'm not cool with that honestly, and those of you that have used my Subversion OS X binary know this.) Not liking that option, I looked into the Trac sources and found a nice little place to inject some code that would make Trac honor the PYTHONPATH environment variable.

If you open up your TRAC_INSTALLAION/trac/versioncontrol/svn_fs.py file, you see that Trac loads the Subversion Python bindings in _import_svn. This is the file we want to hack to make Trac do our bidding. Here is the code I used to make this happen:

# Hack to get Trac to honor PYTHONPATH
import os, sys
 
if os.environ.has_key('PYTHONPATH'):
    sys.path = os.environ['PYTHONPATH'].split(os.pathsep) + sys.path
# End hack

With this code in svn_fs.py (I put it directly after the other imports), Trac will now use my Subversion 1.5 Python bindings without having to remove or replace the Apple-supplied Subversion 1.4 Python bindings.

I hope this helps someone else that runs into this issue. While there is more than one way to do this, the approach above is unobtrusive, since we do not have to alter or remove the Subversion packaged with Leopard, and it allows you to each set a standard Python environment variable to achieve this. Seems like a win-win to me.

Don't you hate it when you build a new Apache module or install a new Apache module only to startup Apache and see the following: httpd: ... Symbol not found ... Well, for those of you using mod_perl, mod_python and many others on OS X Leopard, this may have already become a common occurrence. Unfortunately, it has started to affect me as well since some users of the Subversion OSX Universal Binary started running into this issue as well.

The bad news about this is that it might appear like your new Apache module is broken. The good news is that it probably is not and is in fact a false positive caused by Apache shipped with OS X. The best news of all is that this doesn't only apply to the Subversion Apache modules. Many other Apache modules running on Leopard could possibly be falsely accused of being broken whenin reality, it's a very simple fix to Apache and not to the module in question. Let's shed some light on the situation.

When I first got wind of this issue, I recompiled my binary like 5 times. I just knew the problem was with my build script. After wasting a few hours, I began scouring Google, Apple's Discussion Lists, asking questions on freenode and a few other things. I finally ran out of ideas and started just grasping at straws and that is when I made a huge discovery. If I started Apache using launchd or /usr/sbin/httpd directly, I no longer got the error about missing symbols. In fact, when Apache started up successfully, it also loaded my module successfully and it ran without any errors or warnings. This was the clue that gave me what I needed to find the culprit. So if launchd and direct invocation of httpd worked properly, let's look at /usr/sbin/apachectl.

Initial skimming of apachectl gave me no information. Everything looked fine. In fact, I see that apachectl invokes httpd with no special arguments or anything like that. Then I found this:

.....
# pick up any necessary environment variables
if test -f /usr/sbin/envvars; then
  . /usr/sbin/envvars
fi
.....

Bingo. After looking at /usr/sbin/envvars, I see that DYLD_LIBRARY_PATH is being set to prepend /usr/lib in front of any custom DYLD_LIBRARY_PATH I might have set. Not having a custom DYLD_LIBRARY_PATH, I wasn't sure this was the cause. But since it is the only difference between the working solutions and the broken one, I kept looking. After further analysis, I see that in my case, the Apple Subversion libraries were being loaded instead of my custom Subversion libraries. This would explain the missing symbols mentioned in the thread that started this all up. Subversion didn't have a libsvn_fs_base in 1.4.4, which is the version of Subversion shipped with Leopard, and that would explain why this error was manifesting.

After commenting out the only two uncommented lines in /usr/sbin/envvars, I was able to finally start Apache successfully using apachectl and have it load my Subversion 1.5.1 Apache modules with no errors or warnings. The problem has been found and the solution is now known. So what have we learned?

Apple has done a great job to ship many third-party Apache modules with it's latest operating system offering. The problem is that if you want to build custom versions of any Apache modules shipped with OS X, you may run into the same problem many of us have where apachectl, and envvars, is creating a custom DYLD_LIBRARY_PATH, that will override any one you've setup, and it may cause false errors like: Symbol not found. If you're running into the "Symbol not found" when loading a new module into Apache on OS X, try running httpd directly:

sudo /usr/sbin/httpd -D FOREGROUND

and if your Apache loads fine with this approach, but fails to load using apachectl, there is a good chance you have ran into the same problem many of us have and the solution above might just be your ticket.

As many of you know, I'm the maintainer of the Universal Subversion binary for OSX hosted on openCollabNet. When I released the last version of the binary, I quickly found out that the Subversion Apache modules wouldn't load under the default Leopard installation of Apache. The reason for this is that Apache on Leopard is a 4-way universal binary and when ran on a 64-bit CPU, Apache would run as a 64-bit application and would attempt to load a 32-bit Subversion module. The result was an error about an unsuitable image. As soon as I found out about the issue, I began devising a plan to fix the issue but I did not have access to a 64-bit installation of OSX. Well, due to the success of the OSX binary on openCollabNet, CollabNet decided to hook me up with a new MacBook and my problem was solved. Within a few weeks, since I do this on my own time, I had a new Subversion 1.4.6 build for OSX that is a genuine 4-way universal binary. This means that the new binary works on 32-bit and 64-bit version so of both Tiger and Leopard. This happens to be the only real "feature of the new binary, since the binary already includes everything Subversion has to offer. If you are new to the binary, or just need to know more about the binary before you download it, please see the original release. This link also has the download location as well. Enjoy and please remember to send feedback and/or requests to me at facilitators@svnbinaries.open.collab.net.