ThoughtSpark.org

Don't you hate it when you build a new Apache module or install a new Apache module only to startup Apache and see the following: httpd: ... Symbol not found ... Well, for those of you using mod_perl, mod_python and many others on OS X Leopard, this may have already become a common occurrence. Unfortunately, it has started to affect me as well since some users of the Subversion OSX Universal Binary started running into this issue as well.

The bad news about this is that it might appear like your new Apache module is broken. The good news is that it probably is not and is in fact a false positive caused by Apache shipped with OS X. The best news of all is that this doesn't only apply to the Subversion Apache modules. Many other Apache modules running on Leopard could possibly be falsely accused of being broken whenin reality, it's a very simple fix to Apache and not to the module in question. Let's shed some light on the situation.

When I first got wind of this issue, I recompiled my binary like 5 times. I just knew the problem was with my build script. After wasting a few hours, I began scouring Google, Apple's Discussion Lists, asking questions on freenode and a few other things. I finally ran out of ideas and started just grasping at straws and that is when I made a huge discovery. If I started Apache using launchd or /usr/sbin/httpd directly, I no longer got the error about missing symbols. In fact, when Apache started up successfully, it also loaded my module successfully and it ran without any errors or warnings. This was the clue that gave me what I needed to find the culprit. So if launchd and direct invocation of httpd worked properly, let's look at /usr/sbin/apachectl.

Initial skimming of apachectl gave me no information. Everything looked fine. In fact, I see that apachectl invokes httpd with no special arguments or anything like that. Then I found this:

.....
# pick up any necessary environment variables
if test -f /usr/sbin/envvars; then
  . /usr/sbin/envvars
fi
.....

Bingo. After looking at /usr/sbin/envvars, I see that DYLD_LIBRARY_PATH is being set to prepend /usr/lib in front of any custom DYLD_LIBRARY_PATH I might have set. Not having a custom DYLD_LIBRARY_PATH, I wasn't sure this was the cause. But since it is the only difference between the working solutions and the broken one, I kept looking. After further analysis, I see that in my case, the Apple Subversion libraries were being loaded instead of my custom Subversion libraries. This would explain the missing symbols mentioned in the thread that started this all up. Subversion didn't have a libsvn_fs_base in 1.4.4, which is the version of Subversion shipped with Leopard, and that would explain why this error was manifesting.

After commenting out the only two uncommented lines in /usr/sbin/envvars, I was able to finally start Apache successfully using apachectl and have it load my Subversion 1.5.1 Apache modules with no errors or warnings. The problem has been found and the solution is now known. So what have we learned?

Apple has done a great job to ship many third-party Apache modules with it's latest operating system offering. The problem is that if you want to build custom versions of any Apache modules shipped with OS X, you may run into the same problem many of us have where apachectl, and envvars, is creating a custom DYLD_LIBRARY_PATH, that will override any one you've setup, and it may cause false errors like: Symbol not found. If you're running into the "Symbol not found" when loading a new module into Apache on OS X, try running httpd directly:

sudo /usr/sbin/httpd -D FOREGROUND

and if your Apache loads fine with this approach, but fails to load using apachectl, there is a good chance you have ran into the same problem many of us have and the solution above might just be your ticket.