Using mod_python for Custom Apache/Subversion Authentication/Authorization

As most are aware, Apache is a very modular, highly developer-friendly web server. It still serves more web content worldwide than any other web server. The problem for some is that it is written in C and for you to write custom functionality for Apache, this usually means writing an Apache module in C. What you probably didn't know is that there is an easier way and it is due to mod_python.

mod_python is "an Apache module that embeds the Python interpreter within the server". That is kind of vague, and it's even suggested on the mod_python homepage to read the "Introducing mod_python" article from O'Reilly. To save you the trouble, the important parts in the context we're discussing, mod_python provides the following

  • A handler to the Apache request processing phaes, like authentication (authn) and authorization (authz)
  • An interface to a subset of the Apache API, which means you can call internal Apache functions from Python

Having access to the internal Apache API, or at least some of it, and being able to handle the Apache phases are the parts we're interested in, since a few of those phases are devoted to authn/authz. Before going into any examples or details on how to use these features for creating your own authn/authz module using mod_python, let's go through a very simple series of steps for hooking mod_python into Apache for your application. (We will not be talking about how to install mod_python since there is a plethora of information about this online.) For this example, we'll get talking about how you could use mod_python to create your own custom authn/authz for Subversion.
Hook mod_python into Apache
As usual, there are many ways to do this but since our example is for authentication/authorizing Subversion access, we'll be putting our mod_python stuff within a standard "Location" block. Here is a simple Location block that we will be starting with:

# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/
 
# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION
 
<Location /repos/>
  # Enable Subversion
  DAV svn
 
  # Directory containing all repository for this path
  SVNParentPath /opt/repos/svn
 
  # Enable repository listing when browing the Location root
  SVNListParentPath On
 
  # Enable WebDAV automatic versioning
  SVNAutoversioning On
 
  # Repository Display Name when browsing with the built-in repository browser
  SVNReposName "Your Subversion Repository""
</Location>

The Location block above is one of the simplest ways to expose a Subversion repository. As you can see there is no authn/authz stuff in here meaning right now, if you exposed your repositories using this Location block, you'd have free roam to do whatever you wanted to the repositories and their contents. Now...let's update our Location block to have the necessary bits to require Apache authentication and to make our Python script the handler of the authn/authz Apache phases.
# Work around authz and SVNListParentPath issue
RedirectMatch ^(/repos)$ $1/
 
# Enable Subversion logging
CustomLog logs/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION
 
<Location /repos/>
  # Enable Subversion
  DAV svn
 
  # Directory containing all repository for this path
  SVNParentPath /opt/repos/svn
 
  # Enable repository listing when browing the Location root
  SVNListParentPath On
 
  # Enable WebDAV automatic versioning
  SVNAutoversioning On
 
  # Repository Display Name when browsing with the built-in repository browser
  SVNReposName "Your Subversion Repository""
 
  # Do basic password authentication in the clear
  AuthType Basic
 
  # The name of the protected area or "realm"
  AuthName "Your Subversion Repository"
 
  # Require authentication
  Require valid-user
 
  # Make Apache aware that we want to use mod_python
  AddHandler mod_python .py
 
  # Specify the callable object to handle the authn phase
  PythonAuthenHandler svnauth
 
  # Specify the callable object to handle the authz phase
  PythonAuthzHandler svnauth
 
  # Optionally extend the Python path to locate your callable object
  PythonPath "sys.path+['/opt/repos/scripts']"
</Location>

To better understand the options for the mod_python directives in the Location block, here are a few direct links within the mod_python documentation:

Alright...now that we have Apache ready to use mod_python, we need to create the script used for auth/authz. Since the PythonAuthenHandler and PythonAuthzHandler just specify a module name, we need to have a file named svnauth.py somewhere on the Python path. We can put it into /opt/repos/scripts since we used the PythonPath directive to updated the Python path to include that directory. Now on to the file's contents.
Since we just specified a module name in the mod_python handler directives, we need to follow a specific naming convention for our handler function names. The convention is that you basically strip off the "Python" part of the handler directive's name, and create a function in your module with that name, all lower case of course, that accepts one argument, the Apache request object. Here is a very simple example of our svnauth.py:

#!/usr/bin/env python
# -*-python-*-
#
# Simple example of using mod_python to handle
# Apache's authn and authz phases.
 
from mod_python import apache
 
def authenhandler(req):
  """This function gets called by mod_python to handle Apache's authentication phase"""
 
  return apache.OK
 
def authzhandler(req):
  """This function gets called by mod_python to handle Apache's authorization phase"""
 
  return apache.OK

Pretty simple huh? As the functions stand now, they just return "apache.OK" which instructs Apache that the requested user is authentication and authorized to access the repositories. So...now that you know how to hook up mod_python to handle the Apache phases, like the authentication and authorization phases, where do you go from here? Well, that is up to you. This "article" is only to help get you started. But with me being a nice guy, here are some examples of how you might use this knowledge not only for Subversion but for any Apache-based application where you need to use your own authentication/authorization measures:

  • Making web service calls to authenticate/authorize a user
  • Implementing single sign on for your application
  • Whenever your company stores authn/authz information in a way that is not accessible by conventional Apache modules, like storing your Subversion authz information in a database instead of the typical authz file

The possibilities for using mod_python for authn/authz are endless. Honestly, the possibilities for using mod_python for any of the Apache phases are endless. Sure beats writing C-based Apache modules when it comes to simplicity.
As for what we've learned, I hope that I've taught you enough about mod_python and how it can be used with Apache for creating your own pseudo Apache modules. The example I gave is a real-world example where you can use mod_python to implement your own authn/authz for Subversion but that is just an example. Anywhere Apache is used in your infrastructure, mod_python can be used to make hooking into Apache simple and even provide you the ability to do things that the built-in Apache modules don't provide.

Subversion 1.5.5 Universal for OS X is out

Subversion 1.5.5 has come out and so has the new Subversion 1.5.5 Universal binary. You can download it here: http://www.collab.net/downloads/community/. Here are the Subversion 1.5.5 release notes:

Version 1.5.5
(22 Dec 2008, from /branches/1.5.x)
http://svn.collab.net/repos/svn/tags/1.5.5
 
 User-visible changes:
  * allow prop commits on dirs with modified children (r34487, -92, -94)
  * make Cyrus auth implementation always prefer EXTERNAL to ANONYMOUS (r33866)
  * do not create mergeinfo for wc-wc moves or copies (r34184, -585)
  * do not autoupgrade old BDB filesystems to 1.5 or 1.4 format (r34653, -6)
  * return mergeinfo to prior state during reverse merges (r30257, r33024, -6)
  * remove mergeinfo deleted by merge (issue #3323)
  * make proxy slaves pass through txn GET and PROPFIND requests (issue #3275)
  * merge can now use targets with inconsistent newlines (issue #3262)
  * don't allow empty-string changelists (issue #3344)
  * remove false positive ra_neon mergeinfo errors (r34822)
  * improve performance of 'svn merge --reintegrate' (r34091, -4, and others)
  * fixed: foreign merges keep UUID of foreign repository (r34050, -1, -3)
  * fixed: properly encode diff headers used in conflict resolution (r34171)
  * fixed: segfault in 'svn cp --parents' (r31311, -4)
  * fixed: mergeinfo for '...' maps to empty revision range (issue #3312)
  * fixed: segfault in BDB backend node-origins cache (r34506)
  * fixed: broken merge if target's history includes resurrections (r34385, -93)
  * fixed: invalid mergeinfo created on a subtree during merge (r34560, -2)
 
 Developer-visible changes:
  * fixed: svn_repos_get_logs() chokes on some revision arguments (r33873, -4)

The installer has also undergone some changes. This new version of the installer includes better backup procedures and the preflight/postflight scripts were updated to make fewer environment assumptions, which in some rare cases have lead to some weird installer failures. As usual, let me know what you think about the binary.

Make Trac on Leopard Use Non-Apple Subversion Bindings

I've been running Trac for years now but usually on my Linux box. Since I don't have a web host that has a good way for the full Trac stack (Trac+Subversion), I needed access to a Trac instance from my laptop at all times. That being said, I decided to install Trac on my MacBook running OS X Leopard. After resolving a few issues:

  • Trac doesn't work with Subversion 1.5: To work around this, instead of pointing easy_install to a Trac branch/tag/trunk, I just exported the branch/tag/trunk and pointed easy_install to that local directory.
  • mod_python on Leopard has to be built with x86_64 support: To work around this, I followed: http://matterkkila.com/2007/12/20/building-mod_python-for-leopard/

I had Trac installed but when I tried to create a new Trac project, I kept getting the following error: SubversionException: ("Expected FS format '2'; found format '3'", 160043). I had PYTHONPATH set properly but for some reason, my Subversion 1.5 bindings weren't being used. As usual, I go to Google. I find many suggestions that "work" but they all seemed to suggest overwriting the Leopard-supplied Subversion Python bindings. (I'm not cool with that honestly, and those of you that have used my Subversion OS X binary know this.) Not liking that option, I looked into the Trac sources and found a nice little place to inject some code that would make Trac honor the PYTHONPATH environment variable.

If you open up your TRAC_INSTALLAION/trac/versioncontrol/svn_fs.py file, you see that Trac loads the Subversion Python bindings in _import_svn. This is the file we want to hack to make Trac do our bidding. Here is the code I used to make this happen:

# Hack to get Trac to honor PYTHONPATH
import os, sys
 
if os.environ.has_key('PYTHONPATH'):
    sys.path = os.environ['PYTHONPATH'].split(os.pathsep) + sys.path
# End hack

With this code in svn_fs.py (I put it directly after the other imports), Trac will now use my Subversion 1.5 Python bindings without having to remove or replace the Apple-supplied Subversion 1.4 Python bindings.

I hope this helps someone else that runs into this issue. While there is more than one way to do this, the approach above is unobtrusive, since we do not have to alter or remove the Subversion packaged with Leopard, and it allows you to each set a standard Python environment variable to achieve this. Seems like a win-win to me.

Emacs Tip: Scroll Current Line to the Top of the Window

Emacs is my editor of choice when I'm not in Eclipse. For the most part, I actually think I'm more efficient in Emacs due to there being a key-binding for everything. (Yes...I know there are emacs bindings for Eclipse but I've not had the time to try them yet.) One thing I've never found was a way to make the current line that my cursor is on the top of the window. I started looking through the scrolling section of the Emacs Manual and I found some information about using C-l for scrolling, instead of just clearing the screen. After messing around, I found a simple command that will put the line that the cursor is currently on at the top of the window:

C-u 0 C-l

Presto...no matter what type of file you have open, you can immediately make the Emacs window scroll to make the current line the top of the window.

(Note: I do not claim to be an Emacs expert. I just thought this was useful and since Google was turning up so many false positives, I wanted to preserve this for posterity.)

Subversion 1.5.2 Universal for OSX Released

Sorry for the delay all but I've finally managed to get Subversion 1.5.2 built, tested and packaged for OSX. This new installer doesn't include any new features or bug fixes and is just an updated version of Subversion 1.5.x. (I'm not aware of anything new to add yet and there were no reported bugs to the 1.5.1-2 installer.) As usual, my binaries are hosted by CollabNet, Inc. Here is a link to the download page:

http://www.collab.net/downloads/community/

For the future, I'm thinking of updating the installer to include a Subversion Framework for those software developers wanting to link to Subversion for their projects. I also think it would be cool to be able to strip out any unnecessary architectures from the binary. (For complete CPU support, the Subversion binary is built as a 4-way binary allowing it to run on ppc, ppc64, i386 and x86_64.) I even think it might be useful to take some of the tools in the Subversion sources and packaging them with the binary. Who knows. Feel free to provide any feedback.

Pandora.com Is Amazing

I love music. I really think that if I were to change professions, or go back in life to change something, I would had picked up music as a hobby. Sure I buy music, listen avidly and I own a guitar but I suck at the guitar, at least to this point in my life. While I hope to one day invest in my musical talents, for now, I have to listen to other people's music.

When it comes to what I listen to, I listen to nearly anything. I do not like country all that much, although there are certain artists and songs that I do enjoy, but other than that, I listen to nearly anything. My iPod Touch is busting at the seams due to the amount of music on it. While looking through the Apple iPhone App Store, I kept seeing an application called Pandora in the top 25 with killer reviews. I decided to check it out and ever since, my life has changed.

Pandora is published by pandora.com, whom provides a free service where you can create virtual radio stations based on your favorite artist. After you create a radio station, you listen to music by that artist and other artists with similar music, as identified by the Music Genome Project. (While the Music Genome Project is an extremely brilliant algorithm, I will not dive into it here.) Once you create your radio station(s), you begin to listen either in your web browser or on your iPhone/iPod Touch. Music plays commercial free and the user interface allows you to pause a song, skip to the next song, rate the song (Whether you like it or not) or you can even bookmark that song and/or artist. Pandora even lets you buy the song you're listening to. Pandora.com has literally changed the way I listen to music.

For anyone that loves music and likes music tailored to your likes, with complete control to weed out the things you don't like, Pandora.com is for you. Whether you're listening through a web browser, mobile phone or media device like the iPod Touch, Pandor's Music Genome Project will make sure you have the music you want, commercial free. Pandora.com, you rock.

Apache on Leopard and its "Symbol Not Found" Errors

Don't you hate it when you build a new Apache module or install a new Apache module only to startup Apache and see the following: httpd: ... Symbol not found ... Well, for those of you using mod_perl, mod_python and many others on OS X Leopard, this may have already become a common occurrence. Unfortunately, it has started to affect me as well since some users of the Subversion OSX Universal Binary started running into this issue as well.

The bad news about this is that it might appear like your new Apache module is broken. The good news is that it probably is not and is in fact a false positive caused by Apache shipped with OS X. The best news of all is that this doesn't only apply to the Subversion Apache modules. Many other Apache modules running on Leopard could possibly be falsely accused of being broken whenin reality, it's a very simple fix to Apache and not to the module in question. Let's shed some light on the situation.

When I first got wind of this issue, I recompiled my binary like 5 times. I just knew the problem was with my build script. After wasting a few hours, I began scouring Google, Apple's Discussion Lists, asking questions on freenode and a few other things. I finally ran out of ideas and started just grasping at straws and that is when I made a huge discovery. If I started Apache using launchd or /usr/sbin/httpd directly, I no longer got the error about missing symbols. In fact, when Apache started up successfully, it also loaded my module successfully and it ran without any errors or warnings. This was the clue that gave me what I needed to find the culprit. So if launchd and direct invocation of httpd worked properly, let's look at /usr/sbin/apachectl.

Initial skimming of apachectl gave me no information. Everything looked fine. In fact, I see that apachectl invokes httpd with no special arguments or anything like that. Then I found this:

.....
# pick up any necessary environment variables
if test -f /usr/sbin/envvars; then
  . /usr/sbin/envvars
fi
.....

Bingo. After looking at /usr/sbin/envvars, I see that DYLD_LIBRARY_PATH is being set to prepend /usr/lib in front of any custom DYLD_LIBRARY_PATH I might have set. Not having a custom DYLD_LIBRARY_PATH, I wasn't sure this was the cause. But since it is the only difference between the working solutions and the broken one, I kept looking. After further analysis, I see that in my case, the Apple Subversion libraries were being loaded instead of my custom Subversion libraries. This would explain the missing symbols mentioned in the thread that started this all up. Subversion didn't have a libsvn_fs_base in 1.4.4, which is the version of Subversion shipped with Leopard, and that would explain why this error was manifesting.

After commenting out the only two uncommented lines in /usr/sbin/envvars, I was able to finally start Apache successfully using apachectl and have it load my Subversion 1.5.1 Apache modules with no errors or warnings. The problem has been found and the solution is now known. So what have we learned?

Apple has done a great job to ship many third-party Apache modules with it's latest operating system offering. The problem is that if you want to build custom versions of any Apache modules shipped with OS X, you may run into the same problem many of us have where apachectl, and envvars, is creating a custom DYLD_LIBRARY_PATH, that will override any one you've setup, and it may cause false errors like: Symbol not found. If you're running into the "Symbol not found" when loading a new module into Apache on OS X, try running httpd directly:

sudo /usr/sbin/httpd -D FOREGROUND

and if your Apache loads fine with this approach, but fails to load using apachectl, there is a good chance you have ran into the same problem many of us have and the solution above might just be your ticket.

Subversion 1.5.1-2 Universal Fixes svn+ssh Issue

Well, the Universal Subversion 1.5.1 installer was met with mixed feelings. On one hand, it fixed issues with JavaHL on PPC, included the Subversion 1.5.1 stuff and appeared to finally have SASL support. Little did we know that by including SASL support we broke svn+ssh on Tiger (10.4).

The reason behind this is because we linked against Apple's distribution of libsasl2 and after some research, we found out that libsasl2 from Apple has issues when linked to from third party software. To resolve this, we built Cyrus SASL from source and are now distributing this with the installer.

Please upgrade at your earliest convenience to replace the problematic 1.5.1 installation with the new 1.5.1-2 installation. Other than fixing the SASL stuff, nothing new is in this package.

Here is a direct link: http://www.collab.net/downloads/community/

Subversion 1.5.1 OSX Universal Binary Adds New Features

As Subversion 1.5.1 became available this week, so did the Subversion 1.5.1 OSX Universal Binary. While this is usually a boring little "Downoad the Latest" announcement, this time the binary comes with some improvements. Although the list is short, for enterprise consumers of the binary, this could be big. Here are the noteworthy changes:

Since the binary is already pretty complete, the list above is short and sweet. For those of you using previous versions of the binary, the new version is the most tested version of the binary yet. Feel free to upgrade.

AnkhSVN 2.0 Final(ly) Released

In my last post I mentioned that I had been working feverishly on AnkhSVN 2.0. Well, I'm proud to announce the official release of AnkhSVN 2.0 Final.

For those of you that do not know, AnkhSVN is a Subversion source control provider for Visual Studio.NET. With AnkhSVN, you can do nearly all Subversion client interaction within the IDE using intuitive context menus, tool windows and keyboard shortcuts. AnkhSVN 2.0 is a huge overhaul of the previous AnkhSVN 1.0 line and includes, but is not limited to, the following new features:

  • Visual Studio.NET 2008 Support
  • Subversion 1.5. Support
  • More Subversion Client Features
    • Merge Support
    • Properties Support
    • Enhanced Conflict Resolution Support
    • More tooling
  • End User Documentation

The list above is only a small part of what AnkhSVN 2.0 brings to the table. For a better description of AnkhSVN, the new features in 2.0 and for links to downloads, please read the Official AnkhSVN 2.0 Final Release blog entry on openCollabNet.

Syndicate content