ThoughtSpark.org

I am pleased to announce the release of the Subversion 1.6.3 Universal binary for OS X. This binary has been packaged for Subversion 1.6.3 and it also fixes JavaHL. (There have been reports of JavaHL not working on PPC-based boxes and this has been verified as fixed in the new Subversion 1.6.3 Universal release.) Nothing really spectacular for this release other than updating Subversion and the JavaHL fix.

On the openCollabNet Blog, I released a new article on "Building an OS X Based Subversion Server". While not as comprehensive from an Apache perspective as my "Subversion with Apache and LDAP" entry, it does explain how to install the "Subversion Universal Binary for OS X" and includes a few tidbits about how to make the Apache modules shipped with the binary work with the 64-bit Apache installation on Leopard and its special startup scripts.

Subversion 1.6.1 was just released and right on its heels was the release of the Subversion 1.6.1 Universal binary for OS X. While the binary is just a packaging of the new version of Subversion, which has nothing notable to mention, Subversion has a few noteworthy changes worth mentioning which can be found by viewing its change log.

I just found out that Stanford University is offering its CS 139P - iPhone Application Programming course materials for free. CS 139P is a 10 week course that is being taught by several Apple employees including Evan Doll, Alan Cannistraro and Paul Marcos. After browsing the class' website, I see that you can download the course's syllabus, lectures, handouts and even the assignments. I am very impressed with Stanford. I know they offer a lot to the community but this just blows my mind. I fully intend to follow the course myself and I will do my best to relay my thoughts throughout the semester.

This morning, Subversion 1.6.0 was released. This is a major release and provides quite a bit of new functionality and performance improvements:

• Support for KDE's Kwallet and Gnome's Keyring for authentication credential caching was added
• svn:externals now support single-file external definitions
• Tree conflict support
• Ctypes-based Python bindings
• svnserve logging support
• The full list here: http://subversion.tigris.org/svn_1.6_releasenotes.html

With Subversion's new release, there is also a new Subversion Universal binary for OS X located here: http://www.open.collab.net/downloads/community/. Congratulations to the Subversion team!

The Subversion 1.6.0 release is getting closer and to help get things tested, I'm making Subversion 1.6.0 Universal Release Candidates available for download. Please feel free to install these and run them through their paces to help Subversion 1.6.0 have a stable, quicker release.

I have created a new blog entry on the openCollabnet Subversion Blog explaining the intricacies of exposing Subversion via Apache with LDAP support. Here is the direct link: http://blogs.open.collab.net/svn/2009/03/subversion-with-apache-and-ldap-updated.html. The purpose of this new blog entry was to replace the extremely deprecated and aging entry previously posted on openCollabNet. I hope this new blog entry helps making the use of LDAP for Subversion authentication much easier.

Ramesh Natarajan has, author of "Linux 101 Hacks" is giving his book away FREE if you subscribe to his blog and/or RSS feed. For more details, checkout his blog: http://www.thegeekstuff.com/linux-101-hacks-free-ebook/.

It's contributions to the community like this that make open source so awesome. Thanks Ramesh.

Recently at work, I've been working on writing a Python-based Subversion authorizer. Instead of using mod_authz_svn, which requires your groups and authorization rules be in a text file, we are using something that fits into our product's needs. Anyways, as I was doing this, I really needed a good way to log Subversion's request cycle. Your first thought would be to look at Apache access logs. Well, Subversion's mod_dav_svn uses Apache subrequests, when enabled of course, to communicate and since Apache doesn't log subrequests, I needed something custom.

In a previous blog entry, Using mod_python for Custom Apache/Subversion Authentication/Authorization, I discussed a few reasons why using mod_python was an excellent choice for writing Apache server extensions. That being said, I chose mod_python again for creating a way to log Apache's subrequests.

I will be using my need for logging the Subversion request cycle as the basis for the example code and Apache configuration you see. When using for your own benefit, you'll need to make changes to fit your environment. The good thing is that the example code is well documented so this should be easy. That being said, let's see how we can make mod_python get involved in our Apache communication:

...
LoadModule python_module libexec/apache2/mod_python.so # Your path might be different
...
<Location /repos>
  #Subversion configuration
...
  # mod_python configuration
  SetHandler mod_python
 
  # The interpreter to use
  PythonInterpreter main_interpreter
 
  # The mod_python handlers and their module/script that contains the mod_python handler functions
  PythonLogHandler svn_logger
 
  # Modify the sys.path to be able to locate your module/script
  PythonPath "['/opt/svn'] + sys.path"
...
</Location>
...

The snippet above shows how I took an existing Subversion configuration and put the mod_python bits in there to help out with this venture. The Apache directives for mod_python are pretty straight forward but if you need more details, please refer to the mod_python documentation. Now on to the actual script used to log the Apache requests and subrequests for Subversion.

#!/usr/bin/env python
#
# -*- python -*-
#
# Simple script that uses mod_python to log the Subversion client requests,
# including the subrequests made on the server side.
 
import os, sys
 
def loghandler(req):
  """ mod_python handler method taking an Apache request object. """
  req_type = 'main'
 
  if req.main:
      req_type = 'subrequest'
 
  log = open('/tmp/svn_requests.log', 'a')
 
  log.write("[%s] %s->%s\n" % (req_type, req.method, req.uri))
 
  log.close()
 
  return apache.OK
 
# handler()

That is it. As you can see, the name of the function that you need to implement for mod_python to be happy is the name of the handler but all lower case and without the "Python" in front. (Note: mod_python has a few different ways to specify a handler, even allowing you to specify the method to be called eliminating the need to conform to standards if you want to be such a rebel.) Once we got mod_python's requirements out of the way, the code to log the requests was quite simple. In a production environment, where you don't want this to crash the application if there is a problem, you could easy use a try/except block to catch any errors while opening and writing to the file. Just make sure to return "apache.OK" to let Apache continue doing its thing.

I hope this simple example of using mod_python to log Apache subrequests was enlightening. I know it has helped me tremendously.

This script has become so popular that I have moved it to BitBucket to be able to properly collaborate with the users. Please submit your patches/bugs/requests at: https://bitbucket.org/jcscoobyrs/jw-tools.

Background

When building a Subversion server, people usually go for the setup that requires the least amount of administrative overhead. For many, especially in the enterprise where Active Directory and OpenLDAP rule, this means hooking up Apache to a directory server to authenticate users via LDAP. The reason for such a setup is that you can use the same credentials that log you into your computer, and other internal resources usually, to access Subversion. One less set of credentials for the user to remember and one less user data store for the administrator to maintain. Initially, this scenario is without any real disadvantage. But once you want to start using your groups defined in your directory with Subversion's authorization (authz) mechanism, you find one of the shortcomings of such a configuration.

The Problem

Subversion's authz architecture requires your group definitions to be defined within the authz file. Subversion's authz architecture is also unaware of third-party data stores for users/groups. This means that if you do not define your group within the authz file, Subversion will not know whether a user is a member of said group and will ultimately tell you that you do not have access to a resource. That being said, the current problem is that with this server configuration, either you cannot harness group models defined in your directory server or you have to manually synchronize your group models from your directory server to Subversion's authz file. There are many downsides to doing things this way:

• It's time consuming
• It's easy to forget that changes have been made and need to be mirrored
• It's very easy to make mistakes when doing it yourself
• ...

So while using LDAP for Subversion authentication is a dream, things are not so nice when it comes to reusing your group models for authz that are defined in your directory server.

The Solution

Well, it just so happens that I have a solution. One that is repeatable, loss-less and can be easily setup using the same information you used to configure Apache to authenticate your Subversion users. The solution is the "LDAP Groups to Subversion Authz Groups Bridge" script. (Note: This script, its license and a duplicate of this documentation are attached to the bottom of this article.)

The "LDAP Groups to Subversion Authz Groups Bridge" script is written in Python and, as mentioned before, does its work in a loss-less fashion. This means that while this script will take on the trouble of taking your groups models defined in your directory server and reproducing them in a Subversion authz file, the script will not prohibit you from creating group definitions within the authz file that are not defined in your directory server. Tired of all of the background? Ready to see the script in action? Let's go.

The Implementation

Implementing the "LDAP Groups to Subversion Authz Groups Bridge" is actually simple. If you've already configured Apache to authenticate your Subversion users, which we will not be covering, you can honestly copy/paste pieces of the Apache configuration into the script. Before we go through an example though, there are a few prerequisites that you need to have taken care of before the script will run:

• Install Python (http://www.python.org)
• Install the python-ldap module (http://python-ldap.sourceforge.net)

Now that we have those things out of the way, let's look at the help output of the script, just to get our bearings:

  Usage: sync_ldap_groups_to_svn_authz.py [options]
 
  Options:
    -h, --help            show this help message and exit
    -d BIND_DN, --bind-dn=BIND_DN
                          The DN of the user to bind to the directory with
    -p BIND_PASSWORD, --bind-password=BIND_PASSWORD
                          The password for the user specified with the --bind-dn
    -l URL, --url=URL     The url (scheme://hostname:port) for the directory
                          server
    -b BASE_DN, --base-dn=BASE_DN
                          The DN at which to perform the recursive search
    -g GROUP_QUERY, --group-query=GROUP_QUERY
                          The query/filter used to identify group objects.
                          [Default: objectClass=group]
    -m GROUP_MEMBER_ATTRIBUTE, --group-member-attribute=GROUP_MEMBER_ATTRIBUTE
                          The attribute of the group object that stores the
                          group memberships.  [Default: member]
    -u USER_QUERY, --user-query=USER_QUERY
                          The query/filter used to identify user objects.
                          [Default: objectClass=user]
    -i USERID_ATTRIBUTE, --userid_attribute=USERID_ATTRIBUTE
                          The attribute of the user object that stores the
                          userid to be used in the authz file.  [Default: cn]
    -z AUTHZ_PATH, --authz-path=AUTHZ_PATH
                          The path to the authz file to update/create
    -q, --quiet           Suppress logging information

Anything that has a "[Default:" string in its corresponding documentation is "optional" and can usually be omitted when running against most directory servers. The things that are not optional are things that you can get from your Apache configuration. Now that we have our bearings, let's see an example.

Let's pretend that I have a directory structure like this:

• Release Managers (CN=Release Managers,OU=Groups,DC=subversion,DC=thoughtspark,DC=org)
  • Release Manager One (CN=Release Manager One,OU=Users,DC=subversion,DC=thoughtspark,DC=org)
• Developers (CN=Developers,OU=Groups,DC=subversion,DC=thoughtspark,DC=org)
  • Release Managers (CN=Release Managers,OU=Nested Groups,OU=Groups,DC=subversion,DC=thoughtspark,DC=org)
  • Developer One (CN=Developer One,OU=Users,DC=subversion,DC=thoughtspark,DC=org)
  • Developer Two (CN=Developer Two,OU=Users,DC=subversion,DC=thoughtspark,DC=org)
  • Developer Three (CN=Developer Three,OU=Users,DC=subversion,DC=thoughtspark,DC=org
• Release Managers (CN=Release Managers,OU=Nested Groups,OU=Groups,DC=subversion,DC=thoughtspark,DC=org)
  • Release Manager Two (CN=Release Manager Two,OU=Users,DC=subversion,DC=thoughtspark,DC=org)
• Administrators (CN=Administrators,CN=Roles,DC=subversion,DC=thoughtspark,DC=org)
  • Jeremy Whitlock (CN=Jeremy Whitlock,OU=Users,DC=subversion,DC=thoughtspark,DC=org)

(Yes...I have two groups with the same name but in different locations. This will help showcase how the "LDAP Groups to Subversion Authz Groups Bridge" handles this situation.) I now want to run the script:

  python sync_ldap_groups_to_svn_authz.py \
    -d "CN=Jeremy Whitlock,OU=Users,DC=subversion,DC=thoughtspark,DC=org" \
    -l "ldap://localhost:389" \
    -b "DC=subversion,DC=thoughtspark,DC=org" > svn_auth.txt

You'll notice I didn't specify a password. In this event, the script will prompt you, securely, for a password. After the script runs, as long as you didn't run with the "-q" flag, you should see output like this:

  Successfully bound to ldap://localhost:389
  4 groups found.

Okay, things ran smoothly. (If you notice any "[WARNING]" output, just look at the message along side the warning to see why.) So what does the file look like?

  [groups]
 
  ### Start generated content: LDAP Groups to Subversion Authz Groups Bridge (2009/01/19 23:17:07) ###
  ReleaseManagers = Release Manager One
  Developers = @ReleaseManagers1, Developer Three, Developer Two, Developer One
  ReleaseManagers1 = Release Manager Two
  Administrators = Jeremy Whitlock
 
  ################################################################################
  ###########   LDAP Groups to Subversion Authz Groups Bridge (Legend)  ##########
  ################################################################################
  ### ReleaseManagers = CN=Release Managers,OU=Groups,DC=subversion,DC=thoughtspark,DC=org
  ### Developers = CN=Developers,OU=Groups,DC=subversion,DC=thoughtspark,DC=org
  ### ReleaseManagers1 = CN=Release Managers,OU=Nested Groups,OU=Groups,DC=subversion,DC=thoughtspark,DC=org
  ### Administrators = CN=Administrators,CN=Roles,DC=subversion,DC=thoughtspark,DC=org
  ###############################################################################
 
  ### End generated content: LDAP Groups to Subversion Authz Groups Bridge ###

Here is a list of features, or things to realize, based on the output of this script:

• The "[groups]" section is created but only if necessary
• All generated content is within known "markers" for loss-less regeneration
• The header tells you when the file was last synchronized
• Nested groups, of any level, are supported
• Groups with the same name, but different locations, are supported
• An easy-to-read legend is created for reference reasons

Now that we've seen a very simple example, let's see one more example of how we might take an Apache configuration and turn that into a call to the "LDAP Groups to Subversion Authz Groups Bridge".

Below we have a snippet of the important parts of an Apache configuration using LDAP for Subversion authentication:

  ...
  # The distinguished name to bind to the directory server
  AuthLDAPBindDN "CN=Jeremy Whitlock,OU=Users,DC=subversion,DC=thoughtspark,DC=org"
 
  # The password for the user above
  AuthLDAPBindPassword "myP455w0rd"
 
  # The LDAP query url
  AuthLDAPURL "ldap://localhost:389/DC=subversion,DC=thoughtspark,DC=org?sAMAccountName?sub?(objectClass=user)"
  ...

With the above information, Apache could connect to my theoretical Active Directory server and look for user objects. The way Apache identifies user objects is by the "objectClass" attribute being set to "user". Once a user is found, the "sAMAccountName" attribute is queried to see if it matches the user logging in. Once the user is found, it is authenticated. That being said, let's parse this information and turn it into a successful call to the "LDAP Groups to Subversion Authz Groups Bridge":

  python sync_ldap_groups_to_svn_authz.py \
    -d "CN=Jeremy Whitlock,OU=Users,DC=subversion,DC=thoughtspark,DC=org" \
    -p "myP455w0rd" \
    -l "ldap://localhost:389" \
    -b "DC=subversion,DC=thoughtspark,DC=org" \
    -i "sAMAccountName" > svn_auth.txt

The new things you see in this call that differ from the first call is that we are now looking for the "sAMAccountName" attribute for the username instead of the "cn" attribute. You also see that we can pass the password as a command line argument.

Summary

So now that we've seen how the script is ran, what is necessary to get it running and even a complete example, it's now up to you to get this thing into your Subversion infrastructure. Immediate ideas are to automate this with your operating system's task scheduler and/or create a web interface to kick this script off on an as-needed basis. Regardless of how you use it, the "LDAP Groups to Subversion Authz Groups Bridge" should make the error-prone, tedious and easy-to-forget process of manually synchronizing your LDAP group models to Subversion's authz file much, much easier. It might even get so easy you forget that Subversion doesn't natively support LDAP groups.

Change History

• 2009-01-22 - There was a bug when you didn't specify the -z flag and when the query returned no groups. Version 1.0.1 has been produced as a result.
• 2009-04-15 - There was a bug introduced when reformatting the code that broke the nested groups support. Version 1.0.2 has been produced as a result.