Archive for November, 2010

Implementation Progress

Monday, November 29th, 2010 by hartmans

At the end of September, things were quite exciting as we had our first project meeting. At that meeting those in the room saw a demonstration of the Moonshot GSS EAP mechanism and we discussed a number of open issues and began to plan for our test infrastructure. We’ve made significant progress on the specification front and on explaining Moonshot to important communities since then. However there has been little public progress on the implementation front.

Unfortunately, getting the necessary legal clearance and agreements to release code often takes longer than anyone would like; that is what is happening here. We’re all eagerly awaiting final approval from the lawyers and JANET(UK) management. However, things have been moving behind the scenes. Throughout much of October, Luke Howard and Linus Nordberg were working on their respective parts of the code.

I’ve also been working on putting together the test and build infrastructure. As we discussed at the meeting, we’re going to use Debian and Ubuntu as the basis for our testing. For example, we hope to release virtual machine images for these platforms for the major Moonshot components. Thus the primary build environment for our testing and virtualization will be for Debian. I’ve been putting together that here. Right now, that branch will pull together packages of the SAML infrastructure that we need. I’ve also been looking into virtualized test frameworks and believe I’ve found one that meets our needs. I’ve also put together some primitive build infrastructure that is independent of packaging available here. I’ve set up a buildbot that builds both environments. So, as the code becomes available we’ll be in a good position to start making it available.

Abfab at IETF 79

Monday, November 29th, 2010 by hartmans

The ABFAB working group, which will be standardizing technologies that Moonshot depends on, had its first meeting at IETF 79 in Beijing, China. The meeting was quite productive. Because the meeting was the first of the working group, there were some introductory presentations. A group of authors are putting together a proposed architecture document; we presented the current state of our work. However things have evolved significantly since the working group meeting and I think it will make more sense to wait a couple of weeks to discuss the architecture document.

Most of the time was spent on two presentations. The first was the status of the GSS mechanism. We discussed issues that were discovered while implementing the EAP GSS-API mechanism. Discussion in the room tended to support the proposals made in the slides. A few issues will need to come to the list. We had the most interesting discussion of SAML AAA integration.

Minutes are available.

Review of Git DPM

Tuesday, November 2nd, 2010 by hartmans

I’ve been looking at git dpm. It’s a tool for managing Debian packages in git. The description promises the world: use of git to maintain both a set of patches to some upstream software along with the history of those patches both at the same time. It also promises the ability to let me share my development repositories. The overhead is much less than topgit. My feelings are mixed. It does deliver on its promise to allow you to maintain your upstream patches and to use git normallyish. It produces nice quilt patch series.

In order to understand the down side, it’s necessary to understand a bit about how it works. It maintains your normal branch roughly the way you do things today. There’s also a branch that is rebased often that is effectively the quilt series. It starts from the upstream and has a commit for every patch in your quilt series containing exactly the contents of that patch. This branch is “merged” into your debian branch when you run git dpm update-patches.

The down side is that it makes the debian branch kind of fragile. If you commit an upstream change to the Debian branch it will be reverted by the next git dpm update-patches. The history will not be lost, but unless you have turned that change into a commit along your patches branch, git dpm update-patches will remove the change without warning. This can be particularly surprising if the change in question is a merge from the upstream branch. In that case, the merged in changes will all be lost, but since the tip of the upstream branch is in your history, then future merges will not bring them back. If you either also merge your changes into the patches branch, or tell git dpm about a new upstream, then the changes will reappear at the next git dpm update-patches.

The other fragility is that rebasing your debian branch can have really unpleasant consequences. The problem is that rebase removes merge commits: it assumes that the only changes introduced by merge commits are resolution of conflicts. However, git dpm synthesizes merge commits. In particular, it takes the debian directory from your debian branch, plus the upstream sources from your upstream branch, plus all the patches you applied and calls that your new debian branch. There’s also a metadata file that contains pointers to the upstream branch and the quilt patch series branch. In addition, debian/patches gets populated. Discarding this commit completely would simply roll you back to the previous state of your patches. However, this loses history! There is no reference that is left pointing to the patches branch; the only reason it is still referenced is that it’s a parent of the commit rebase is throwing away. This isn’t really all that unusual; if you merged in any branch, deleted the branch and rebased away both the merge and the commits from the branch, you’d lose history. I do find it a bit easier to do with git dpm.

However, it’s more likely that you’ll manage to rebase and keep the commits from the patches branch. In particular, if the horizon of your rebase includes a merge of the patches branch,then for every patch that involves changes to the debian branch from the point where rebase starts rewriting history, you will see a commit included by your rebase. My suspicion is that this commit is more likely to generate conflicts than usual because it’s a patch on top of the upstream sources being applied to a partially patched debian branch. However I have not fully worked through things: it may be that such a patch is exactly as likely to cause conflicts as it would in a git dpm update-patches operation. If that succeeds and you do not manage to fail to pick one of these commits, you’ll end up with roughly the same debian sources you started with. There are two key exceptions: the git dpm metadata file is not updated and your debian/patches is not consistent with your sources. The first means you’re in the state of the previous paragraph: git dpm update-patches will blow away your changes. The second means that any source package you produce is going to be kind of funky if not out right broken.

It is possible to recover from git dpm plus rebase. I think if you git dpm checkout-patches, then cherry pick the patch related commits from your debian branch that were introduced by the rebase, then git dpm update-patches, you’ll be in a sane state. Obviously you could also recover by finding the pre-rebase commit and resetting there. I’m assuming though that you actually needed to rebase for some reason.

Of course git dpm does involve a space penalty. You generate a commit for every patch in your quilt series for every version of the upstream sources you deal with. Also, you will introduce and store every version of the quilt series directly in your debian directory. The extra commits probably aren’t a big deal: the blobs probably delta compress well. I’m not sure though that the quilt patch blobs will compress well with other things. In my opinion the ability to get nice quilt patches is well worth the space penalty.

In conclusion, git dpm is a sharp tool. Actually, I think that’s understating things. Git dpm involves a bunch of rotating blades. You stick your face and foot between the blades, and if you use it correctly, there’s exactly enough clearance that nothing gets damaged and you get some fairly neat results. Other outcomes are available. I’m still trying to evaluate if it is worth it. I think the tradeoff might be different for something like the krb5 package which is maintained by two very experienced maintainers than say for Shibboleth which seems to involve several more maintainers. I’m not sure that there is much git dpm can do to make things better. Even detecting loss-of-foot events might be kind of tricky.