Archive for the ‘Debian’ Category

Debian: Init Interfaces–Our Users *and* Free Software

Saturday, February 8th, 2014 by hartmans

Recently, I’ve been watching the Debian Schadenfreude related to init systems. For those who do not know, Debian is trying to decide whether Systemd or Upstart will be used to start software on Debian. There are two other options, although I think Systemd and Upstart are the main contenders. Currently the Technical Committee is considering the issue, although there’s some very real chance that the entire project will get dragged through this swamp with the general resolution process. This is one of those discussions that proves that Debian truly is a community: if we didn’t have something really strong holding us together we’d never put up with something this painful. Between the accusations that our users now know the persecution European pagans faced at the hands of Christians as the Systemd priests drive all before them, a heated discussion of how the Debian voting system interacts with this issue, two failed calls for votes, a discussion of conflicts of interests, and a last-minute discussion of whether the matter had been appropriately brought before the Technical committee (and if so, under what authority), there’s certainly schadenfreude to go around if you’re into that sort of thing.

However, through all this, the Technical Committee has been doing great work understanding the issues; it has been a pleasure to watch their hard work from the side lines. I’d like to focus on one key question they’ve found: how tightly can other software depend on the init system. Each init system offers some nice features. Upstart has an event triggering model. Systemd manages login sessions and at least in my opinion has a really nice service file format. Can I take advantage of these in my software. If I do, then users of my software might need to use a particular init system. Ian Jackson argues that we should give our users the choice of what init system to run. He reminds us that Debian is a community that supports diversity and diverse goals. We support multiple desktop systems, web browsers, etc. Wherever we can, we support people working on their goals and give developers the opportunity to make it work. When developers succeed, we give our users the opportunity to choose from among the options that have been developed.

I think of Ian’s argument as an appeal to part of our Social Contract. Clause 4 of the social contract begins:

Our priorities are our users and free software.
We will be guided by the needs of our users and the free software community. We will place their interests first in our priorities. We will support the needs of our users for operation in many different kinds of computing environments.

Ian is right that our users will be served by choice in init systems. However, this trade off needs to be balanced against the needs of the free software community. Diversity is an important goal but it should not come at the price of stagnation and innovation. If I want to avoid using init scripts because they don’t provide restart on failure, because they are hard to write correctly, and because they don’t provide access to modern security isolation techniques, I should be able to do that. If Systemd service files provide a superior solution, I should be able to work toward using them. If the desire for init system diversity shuts down my ability to find like-minded people who want to take advantage of improvements in init systems and work towards that goal, then we’ve significantly damaged Debian as a forum for technical cooperation.

Early in my work as a Debian Developer, Anthony Towns taught me an interesting principle. Those who do the work get significant influence over how and what gets done. Ian’s right that we need to enable people to work towards choice in init systems. Those like-minded people need to be given a chance to find each other and pursue their goal. However the cost of success should be on the shoulders of those who value choice in init systems. It should not come at the cost of preventing people from depending on improvements in init systems.

The best proposal I’ve seen for balancing this is to enumerate stable interfaces that people want to use. Things like the logind interface or my favorite the service file interface. It needs to be possible to make another implementation of the interface. The interface needs to be stable enough that a dedicated team could have a chance of catching up with an implementation. At some point during the release cycle such interfaces would need to be frozen. However, I don’t think it is reasonable to mandate that there are multiple implementations of the interface, only that there could be. The point is to give people a chance to work towards diversity in init systems, not to force it down peoples’ throats kicking and screaming until they leave the project or ignore our processes. Steve Langasek and Colin Watson seem to be working towards this.

It’s possible there’s another approach besides interfaces. My main concern is the same as Ian’s: maintain Debian as a forum for people to pursue their goals and work together. I suspect we see the conflict in these goals differently. I hope that we as a project can explore this conflict and figure out where we have common ground. I commit to exploring how we can work towards init choice in my frameworks; I ask those who prioritize init choice to commit to explore how we can take advantage of new features in their framework.

Dream Plug

Monday, June 27th, 2011 by hartmans

As part of trying to help with Freedom Box, I gained access to a Dream Plug The Dream Plug is one of the leading possible platforms for Freedom Box.

Computationally it’s sexy for a low-power device. There’s a 1200 mhz arm with 512m of RAM. As a platform to enable people to create novel applications, it’s great. It has multiple USB ports, two ethernets, a built-in 802.11B/G access point, bluetooth, audio, ESATA, Micro SD and full-sized SD. So, whether you application needs storage, networking, audio, or some interesting side device, you’re covered. With the optional JTAG adapter it’s even fairly friendly to developers: there are options for recovering from most failures and full console access. If you’re looking at a reference platform that’s still nominally embedded, but that allows you to play around thinking about what your application could do, this is a great option.

However, especially for Freedom Box, it’s important to remember that the reference board a developer wants is not the same as a cost-reduced product on which to actually deploy something for people to buy. The Dream Plug is wrong for every actual deployment I’ve imagined. First, it’s huge. If you happen to have free power plugs on your wall with lots of space above and below them, it might be an option. If however, you’re like everyone I know and have power strips, constrained spacing, etc, then the industrial engineering will disappoint at every turn. I was hoping for something that kind of looked like an Apple Airport Express. It’s significantly larger than that in every dimension. Also, the plug is oriented the wrong way for minimizing the fraction of the power strip it takes up. The power supply part of the device can be detached, although even that is way too huge for a power strip, and the cable between the power supply and the computer is fairly short. You can reconfigure the device without its plug nature: running a cord from a normal power outlet to the power supply. But then if you have to detach the power supply for heat management reasons, you get a cord from the outlet to the power supply and another cord from the supply to the device. Add a few USB or ESATA devices and the octopus of cables begins to resemble some arcane mechanism. (So far, no elder gods have appeared though.)

The other issue is that you almost never want all the functionality. You pay for the bluetooth, audio and ESATA in terms of cost, heat and space regardless of whether you use them. I don’t have a lot of applications that really take advantage of the full array of hardware.

The firmware update mechanism is decidedly not targeted at end-users. The version I obtained had a hacked copy of Debian lenny. No mechanism was provided for replacing the image in a safe manner that did not potentially require the optional highly-non-end-user-compatible JTAG board if something got interrupted. You could either unscrew the device and get to the micro SD card containing the image, or run software to replace the image from within Debian lenny. It’s possible to configure the boot loader to run an update image off USB or SD until that succeeds, but doing that is also a non-end-user operation.

In conclusion, the name is perfect. This is exactly what hackers need to dream about the power of small computers everywhere. However we must not forget that there’s a step required to turn dreams into reality. Just as with any fully proprietary product, Freedom Box will require cost reduction steps, semi-custom boards and actual OEMs to truly be usable. The claim in the previous sentence that the Freedom Box may have proprietary elements is disquieting to some. I think we can put together a software stack that is free with the possible exception of some device firmware. However, my suspicion is that anyone who turns that into a fully realized end-user product will add proprietary elements. I suspect some of the results of the cost reduction process such as resulting semi-custom boards will be proprietary. In many cases though, I suspect some proprietary software elements will be introduced.

Review of Git DPM

Tuesday, November 2nd, 2010 by hartmans

I’ve been looking at git dpm. It’s a tool for managing Debian packages in git. The description promises the world: use of git to maintain both a set of patches to some upstream software along with the history of those patches both at the same time. It also promises the ability to let me share my development repositories. The overhead is much less than topgit. My feelings are mixed. It does deliver on its promise to allow you to maintain your upstream patches and to use git normallyish. It produces nice quilt patch series.

In order to understand the down side, it’s necessary to understand a bit about how it works. It maintains your normal branch roughly the way you do things today. There’s also a branch that is rebased often that is effectively the quilt series. It starts from the upstream and has a commit for every patch in your quilt series containing exactly the contents of that patch. This branch is “merged” into your debian branch when you run git dpm update-patches.

The down side is that it makes the debian branch kind of fragile. If you commit an upstream change to the Debian branch it will be reverted by the next git dpm update-patches. The history will not be lost, but unless you have turned that change into a commit along your patches branch, git dpm update-patches will remove the change without warning. This can be particularly surprising if the change in question is a merge from the upstream branch. In that case, the merged in changes will all be lost, but since the tip of the upstream branch is in your history, then future merges will not bring them back. If you either also merge your changes into the patches branch, or tell git dpm about a new upstream, then the changes will reappear at the next git dpm update-patches.

The other fragility is that rebasing your debian branch can have really unpleasant consequences. The problem is that rebase removes merge commits: it assumes that the only changes introduced by merge commits are resolution of conflicts. However, git dpm synthesizes merge commits. In particular, it takes the debian directory from your debian branch, plus the upstream sources from your upstream branch, plus all the patches you applied and calls that your new debian branch. There’s also a metadata file that contains pointers to the upstream branch and the quilt patch series branch. In addition, debian/patches gets populated. Discarding this commit completely would simply roll you back to the previous state of your patches. However, this loses history! There is no reference that is left pointing to the patches branch; the only reason it is still referenced is that it’s a parent of the commit rebase is throwing away. This isn’t really all that unusual; if you merged in any branch, deleted the branch and rebased away both the merge and the commits from the branch, you’d lose history. I do find it a bit easier to do with git dpm.

However, it’s more likely that you’ll manage to rebase and keep the commits from the patches branch. In particular, if the horizon of your rebase includes a merge of the patches branch,then for every patch that involves changes to the debian branch from the point where rebase starts rewriting history, you will see a commit included by your rebase. My suspicion is that this commit is more likely to generate conflicts than usual because it’s a patch on top of the upstream sources being applied to a partially patched debian branch. However I have not fully worked through things: it may be that such a patch is exactly as likely to cause conflicts as it would in a git dpm update-patches operation. If that succeeds and you do not manage to fail to pick one of these commits, you’ll end up with roughly the same debian sources you started with. There are two key exceptions: the git dpm metadata file is not updated and your debian/patches is not consistent with your sources. The first means you’re in the state of the previous paragraph: git dpm update-patches will blow away your changes. The second means that any source package you produce is going to be kind of funky if not out right broken.

It is possible to recover from git dpm plus rebase. I think if you git dpm checkout-patches, then cherry pick the patch related commits from your debian branch that were introduced by the rebase, then git dpm update-patches, you’ll be in a sane state. Obviously you could also recover by finding the pre-rebase commit and resetting there. I’m assuming though that you actually needed to rebase for some reason.

Of course git dpm does involve a space penalty. You generate a commit for every patch in your quilt series for every version of the upstream sources you deal with. Also, you will introduce and store every version of the quilt series directly in your debian directory. The extra commits probably aren’t a big deal: the blobs probably delta compress well. I’m not sure though that the quilt patch blobs will compress well with other things. In my opinion the ability to get nice quilt patches is well worth the space penalty.

In conclusion, git dpm is a sharp tool. Actually, I think that’s understating things. Git dpm involves a bunch of rotating blades. You stick your face and foot between the blades, and if you use it correctly, there’s exactly enough clearance that nothing gets damaged and you get some fairly neat results. Other outcomes are available. I’m still trying to evaluate if it is worth it. I think the tradeoff might be different for something like the krb5 package which is maintained by two very experienced maintainers than say for Shibboleth which seems to involve several more maintainers. I’m not sure that there is much git dpm can do to make things better. Even detecting loss-of-foot events might be kind of tricky.

Debconf 10 Enterprise Track

Wednesday, April 28th, 2010 by hartmans

I’ve been asked by the Debconf 10 talks team to coordinate a Debian Enterprise track at the upcoming Debconf 10 conference. I’m really excited about this, and I could use your help. From my standpoint, this all started with a BOF proposal looking at better coordination and what is missing in Debian enterprise integration. The track will also include talks and other events on what exciting things are happening with Debian in the enterprise. I need your help in the form of suggestions for talks, panels and the like (especially if you would be willing to give the talk or coordinate an activity). We’re under a fairly tight deadline; ideally proposals would be in by May 1, but if they are not, it’s still definitely worth discussing with me. From my standpoint, topics include:

  • Managing groups of machines together
  • Working with Active Directory
  • Open source answers to the use cases of Active Directory if you aren’t a Windows shop
  • Responses especially in the form of working code or ideas for getting there to my concerns about how we’re letting Microsoft drive the evolution in the enterprise
  • Federated authentication, authorization and personalization
  • Integrating with asset tracking, ERP, and other enterprise processes and how Debian fits in.
  • Bringing enterprise players into the community as contributors to the project

Obviously, many other things could fit in and I’d be interested in your ideas as well.

Federated Authentication discussion tonight at 9 PM Pacific

Thursday, March 25th, 2010 by hartmans

The federated authentication bar BOF will be held tonight at 9 PM US Pacific time in the Manhattan room at the IETF 77 meeting.. Here is information for participation.

Reading List

Remote Participation

  • Join our audio stream during the session
  • Join our jabber chat room at
  • Join our mailing list
  • Linux defeats Samba

    Wednesday, July 22nd, 2009 by hartmans

    I hoped to come here and work on Samba4 with MIT Kerberos. Unfortunately the code for that project isn’t as far along as I had hoped. It definitely seems like individuals from Redhat, the MIT Kerberos team, and the Samba time are excited about progress on that front. Amusingly, everyone I’ve talked to is concerned about messy politics. However, I think I’ve talked to people involved in each team at this point. Everyone is very dedicated to the project but is concerned that others may raise political issues.

    At this point, I think the only thing to be concerned about on that front is the fear of politics: the desire to do something hackish to avoid discussing something with some other group because they might disagree with it. Yes, we’re going to have design tradeoffs. Yes, we will have to convince each other of things. Yes, time-to-market will influence what solutions are possible and what is too expensive. Perfect may be the enemy of success. However I firmly believe everyone s open to change, open to listening to other parties, and everyone wants to make this happen. When people have brought up specific issues they are concerned other parties might not like, I’ve talked to those other parties and generally found that it won’t be a big deal.

    With Jelmer’s help I tried to set up Samba4 and some Windows test environments. It took several days, and I was hugely frustrated. I eventually run out of the time I wanted to dedicate to it, although I did have some success. I may get back to it before returning to the US. Interestingly, Samba4 was not the big problem. I did find bugs—it’s alpha software—but also did get it up and running.

    • KVM proved quite frustrating. I wanted to find a simple way to bridge all my VMs onto the Debconf network along with the host. I found a way to do that, although I would not describe it as simple. (Create a bunch of tap interfaces, add the host to a bridge, add the taps to that bridge). Sadly, it didn’t work. The VMs cannot receive DHCP traffic.
    • I ran into a Debian specific kernel bug I have a thinkpad. In order to use my external drive bay, I really need to be using the ata_piix driver rather than the older piix driver. Unfortunately, Debian has decided that when a chipset supports both, we will disable the ata_piix support and force piix. (There is some concern about a race condition). Several people have proposed solutions that would address this, other distributions including Ubuntu have addressed the problem, but the only statement from the kernel team is a a statement that they won’t fix it. There has been no willingness to engage in dialogue.
    • Even when I get my DVD working, there is some interaction with KVM and the DVD where the system hangs consuming all CPU or IO in the DVD access. X, the VM, everything becomes unresponsive. It’s a soft hang; if I physically remove the DVD, it all turns into errors and I get the system back. However it makes it hard to install software.
    • Getting KVM to recognize my windows key is difficult. Works fine with VirtualBox, native applications, etc. The eventual solution (sendkey 0xdc-u for pressing Windows-u leaves something to be desired from a usability standpoint. (There seems to be no way to get the effect of windows-u at the login screen other than actually pressing that combination).
    • Of course, even once you manage to hit Windows-u, Vista’s implementation of Narrator is shall we say unresponsive. It’s fairly good at reading what you type. It often reads window titles. I’ve sometimes heard it read part of the contents of a Window. However it’s completely unusable on my system.
    • Subversion is slow. Setting up a build environment for the Debian Samba packages took a day.

    A lot of people were very helpful in this mess. Joey Hess and Martin Krafft both helped keep me sane.

    Redebconfiscation of krb5-config

    Wednesday, July 22nd, 2009 by hartmans

    I’ve been at Debcamp since July 16. The people have been great; the food is reasonable . I’m reminded how wonderful Debian is as a community. There are a huge variety of interests here: conversations have ranged from how to balance embedded system constraints against abstraction, security, computability and complexity of dependency management, sex, drinking and politics. I had been under the impression that the Debian community was very young. While there are a lot of young people here, there are definitely people older than I am, and the depth of experience is amazing. People have been very helpful bouncing ideas around, debugging, or just helping when things get too frustrating.

    Yesterday, I redebconfiscated krb5-config package. The previous package had few heuristics for configuring Kerberos; it assumed that the Kerberos realm name mapped directly on the DNS domain name. It also assumed people generally wanted to specify Kerberos servers locally rather than relying on DNS. The new version will hopefully never interact with most users. It supports all the automated mechanisms I’m aware of for determining the Kerberos domain name . If the script can find a plausible realm and find KDCs for that realm, it will not bother most users. If it cannot find KDCs, then it is quite noisy until you help it along. There were a few tricks to the implementation:

    • It’s tricky to write a debconf script that reads a user config file to pre-populate debconf, allows the user to change things in debconf, and keeps the file and debconf in sync while giving preference to the file. One thing that makes this tricky is that the config script will be run twice between runs of the postinst script if preconfiguration is involved. The hardest situation to deal with correctly I’ve found is:
      1. Install the package at a relatively high debconf priority so you see few questions.
      2. Lower your debconf priority so that if the package were installed you’d see more questions.
      3. Wait for an upgrade of the package to come out and go through preconfiguration. Does the package behave correctly?
    • I ended up having to auto-generate the config script because I need a fair bit of static configuration data that won’t be available until the package is actually installed.
    • I think I may have over used the Perl ... operator.

    Debconf and Debcamp

    Sunday, June 7th, 2009 by hartmans

    I will be attending Debconf 9 in Spain from July 23-30. I will also be attending debcamp the previous week. I’m hoping to build contacts and increase my involvement in the Debian community, and the previous debconf I attended was an interesting window into what was going on in Debian and Linux.

    I’m still lining up things to do at Debcamp. Jelmer Vernooij will be there; he’s interested in working with me on Samba 4 support for MIT Kerberos in Debian. I’m interested in working with him on making the user experience good for people who use both Samba 4 and other Kerberos applications.

    As I wrote at the bottom of this post, I believe it is critical that the open source community not just follow what Microsoft is doing in the Enterprise space. I also think it is important that we maintain avenues for our own innovation. To that end, I want to look at what we can do to use enterprise infrastructure independent of AD-look-alike projects like Samba as well. So, I’ll be looking at making what I can do to help this in Debian. Areas of interest include:

    1. Easy set up of Kerberos to use an LDAP database
    2. Easy configuration of libpam-krb5 and libpam-ldap together using Kerberos for authentication and LDAP for authorization but not authentication.
    3. Support for FAST integrated into Debian systems so we can gain better protection against weak passwords. As I promised, more about this in its own post.
    4. Better support for PKI/smart cards for network authentication.

    These are all projects I think I could make headway on myself. However the value of debcamp is the other people there. I’ve never been to a debcamp before and so I don’t know what it will be like. I do know that I will give higher priority to projects that will benefit from close cooperation over a week. So, if you’re there and want to try to recruit me to your project, feel free. I’m interested in enterprise infrastructure, VOIP, IPv6, network security and making complex infrastructure easy to use.

    Kerberos 1.7

    Thursday, June 4th, 2009 by hartmans

    MIT Kerberos 1.7 is released. I think this release really takes MIT Kerberos forward both for end sites and for system integrators. There are a lot of code quality improvements and bug fixes. For sites, this release allows changes to flow from one KDC to another on an ongoing basis rather than waiting for periodic refreshes. In addition, the domain-realm referral project allows information mapping hosts to domains to be configured in one place rather than on each client.

    I already wrote about Active Directory enhancements. Painless Security was also involved in a project to secure Kerberos against offline dictionary attacks. I’m very happy that this project made the 1.7 release. To be truly useful, it will require integration from OS vendors into PAM modules and the like. I’ll discuss my plans for doing that in Debian in a future post.

    Despite a lot of new features, initial signs are that 1.7 is going to be a relatively stable release. It has been in Debian unstable for over a month and at this point is working quite well.