Painless Security

At IETF 77, we’re having a get together to discuss federated authentication beyond the web. The meeting will be in the Mahattan room starting at 9 PM US Pacific time. I think audio streaming will be available; I will post a link closer to the meeting time.

In the last entry, I mentioned that a preliminary spec would be available; see the preliminary EAP GSS-API mechanism. A use case paper and slide set are being reviewed internally and should be ready early next week. We may even have preliminary versions of the binding between RADIUS and SAML available before IETF.

There have been a number of great discussions on the moonshot-community list and with others interested in the broader area.

Recently, I’ve been working on an exciting project with JaNet(UK) on a project to bring federated authentication to non-web applications. I’ve worked on authentication projects a lot, although this is the first federation project I’ve worked on. The big difference appears to be an emphasis on credential independence: the subject (person trying to authenticate) and service will not share a common credential type. Within their organization, the subject and their identity provider share a credential. Then, the federation has some credential mechanism such that the user’s organization and the service share some (probably completely different) type of credential. The other emphasis is on providing personalization.

For web applications, there are a lot of options to achieve this: Information Card, Open ID, SAML, and OAuth all provide solutions in this space. However there are not good options for non-web applications. If you out-source your mail and chat infrastructure but want to use your own chat client or IMAP client, then you will not get the same federation benefits you can get with the web. If you’re using usernames and passwords and don’t mind the potential problems with your out-sourcing provider being able to impersonate all your users, you can simply synchronize usernames and passwords. Within an enterprise, you can do better using Kerberos. JaNet(UK) runs the UK Access Federation, which is a SAML-based web single-sign-on federation. In order to better meet the needs of their customers they’d like to expand this offering to non-web applications. This demand is apparently shared across the European academic community. I suspect there is also some demand in the US academic community and in enterprise situations.

With the web, it turns out that you have a convenient platform for interactions with the identity provider: you can simply direct a web browser to the identity provider and need not specify the user interaction with the web browser at all. This is seen as a significant branding and usability advantage. With other environments, it becomes necessary to specify the interaction with the identity provider. Consider an automated client that wishes to examine a mail box and provide advanced mail sorting or aggregation. That automated client cannot directly use a web browser. OAuth solves this issue with an enrollment step that does typically involve a web browser that produces a consumer key and an authentication step that does not. However for non-web clients it seems like avoiding reliance on the browser authentication will be important. It turns out we already have widely used technologies that do this: the Extensible Authentication Protocol (EAP) mediates the interaction between a subject and identity provider for obtaining network access. It also turns out that we have fairly good technologies for abstract security services within non-web applications: thanks to Kerberos and Active Director, many application protocols and a fair number of applications support GSS-API. JaNet(UK) proposes to combine these technologies with SAML in order to produce a solution for federation beyond the web.

I prepared a feasibility analysis of this proposal. At a technical level, the proposal is sound. There’s a lot of standardization and implementation work, but there appears to be sufficient motivation to form the seeds of a standards activity and put together a proof-of-concept implementation. However, the big question is “Will anyone use it?” In particular, to be useful beyond fairly small communities, support from client vendors and application framework vendors will be needed. It’s taken massive money and around 20 years to get Kerberos support to a point where it is effective within an enterprise. Moonshot can leverage that work to a large extent, but moonshot may also have greater usability and penetration goals.

It’s interesting that I’m advocating EAP for application layer authentication. When I was a Security AD, I made a strong statement that EAP must only be used for network access. I’ve been fairly consistent about that since then. I think there are two huge problems with using EAP for application authentication. The first is that EAP only authenticates the home realm; it does not authenticate what service you’re going to. So you might try to connect to your e-mail and end up giving something access to your stored files and pictures instead. That is, EAP has a phishing exposure in the federated context. If the only thing you can get by using EAP is network access, that exposure is only moderate. However in a fully federated environment that is a huge exposure. Moonshot will address this problem by using EAP channel binding and by doing the necessary work to make that a viable solution. The second concern is that interoperability is reduced when you have multiple authentication approaches for the same problem. If EAP is going to be used for application authentication, we need to understand how it relates to the rest of the application authentication metasystem. Moonshot proposes such a relationship, addressing my objection.

Moonshot is designed to work well with the objectives of the Identity Metasystem and its laws of identity. It uses a different technology, but does have an approach for dealing with claims-based identity and hopefully will have a user experience very similar to the identity metasystem. It uses a different underlying technology. However one of the main beliefs behind the identity metasystem is that is the user experience and universal interoperability that is important, not any specific technology. In its domain, the technologies Moonshot selects will be a better fit than a web services stack.

It’s strange not to be working on Kerberos; Moonshot uses some Kerberos technology, but its core is definitely not today’s Kerberos. In some ways it is fun to be working on something new. There’s one aspect of Kerberos I really miss: Moonshot has nothing like tickets. There’s no place to remember state or exchange to directly involve the client in what the server learns. My analysis talks about ways to make Moonshot more like Kerberos; there are some potential advantages, but so far, the tradeoffs do not justify changes.

We’re hoping to have a bar BOF at IETF 77 and a BOF in the summer at IETF 78.

I’m developing a huge backlog of things to write about, going back as far as a couple of posts inspired by the Kerberos conference in October. However, those require more effort putting together the post, so I’m going to focus on something more recent.

Part of running a business—even a small business like Painless Security—is dealing with the administrative and bookkeeping. The normal solution seems to be Intuit’s Quick Books. When I set up the company, I looked into that. However, Intuit’s accessibility story starts with “give up,” and goes down hill from there. Apparently, screen reader vendors have offered to work with Intuit to help them, but have been turned down, or at least that’s the strong implication I get from reading blogs from these accessibility vendors. So, I’d definitely rather not give Intuit my money. I’m also looking for a bit more than the minimum in accounting software. Keeping the books in order enough to pay taxes would be easy. However I want to be able to understand where I’m making money; I want to be able to figure out if the fixed price contracts I enter into end up being good ideas. I want to understand how expensive community projects Painless Security gets involved in like Debian and IETF work are both in terms of direct costs and opportunity costs. I want to understand what sorts of work ends up being the most profitable. All of these are fairly typical management questions; solutions to varying degrees are understood. however it means I’m actually going to use an accounting product more than just to track my invoices and prepare taxes.

I decided to see what the open source world was like in this space. I started with Ledger SMB. Ledger SMB’s main claim to fame is that it tries to be better than SQL Ledger. Being better than SQl Ledger is definitely a good thing. It “worked” well enough to generate invoices and income statements. It nominally had facilities to track the sort of per-project information I’m looking for, but the facilities are not near good enough. Also, there were some issues—things like the fact that total debits didn’t particularly need or tend to sum to total credits got old after a very short while. Also, facilities for correcting mistakes were unfortunate. You could either operate in a mode where you could delete a transaction, or a mode where you reverse transactions. Deleting transactions in practice meant deleting most of a transaction; balances became out of sync and half of the transaction tended to stick around using some interfaces but not other interfaces. Database cleanup was almost always required. In principle, reversals are better. However, there is no facility to indicate that you’re not expecting payment on a reversed invoice. The receivables/payables account balance out because of the reversing transaction, but both the reversing and reversed transaction end up becoming over-due. “Hey, can you please send me some anti-money to clean this up?” The code was dreadful; the goal was to have better abstraction than SQL Ledger. Perhaps that was achieved, but man that leaves a lot to desire.

Now, I’m playing with Open ERP. It wants to grow up some day to be a competitor to SAP R3. In a way that sounds good, although it does mean there’s a high complexity cost. There’s a lot of functionality. There’s reasonably good separation between view and model (and possibly even controller). The code is often clean, although the random blocks of commented out code with no explanation cause me to cringe. Many aspects of the system seem incredibly well designed; you can approach the system through a web client, graphical client, two different RPC mechanisms, XL and Open Office plugins.

However, there’s this mix of missing stuff and completely broken that causes me to wonder whether I’m going to be sad. First, there appears to be basically no support for the US. All addresses are European format and are hard coded. Each individual report needs to be changed. Recently, I found cases where as best I can tell debits and credits are just reversed. I’ve found a case where backing up the database succeeds but generates a zero length file. I almost lost data through that last one. There’s a complex set of mechanisms to deal with units-of-measure for products—for example, some jobs billed in hours, some are billed in days. However other parts of the code just add quantities.

Playing around with this I am reminded again that I really enjoy thinking about these sorts of problems. An interesting ERP project could be fun to work on. for example, I bet handling ERP needs for some cloud-centered company would be a lot of fun.

Also, as part of looking at Open ERP, I’ve more or less developed the necessary scripts to migrate a simple service company from Ledger SMB to Open ERP. There are limitations of course, but if this would be useful to you, drop me a line.

One of the more annoying aspects to deploying Kerberos and GSS-API is making sure that clients have the correct name for the server they’re talking to. CIFS, the Windows file-sharing protocol, provided the identity of the server to the client. Windows used this to make a few things easier with NTLM but does not use this information with Kerberos.

I keep finding myself in conversations where someone has the bright idea of making this problem easier by generalizing this mechanism and having the server tell the client its identity. The client can then authenticate to that identity. There’s definitely an implementation advantage: you remove all the complexity of name mapping. The problem of course is that it matters what server the client is talking to; the client actually needs to make a decision about how much to trust the server. Authentication to the bad guy is just as bad as the bad guy being able to subvirt your authentication to the good guy.

I was talking recently to another implementor who has similar experience with their customers. Frustrated, I was looking for an analogy simple enough that people could understand the mistake here. I was running through nursery rhymes and other childrens’ tales in may head until I came to the Three Little Pigs. It’s perfect!
The pigs do not want the results of the “Let me in,” service from the big bad wolf. There’s no way that the situation could be made better by more authentication. Asking the wolf for his PIV Card (when have you met a big bad wolf who was not a federal contractor) will not help the pigs decide to let the wolf in. Because the pigs think to consider who they are getting a service from, they decide not to trust it. Of course, physical security is something that first two pigs should have worked on. However even their brother would not have been safe if he’d taken an approach similar to the one proposed for GSS-API of asking the bad guy what name to trust before authenticating to see if the bad guy in fact has that name.

There may be things we can do to make name mapping easier. However we cannot provide security without making a trust decision about who we talk to, and whenever we talk about “who” and trust together, we must consider the security of the mapping.

Of course, as Little Red Riding Hood could tell you, sometimes it is all about authentication. Sadly, her grandmother was unable to get better than level of assurance 1 for her identity as “Little Red Riding Hood’s grandmother,” and the wolf was able to claim that identify for himself.

I hoped to come here and work on Samba4 with MIT Kerberos. Unfortunately the code for that project isn’t as far along as I had hoped. It definitely seems like individuals from Redhat, the MIT Kerberos team, and the Samba time are excited about progress on that front. Amusingly, everyone I’ve talked to is concerned about messy politics. However, I think I’ve talked to people involved in each team at this point. Everyone is very dedicated to the project but is concerned that others may raise political issues.

At this point, I think the only thing to be concerned about on that front is the fear of politics: the desire to do something hackish to avoid discussing something with some other group because they might disagree with it. Yes, we’re going to have design tradeoffs. Yes, we will have to convince each other of things. Yes, time-to-market will influence what solutions are possible and what is too expensive. Perfect may be the enemy of success. However I firmly believe everyone s open to change, open to listening to other parties, and everyone wants to make this happen. When people have brought up specific issues they are concerned other parties might not like, I’ve talked to those other parties and generally found that it won’t be a big deal.

With Jelmer’s help I tried to set up Samba4 and some Windows test environments. It took several days, and I was hugely frustrated. I eventually run out of the time I wanted to dedicate to it, although I did have some success. I may get back to it before returning to the US. Interestingly, Samba4 was not the big problem. I did find bugs—it’s alpha software—but also did get it up and running.

• KVM proved quite frustrating. I wanted to find a simple way to bridge all my VMs onto the Debconf network along with the host. I found a way to do that, although I would not describe it as simple. (Create a bunch of tap interfaces, add the host to a bridge, add the taps to that bridge). Sadly, it didn’t work. The VMs cannot receive DHCP traffic.
• I ran into a Debian specific kernel bug I have a thinkpad. In order to use my external drive bay, I really need to be using the ata_piix driver rather than the older piix driver. Unfortunately, Debian has decided that when a chipset supports both, we will disable the ata_piix support and force piix. (There is some concern about a race condition). Several people have proposed solutions that would address this, other distributions including Ubuntu have addressed the problem, but the only statement from the kernel team is a a statement that they won’t fix it. There has been no willingness to engage in dialogue.
• Even when I get my DVD working, there is some interaction with KVM and the DVD where the system hangs consuming all CPU or IO in the DVD access. X, the VM, everything becomes unresponsive. It’s a soft hang; if I physically remove the DVD, it all turns into errors and I get the system back. However it makes it hard to install software.
• Getting KVM to recognize my windows key is difficult. Works fine with VirtualBox, native applications, etc. The eventual solution (sendkey 0xdc-u for pressing Windows-u leaves something to be desired from a usability standpoint. (There seems to be no way to get the effect of windows-u at the login screen other than actually pressing that combination).
• Of course, even once you manage to hit Windows-u, Vista’s implementation of Narrator is shall we say unresponsive. It’s fairly good at reading what you type. It often reads window titles. I’ve sometimes heard it read part of the contents of a Window. However it’s completely unusable on my system.
• Subversion is slow. Setting up a build environment for the Debian Samba packages took a day.

A lot of people were very helpful in this mess. Joey Hess and Martin Krafft both helped keep me sane.

I’ve been at Debcamp since July 16. The people have been great; the food is reasonable . I’m reminded how wonderful Debian is as a community. There are a huge variety of interests here: conversations have ranged from how to balance embedded system constraints against abstraction, security, computability and complexity of dependency management, sex, drinking and politics. I had been under the impression that the Debian community was very young. While there are a lot of young people here, there are definitely people older than I am, and the depth of experience is amazing. People have been very helpful bouncing ideas around, debugging, or just helping when things get too frustrating.

Yesterday, I redebconfiscated krb5-config package. The previous package had few heuristics for configuring Kerberos; it assumed that the Kerberos realm name mapped directly on the DNS domain name. It also assumed people generally wanted to specify Kerberos servers locally rather than relying on DNS. The new version will hopefully never interact with most users. It supports all the automated mechanisms I’m aware of for determining the Kerberos domain name . If the script can find a plausible realm and find KDCs for that realm, it will not bother most users. If it cannot find KDCs, then it is quite noisy until you help it along. There were a few tricks to the implementation:

• It’s tricky to write a debconf script that reads a user config file to pre-populate debconf, allows the user to change things in debconf, and keeps the file and debconf in sync while giving preference to the file. One thing that makes this tricky is that the config script will be run twice between runs of the postinst script if preconfiguration is involved. The hardest situation to deal with correctly I’ve found is:
  1. Install the package at a relatively high debconf priority so you see few questions.
  2. Lower your debconf priority so that if the package were installed you’d see more questions.
  3. Wait for an upgrade of the package to come out and go through preconfiguration. Does the package behave correctly?
• I ended up having to auto-generate the config script because I need a fair bit of static configuration data that won’t be available until the package is actually installed.
• I think I may have over used the Perl ... operator.

I will be attending Debconf 9 in Spain from July 23-30. I will also be attending debcamp the previous week. I’m hoping to build contacts and increase my involvement in the Debian community, and the previous debconf I attended was an interesting window into what was going on in Debian and Linux.

I’m still lining up things to do at Debcamp. Jelmer Vernooij will be there; he’s interested in working with me on Samba 4 support for MIT Kerberos in Debian. I’m interested in working with him on making the user experience good for people who use both Samba 4 and other Kerberos applications.

As I wrote at the bottom of this post, I believe it is critical that the open source community not just follow what Microsoft is doing in the Enterprise space. I also think it is important that we maintain avenues for our own innovation. To that end, I want to look at what we can do to use enterprise infrastructure independent of AD-look-alike projects like Samba as well. So, I’ll be looking at making what I can do to help this in Debian. Areas of interest include:

1. Easy set up of Kerberos to use an LDAP database
2. Easy configuration of libpam-krb5 and libpam-ldap together using Kerberos for authentication and LDAP for authorization but not authentication.
3. Support for FAST integrated into Debian systems so we can gain better protection against weak passwords. As I promised, more about this in its own post.
4. Better support for PKI/smart cards for network authentication.

These are all projects I think I could make headway on myself. However the value of debcamp is the other people there. I’ve never been to a debcamp before and so I don’t know what it will be like. I do know that I will give higher priority to projects that will benefit from close cooperation over a week. So, if you’re there and want to try to recruit me to your project, feel free. I’m interested in enterprise infrastructure, VOIP, IPv6, network security and making complex infrastructure easy to use.

MIT Kerberos 1.7 is released. I think this release really takes MIT Kerberos forward both for end sites and for system integrators. There are a lot of code quality improvements and bug fixes. For sites, this release allows changes to flow from one KDC to another on an ongoing basis rather than waiting for periodic refreshes. In addition, the domain-realm referral project allows information mapping hosts to domains to be configured in one place rather than on each client.

I already wrote about Active Directory enhancements. Painless Security was also involved in a project to secure Kerberos against offline dictionary attacks. I’m very happy that this project made the 1.7 release. To be truly useful, it will require integration from OS vendors into PAM modules and the like. I’ll discuss my plans for doing that in Debian in a future post.

Despite a lot of new features, initial signs are that 1.7 is going to be a relatively stable release. It has been in Debian unstable for over a month and at this point is working quite well.

A while ago I was working with a client in the financial services sector. They had an online banking product. After a software upgrade, they got a confusing customer service call. A customer, we’ll call her Nancy, called up and reported that she could no longer access the online billpay portion of the site. However, she was sure that she had access because her bills kept paying. After some investigation it turned out that what she meant is that when she tried to click on the bill payments section of the site, she received a page indicating that she didn’t have the online bill pay feature. However, her periodic payments were still being deducted from her account. According to the provisioning system, she did have bill pay access, and audit records confirmed her claim that her payments were being made.

The client was concerned. Taking money out of end-user accounts without giving the user visibility into what was going on or a way to change it was problematic. Besides, she was paying for a service, and the client wanted to provide it. I got involved when the other developers were unable to reproduce the problem in the development environment. When the data set including Nancy’s information was loaded, they could see that she did in fact have access. If someone logged in as her in the development environment, then the bill payment system was available.

When the second user called with the same problem, the issue was escalated. The problem was not universal, but a small number of users reported trying to use online payments and getting the page indicating they did not have the service even when they did.

We still were not able to reproduce on the development environment. We were focusing on whether there was some sort of release engineering error that had caused a problem to creep into the production environment. We could easily create a fresh instance of the development code (although not the supporting database) and that didn’t cause the problem to appear. Our system was interpreted and the access control logic was isolated from the operating system or hardware. So, while we acknowledged the possibility of a problem in that area, we didn’t think that development running on Alpha while production was running on newer Sparc hardware would be the issue.

We looked very closely at the code that decided whether to display the online payments page. We did find a few problems, but none should have caused this bug. For example, the system had a concept of an individual account and a business account. an individual account could be given access to a business account for auditing and dual control reasons. However we did not support using an individual account to access the payments section of a business account. So, there was logic to make sure that the login ID of the user matched the login ID of the resource being accessed. This check would always return true because it used a numeric comparison rather than a string comparison.

While going over the situation someone joked that we could just ask people to change their name: the problem only seemed to happen for users named Nancy. I heard this description and looked at the users who had reported the problem. Sure enough, all named Nancy. How do you get a name dependent bug in access control? Surely this was just sampling error.

What’s special about Nancy? Well, it starts with “nan” as in not-a-number. Surely, though, that shouldn’t affect anything. Unless . . . I logged into a development and production server. Sure enough, on production, “nan+0″ evaluated to “nan”. On development, “nan+0″ evaluated to “0″. Apparently whatever the interpreter was using to read numbers respected nan on Sparc but not alpha. Still, how could this be our issue?

Then I remembered that numeric comparison instead of the string comparison. You see there are two cases where using a numeric comparison to compare strings is not true. The first is when the string represents the number zero. The second is when it represents nan. Sure enough, with a two character change and a lot of paperwork, the Nancies gained access to their payments.

I really love the story of this bug, so I’m sharing it here. People have tried to find a moral in “No Bills for Nancy,” over the years. “This shows the value of static type checking!” some have said. “This shows the critical importance of having identical test environments,” others have said. “If you had the right development methodology, this wouldn’t happen!” others have said.

In some sense, that’s all true. However I’ve found that no matter how good your testing, no matter how good your practices, Nancy is out there lurking, ready to demonstrate that there is some facet of the system that we do not understand. Really, though, Mark Twain had the right answer. It’s a good story; enjoy it for itself.

The Kerberos Consortium, Padl Software, Interisle and Painless Security have been working on adding support for various Active Directory features into MIT Kerberos’s upcoming 1.7 release. I think this project will bring a lot of much needed functionality to MIT Kerberos, and will support the use of Kerberos as a tool in other larger systems.

The project has brought together a lot of players: it wouldn’t have been possible without the efforts of Microsoft, Samba, Novell and several others I’m probably forgetting. It’s great to see such an interest in interoperability that all these parties can work together.

For me, it has been a different approach. I’m used to doing a fair bit of design work up front, understanding what is being delivered, and then working on the code. For a variety of reasons we took a different approach here. Every morning I’d wake up to a new chunk of code to review, evaluate and present to the Kerberos development community. I’d describe the design of the code in order to seek comments and if changes were justified, we’d work to make them. For much of the project, code was coming in faster than I could evaluate it. This meant it was a high-stress and exhilarating project. In other words, it was great fun!

There’s one thing that worries me about this focus on Active Directory. Sure, everyone needs to work with Microsoft. First, it is a market reality. Secondly, Microsoft has brought some great innovative thinking to the realm of network security and we should all take advantage of it. However, it seems that most of the players are only focused on supporting Microsoft features and are ceding the entire space to Microsoft. No one else is working on open standards for expressing authorization. The entire PAC structure, how entities are named, how they belong to groups and how this all interacts in a directory is defined by Microsoft. As a result, Microsoft is in a position to add new technology. However with the current approaches, no one else has this ability. That means, Microsoft will always be one step ahead.

I think it is important that that we all look at how we can embrace and extend Microsoft technology, while maintaining the ability to work together and to work with Microsoft. Doing this is going to require a lot of work but is essential for the continued innovation of network security.