Scott James Remnant

There’s been a meme going around the open source community for a while now.  That programming in C is somehow dirty, distasteful and worst-of-all inefficient compared to programming in a high-level language such as C# or Python.

Its detractors will tell you how it takes much longer it takes to program anything in C.  They’ll point at how much C code it takes to do something as simple as create a GObject sub-class compared to the equivalent in Python or C#.  They’ll also probably complain that everything in C has to be compiled first, which takes even more time up.

No argument would be complete without them pointing out that C has no standard types for strings, let alone linked lists, binary trees, associative arrays, etc. and that you have to spend all that time implementing your own.  They’ll probably make a point about how C’s static type system means that even if you have a array type, you need to know in advance what types you’re going to put into it and can’t just mix and match.

Don’t feel tempted at this point to counter with a discussion about how great and flexible pointers are.  You’ll receive a lashing about how they’re even more evil than people who talk in the theatre.  The rant about the C problems of uninitialised memory, out of bounds pointer errors and segmentation faults is a timeless classic.  Especially when they get to the bit about how much time was lost debugging them.

And do you know what?

I simply do not agree with them.

I cannot think of a single project where the majority of time was wasted writing GObject header files compared to the single line of Python I needed.  I can think of lots where I’ve sat for hours trying to figure out which class I needed to derive from, or reworking the code after I realised I derived from the wrong class to begin with.  The high-level language doesn’t make this any easier.

As to the number of projects where I’ve needed to write a linked list or hash table implementation because C lacked a convenient dynamic array or associative array type like Python?  If it takes you any time to write that kind of code, you’re doing it wrong.  I’ve spent far more time realising that the structure is a performance bottleneck, and planning on the whiteboard a faster alternative.  Neither language helps with this whiteboard time.

And all those pointer issues?  This comes down to the tools that you’re using.  If you’re writing in a language and not using its development environment properly, then it’s little surprise that you’re not being as efficient in it.  gdb, valgrind, gprof and gcov are your friends.  Use them well.  I’ve spent just as much time dealing with other language-specific issues to make me believe that pointers aren’t any more evil as (for example) monkey patching.

The vast majority of my time on any new project is first of all spent thinking, and on a more mature project its figuring out what I did wrong and how I need to rework it.

Yes, the next biggest use of my time is working out what the best way is to express that.  If I’m writing in C, that means I’m deciding whether it needs a linked list, or a hash table, or some other fancy structure.  But if I’m writing in Python, believe me I spend just as much time normalising my class structure and coming up with all sorts of insane Pythonic ways of doing things.

If you’ve ever written in Perl and not lost a day or two to optimising your regular expressions, or eliminating code to arrive at the shortest possible expression, you’ve never written in Perl.

Refactoring takes you just as long in Python as it does in C.  Just because when you do it to C code you end up setting segfaults doesn’t mean that when you do it to Python code, suddenly your class structures don’t match anymore.

Proponents of test-driven-development, AGILE, LEAN and ANEMIC programming methodologies will probably argue that it’s easier to practice their religion with a high-level language.  I’m not buying that either, I’ve managed to write several large software projects in C that have a comprehensive test suite – including testing for allocation failures.

Ah, Rapid Application Development I hear you say?  Well, the only people I’ve ever heard say how great RAD is are people who’ve never had to support the software that was written rapidly, or debug issues with it years later.  RAD is great when v2 is going to be a complete rewrite, and v3 a complete rewrite again.  Very few websites have upgrades without announcing a completely new codebase.

It’s certainly true that it’s faster to mash up some code in a high-level language.  I use shell scripts and bits of Perl for this kind of thing all the time, and I frequently even do basic mock-ups or essays in Python.  But ultimately it all tends to be throw-away code, that I don’t really ever intend to take seriously or attempt to support later on.

For larger projects, I just don’t see any difference in the time it takes to write.

I’d like to cite an example.  The GIT and Bzr revision control systems are about the same age, one of them is written in C and one of them is written in Python.  It hasn’t taken them any less time to write the one in Python than it’s taken the others to write the other in C.  The one in Python doesn’t have extraordinary features that the one in C lacks.

C# fans would point out how much faster it is to UI code.  Really?  Then why isn’t Banshee that much dazzingly better than Rhythmbox?  Sure they’re different, but there’s nothing there that suggests one language is better than the other.

And do you know what?  I trust code written in C far more than I do any higher level language.  No, that’s probably not fair.  I trust C programmers far more than I do programmers of other languages.  If you tell me I have the option of choosing a program or library written in C over one written in Python or C#, I’ll take the C one every time.

I’d like to propose a new meme for the Planets, something a little more interesting than the third paragraph in your nearest bowl of Alphabet Soup.  Something, in fact, a little creative.

I wonder if you’ve heard of a Drabble?

A drabble, simply put, is a story, normally science fiction or fantasy that is exactly one hundred (100) words in length.  No more, no less.

Such things were often written in a contest, and so I’d like to propose such a contest.

To participate, all you have to do is write a story in 100 words and post it on your blog.

Here’s my attempt, entitled The Siege.

‘The city, sir!  It’s under siege!’
The king put down his goblet and stared at the sergeant.
‘Who is it this time?
‘It can’t be the dwarfs, they can’t reach the ramparts.
‘The trolls just don’t have the stamina.
‘The elves get drunk and wander off singing.
‘The ents are easily dispatched with fire.
‘And as for the orcs, wait long enough and they kill each other.’
‘So who is it?  Hmm?’
The sergeant looked sheepish, as the king awaited his answer.
‘It’s our own people, Sir!’
Ah, this was going to be a problem.

My recent series of posts in which I had some difficulty with GIT (1, 2, 3) have garnered a lot of interest, and generated some discussion.  Many of the replies saw the funny side, which is nice as I was hoping to be at least amusing (rants are, after all, a form of entertainment) and more than I expected sympathised.  Many tried to help by giving me a sequence of commands that might work, ironically often the exact same commands I used in my original post.

And many told me I was an idiot, and that they had no problem; and that I should learn how to use GIT instead of just stabbing at guesses.

Those made me realise what the fundamental difference between myself and them are.

I don’t want to use a revision control system.

It follows that if I don’t want to use one, I certainly don’t want to waste time learning one!  I don’t want to use a text editor either, again it’s just something that gets in the way of what I really want to do.

I want to write code, and I want to fix bugs and test the fixes.

Actually, I don’t really want to fix bugs; but it’s one of those things you have to do as a responsible software author and maintainer.  Now, I sadly can’t just write code by thinking about it.  Even though I use zsh, holding down the TAB key just doesn’t write my code for me, so I have to use a text editor.

And I hate them, especially emacs, which is the text editor I use.  I keep hoping that one day Anjuta or something like MonoDevelop will be usable for writing and debugging C – but alas, that day has not yet come.  (Incidentally, if a developer would like some constructive criticism from my side, I would be perfectly happy to do give it – just grab me on IRC).

Now I don’t have to use a Revision Control System, unlike a text editor which is somewhat essential, but there are a lot of advantages to using one.  In particular, I think I have two primary use cases:

1. Keeping a history of my changes
2. Sharing my work with others, and allowing others to share their work with me.

Since I don’t want to use a Revision Control System, it should be as simple and painless to use as possible.  I have probably a dozen common actions that I do; so each of these actions should be just one command, with no options or flags to make them do the right thing.  For bonus points, these commands should be obvious and ideally have aliases for the similar commands in other systems for easier learning.

Here’s the first set:

• get a copy of some source code, to which I can make my own changes if I wish
• save my current set of changes to the history
• put a copy of my source code, with changes, somewhere others can get at it
• update my source code with changes made to where I got it from

We could probably call these get, commit, push and pull. Note that I’m not allowing for any kind of add, move or remove command.  Such things are a pain in the neck; my most common preference is just for any changes I’ve made to be committed.  This is probably one big reason why GIT and I don’t get on so well; it’s written for somebody else’s use case where the most common use case is committing nothing.

Since I pretend to like other people working on my software, I’ll need a couple of commands for doing that:

• review the changes somebody else’s source code makes to mine
• add the changes they’ve made to my source code.

I’ve yet to see a revision control system that makes the first one easy, which is sad, because it’s just about the most common operation I do on other people’s branches/repositories/clones; I’d call that review, and then we can call the second one merge.

And then for the last set:

• look at the history of changes
• examine the difference between two versions
• extract a single change, in a format compatible with mailing

We can call these log, diff and show (though ideally diff and show would be just the same command with a revision range or single revision).

I’ve probably forgotten something important, but in general, those are the operations I try and do with a revision control system.  Thus I can create a Revision Control System Hate Index.  The more commands, options or flags it takes to do any of these operations, the higher the Hate Index and the more I’ll loathe it.

GIT scores badly, very few of these operations are a simple command.  Many require pre-configuration, many require options and flags to do the right thing.  And a couple of them are actually multiple commands.

In fact, GIT’s hate index gets increased further, because after a while you realise that most of the top-level commands very rarely do the right thing.  Instead, you start to break down the commands to their fundamentals.  If you’ve ever found yourself using git fetch you know what I mean.

This means that I have to stop thinking about writing code, and start thinking about using the revision control system and what the best way to accomplish the operation that I want is.  This is a fail that only results in hate.

And just for the peanut gallery, I’d like to point out that Bzr has a non-zero Hate Index as well.  In fact, there are misfeatures and bugs in Bzr that make my blood boil just as much as GIT, sometimes even more since Bzr actually claims to always try to do what I mean.

You know what would make me happiest?  If I had a text editor that took care of revision control for me.  Sometimes I wish there was a Linux version of the good old Borland IDE or at least VisualStudio.

Unfortunately I have to make a correction to my previous post.

In that post, I suggested that the following git command would work:

quest util-linux% git clone --bare . ssh://kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott/util-linux.git

I wholeheartedly apologise.  It turns out that what this command does is make the following local directory: $(pwd)/ssh:/kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott/util-linux.git.

Unfortunately this means that normal service of hating git, while still being utterly unable to find any reasonable command to do what I want, will have to resume.

All I want to do is run a command to take the branch I have in my local working tree, and put that on a remote server so that other people can get at it.

Failing that, I want sharks with frikkin’ laser beams on their foreheads!

Is either so much to ask?

Jason Clinton replied to my virtiolic rant about GIT with a well researched and thought out reply.  Sorry, that’s not what I mean.  He replied to my rant against a piece of technology with a personal attack.

I feel that there’s a few points that I need to reply to.

Romeo Tango Foxtrot Mike

I did, in fact, the vast majority of my post was excerpts from the Foxtrot Mike.  It was the Foxtrot Mike that was half of the problem.

and because the company you work for has a horse in this race and they can do no wrong

What does this having anything to do with anything?  The company I work for develops many pieces of software, and a quick bit of searching will find people who work for the same company (including myself) complaining just as loudly about them.

Sometimes, sadly, it seems that people in the open source world are unable to separate other people from the companies that they work for.  Ironic, given the high turnover, and frequent moves between the different companies that most people seem to go through.

I am not Canonical.  I am not on the Bazaar team.  I have been involved in open source since long before I joined Canonical, and I will almost certainly be involved long after I leave Canonical.

Most importantly, I am able to hold my own opinions, and I am able to form those opinions without influence from my employers.

If you don’t believe me, just go back a couple of years and look at some of my rants about Arch, and the first Bazaar.  Ask for a copy of my book, “Arch is easy, and other lies the developers may have told you.” It has a cool cover.

GIT reminds me a lot of Arch.

My personal opinion about this is that Arch (and now GIT) is the first distributed revision control system that people try, and then they get it.  They understand why distributed revision control is so awesome, and they attribute this awesomeness to Arch (and now GIT) rather than realising that it’s an inherent property of any such system.  The learning curve is pretty damned steep, so there’s a lot of investment to learn Arch (and now GIT) and once people have made an investment in something, and received an epiphany as an award, they become very attached to it and very aggressive about attacks on it.

and because you can’t be bothered to look up what you want to do first

Again, it should be clear from the post that I had spent a very long time trying to look up what I needed to do and found no help.

From the very same Git Tutorial that you were about to link to in a giant section titled Using git for collaboration you could, you know, look at the command you think you might want:

Have you actually read that?  I have, yesterday as it happens.  All of the examples are with bob and alice working on the same machine, it makes a brief reference to the two being on different machines but then assumes that bob can access alice’s workstation.

Utterly useless.

And since you’re a very smart person with a history of working with VCS’s and a resume a mile long in the FOSS community, you know that that’s a giant red flag that you’re about to force the tool you’re using in a mode that is not distributed.

And as you can clearly see, I was trying not to do so.  I was simply trying to push changes from my workstation to a server from which alice, err sorry, lamont could pull them.  If I am picking the wrong commands because of familiarity with other systems, then frankly GIT is being perverse in deliberately using commands in a different way.  Even BitKeeper used push for “make your changes public”.

You say that I’m using the wrong command?  Well, let’s try and figure out from first principles what command I actually want.  git help should tell us:

The most commonly used git commands are:
   add        Add file contents to the index
   bisect     Find the change that introduced a bug by binary search
   branch     List, create, or delete branches
   checkout   Checkout a branch or paths to the working tree
   clone      Clone a repository into a new directory
   commit     Record changes to the repository
   diff       Show changes between commits, commit and working tree, etc
   fetch      Download objects and refs from another repository
   grep       Print lines matching a pattern
   init       Create an empty git repository or reinitialize an existing one
   log        Show commit logs
   merge      Join two or more development histories together
   mv         Move or rename a file, a directory, or a symlink
   pull       Fetch from and merge with another repository or a local branch
   push       Update remote refs along with associated objects
   rebase     Forward-port local commits to the updated upstream head
   reset      Reset current HEAD to the specified state
   rm         Remove files from the working tree and from the index
   show       Show various types of objects
   status     Show the working tree status
   tag        Create, list, delete or verify a tag object signed with GPG

Any of those strike you as doing what I want?  I imagine that getting changes from your workstation to a web or git server is a sufficiently commonly used command, after all, even the kernel developers all publish their branches on master.kernel.org.

Well, it’s clearly not add, or fetch.  merge? pull? no. reset? status? no.

You know, reading that, I really might think that the right command is push.

The git(1) manual page gives even more commands, one of them looks really appropriate:

git-send-pack(1)
Push objects over git protocol to another repository.

The manual page on this instantly refers us back to push.

DESCRIPTION
Usually you would want to use git-push, which is a higher-level wrapper
of this command, instead. See git-push(1).

If push really isn’t the right command, perhaps the Foxtrot Mike could stop referring to it everywhere?

Now, it happens that I received some helpful replies as well.  You clearly know the right command, and even manage to mention it, but not in any kind of helpful way.  Another kinder soul was much more helpful.

clone is the right command, it can take two arguments and the first argument can be “.” while the second can be a remote URL:

quest util-linux% git clone --bare . ssh://kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott/util-linux.git

was what I actually wanted.  Curiously nobody in the #git channel seemed to know about this when I was asking for help.

Users coming from other revision control systems will probably equate clone to branch, or checkout or some other similar operation.  It probably won’t occur to them that they can reverse the arguments.  This should definitely be prominently documented.

And no, the GIT tutorial never gives this example.  It’s always clone remote to local.

Random aside, I discovered that bzr supports bzr branch local remote as well; neat.

Someone else tried to be helpful, but ended up simply being amusing.  They suggested that I use git format-patch to extract my patches and then send them by e-mail.

Can I just say that if your revision control system is so bad for collaboration that patches must still be sent by e-mail, then something is deeply, deeply wrong.

To finish up, in a comment on your own blog post you say:

All the documentation in the world won’t help if even smart people like Scott decide not to read it and demand that git work exactly like bzr does. Which is exactly what happened here. Again.

A quick bit of checking would show that the example I gave wouldn’t work with bzr either.  Bzr needs a silly “bzr+” on the front of the URL, there’s a lot of fail there too.

I do not like git.

Let’s take the most mind-numbingly trivial of operations, I want to put a branch I have somewhere so somebody else can get it.  That’s the whole point of distributed revision-control, collaboration.

That’s too fundamental to be covered in the git tutorial, after all, it wouldn’t be fun if it were that easy.

Happily, git has built in manual pages.  And knowing a few other revision control systems, we can guess that the command might be push.  And indeed, there’s a push command:

   push       Update remote refs along with associated objects

Maybe this isn’t going to be so hard after all, a quick git push –help and we’ll be laughing.

DESCRIPTION
Updates remote refs using local refs, while sending objects necessary
to complete the given refs.

Ok, err, not quite sure what that means; but it sounds like it’s doing something over there with what I have here.  Probably the tool I want.

You can make interesting things happen to a repository every time you
push into it, by setting up hooks there. See documentation for git-
receive-pack(1).

I’m not sure I want to know about making interesting things happen right now, I’ll just settle for some boring making-things-public.

OPTIONS
<repository>
The “remote” repository that is destination of a push operation.
See the section GIT URLS below.

Aha!  Now we’re getting somewhere.  I give it an argument saying where I want to push to, that all makes perfect sense.  A quick skip down to the URLs bit tells me I can use ssh, which is what I want.

There’s one other mandatory argument though.

<refspec>…
The canonical format of a <refspec> parameter is +?<src>:<dst>;
that is, an optional plus +, followed by the source ref, followed
by a colon :, followed by the destination ref.

Err?

*blink*

Whuuuuh?

My brain seems to have fallen out of my head, let me pop it back in and read that paragraph again.

<refspec>…
The canonical format of a <refspec> parameter is +?<src>:<dst>;
that is, an optional plus +, followed by the source ref, followed
by a colon :, followed by the destination ref.

It didn’t get any better the second time.

Some faffing around, guess-work and cargo culting what other people do seems to suggest it wants the branch name there.  Well, why didn’t it say so?

Ok, that should be easy then.

quest util-linux% git push ssh://kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott/util-linux.git ubuntu
fatal: '/srv/kernel.ubuntu.com/git/scott/util-linux.git': unable to chdir or not a git archive
fatal: The remote end hung up unexpectedly

I’ve had a few ex-boyfriends hang up on me before, and they were gits too.

There’s nothing in the manual page about this.

There’s nothing in the tutorial.

What the holy crap is going on?

Turns out, you need to create the other side first.  Why the buggering bollocks can it not to that for you?

So, let’s try that.  There’s some clues about this in the git init manual page, but I have to be honest and I begged for help (and mercy) at this point:

quest util-linux% ssh zinc
zinc scott% cd /srv/kernel.ubuntu.com/git/scott
zinc scott% GIT_DIR=util-linux.git git init
Initialized empty Git repository in /srv/kernel.ubuntu.com/git/scott/util-linux.git/
zinc scott% exit
quest util-linux% git remote add zinc ssh://kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott
quest util-linux% git push zinc ubuntu
Counting objects: 10652, done.
Compressing objects: 100% (2545/2545), done.
Writing objects: 100% (10652/10652), 19.10 MiB | 12192 KiB/s, done.
Total 10652 (delta 8075), reused 10534 (delta 8013)
To ssh://kernel.ubuntu.com/srv/kernel.ubuntu.com/git/scott/util-linux.git
 * [new branch]      ubuntu -> ubuntu

Hurrah!  At last!

No.

Not at all.

Firstly, pushing to a remote branch doesn’t make it visible in anything like gitweb.  You have to do enable that.

quest util-linux% ssh zinc
zinc scott% cd /srv/kernel.ubuntu.com/git/scott/util-linux.git
zinc util-linux.git% chmod +x hooks/post-update
zinc util-linux.git% git update-server-info
zinc util-linux.git% exit

Ok, now does it work?

No.

Looks all ok, but if somebody tries and checks that out:

wing-commander scott% git clone git://kernel.ubuntu.com/git/scott/util-linux.git
Initialized empty Git repository in /home/scott/util-linux/.git/
warning: remote HEAD refers to nonexistent ref, unable to checkout.

What the hell?

How can a push command succeed while leaving an utterly useless branch?

Don’t bother trying to find this one in the manuals:

quest util-linux% ssh zinc
zinc scott% cd /srv/kernel.ubuntu.com/git/scott/util-linux.git
zinc util-linux.git% rm HEAD
zinc util-linux.git% ln -s refs/heads/ubuntu HEAD
zinc util-linux.git% exit

Now it works.

In all that is holy, what, the, fuck.

Seriously?

My quote of the day:

“Git is hard to use if you are not used to its workflow.”

No.

Git is hard to use.

Here’s what I wanted to do:

quest util-linux% revision-control-system push ssh://host/path/somewhere

If it takes anything more than that, it’s fucked.

A complete surprise to me, from slides of today’s OSiM Maemo Developer Session it appears that Maemo (the Nokia open source Internet Tablet platform) has adopted Upstart.  Does anyone know whether they are using native jobs or still using SysV compatibility?

In response to my blog post “Whatever you do, don’t fix the kernel!“, David Zeuthen (prominent plumber, the maintainer of HAL and author of DeviceKit) wrote:

Scott, here’s why you’re wrong. It’s very simple and comes down to two points

- you obviously agree we can’t break huge amounts of userspace by changing DEVPATH

- having two names emitted from the kernel (_just_ because lots of user space is
broken) is just wrong and confusing
=> much better to fix up things in user space

Besides, what’s in a freaking name _anyway_? Apps should be using stable symlinks or, gosh, a device enumeration framework like HAL or the upcoming DeviceKit.

He makes, I think, an interesting point.

Why do we have two names for devices?

The kernel maintains its own namespace for devices which is based on its tree of internal objects and exported via the sysfs virtual filesystem.  My mouse’s object path is /devices/pci0000:00/0000:00:02.0/usb2/2-4/2-4:1.0/input/input2/mouse1 (convention is to omit the /sys prefix), and a class device link also exists as /class/input/mouse1 for easy access.

Properties of the device object are exported into user space via the sysfs filesystem, at that path under its mount point, and announcements of new objects, and significant changes to or removal of existing objects are made through the uevent system.  One of those properties is details about the device node that needs to be created; the dev file contains the major and minor number, and these are also present in the MAJOR and MINOR keys of the uevent.

The udev daemon listens out for these uevents and creates device nodes under the /dev path for userspace to use. These device nodes have a naming scheme that is mostly flat, with some sub-directories used for grouping. It records both the kernel object name and device path in its database so that lookups can be performed on device removal, and queries by applications using DeviceKit.

It also passes on the uevent to HAL, which stores the mapping in its own database and performs its own actions.

Applications can then use DeviceKit or HAL to enumerate devices by type, or walk the tree of devices, and lookup the actual device node path from that. They may also use them to lookup the object information for a given device node path.

That’s quite a lot of work going on behind the scenes already to map between two different names for a single device, all my proposal tried to do was reduce a little bit of the user-space side of that work by harmonising the names.

But maybe David has a point, the real problem is that we have two names in the first place!

Obviously the /dev paths are necessary for the vast number of userspace applications that still require them. But why do applications that use DeviceKit or HAL need them?

If the kernel placed a device node inside the sysfs filesystem, we wouldn’t need to do any path lookup, we’d just append the fixed name of the node to the object name.

udev would only need to make symlinks to those devices in /dev for legacy applications.

The automotive industry, with its particular emphasis on efficient workflow and practices, has had a lot to teach the software world over the years.  From the process of requirements, specification and design through to LEAN development practices, it is difficult to argue that we haven’t learned anything from them.

I think that there’s another practice from that industry it might be fun to adopt: the Concept Car (sometimes known as a Show or Halo Car).

These are cars where the designers and engineers have been allowed to let their imaginations run wild, and build something that shows off the limits of what’s possible.  Often they’re also used to explore new technologies or ideas without having to commit to standards of production that would be required for a marketable road car.

And that’s pretty much the key point about these cars, they normally build just one or two and take them around the car and motoring shows for everybody to look at and talk about.

Obviously I’m not suggesting that we build strange and outlandish cars, and drape them in fancy lights and scantily clad people on a slowly rotating podium; but I think the idea can translate to our world.

Thus I’d like to humbly introduce my idea of the Concept Distro.

The Concept Distro would be an engineering project to allow developers and maintainers to let their imaginations run wild.  It’d be released, probably to demonstrate at a major event, and would explicitly not be supported.  Not even basic security support, or a bug tracker, or even answering questions about why things don’t work.

On a Concept Car, it’s entirely normal that half of the doors don’t even open; likewise in the Concept Distro, it would be entirely expected that half of the icons were just placeholders and didn’t do anything if you clicked on them.

After release, engineering effort could be focussed either on integrating the successful technologies into Linux distributions proper, or on working on the next Concept Distro for the next big event a year or two down the line.

In the early days of Ubuntu, when we had two different CDs, we had a plan to do this kind of thing with the Live CD.  Since that didn’t have an installer, it could be a little more experimental and a little more risqué.  It was a good place to try out Network Manager before we integrated it with the distribution proper, and the intent was that the naked people would have had even less clothing (I didn’t mind the loss of this, the male model they picked was not the prettiest of the options).

Assuming we don’t resurrect the naked people, what kinds of things would we do with the Concept Distro?

It’s a chance to make some fundamental changes without having to worry about the support or upgrade implications of them.  I’d like to see what we could do by assuming that the filesystem is a single mount of ext4 on LVM on RAID, which we grow onto additional disks as they are made available.

And since we wouldn’t have to worry about partitioning, it might be interesting to look into rearranging the hierarchy.  Maybe having /System, /Software and /Users really is better than /bin, /usr and /home.

If we went down that route, we could throw out the traditional package manager and experiment with some new approaches.  What better way to upgrade the operating system than:

cd /System
bzr update

or switch to a new version with bzr switch?  It works well enough to upgrade my WordPress installation, after all.

From a technology fetishist point of view, there’s plenty to play with and try out.  Would we use ALSA and dmix instead of PulseAudio?  Assuming we didn’t use the Concept Distro to try out going fully volume control per application, of course.  It’d be a great place to see what we can do with Upstart, udev, D-Bus, DeviceKit (replacing HAL)  and other plumbing-layer components.

In the desktop library layer, the bling guys could play with Multi-Pointer X with kernel-mode setting support and a resolution independent GTK+.  Rendering could be fully indirect or entirely direct GL based, depending on preference.

And for the desktop itself, the user experience and interface designers have a completely blank canvas to play with.  Since it’s just a Concept Distro, one needn’t worry about the ability of users to transition to new ways of working.  Instead you can see how they react to seeing new ways of working in a demonstration or talk, perform usability testing in the lab and even see how they get on in the field.

It would be a very fun and exciting project.

Unfortunately, unlike the car world, there’s not necessarily the funding for such a thing.  Who would want to finance an ongoing software development project that was explicitly intended to have no users?

In the automotive world, the Concept Car from a development point of view is important since companies cannot, for example, experiment with new engine technologies and expect their customers to be able to drive them on the road.  In the software world, such “lab” projects are much easier to develop in isolation and tend to remain on our own workstations.

The Concept Car can also serve as a marketing tool, it draws potential customers to your show stand and while looking at the sexy car on the stand they’re ripe for being sold a somewhat more pedestrian road car.  It also aids towards customer loyalty, since you’re more likely to buy another car from a manufacturer who is showing off the most advanced concepts.

In the Linux world, while we appear to have direct competitions between the distributions, the reality is that we co-operate far more than you might expect unless you’re involved with development.  A Concept Distro would need upstream work from just about everybody.

And would such a thing help convert people from Windows or Mac OS?  If it would, maybe it’s a good idea after all.

As you may have read in LWN (subscription required, and strongly recommended anyway), there’s been some argument on the linux-hotplug mailing list, the historically named home of udev development, about device naming.

The key threads are “default udev rules” and “Patches for device names“.

It all started when Kay reminded everybody that distributions should attempt to drop their own udev rules in favour of those supplied by upstream.  For those not familar with udev, the rules are a language that creates device nodes and performs other actions based the information about that device from the kernel.  A typical rule to put all devices from the “sound” subsystem into the “audio” group looks like:

SUBSYSTEM=="sound", GROUP="audio"

Sometimes these rules also change the names of the devices.  For example rules such as the following are automatically generated to keep the name of your ethernet devices the same between reboots:

SUBSYSTEM=="net", ATTRS{address}=="00:11:22:33:44:55", ATTR{type}=="1", NAME="eth0"

Ironically, perhaps, none of the argument is about the names of the devices, the permissions assigned or the groups they’re placed in.  We’re all pretty much in agreement about that.

Every major distribution pretty much follows the plan laid out in the devices.txt file found in the kernel’s Documentation sub-directory.  This is maintained by the Linux Assigned Names and Numbers Authority, and up until 2.2, was included by reference in the Filesystem Hierarchy Standard (FHS).  Nobody really knows why the reference was removed, I guess the LSB didn’t like having standards everybody agreed on

So what is the argument about?  Marco d’Itri, the Debian udev maintainer, is arguing because he’s spent a lot of time and effort making their rules readable and elegant in their operation.  The upstream rules are, in his opinion, somewhat scraggy.  I don’t really see this as a problem, we can fix the upstream rules to be more elegant easily enough.

My argument is different, and is a little more fundamental.

While most of the rules do udev-specific things like permissions, groups, run callouts to gather more information and perhaps run programs after device creation, we have many rules such as this:

KERNEL=="hw_random", NAME="hwrng"

What that says is:

Rename the kernel device “hw_random” to “hwrng”.

This makes the device name correct according to devices.txt.  What irritates me about this is that this rule should be entirely unnecessary!  It would be a one line patch to the kernel to cause it to name the device properly in the first place.  Then we wouldn’t need to spend the resource and CPU time changing the name every single time every Linux machine around the world boots.

There’s another set of rules that annoys me:

KERNEL=="device-mapper", NAME="mapper/control"

The kernel object for the device mapper control node is /sys/class/misc/device-mapper, but the device name according to devices.txt should be /dev/mapper/control – in a sub-directory. The kernel and udev have a mechanism to deal with this, the kernel object could be named /sys/class/misc/mapper!control and the right thing will happen.

Another similar class of devices needs udev to rename them:

KERNEL=="mice", NAME="input/mice"
KERNEL=="mouse[0-9]*", NAME="input/mouse%n"

The first one seems straight forward, but the kernel object is named /sys/class/input/mice so if we used the ! trick, it would become /sys/class/input/input!mice. I can appreciate that it’s ugly. Similarly for the mouse case.

I’ve suggested a fix for this though, and this fix also alleviates any concerns about backwards compatibility with sysfs names. The uevent from the kernel for the “mice” device looks like this:

ACTION=add
DEVPATH=/devices/virtual/input/mice
SUBSYSTEM=input
MAJOR=13
MINOR=63

I’ve suggested where the device ends up in a sub-directory, adding an extra field to this:

DEVNAME=input/mice

When present, udev would use this instead of the last part of the sysfs path as the kernel name. The extra cost to the kernel is a single %s in an existing sprintf() call, the result, a vast saving in userspace time.

This fix would also let us deal with the raw USB devices, and other things like the DVB devices, where we have to construct the device names. For example, the following rule is used to name DVB devices:

SUBSYSTEM=="dvb", PROGRAM="/bin/sh -c 'K=%k; K=$${K#dvb}; printf dvb/adapter%%i/%%s $${K%%%%.*} %%{K#*.}" NAME="%c"

That means that for every DVB device, on every computer, every time Linux boots, we have to fork and exec a shell, do some string pattern matching, fork and exec printf and apply more string pattern matching to the format string to name the device.

This could be avoided by doing that printf in the kernel, and setting DEVNAME for that device.

Device names are set down in a standard. That standard is shipped inside the kernel’s own source tree. Most distributions are already following that standard. The udev default rules follow that standard. Most distributions are likely to adopt the default udev rules. This is, for all intents and purposes, as official naming policy as you can get.

For those devices where the name is static, or constructed entirely from information from the kernel (ie. not persistent storage, input, network, etc.); why do we waste resource and CPU time every single boot changing the name that the kernel exports to match the standard?

To me this is obvious, fix the kernel to export the right name in the first place.

To kernel developers, such as Greg K-H, this is not so obvious:

“Wait, why do this at all?”

and

“Can’t you live with input devices having a few rules in udev? Is it really that hard to maintain? ”

While patches were apparently welcome in the first thread, by the second thread when it was clear that patches were going to be done, they didn’t seem quite so welcome after all.

This isn’t the first time that I’ve seen kernel developers claim that it’s better to work around the kernel in userspace than it is to fix it. I could understand this if we didn’t have the source code to our own kernel, but we do.

The kernel isn’t sacred and it isn’t a separate part of the system. It needs to be seen as just one component of a fully integrated system, especially by its developers.

That 12ft-high wall between “kernel space” and “user space” needs to come down.

As LWN notes, we have a lot to talk about at the LPC in September.