Upstart 0.5: Events
In the previous posts, I’ve covered the various features that make Upstart a good service manager, but these are things you’ll find in most others as well. It’s now time to cover that which is singularly unique to Upstart, Events.
Start and Stop
You’ve already seen the start and stop commands, which do somewhat unsurprising things to jobs. The important thing to remember about these is that they are not events. I just wanted to clear that up before we start, since it’s often been a source of confusion not helped by the design of some earlier versions of Upstart.
start and stop operate directly on jobs, and the command will not normally return until the operation is complete or otherwise interrupted. Services are considered complete when they are running, Tasks are considered complete when they have stopped again; in both cases the stop command is complete when the service or task has actually stopped.
This is important since it provides a common-sense behaviour, ensuring that the following operation is not a race condition:
# start apache
apache running (start), process 3591
# wget http://localhost/
Solving race conditions is one key part of Upstart’s purpose.
Both commands may also set environment variables, those set by the start command form part of the environment of the job itself and those set by the stop command are available to the pre-stop script.
# cat /etc/init/jobs.d/getty
instance $TTY
env SPEED=38400
exec /sbin/getty $SPEED $TTY
# start getty TTY=tty1
getty (tty1) running (start), process 4152
Events
As described above, the start and stop commands are admin instructions that act directly on named jobs. Events have many similar properties: they carry environment variables that end up in the environment of jobs they start, and they are not complete until the jobs that they affected have been started or stopped as appropriate.
The difference is that the start and stop commands are targeted at specific jobs, whereas events have no such targetting and instead it is jobs that specify which events they are interested in.
In the Upstart world events serve three general purposes: they act as signals of state changes that jobs can react to (e.g. hardware going away), as method calls to automatically start or stop jobs (e.g. shutdown) and as a way of passing information between jobs.
Events are identified by their name and have a different namespace to that of jobs. They are emitted by a D-Bus call or by using emit on the command-line, naming the event and providing any associated environment variables you wish:
# emit interface-up IFACE=eth0 ADDRFAM=Ethernet ADDRESS=01:23:45:67:89:0a
Jobs may match them on this name and any number of their environment variables, specifying whether the event would automatically start or stop the Job.
start on interface-up IFACE=eth* ADDRFAM=Ethernet
As a short-hand, where the order of the variables for an event is fixed, the names may be omitted:
start on interface-up wlan*
When a job is started by an event, the environment for that event forms part of the environment for the job and may be used when matching events that can automatically stop the job. Harking back to our getty job from previous posts, we can bind this to the lifetime of the underlying device.
start on tty-added
stop on tty-removed TTY=$TTY
instance TTY
exec /sbin/getty 38400 $TTY
We can also match multiple events, either requiring that both occur or either using unsurprising operators:
start on a-up and b-up
stop on a-down or b-down
In these situations, once stopped, both the a-up and b-up events must happen again for the job to be restarted.
Upstart Events
Upstart itself only emits a few events, leaving the rest up to application authors to define. The startup event is the most interesting of these, and is ultimately what nearly all jobs get chained from.
Job Events
As jobs are started and stopped, Upstart emits events on their behalf for four key points in their lifecyle.
- starting is emitted when the job is first starting, and the job will not actually be started until this event completes.
- started is emitted once the job is fully running.
- stopping is emitted when the job is stopping (after the pre-stop has completed), the job will not actually be stopped until this event completes.
- stopped is emitted once the job is fully stopped.
All of the events have the name of the job in the first variable, JOB and the instance of the job (if applicable) in the second variable, INSTANCE. The stopping and stopped events then have a series of variables indicating the reason for the job stopping: RESULT indicates whether it was a normal stop or a failure then if it failed, PROCESS will say what failed and EXIT_SIGNAL or EXIT_STATUS will contain the terminating signal or exit code.
For example, we can take action to backup a database if the server crashes:
start on stopping hersql RESULT=failed EXIT_SIGNAL=SEGV
task
exec hersql-backup
Jobs can also export variables from their own environment to others through these events by using the export stanza:
start on interface-up
stop on interface-down $IFACE
instance $IFACE
export IFACE
exec ...
Another job may then be started along with this one, and know what interface it’s bound to:
start on started JOBNAME
stop on stopping JOBNAME
instance $IFACE
We’ll look at the various powerful forms of dependency that these events allow us to express in the next post.
The only annoying this about upstart as-is (I can’t comment on upstart 5 which is upstart as-it-will-be) is I have literally no clue which of these keywords are in the version of upstart in my distro (which is version 0.3.8), or indeed which other keywords NOT described are there, or what they might do.
While there is a very good getting started guide on this webite, it is pretty much the only documentation that exists, and it _IS_ only useful for getting started, not finished. “man event.d” does nothing, “man upstart” does nothing, “man init” leads via the “see also” section to initctl, which is interesting but also doesn’t help in writing the things. The example jobs are of the very simple kind and are similarly unhelpful. (The existance of a pre-stop script and a post-start was inferred from the existing examples, but aren’t explicitly mentioned anywhere)
Just from what has been casually dropped in here and on I know there’s a “task” keyword, but have no idea what that means. There are tantalising hints to some kind of logging facility (which is what I’m tearing my hair out over at the moment) but I have no idea how to invoke it, or even to redirect output somewhere sensible, other than redirecting at the end of the line calling the program (which then triggers that app’s very annoying let’s-buffer-1-screen-at-a-time mode). If I put the app at the beginning of a pipe, stopping the job kills the shell that’s calling the program without killing the program itself and I have no idea how to fix it. I have no clue what events might fire on my system, or what might decide to fire them, and have no systematic way of finding out. I have no idea how to carry state information forward from pre-start scripts to the start script itself, other than creating tons of temp files.
In short, I need documentation
Hi,
I want to trigger a script when a device (.f.ex. usb dongle, usb bluetooth etc) i removed. I read that upstart can be triggert on these events (emitet by the kernel, hal or dbus) but i do not know the name “?” of the event to trigger on. Could you help me out there? /giko
Thanks, Scott, for the great introduction! I’m using upstart to run various fastcgi processes in Hardy.
Is this the only/best documentation available for writing jobs (other than the source I mean)?
All of this is very much like the room full of mouse traps and balls. Great technology, but almost non-deterministic in how whatever happens. I’m thrilled that it mostly works. However, some of us have parts that don’t work. Sadly, we lack enough information to (1)instrument our situation, (2) gather details about whatever is going on, (3) seek help by sharing what we know and asking questions, and (4)fix our individual problem and share the solution.
I’ve done some very sophisticated embedded system event driven applications. For what I know about upstart, it will be a wonderful improvement. We need help to watch the bouncing balls and work with this thing.
~~~ Saint 8d;-)
Hi Scott,
A question. Suppose I have a service that uses lots of wrappers to start. I use a “script” instead of an “exec” to start it, and from that script I call the topmost wrapper. So it is going to fork a few times before it actually starts the service. When I call stop, the “script” gets killed, but not the service.
Worse still, my service requires special handling to be stopped. I can’t just send it a kill, I need to run a command. Suppose it’s a kernel module or a driver that I’m running.
How can I specify the means to stop the service? Is there a “stop” section for that, as there is “script” in addition to simply “exec” to specify how to start?
Or should I let Upstart simply kill the “script” section and do the actual killing in post-start?
I don’t want to employ dirty hack. What is the intended way?
Thanks!
Alex
In this corner, we have everything that someone has programmed or configured our current box to respond to. This is represented by the config and script files under /etc and in some cases under /opt. [If elsewhere, please explain.]
In the other corner, we have “events” that are POSSIBLE because of (1)our hardware, (2) our behavior, (3) installed software packages, (4) our behavior, er — use of installed software.
QUESTION: How do we learn — other than source — about the list of POSSIBLES? Are we forced to read source code for device drivers? Can ’strings’ or similar help us inspect binaries? If so, what are we looking for?
RATIONALE: I have this workstation. It has all sorts of knobs and buttons. Some fire “events”. Others do not fire “events”. What causes/configures Fn+F7 to throw an “event” while “Alt+F7″ does nothing?
Wanting to learn. Willing to write!! Needing information.
~~~ 0;-s
The current lack of documentation is not helped by an unfortunate confusion between events and jobs. When running “ls /etc/event.d/” one would expect to see a list of events, don’t you think? Instead one sees the list of jobs.
In man events: “upstart works on the basis of event files that describe jobs that should be managed”
How come a file holding the entire description of one and only one job is called an “event” file? Wouldn’t be “job file” more intuitive?
I can imagine that changing a directory name would be too disruptive. But please at least fix the terminology in the documentation.
I’m wondering what the “best practices” approach is to running scripts _for_ other users?
I went to a great deal of difficulty to get a device that doesn’t require me to be root to init it…and I’m seeing several different ways suggested to pull it off.
Is there a facility for a user’s job (as opposed to a root job) to be called out there?
@Brian Fahrlander:
I had to implement something similar (I wanted a web interfact to start/stop certain jobs – namely, multiplayer game servers)
I wrote a helper script that would accept 2 parameters (jobname, and “start” or “stop”) and, if the jobname matches a list, would run the specified command. You can’t make scripts setuid, so instead I added the following line to /etc/sudoers:
%www-data ALL=NOPASSWD: /path/to/script
that lets anyone in the www-data group run my script as root using sudo, without a password (I needed that because it was being ran by web scripts; you probably don’t)
I’d imagine this is less than idea for your purposes (I assume you want users to be able to add their own jobs, and you also want the jobs to run as that user, rather than root)
Frymaster, I have Fedora 12 with upstart version 3.11 (and a bunch of fedora-specific patches). I had a look at the code and understood that actually, the present documentation says it almost all. Disclaimer: I only had a quick look at a few files, and may well have misunderstood things.
The upstart program (“init”) emits a short list of events:
startup
stalled
control-alt-delete (formerly ctrlaltdelete or something similar)
kbdrequest
starting
started
stopping
stopped
(The last four relate to jobs. The first, startup, relates to init itself.)
The telinit program emits events named “runlevel”. (Formerly runlevel-/n/.)
The initctl program emits events with arbitrary names specified on the command line. This can be used in various scripts under /etc, e.g. on my Fedora12,
/etc/gdm/Init/Default:initctl -q emit login-session-start DISPLAY_MANAGER=gdm.
Events don’t just have names, they have also arguments – a list of arbitrary strings, and “environment variables”. Job files specify “start on xxx” and “stop on xxx”, where xxx are specifications that can match a number of events. I suppose that when an event happens init checks all such start/stop on specifications to see if they should be triggered. The name must match exactly, the following words are “file globs” that are matched against successive arguments of the event. Events with too few args do not match.
There is just one thing I have found in the code that I had not become aware of reading the docs: If an event emission is considered “failed” (I am unsure what exactly the criteria are) a new event is automatically emitted with “/failed” appended to the name, and with the same arguments and environment. (If the “name/failed” event fails no new event is emitted.)
I suggest something is added to the documentation in the spirit of this reply, to give the reader a feeling of having a handle on the set of possible events and where to look for user-defined events.