Many thanks to racarr for posting about this earlier today.  I had seen posts on the ‘Do-ifying’ of GTK but they had passed me by. ‘Do-ifying’ applications is a fantastic idea. We shouldn’t limit ourselves to GTK apps though.

The bit that caught my interest was that racarr was thinking of exposing a UI heirarchy over D-Bus to allow Gnome Do to find key-bindings and make them available. I suggested that AT-SPI would be a better way to do this, as all key-bindings are already exposed. I come to no conclusions over whether A11y is the best way. I’m biased. 🙂 It did get me thinking about how A11y technologies could be used to provide general application assistance with Gnome Do.

Context sensitive ‘Do’ commands

Orca, the gnome screen reader, uses a11y mainly by responding to ‘focus’ events which are emitted when input focus moves to a new widget. Gnome Do could use ‘focus’ events to provide context-sensitive commands. By knowing which application, and which part of an application the user is currently focused on it might be possible to provide a more tailored selection of commands.

Perhaps the choice of ‘Do’ commands for each application would depend on finding out what D-Bus interfaces it supports.

Application generated ‘Do’ commands

Obviously for application generated commands we need methods of exposing these to Gnome Do. My preference would be to do this via D-Bus, which again a11y could help with. All ATs (Assistive technologies) need to know when new accessible applications appear on the desktop. Towards this end, D-Bus AT-SPI has a daemon that acts as an application registry and informs ATs when new applications are added. This registry could be made more generic so that Gnome Do would listen for new applications and register their ‘Do’ commands when they are started.

I can’t describe the registration interface here. I don’t know Gnome Do. I’m imagining a list of actions with a command name and descriptions, but it could well be much much more complicated.

Key Bindings

As racarr suggests, it should be possible to inspect the applications UI hierarchy to find all the key bindings and present them and their descriptions to Gnome Do. GOK, the gnome on screen keyboard, already does something VERY similar to this by inspecting an application and presenting all the actions of the currently focused window in a simple manner. There may be other things we can infer from the UI hierarchy, but I wouldn’t want to get too ambitious here.

I’ve often thought that a11y technology was under utilized. As applications on our desktop are already merging together a little by providing and accessing more D-Bus services, a11y seems like it could be well placed to enable some really innovate interfaces for the general user. If this happens it could really help experiences for accessibility users too. More bugs fixed, more descriptions added to widgets with key-bindings. 🙂