And if you're an application/toolkit developer, it's somewhat your problem.
You guys really seemed to enjoy my previous article about how I reject artificial intelligence. Unfortunately this article won’t be quite as spicy. But I still do have a slight bone to pick about something. So if you’ll entertain me, I’d like to talk about flashing rectangles this time.
Let’s zoom in for a moment
And I mean quite a lot, and quite literally.
Lots of desktop environments and operating systems provide some kind of zoom feature. Even game consoles have it now. It’s a really awesome feature for those of us with severe sight loss, when done well.
Generally, on desktop, zoom is for the most part controlled with your mouse. The
problem I’ll describe is worse on mobile devices for different reasons, but I’d
like to focus just on the desktop for now. In general you zoom in and out using
a keyboard shortcut (Meta+= and Meta+- on KDE Plasma, Meta+0 to
reset it). You move around the zoomed desktop by moving your mouse. Set it up
right, get used to the controls, and it can be really natural.
There is a problem with needing to zoom in lots. It should be obvious once you’ve done it, but the closer you zoom in, the less of the screen you’re able to see at once. For me to be comfortable, I need to zoom in almost to Plasma’s maximum zoom level. Unfortunately, for me to get sharp and readable text, I have no choice but to sacrifice only being able to see a small part of what’s actually on my screen.
Which begs the question… what is typing up one of these articles like?
I can type one-handed
They tried to teach me standard two-hand touch typing at school, but I just never picked it up properly. It never felt natural to me. This is because I’m used to resting my right hand on Backspace and Shift, and typing most letters my left hand.
While this was just the way I naturally learned how to type, when my sight got worse it became genuinely necessary to type like that.
At the time, the Windows magnifier wasn’t capable of caret tracking. It couldn’t follow the typing cursor. So I got used to needing to use my right hand to pan around as I type.
I don’t like that. It’s annoying.
Where I’m at with caret tracking today
As of Windows 10, Magnifier supports caret tracking and does it by default. Great. But the weird thing is, I always turned it off.
At first it was because I just didn’t need it. I was used to my weird typing and panning habbit.
But lately I’ve been working on KDE Plasma’s caret tracking in the zoom effect. It is honestly so much better for me, though…really needs some love.
On Windows, caret tracking causes the magnifier to center directly on the text cursor. In other words, the text cursor always stays in the exact center of my screen. That sounds reasonable, but for me it means I can’t see as much of what I just typed. I like to proof-read as I write, to make sure I didn’t make typos. It is best if the word I’m working on is in the middle of my view.
In Plasma, the way caret tracking works depends on what your current mouse tracking behaviour is set to. If mouse tracking is set to Proportional, then so is caret tracking. And if mouse tracking is set to Centered or Strictly Centered, then caret tracking behaves like Windows. And if you have mouse tracking set to Push, then caret tracking only kicks in when the cursor goes off-screen. I think that it is a very good idea to have different caret tracking styles, but having it depend on the mouse tracking setting is something I’d really like to change.
Where it’s your problem (if you develop GUIs)
Just like screen reader support and labelling, caret tracking isn’t magic. It is on your application’s UI system to provide caret tracking information to the desktop environment.
And there are ways to get it wrong. I’d know, because me fighting them is why I’m writing this.
An in-my-opinion-naive approach to implementing caret tracking is telling the operating system every time the text caret moves visually. This is what Rider and CLion do on my computer, and it often causes problems.
There are generally three actions I can perform that cause a text caret to visually move. The first one, obviously, is typing - you want that. The second is clicking somewhere in the document with a mouse or tapping with a finger. The third action, and the one that causes me problems, is scrolling the page.
You generally always should report caret tracking changes when the user types something or otherwise moves around the page with their keyboard, That would be the entire point of caret tracking; so we don’t need to take our hands off the keyboard.
Caret tracking on a mouse click or finger tap is redundant. Whether you report the movement or not will still change how the OS reacts, and you should report the position when a widget receives text focus (for a different reason), but… generally if I’ve clicked on a thing, it is already on my screen.
You should never report caret position changes when the page is scrolled. If your application or toolkit does this, it is a bug in your user interface. This is because, if the caret goes offscreen or otherwise triggers caret tracking, scrolling will move the zoom area. It is as though the screen magnifier becomes glued to the text cursor, preventing me from seeing where I’m scrolling toward.
These are things we can filter for in KWin, but not every compositor is KWin.
There are also many apps that straight up do not provide caret tracking support for whatever reason. While I don’t expect every bedroom GUI toolkit to be fully accessible to the blind because it is genuinely a difficult problem to solve, I would really appreciate the FOSS community endeavouring to improve the current state of caret tracking on Wayland. KDE applications can do it, so can my IDEs, but my browser and many Electron applications simply don’t support it properly.
Multiple monitors
Did you know desktop zoom with multiple monitors is extremely hard to get right? Do it poorly, and panning around the desktop becomes very difficult when your monitors aren’t exactly lined up.
Caret tracking is absolutely no exception.
Screw up the position of a text caret in relation to a popup window parented to a toplevel? You can send the caret tracking off into fuck knows where and I can’t see what I’m typing at all. Even if I move the zoom back to where the input field is, the moment I type? Back into the void I go.
Windows deals with the multiple-monitor caret tracking problem by picking one monitor and keeping the caret in the middle of that monitor. You can’t change which monitor it is easily if at all. On my Windows system, it chooses my second (smaller, harder to read) monitor instead of my much larger primary monitor. The solution Microsoft went with for that, in my opinion, is incorrect.
The exact opposite end of that spectrum would be KWin, which doesn’t do anything special with respect to caret tracking and multiple displays. With Push mode, if your goal is to keep things on a larger display like me, caret tracking can be…somewhat problematic. I’m working on that.
My idea is to keep caret tracking focused on the monitor the active window would normally be visible on. That is, if your browser is on the left monitor when you zoom out, and you focus the address bar, then the address bar should be visible on the left monitor when you zoom in and type inside it.
It’s worse on mobile
It may be a surprise, but…phone screens are really tiny. For someone with sight loss, this might mean needing to use a screen reader when we can otherwise get away with huge fonts on desktop.
But…you can still zoom in on most phones, and I find that incredibly helpful when typing.
The problem is we tend to hold our phones vertically, and so the screens are tall rather than wide. This means that, at maximum zoom level, which is where I need to be when typing… there’s even less room to work. So caret tracking is a billion billion times more important to get right.
For one, touch screens are a lot more sensitive to accidental wrong moves. To pan the zoom on my phone, I have to drag around the screen with two fingers. To zoom in or out, it’s a standard pinch gesture like you’d use in the browser. To toggle zoom on and off, I tap the screen three times really fast. The touchscreen often misinterprets these gestures and it’s been like that on every phone and every tablet I’ve ever had. So the more I have to mess with the zoom, the more likely I’m to accidentally click something or otherwise screw up whatever I’m trying to work with.
And yes, even on mobile, caret tracking is spotty and often broken. I often have problems with React Native applications.
The weird world of game engines
Aside from accessibility stuff, my main interests and most of my skills involve video games.
Since this article was about caret tracking and desktop zoom, this is an open letter to all video game user interface developers.
Please do not implement in-engine mouse cursors. Games often like to capture the mouse cursor and hide it, which makes sense during gameplay so you can control the camera. But in menus, and in point-and-click games, developers often enjoy implementing custom cursors. If you do, please use your platform’s API to show a custom cursor instead of trying to re-invent a mouse cursor in your game’s engine.
There are 4 titles that come immediately to my mind that are fundamentally broken when using a screen magnifier, to the point where I can no longer properly play them on PC.
- Prototype (2009)
- Prototype 2 (2011)
- Grand Theft Auto V (2015)
- Grand Theft Auto: San Andreas (2004)
These games prevent screen magnifiers including KWin’s from knowing that the mouse cursor has moved, preventing us from panning around the screen while the game window has focus. You therefore can’t navigate menus or read HUD elements.
There are 4 games that come to mind as being very mouse-heavy whilst NOT breaking assistive technology:
- Hacknet
- Orwell
- Cities: Skylines
- The Sims 4
I’m also really sad Cyberpunk 2077 is one of the ones that belong on that first list… It’s my favorite game. I miss playing it.