This resource explores various AI tools for creating image descriptions.
Subject(s):
Artist Resources, Image Descriptions
Resource Type(s):
Tools
Audience:
Introduction
Before reading this, you might want to read:
Generative artificial intelligence (AI) has exciting implications for digital accessibility. As the technology develops, we expect AI to address a variety of issues that impact users with disabilities. However, with any new technology, it’s important to understand the limitations — particularly when you’re trying to improve experiences for real-life users.
Recently, OpenAI began rolling out image recognition capabilities for its flagship product, ChatGPT. Users can upload an image and ask the chatbot to describe the image. ChatGPT describes images with remarkable accuracy. Eventually, the tool could allow content creators to quickly generate alternative text (also called alt text) for thousands of images.
But there’s a catch: From a digital accessibility perspective, simply describing an image isn’t always sufficient.
AI chatbots can describe most images accurately and even read images of text. But once again, alt text isn’t just about accuracy; it’s about context. For example, let’s say you’re writing alt text for a photo of a woman holding a cat. Generative AI tools might describe the image like this:
Woman in a white shirt holding an orange tabby cat with a white background.
There’s nothing wrong with that alt text — unless it’s on a veterinary website. If that’s the case, a sighted user might immediately identify the woman as a veterinarian. That’s important context that should be part of the alt text. Better alt text might read:
Veterinarian holding an orange cat.
This description contains fewer details, but it provides more useful information for real human users. Most current AI alt text generators have trouble deciding what users need to know.
AI tools may also describe every part of an image without considering which components are the most important. In other words, their descriptions might be…well, too descriptive. For example, this alt text is accurate, but overwhelming for a screen reader user:
A red apple with a short stem and one green leaf with a serrated edge. Dew drops roll down the side of the apple. The background is clear. The apple casts a shadow.
Depending on the context, “red apple” might be perfectly fine alt text—there’s no need to go into extreme detail.
Of course, you could train generative AI tools to recognize the context of each image, but that requires human work.
Generative AI will become better at recognizing alt text, and current tools are still quite useful. In the near future, screen readers might use generative AI to fill in the blanks when content creators leave out alt text.
And since somewhat accurate alt text is arguably better than no alt text, there’s a defensible use case for generative AI, particularly when websites have thousands of untagged images.
Things that AI cannot discern:
AI does not achieve any of these at the moment. Human intervention is always required to make edits/corrections, so the time-saving benefits are not really there.
This Scribely blog post has a really interesting example of how AI treats an image with a lot of data and how it misinterprets the data. While AI might be useful for relatively simple images — maybe! — using it for more complex images is most definitely dangerous.
We ran some tests with the following image with mixed results.
This is what Azure gave me:
A woman taking a selfie in front of a sign.
A woman taking a selfie.
Close up of a woman wearing glasses.
A large circular object with a blue and white design.
A building with many windows.
A building with a blue sky and clouds.
A circular object with a green turtle on it.
Close up of a person’s nose and nose wearing glasses.
Close up of a person’s lips.
A blurry image of a red and white heart.
This was not a good result. It’s a kind of nonsensical word salad. A person’s nose and nose wearing glasses? Thanks Azure, very helpful.
This was a far more useful description.
A woman with gray hair and glasses smiles in a selfie at Nathan Phillips Square in Toronto. Behind her, the large “TORONTO” sign and city buildings are visible under a partly cloudy sky.
So useful! It managed to situate this image geographically which is impressive. (I mean, Toronto city hall is fairly distinctive. But still.)
This description might certainly be the basis on which an actual description is based, re-thought for the context. If this photo is used to show the “Toronto” sign, then more detail about the sign should include someone sitting in the second “O”, the turtle island reference in the third “O”, the partial maple leaf at the end, and the reflecting pond, for example.
This flavour of AI is, well, incoherent.
The alt text reads: “arafed woman with glasses looking up at the sky in front of a sign, a photo inspired by Louise Catherine Breslau, flickr contest winner. Regionalism, facing sideways, upside-down building, front-facing perspective.” For the record, Louise Catherine Breslau was a Swiss painter in the late 18th century and, so far as we know, had never even heard of Flickr.
This tool gave me far too much detail:
Close-up of a woman with gray hair, wearing glasses and a light beige or cream-colored coat, standing in front of a large sign that reads “TORONTO” in large, stylized, illuminated letters.
The sign is placed on a low concrete structure beside a reflecting pool that extends into the background.
The woman’s expression appears pleasant and friendly. She is positioned in the foreground, focusing attention on her.
The background includes modern buildings of various heights and architectural styles. One of the taller buildings is a glass-fronted skyscraper. A light blue sky with scattered white clouds sits behind the buildings.
The overall impression is of a sunny day in a city setting, with a focus on a person interacting with the city’s landmarks.
Because alt text is meant to be a short and meaningful description that captures the essence of an image and interprets it for the context in which it’s being used, and therefore cannot be formulaic or wooden. If nuance and accuracy is what you need from your image descriptions, use AI with great care.
Note: This resource was authored by Laura Brady. Laura (she/her) is an inclusive publishing expert whose priority is always to put users first. She has more than 25 years of trade publishing experience, creating and converting ebooks, training publishers on accessible workflows, writing a blog helping developers work more accessibly, and consulting for services organizations about how to publish inclusively while worrying about everyone’s reading experience. She is on the board of the Accessible Books Consortium.
