Preference Test Results

Independent Redesign
Duolingo's Error Page for Desktop

Project
Independent Project
Role
Product Designer
Timeline
2 Weeks
Duolingo is a language-learning app and website that has over 500 million registered learners, according to a November 2020 article.
Guiding Questions
Does the error page offer suggestions to get around the error?
Is the error page easy to understand?
Is anything inaccessible?
What can be done better?
Note
This project was self-initiated, and it was not commissioned or implemented by Duolingo. I was not privy to the information, data, and business requirements that Duolingo designers had, so our design ideas may differ.
Duolingo's Original Design
My Initial Redesign


Redesign Outline
I changed the technical wording 'Error 404' to a more easy-to-understand wording 'Page not Found'.
Instead of a paragraph form, I presented the workarounds with bullet points.
I offered 2 more suggestions: reminding users that they type in the correct web address and that they copy the correct web address, following the best design practices of the British government: https://design-system.service.gov.uk/patterns/page-not-found-pages/.
I changed font colour #AFAFAF to colour #3C3C3C in order for the page to be accessible while adhering to Duolingo's design language.
I had the links to be always underlined regardless if the mouse cursor is hovering over them.


A/B Test
I ran an unmoderated A/B test from 3 February 2022 to 11 February 2022. Thirty-six participants took part in it, with 39% of participants from Germany and 31% from the USA. I recruited the participants from CareerFoundry's Slack.
Test Objectives
Find out how learners feel about the Duolingo error page for desktop
Determine learner preferences regarding my proposed design elements, such as layout, content, etc.
Two designs were used for the first A/B test:
Version 1, Duolingo's original design
Version 2, my proposed design
After choosing their preferred design, the participants were provided the option of answering the following 2 questions:
Why did you choose this design?
Is there anything else that you would like to add?
They were free to skip the questions.
Test Limitations
The participants may have not been active learners of Duolingo.
I could not ask follow-up questions since the test was unmoderated.

Key Insights
Twenty-two participants (61%) preferred my proposed design.
Fourteen participants (39%) preferred Duolingo's original design.
This result is 90.0% likely to be statistically significant.
Among the comments that preferred my preferred design included 'The language here is more empathetic with the user, trying to give solutions', 'It provides a clear explanation on why I might have not been able to see the page I was looking for, while the other one is vague and does not provide any type of help.', and 'because that error 404 message is irritating'.
Among the comments that preferred Duolingo's original design included 'It is shorter and to the point', 'It was more simplistic, compact and easier on the eye although I did like the underlines links in the other design.', and 'I think this one is less abrasive. I think you might want to increase the contrast though here with the light grey bc to me it doesn't seem accessible for the size of the copy. (But not necessarily black like the other one, maybe just a darker grey.)'
General comments included 'I can read it better, but the dots [of the proposed design] are too big', 'I like that it doesn't say error. When I read error I didn't feel good. I especially like this one because it gives you options on how to fix the issue.', and 'The font [of the proposed design] is clear and easy to read even at a small type size.'
Based on the results of the A/B test
Participants overwhelmingly preferred my preferred design over Duolingo's original design. Based on the comments that I received from the test, I am going to adjust slightly certain design details of the proposed design.
I am going to make the bullet points smaller.
I am going to make the text horizontally-aligned with Duo, which is Duolingo's green-bird mascot.
I am going to reduce the indentation of the bullet-point text.

Duolingo's Original Design

My Proposed Design
My Recommendations
Change the technical wording 'Error 404' to the more user-friendly wording 'Page not Found'.
Present the workarounds with bullet points instead of in a paragraph format.
Offer 2 more suggestions:
Ask users if they typed in the correct web address.
Ask users if they copied the correct web address.
Change the font colour #AFAFAF to colour #3C3C3C in order for the page to be accessible while adhering to Duolingo's design language.
Underline the links.
Future Consideration
Duolingo uses the typeface DIN Next Rounded for body text. Since I did not have it during this case study, I used Nunito as a substitute, as recommended by Duolingo's design language for people who do not have DIN Next Rounded. To align more with Duolingo's design language, I would prefer to do future redesigns using the DIN Next Rounded.