A usability testing framework for the Istanbul Metro Map
The Istanbul transport network presents a demanding wayfinding problem. A visitor may move between metro lines, buses, trams, Marmaray and ferries, then finish on foot; a regular passenger may instead check a transfer, service disruption or exit. A geographically accurate map can still fail when labels, colours or interaction sequences do not match the decisions users need to make.
A useful evaluation combines observation with explanation. It asks whether people can identify the right station, understand where a line goes, follow transfers and recover from mistakes across paper and digital versions. The central question is not whether testers prefer a particular style, but whether they can complete realistic journeys accurately, efficiently and confidently.
For Australian users, include crowded stations, outdoor glare, quick phone checks and long first- and last-mile trips from outer suburbs. Sydney and Melbourne travellers know Opal and Myki, integrated signs and interchange planning; Istanbul adds a denser mix of modes and the need to combine rail, ferry and walking information.
Frame the evaluation around real journeys
Begin with a brief that fixes the decisions under examination. Separate orientation tasks, such as finding a station and exit, from journey tasks, such as changing lines with a ten-minute connection. Distinguish pre-trip planning from an on-street check during a disruption. This prevents attractive but irrelevant features from dominating the study.
Choose participants, tasks and measures early. Effectiveness can mean completion without help; efficiency can include time and erroneous taps; satisfaction can be rated confidence. These dimensions align with ISO 9241-11, while map-specific measures include transfer errors and label look-ups.
Document hypotheses rather than expected winners. One might be that direct station labels outperform a detached legend for short journeys, or that animated service alerts are noticed only near the relevant line. Each hypothesis needs a route, a device condition and a threshold for successful performance.
Compare the formats users actually encounter
No single presentation suits every stage of a trip. Compare realistic alternatives while keeping the underlying network and route data constant.
| Presentation | Best moment to use it | What it can support | Likely weakness |
|---|---|---|---|
| Printed schematic | Carrying or reading before entry | Rapid line recognition and a durable backup | Fixed labels and no live updates |
| Desktop web map | Detailed planning at home | Zooming, filtering and route comparison | Less suitable for a phone used outdoors |
| Mobile web map | Planning and checking while travelling | Search, location and disruption notices | Small labels, glare and accidental taps |
| Station diagram | Moving through a complex interchange | Transfer sequence and exit orientation | Limited geographic context |
Test each format with the same tasks, then examine how users move between them. A commuter may use a printed schematic to recognise a line, a phone to check disruption, and platform signs to make the final transfer. Usability may therefore lie in a smooth hand-off, not one perfect product.
Include visual-accessibility variants in the comparison. Colour should not be the only cue for a line or interchange; line numbers, symbols and direct labels should remain legible. Turkish and English wording, text resizing and screen-reader order also need scrutiny, especially where several routes converge.
Recruit participants who travel differently
Recruit several participant groups rather than a single convenience sample of map enthusiasts. Prioritise occasional visitors, daily commuters, older travellers, people with low vision or limited dexterity, wheelchair users, and people who navigate using Turkish and English to different levels. Airport passengers and ferry users add another perspective because their route often begins or ends outside the metro.
Recruit in Australia and validate the findings in Istanbul. Australian participants can test advance planning, airport transfers and multi-operator itineraries before travel; Istanbul participants can expose assumptions about familiar interchanges and actual crowding. Include people familiar with Australia’s mix of state agencies, local councils and private operators.
Use community partners to reach participants often absent from product panels. Methods associated with self-organisation networks can help distribute recruitment, facilitation and accessibility-review roles among community groups. Obtain informed consent and de-identify recordings. For personal information collected in Australia, align the process with the Privacy Act 1988 and the Australian Privacy Principles rather than treating usability data as anonymous by default.
Test tasks in realistic operating conditions
Use a moderated laboratory round for controlled comparisons and a field round for environmental effects. In the laboratory, provide realistic route options and use think-aloud prompts. In the field, test near interchanges during busy periods without recording bystanders. A task completed at a quiet desk can conceal the small type, glare, noise and hurried finger movements that affect actual use.
Build a task set that spans the network. Ask a first-time visitor to locate the correct platform for a rail connection, identify which direction to take, find the exit nearest a ferry, and determine whether a walking link is required. Give regular passengers a disrupted-service scenario requiring an alternative. Include departure and arrival times, not just topology, so the study reflects whether the map supports a timed connection.
For a web round, test a staging deployment of the Istanbul Metro Map rather than relying on the current hosting placeholder. A moderator records navigation choices, incorrect taps, backtracking, uncertainty and help requests without coaching. Randomise equivalent route pairs across formats to reduce learning effects, and ask open follow-up questions after each task rather than interrupting with explanations.
Measure comprehension, errors and confidence
Score more than task completion. A short route may be completed quickly yet reveal a serious misconception, while a difficult interchange may produce correct performance with little confidence. Track first-attempt accuracy, completion time, sequence errors, help requests, backtracking and the point at which participants stop trusting the map. Add a confidence rating so apparently smooth behaviour is not mistaken for comprehension.
Classify errors by cause: illegible label, ambiguous direction, missing transfer, unexpected interaction, outdated service data or weak geographic orientation. This taxonomy directs fixes more precisely than a low satisfaction score. It also shows whether one format reduces errors or merely moves them, such as hiding difficulty behind more taps.
Assess the digital experience against accessibility expectations and the intent of the Disability Discrimination Act 1992, treating legal compliance as a baseline rather than proof of usability. Check text scaling, keyboard operation, screen-reader labels, focus order and colour-independent cues. Compare results by participant group and trip type so a strong average does not conceal serious barriers.
Build a repeatable improvement cycle
Use the evidence to prioritise revisions and test again under the same conditions.
- Define success measures, route scenarios and decision thresholds before each round.
- Recruit by travel behaviour and access need, recording relevant segment characteristics without collecting unnecessary identity data.
- Test a balanced mix of planning, disruption, interchange and last-mile tasks on the formats people realistically use.
- Record each failure as an observable action and its consequence, then distinguish interface faults from information gaps.
- Prioritise changes by frequency, severity and affected users, and rerun the most discriminating tasks after each release.
A revision might strengthen transfer labels, retain line colours while adding patterns, expose the walking portion of a journey earlier, or explain service alerts in plain language. These are possibilities, not assumptions: the study should reveal which changes matter. Keep a decision log linking each change to evidence, unresolved risk and the next validation round.
The lasting point is that the Istanbul Metro Map should be judged by the journeys it helps people complete, not by visual polish alone. Remember the line between a successful route and a lucky guess, and the difference between general accessibility and confidence across every kind of traveller.