Sample work · Benchmarking study
Harbor UX Benchmark — 10 core tasks
- 78%
- Mean task
success rate - 68
- Mean SUS
(SD 12) - 5.2
- Mean SEQ
(1–7 scale) - +8
- NPS
(n = 32)
1 · Executive summary
Core creation flows are in good shape; settings and retrieval flows drag the benchmark down.
We benchmarked 10 core Harbor tasks with 32 participants. Onboarding and creation tasks (T1–T4, T7) meet or approach target: success ≥ 81% and SEQ ≥ 5.4. Three tasks fall well below the 75% success target — notification preferences (T8, 59%), board filtering (T6, 63%) and PDF export (T10, 69%) — and account for most observed errors and low ease ratings. Overall SUS of 68 sits at the industry average, consistent with a product whose everyday paths work but whose secondary paths need repair.
Wave 2 targets: ≥ 85% mean success, SUS ≥ 75, and no task below 75% success. The highest-leverage fixes are making notification settings and board filters discoverable without navigation rework.
2 · Method
Design. Remote moderated sessions, ~45 minutes each. Task order counterbalanced in two blocks (creation block T1–T5, management block T6–T10) to control for fatigue. Think-aloud during tasks; SEQ after each task; SUS + NPS + background questionnaire at the end.
Participants. N = 32 adults who manage projects in teams of 3–20 and use a collaboration tool at least weekly. Recruited to mix current-tool experience (11 Asana/Monday, 9 Jira, 12 spreadsheets/chat only). 16 desktop Mac, 16 desktop Windows; Chrome or Edge. No prior Harbor exposure. Incentive: $60 gift card.
Metrics. Task success (binary, strict: completed without moderator help), time on task (seconds, successful attempts), errors per attempt (wrong path, slip, or backtrack), SEQ (Single Ease Question, 1–7), plus study-level SUS and NPS. 95% CIs: Wilson for rates, t-based for means.
Analysis. Python (pandas/scipy): descriptives with CIs, task × experience-segment comparisons (ANOVA), SEQ–success correlation. Qual: affinity mapping of 214 observation notes into issue themes, severity-rated by frequency × impact on success.
3 · Tasks
Ten tasks covering onboarding, creation, collaboration, retrieval and settings.
| # | Task (as prompted) | Starts from | Success = |
|---|---|---|---|
| T1 | Create a Harbor account with this email invite | Invite email | Landed in empty workspace |
| T2 | Start a “Website relaunch” project from a template | Empty workspace | Project created from template |
| T3 | Invite a teammate to the project by email | Project home | Invite sent to correct address |
| T4 | Create a task “Draft homepage copy”, assign it to Sam, due Friday | Project board | Task exists, assigned, dated |
| T5 | Attach the file brief.pdf to that task | Task detail open | File attached and visible |
| T6 | Show only tasks assigned to you on this board | Project board | Filter applied, correct subset |
| T7 | Leave a comment on the task mentioning Sam | Task detail open | Comment posted with @mention |
| T8 | Turn off email notifications for weekends | Project board | Setting changed and saved |
| T9 | Find the keyboard shortcut for creating a task | Anywhere in app | Correct shortcut reported |
| T10 | Export a one-page PDF summary of this project | Project home | Correct PDF downloaded |
4 · Results by task
N = 32 per task. Target: ≥ 75% success and ≥ 5.0 SEQ. Three tasks miss both.
| # | Task | Success | 95% CI | Mean time | SEQ | Errors/try | Status |
|---|---|---|---|---|---|---|---|
| T1 | Sign up | 97% | 84–99 | 1:24 | 6.4 | 0.2 | Meets |
| T2 | Project from template | 91% | 76–97 | 2:05 | 6.0 | 0.4 | Meets |
| T3 | Invite teammate | 84% | 68–94 | 1:48 | 5.6 | 0.5 | Meets |
| T4 | Create + assign task | 88% | 72–95 | 2:31 | 5.8 | 0.5 | Meets |
| T5 | Attach file | 72% | 54–85 | 2:57 | 4.9 | 1.1 | Below |
| T6 | Filter by assignee | 63% | 45–78 | 3:12 | 4.2 | 1.6 | Below |
| T7 | @mention comment | 81% | 65–91 | 1:56 | 5.4 | 0.6 | Meets |
| T8 | Weekend notifications off | 59% | 42–75 | 3:41 | 3.9 | 1.9 | Below |
| T9 | Find shortcut | 75% | 58–87 | 2:44 | 5.0 | 0.8 | Meets |
| T10 | Export PDF summary | 69% | 51–83 | 3:05 | 4.6 | 1.3 | Below |
Success rate by task (target ≥ 75%)
Study-level scores
SUS 68 (SD 12, 95% CI 64–72) — at the industry average of 68; adjective rating “OK”. Promoters of ease (T1, T2) and detractors (T6, T8) split the scale: SEQ correlates strongly with task success (r = .71), so lifting the three failing tasks should move SUS directly.
NPS +8 (9 promoters, 15 passives, 8 detractors). Detractor comments cluster on “hard to find things” (6 of 8) rather than missing features — a discoverability problem, not a scope problem. No significant success differences by prior-tool segment (ANOVA, p = .34), so findings generalize across newcomer types in this sample.
5 · Key findings
Five issues explain 71% of all observed errors. Severity = frequency × impact on success.
P0Notification settings are effectively hidden (T8)
13 of 32 participants never found Preferences → Notifications → Schedule; 7 gave up in the project-level settings, which look identical but lack the weekend toggle. Mean 1.9 errors/attempt, lowest SEQ (3.9). Quote: “I checked every settings page — I swear this toggle doesn’t exist.”
P0Board filter icon is not recognized as a filter (T6)
12 failures. The funnel icon reads as “sort” or decoration; 9 participants scrolled the full board manually instead. Spreadsheet-only users failed at twice the rate of Jira users, suggesting the icon relies on learned convention.
P1“Export” is labeled “Share snapshot” (T10)
10 failures. Participants looking for Export/PDF skipped the Share menu entirely; 4 exported a CSV from the wrong surface and counted it as success until debrief. Label mismatch, not flow length, is the cause — those who opened the menu finished quickly.
P1Attach affordance appears only on hover (T5)
9 failures. The paperclip reveals on row hover, so trackpad and keyboard-first users never saw it; 5 tried drag-and-drop onto the task, which is unsupported. Errors concentrated in the first 60 seconds, then success — a classic discoverability cliff.
P2Help search ranks marketing pages above shortcuts (T9)
8 failures. Searching “shortcut” returns the pricing-tour page first; the shortcut list is the fourth result. Participants who used the in-app “?” menu succeeded (92%); those who used global search succeeded at 58%.
6 · Recommendations
Ordered by expected benchmark lift per unit of effort.
| # | Recommendation | Fixes | Expected lift |
|---|---|---|---|
| R1 | Add “Notifications” link to the avatar menu surfacing the weekend toggle directly | T8 | T8 success → 80%+, SEQ → 5.0+ |
| R2 | Replace filter icon with labeled “Filter” button; persist last filter per board | T6 | T6 success → 80%+, errors halved |
| R3 | Rename “Share snapshot” to “Export / share” and add PDF as first option | T10 | T10 success → 82%+ |
| R4 | Always-visible attach button on task rows; support drop-to-attach | T5 | T5 success → 85%+, time −20% |
| R5 | Pin shortcut list as first help-search result for “shortcut/keyboard” | T9 | T9 success → 85%+ |
If R1–R4 ship, projected mean success rises from 78% to ~87% and SUS to the mid-70s — revisit in Wave 2 with identical tasks and a fresh N = 32 to confirm.
7 · Appendix & limitations
Task scripts (verbatim prompts). Each participant received the ten prompts in §3 with starter materials (invite email, brief.pdf, seeded “Website relaunch” board with 24 tasks across 4 assignees). Moderators answered only with “do what you would normally do” until success, give-up, or the 5-minute cap.
Scoring. Two raters scored success independently from recordings (agreement 96%, κ = .89); disagreements resolved by the strict criterion in §3. Time on task covers successful attempts only; capped attempts excluded from means.
Limitations. Desktop web only — mobile is untested. Sample skews toward experienced tool users, so absolute success likely overstates first-week adoption. Illustrative data: figures demonstrate reporting format, not a real product readout.
← Back to jamie-li.com · Questions: [email protected]