STARWEST Without Me: Letting AI Do the Testing

Since I can’t make it to STARWEST this year, I figured the most useful thing I could do was put my AI testing agents to work — finding issues and testing the apps of the sponsors and speakers for them. 🙂

That’s not meant as a jab — just a reminder that even the most prominent names in testing can miss things. Every bug or design flaw uncovered here is an escaped issue — something that slipped past internal QA teams. And that’s exactly where AI testing agents shine. They don’t replace human testers, but they catch what slips through the cracks, providing a complementary layer of coverage.

The power of AI in testing is hard to overstate. A few years ago, I could barely manage the testing of a single app. Today, I can point AI-driven testing agents at thousands of apps — and even benchmark their quality side-by-side. Testing has shifted from local and manual to global, scalable, and comparative.

The Results

Here’s the Quality Quadrant I generated by testing the STARWEST ecosystem:

This chart plots each site across two dimensions:

• Quantitative Quality (y-axis): bugs, issues, and their severity.
• Qualitative Quality (x-axis): user impressions, usability, and emotional resonance.

What We Found

• Top Right (High-High): Giants like Amazon, Selenium, and Edelman Financial scored well across the board. Not flawless, but their investment in digital polish shows.
• High Quantitative, Low Qualitative: Oracle Cloud and CSAA Insurance were technically solid but felt uninspired to testers. Functionality worked, but the experience didn’t stick.
• High Qualitative, Lower Quantitative: City of Hope was beautiful, emotionally resonant, and highly usable — but agents flagged bugs like misleading future-dated content.
• Struggling Zone: Sites like Acorns, Banner Health, and TrustCloud had more issues and weaker user impressions. Not catastrophic, but surprising for prominent names.

A Few Quality Nerd Highlights

Remember, these sites are tested by great test teams that are nerdy enough about testing to attend a conference on the topic!

• Selenium — Great resource, but testers found it intimidating for newcomers: dense explanations, not very beginner-friendly
• ChurchOfJesusChrist LDS — Well-structured and provides valuable information, but it could benefit from improved accessibility has 404 errors.
• Tricentis — Still advertising past events as if they were upcoming, which could confuse new visitors
• Tricentis — Looked modern and professional but had event registration links pointing to already-expired events
• Amazon — Solid experience overall, but flagged for potential third-party tracking without explicit consent
• Pokémon — Fun and content-rich, but agents uncovered malformed URLs being requested multiple times (likely developer leftovers)
• Guardant Health — Strong mission, but quietly running BugHerd tracking scripts without disclosure — classic escaped privacy issue
• Vanguard — Flagged for a missing CSRF defense; not the kind of vulnerability you want on a financial site
• CSAA Insurance — Site forced a zip code check immediately on entry — usable but unnecessarily intrusive. Also lacked attribute, breaking accessibility standards
• UKG — Beautiful HR platform homepage, but the “Skip to main content” link failed color contrast checks — hurting accessibility
• Indeed — Resource request blocked due to CORS misconfiguration. Nothing kills job hunting like broken scripts
• PwC — “Failed to load resource: net::ERR_NAME_NOT_RESOLVED” errors — corporate site, but still shipping DNS/configuration bugs
• University of Memphis — DNS failures breaking resource loading
• Dell — Missing attribute, making the site less accessible to screen readers
• Envorso — Tracking scripts from Google Tag Manager and others ran with no user consent banner — GDPR gap
• JetBrains — Strong dev-tool site, but flagged for loading Google Tag Manager without explicit consent
• Intuit — Lacked a proper cookie consent banner; only had passive footer links
• TrustCloud — CSS versioning all tied to one shared hash — kills cache efficiency
• Women Who Test — Inspiring community site, but cookie banner text contrast too low, failing WCAG
• Hitachi Digital Services — Consent banners present but lacked granular controls; GDPR compliance incomplete
• Capco — Multiple resource loading failures due to DNS issues — page broke visually and functionally
• StoneX — Footer text contrast too weak, violating WCAG
• Purdue University — Beautifully detailed, but dozens of images lacked alt text — serious accessibility gap

Why This Matters

Testing isn’t just about defects anymore — it’s about quality across multiple dimensions:

• Reliability — does it actually work?
• Accessibility — can everyone use it?
• Usability — is it intuitive?
• Trust — are data and privacy handled responsibly?
• Emotion — does it leave users feeling confident or frustrated?

By scanning all STARWEST sponsors and speakers, we get a snapshot of the industry’s digital health. Some companies are leading by example. Others reflect the common pitfalls that every team — ours included — runs into along the way.

And that’s the point: software quality is universal, and nobody is immune.

Closing Thoughts

Even though I can’t be at STARWEST in person this year, this project gave me a way to still contribute. Instead of hallway chats, I’m sharing quality data. Instead of a keynote, here’s a snapshot of how well (or not so well) our industry is doing when it comes to its own front doors.

👉 If you want a quick analysis of your own homepage quality — like the ones I ran for STARWEST sponsors — just reach out to me on LinkedIn, or sign up directly at testers.ai.

👉 Or, if you’d rather not learn or wrestle with AI at all, check out IcebergQA. Expert testers there can run these AI-powered checks for you — no setup required, just instant-on results. Since AI is the hot topic at STARWEST, this is the fastest way to add AI to your testing strategy and start catching escaped issues today.

Here’s to raising the bar — see you at the next one.

— Jason Arbon, testers.ai