Problems and goals

Problem 1: Low usage

Q&A had been a feature in Power BI for years, but with extremely low usage. My researcher and I wanted to discover why usage was so low and solve any easy issues.

Problem 2: Inconsistent experiences

Over the years, different, disconnected Q&A experiences had been added to Power BI, which customers cited as a point of confusion. Our goal was to create one straightforward way to use Q&A.

Problem 3: Inaccurate results

Originally, Q&A could only answer a question accurately if it was formed with terms from the underlying metadata. But we learned that most Q&A users, like CEOs or delivery drivers, knew nothing about their metadata. To account for this, we made a way for data analysts to train Q&A to understand how their end-users naturally ask questions.

Personas

Q&A is intended for use by two personas, Ash and Nat. Ash is a data analyst who creates reports for others to consume. Ash knows the data well, and might use Q&A to quickly create visuals on a report page.

Nat is a CEO, delivery driver, or anyone who relies on Ash's reports to do their job. If Nat has a data question, they typically reach out to Ash -- a huge inconvenience if an urgent question comes up in the boardroom or out in the field. Q&A lets Nat get answers about their data without involving Ash.

Identifying problems

Through heuristic analysis, I found a number of UI-related usability issues. James confirmed many of these through user research, plus he discovered another huge pain point: the words Nat used to ask questions rarely matched the metadata used by Q&A to find answers.

For example, Nat might ask a question in natural language:

What were my sales in Q4?

but if the desired data were stored in a column called "US_prod_sales" (instead of "sales," as asked) Q&A might produce an unexpected answer.

Over and over, we met customers who were very excited by the concept of Q&A, but the feature's usability and language issues made it essentially useless.

Multiple experiences, none of them good

Over the years, disconnected Q&A experiences were added to different parts of Power BI. Each version was confusing in its own way and, understandably, users felt lost.

• Clicking a search box above a dashboard led to a cluttered, full-page Q&A experience.
• A data analyst could add a button to their report, which would open Q&A in a modal.
• End-users could double-click blank space on a report to open a Q&A pop-up.
• Q&A settings were accessed from four different places.

Process

The Q&A visualization was one of the breeziest high-impact projects I've worked on, coming together in a matter of weeks, mostly through whiteboarding sessions with my engineer, Eeshan.

Eventually, my work moved from whiteboarding to redlines. Power BI's style guide was in flux at the time, so it was important to call out stylistic details. Sitting with Eeshan, I gave design guidance on UI, interactions, animations, and content, making sure everything was consistent with other Power BI visualizations. Eeshan rocked the rest.

Process

For months, my content designer, researcher, and I met with our engineering team to work through the complex scenario before us: how to guide data analysts through training a natural language search engine. We did hours of whiteboarding, brainstorming, and debating. I built wireframes and prototypes, and collaborated with my new researcher, Carly, to design and run user studies.

We divided the giant scenario into parts that could be released independently:

• a feature that lets data analysts (our persona Ash) teach Q&A to understand specific terms that people at their company (persona Nat) might use,
• a feature that lets Ash review questions Nat has asked Q&A, to see where errors have occurred and discover important terms to define,
• a way for Ash to review or edit terms they've already defined for Q&A,
• and a way for Nat to request that Ash add a specific term to Q&A.

Figure 7. Exploratory sketches of how Ash could train Q&A to understand new words.

Wireframes

Figure 8. An early wireframe, showing the process by which Ash trains Q&A to understand new terms.

Figure 9. A later wireframe showing the same process, refined by user testing, design reviews, and technical reviews.

Figure 10. Open the image larger in a new tab. A user flow showing how Ash can combine two separate terms into one term. Our team had many conversations about linguistic nuances like this, e.g. how can Ash define whether "white house" refers to any house of that particular color (an adjective modifying a noun) or to the White House (a two-word proper noun)?

Figure 11. An early wireframe, showing how Ash could review questions Nat had asked Q&A and define any unknown terms inline.

Figure 12. A later wireframe, showing questions Nat had asked Q&A, for Ash to review. We discovered through user testing that people were confused by the ability to edit terms in multiple places, so now terms can be edited in just one place -- Teach Q&A. Clicking the edit icon on this screen redirects there.

Outcome

Q&A is among the pioneering technologies that could democratize data, making accurate data analysis accessible to more people, even those without extensive training.

We got huge cheers when we demoed [how to train Q&A] and it's awesome to see us on the right track.
- Mo, our Q&A PM, after the 2019 Power BI MVP Summit

The first parts of Q&A training were released as public preview in October, 2019. The next step will be to gather sentiment, usage data, and customer feedback, so the feature can be improved before general availability.

This whole project was a fantastic collaboration between design, engineering, and PM. My content designer, researcher, and I had seats at the table alongside our engineers, and everyone valued each other's contributions. I couldn't have asked for a better team.

Figure 13. Q&A training was a major component of the work Power BI shared with Gartner last fall, and once again we're rated as the leader in Gartner's Magic Quadrant for Analytics and Business Intelligence Platforms.