Mistakes I made when doing User Tests for a WordPress plugin

I recently started user tests for our WordPress plugin Advanced Ads. I find watching the videos from testers addictive and helpful. The insights help us to hone the edges so that we can quickly onboard new users.

Doing user tests has a learning curve to itself. I made plenty of technical mistakes when preparing the tests or briefings. That process cost me a significant amount of time and money.

I now keep a log of such mistakes to help others to prevent them.

Even though I am user testing a WordPress plugin, most mistakes are not related to a specific test scenario.

Not doing user testing

For many years, I thought that whatever users are struggling with, it will come up in support so we have a chance to fix it.

The truth is, that no one is going to tell you if your navigation is confusing or a link to a manual is not working.

Especially with free WordPress plugins, users try several of them and leave quickly if they don’t find what they are looking for.

People struggle with a lot of things that you, as the developer and everyday user, believe are a no-brainer or “just” or “simple”. Don’t trust yourself!

The even sadder truth in my case is that I knew that. From time to time, we helped users to onboard via video call.

There wasn’t a single call without an aha moment. Every user struggled somewhere, where we found the solution obvious.

Still, it took me years to do user tests pro-actively and get to fixing flaws in our plugins workflows.

Setting up user tests myself

This one was the most costly mistake. And the one that prevented me from doing user tests for a while.

Our first attempt at doing user testing was implementing the whole setup and user acquisition ourselves.

It looked a bit like this:

• set up a new WordPress installation for each user test
• look for, screen, and hire individual freelancers through upwork
• send them instructions for tests
• telling them to record a video in a tool of their choice
• provide some website space where the videos can be uploaded
• ask for feedback via email or chat or whatever comes up
• send payment
• rate the freelancer
• handle freelancer complains, feedback, or outreach for more work

That wasn‘t smooth at all. Neither was it scalable.

If my memory serves me well, it took us weeks to get 3-5 testers to record suitable videos.

I always found user test services too expensive. Looking back, the individual freelancers we hired before asked for the same price as many user testing services do. Plus, we spend a lot of time managing them.

A dedicated user test tool takes care of acquiring, managing, and paying the user testers. They also record and store the videos.

On the day of writing this post, I started and reviewed 5 user tests in little breaks between other tasks.

This is not an advertising post, so I am not mentioning the service I am using right now. But they know that I am happy. I keep telling them whenever I have the chance.

Looking for WordPress users

Next to costs, another reason to not use a user test service earlier was that none provided dedicated WordPress user tests.

I was looking for someone that can guarantee that their testers have a basic knowledge of WordPress so that they can jump into testing right away.

I learned though that unless it is my intention to test WordPress (see the next section), screening for WordPress users isn’t that important.

The user testing service I am using doesn’t offer any specific screening regarding WordPress user testing, but it turns out that many of their users are already good with using (online) software in general.

Testing WordPress

When you are not 100% certain that your testers know WordPress and its lingo, then be specific in your instructions. Unless you want to test if your wording is understandable, use terms visible on the screen.

It is so natural for me to jump from WordPress backend to frontend by clicking on the site title link at the top left in the admin toolbar. As it turned out, that many users failed, when I wasn’t explicit about it in my instructions. It is not their fault.

An experienced WordPress user also knows, that plugin settings are likely in their own menu item or under “Settings”. I have seen genuine user testers fail here as well. Again, I don’t blame them. It is my fault that they spend significant time looking through every menu.

I only want to test our software, so I now make sure that I am specific about how users find the pages they are supposed to work on.

Of course, I still keep them in the dark when I want to test specific wordings and positions of items.

User testers also find bugs in WordPress. Well, they normally don’t know that it is a bug. They might believe something is a feature. Just now, I saw a video of a user clicking on a button that I, as an experienced user would never click in this situation. The user did and some boxes vanished. That was never intended.

My tip. If you are running a user test in WordPress admin, install an autologin plugin. This saves users time and prevents issues when logging in.

Asking your existing users

For a while, I tried to find a way to engage our existing users in testing. That also stopped me from moving forward with user testing in general.

Long-term users are hard to reach for projects they don’t benefit from directly. We tried that on various occasions when developing new features.

We now ask new email subscribers to send us video reports if they discover an issue or have a question, but that also doesn’t provide a volume. It definitely doesn’t provide it in areas we are currently looking forward to improving.

The idea I have left is to reach completely new users before they start doing anything with it and offering them an incentive when they make a video of their first attempts.

Would I participate in this when I tested a new plugin? Probably not. And while I shouldn’t assume that everyone is like me, I have a pretty strong gut feeling about this one.

Invest too little into the setup

While a user test provider removes a lot of burdens, you still need to invest some time into setting up the test.

I had to create a new test site and set up not only our plugin, but also a user with the correct role, and login credentials.

The biggest investment though was in paying for a few tests knowing that they will likely fail in terms of completion. I also had to get acquainted with the user testing platform, even though it was fairly simple.

It took me about 5 user tests to get to a technical setup and a task list that users were able to complete.

When I start a new test with all the knowledge I have now, I still expect a few tests to “fail” due to technical issues to confusing instructions on my end.

Not going through the test myself

I would say that half of the unsuccessful – in terms of uncompleted without the user’s fault – tests were about a technical issue.

Here are a few things that happened to me:

• I created an editor role in WordPress, but didn’t set the capabilities right, so the user couldn’t see the menu item she was supposed to use
• The auto login link was not working

I could have prevented this if I had just opened a private browser window, used the auto-login link myself, and check the site as if being a user.

I am doing that whenever I make even the slightest change in the test setup.

Testing in bulk

Using a dedicated user test service makes it easy to scale testing and do multiple user tests at once.

I only have one test site and can only run one user test at a time. That is a lucky and good thing. It prevents me from running too many failed tests at the same time.

I still have occasional user tests where the instructions proved to be confusing to the testers. Or the test site is broken, or I forgot to remove old data, etc.

Going through tests individually allows me to fix these things before too many users run into them.

I also find it a great diversion between other tasks to look at the latest user test results.

Thinking, that testing is too expensive

I already explained that in our first attempt at organizing user tests, our internal costs were a lot higher than the fees we paid to participants.

When I first looked at user test providers, I thought they are also charging too much. It was up to $50 for not even half an hour of testing.

But the truth I learned is that just a handful of user tests is already extremely fruitful. A few user tests are enough to find major flaws. I like to fix them quickly and then start new tests to see if the change did some improvement.

The costs of implementing insights from a user test are also more expensive than the test itself.

Unless you are in a very early project without any money to spend, user testing is cheap compared to what you are losing when new or existing users fail to use whatever you are building.

I know I will forget about this from time to time and test too late. That’s life.

A decision I made in terms of costs was to choose one of the very few user test platforms that allow paying per test. Most of them come with monthly subscriptions that include a specific number of tests, but I didn’t want to be forced to doing tests regularly just because my service provider tells me so.

As much as I love user testing, I know that there will be a time when I won’t do it.

Pro tip: whenever I am sure that the test wasn’t successful because the tester ignored specific instructions, like clicking on a menu item that was visible, I report the test. The user test service is very generous in giving me an extra credit to use for another test.

Confusing user testing with A/B tests

We do A/B tests on our plugin website. They are a numbers game. Two groups with thousands of visitors see two different versions of a website. We then measure which version creates more sales or clicks on a call to action button.

A/B tests are great if you have an idea about the possible impact of a specific change you made.

User tests are different. They can show you things you didn’t know. As long as your testers are reflecting on their experience and talk while testing, you will end up with more ideas than you had initially.

I also don’t need thousands of users to perform a test. That wouldn’t scale, since I had to spend time on each of them. The obvious issues show up quickly and often.

Testing too little or too much

The service I am using for testing is paid a fixed sum per test. The time frame is limited to 20 minutes.

It is good advice to not test too many things at the beginning to see where your testers normally end.

My current test is about a setup I can finish within 1 minute. Most testers need up to 10 minutes. That is ok. Remember, they might not know WordPress and especially not your product.

When it comes to testing variations in my WordPress plugin, I dare to change a lot more than what I would in normal A/B testing.

Every user test is different, but this doesn’t mean that there aren’t patterns. Large and bold changes can help to see them.

Here are variations I bundled together when testing what it needs users to fail when setting up conditions in Advanced Ads.

• remove the condition options from the setup wizard
• removing any descriptive texts
• changing the meta box title

I did that after running 5 user tests before in which everyone successfully set up conditions. I believe that 5 more user tests with these changes should be enough to see if I removed too much.

Keeping user tests to myself

I find watching user tests fascinating. I enjoy learning things and generate ideas for improvements from them.

We recently had an all-in meeting for a few days. I started one morning by showing outtakes from user tests to the team. It was great to see how they became agitated and ideas filled the room. Whoever implements a GitHub ticket generated out of these videos now has a better idea of why they are doing it.

While I don’t think it is possible and necessary to show every video to everyone involved, sharing outtakes where it matters, maybe as a comment in a related GitHub ticket, can help.

Inactionable instructions

I wrote above that one should be specific about where to click and what to look for to prevent frustration on both ends.

I also found that too wordy instructions can keep users from succeeding. Among them are instructions that are not actionable.

Here are a few things I removed over time:

• Giving login instructions when using an auto-login link. It caused some testers to try to re-login
• Ask them to use a desktop device, while this was already quaranteed by the user testing provider
• Asking users to abort, if they don’t succeed. I thought this might help me save time with users who just don’t get it, but in the end caused some who might have succeeded to abort early.
• Using empty sentences like “We are not testing you. You cannot fail…“. Regular user testers know that. I later found that the user testing provider used almost the exact wording in their manual as well.
• Telling testers that this is WordPress. I though this might help filter testers by experience. I later learned that the user test service does not show the briefing text to testers before they start. So there was no use in mentioning WordPress.

I still like to use some introduction sentences to set the mood though. Something along the line of “You are a website owner who wants to publish an ad on their website.”

Don’t hesitate to update your instructions on the fly. I am fine-tuning a lot. For example, only the eights test showed a user with an ad blocker. They could only fail. So I added a specific note about disabling them while testing.