WP Briefing: Episode 40: All Things Testing with Special Guests Anne McCarthy and Brian Alexander

[Wordpress]

In the fortieth episode of the WordPress Briefing, Josepha Haden Chomphosy sits down with special guests Anne McCarthy and Brian Alexander to discuss the Testing Team and how to get started with testing in the WordPress project.

Have a question you’d like answered? You can submit them to wpbriefing@wordpress.org, either written or as a voice recording.

Credits

Editor: Dustin Hartzler
Logo: Javier Arce
Production: Santana Inniss
Song: Fearless First by Kevin MacLeod

Guests

Anne McCarthy
Brian Alexander

References

WordPress 6.1 Testing
Testing Reports w/ Template
Week in Test Series
Reporting Bugs Handbook Page
Fullsite Editing Outreach Program
FSE Outreach Experiment Slack Channel
make.wordpress.org/test
WordPress.org/news
Learn.wordpress.org
#WPDiversity Speaker Workshop for Women Voices in Latin America

Transcript

[Josepha Haden Chomphosy 00:00:00] 

Hello everyone. And welcome to the WordPress Briefing, the podcast where you can catch quick explanations of some of the ideas behind the WordPress open source project and the community around it, as well as get a small list of big things coming up in the next two weeks. I’m your host, Josepha Haden Chomphosy.

Here we go. 

[Josepha Haden Chomphosy 00:00:42] 

So I have with us today on the WordPress Briefing a couple of special guests. I have Brian Alexander, as well as Anne McCarthy. I’m gonna ask you both to tell us a little bit about yourselves, if you can tell us what you do with the WordPress project, maybe how long you’ve been with WordPress, and if there are any particular teams that you contribute to, that would be great. 

Brian, why don’t you get us started?

[Brian Alexander 00:01:02] 

Hi, I’m Brian. I work on the WordPress project as a full-time contributor, sponsored by Automattic. And I am one of the Test Team reps, so I help promote testing across the project. And that’s not just in Core, but it could be for Themes, Performance, feature plugins, what have you. So try to make that stuff move forward and wrangle as many people as we can to get on board and help.

[Josepha Haden Chomphosy 00:01:32] 

Excellent. All right, and Anne, what about you?

[Anne McCarthy 00:01:36] 

I spearhead the Full Site Editing outreach program. I am a sponsor contributor for Automattic as well, and so I contribute across a couple of different teams depending upon what the outreach program needs as well as various release squads I have been a part of. So for 6.1 coming up, I’m one of the co-Core Editor triage leads. 

Brian is also on the squad as the co-Test lead, which is very exciting. So it’s been fun to work with him and be on the podcast. And I’ve been around the WordPress project since about 2011. But this is, the last couple of years, the first time I’ve been able to be sponsored by Automattic and be a part of giving back to the community that’s given me so much.

[Josepha Haden Chomphosy 00:02:13] 

Amazing. All right. For folks who’ve been listening to the WP Briefing for a while, you know that I’ve been saying for like a full year that I think that testing is one of the best onboarding opportunities we have. And then also I really like to bring in our co-creators of WordPress through that testing program. Because we don’t know whether we’re right or not unless people tell us that we’re right or not. And we would like to hear so much from the users who, you know, use it and don’t necessarily have an opportunity, that privilege to kind of build on it or build the CMS itself.

So I just have a few questions since I’ve got a couple of our strong testing wranglers here. The first thing I have is what are you doing? Or, do you have any advice for getting people outside of our active contributor base and the community to participate in testing?

[Anne McCarthy 00:03:03] 

I can kick this off. Just thinking about the Full Site Editing outreach program model. So just for context, there are various calls for testing in different formats. So everything from really procedural where you’re following exact steps to follow, to very open-ended calls for testing, as well as we recently did usability testing.

And one of the things that come to mind immediately just for getting different contributors is to have very specific, fun, engaging, relevant tests that can draw people in. So if you have a call for testing that really speaks to someone, they might be more willing to participate. As well as just different formats.

So someone may not want to, you know, follow 30 steps, but they might want to follow something more open-ended. They might want to answer a survey rather than opening a GitHub link. And so I think a lot of facilitation with the outreach program has served us really well to bring in different folks as well as explicitly reaching out.

So I’ve done a number of talks in different WordPress related spaces and non-WordPress spaces to try to tell people about what we’re up to and really go meet them where they are. Because I think that’s ultimately, especially with Covid and the pandemic, there was a really unique opportunity to do that and to join the random online meetup that was happening and talk about the program and talk about ways that people could get involved and feel heard. 

[Anne McCarthy 00:04:12] 

And the last thing I’ll mention is translations. The program that’s culture testing that I write is written in English, but I’m very fortunate to have people who translate those. And so that’s a huge way that I cannot contribute but that other people have. And so I really want to highlight that and call that out because it’s been hugely impactful to have these calls for testing in a way that people can more readily access. 

[Josepha Haden Chomphosy 00:04:32] 

Yeah, absolutely.

[Brian Alexander 00:04:35] 

Yeah, I was going to add in, in addition to the calls for testing that are, as Anne said, structured such to isolate so that someone can just kind of go through a list of steps to do rather than just being exposed to Trac or GitHub and have kind of snow blindness with, with everything that’s happening.

We also have a Week in Test series of posts that goes out about every week. And what we try to do with that series is to curate a list of posts that might be a good starting point. So we try to find one that, in each type of testing example, is something that would, a more novice contributor might be able to start with. Things for more intermediate and then also advanced ones that, for testers who may need to have a development environment and the ability to make some pretty deep or type of customizations to their WordPress project in order to test a patch or reproduce a particular issue that might be happening.

So that’s a good springboard for someone to come in where there’s just a small thing that they can kind of look at and then dive into the larger process.

[Josepha Haden Chomphosy 00:05:46] 

Absolutely. That’s very smart. It’s hard to figure out how to get started in WordPress at all, let alone as a contributing by testing things sort of area. That feels new to WordPress even though the team has been around for a long time. And so I think that’s excellent. 

Brian, you mentioned in your note about who you are and what you’re doing that you’re helping with testing not only in the test section in the Test Team but then also across the project. So, I have a follow-up question for you. What can developers do to create better tests for their software?

[Brian Alexander 00:06:18] 

There are sections within the Core handbook that kind of go into detail about the types of tests that should accompany individual contributions. A lot of those require kind of an extra step, and some developers maybe don’t have as much experience there. So hopefully, the Core handbook can provide a little bit of that guidance.

We also have a lot of contributors who are interested in things such as unit testing, E2E testing, which is end-to-end testing, and testing in JavaScript or in PHP. So there’s a wide variety of the types of tests that you can actually contribute to. And I would say maybe about 50% of the tickets that I’ve triaged, personally, the contributor who brought in the patch was unable to or was not familiar with providing unit tests. So that is a very good opportunity for someone to come in who maybe is not as well versed in the depth of what the patch was involved with. But by contributing a user test, they get an opportunity to look very focused at a particular piece of code, what was modified, and then create unit tests based on that.

[Brian Alexander 00:07:40] 

And then once that unit test has been submitted and starting to be reviewed, other reviewers, Core contributors, or Core committers, I would say, they’ll start looking at that and if there are additional details that should be there, expanding the tests or little modifications. Then that also is feedback to that test contributor so that the next time they come in, they’re more prepared for it. They’re learning more about Core, and eventually, maybe they’ll also become a Core contributor.

[Josepha Haden Chomphosy 00:08:07] 

Excellent. We will include links to these handbook pages and documentation in the show notes if you’re listening to the podcast on your favorite podcasting platform, Pocketcasts, or it’s somewhere else. I don’t know where people listen to podcasts, but if you’re listening to it somewhere that’s not on the website, you can come to get that on wordpress.org/news.

Okay, the next question that I have, and I think this is for both of you, Brian, it sounds like you partially answered it, but I bet there are more answers from Anne as well. What advice do you have for those submitting bug reports?

[Anne McCarthy 00:08:38] 

I’ll chime in to start, and then Brian, I’d love to hear your unique take because I also think you do an excellent job whenever I’ve engaged with you in various places of providing really good replication steps. And so I love that, and I wanna offer things specific to WordPress itself and something that I’ve noticed that’s more cultural rather than necessarily like steps to follow.

And one of the things I’ve noticed that I think has started to come up partially with Covid is people, you know, you start talking at WordCamps or at a meetup, and a bug comes up, and you find someone who knows where to put it, and that kind of connection is has been frayed in the last couple years.

And so one of the things I feel like I’ve been saying to a lot of different people at this unique point in time is that it doesn’t need to be perfect. Don’t let perfect be the enemy of good. And so if it means you just need to drop it in a Slack channel and you just are like, I don’t know where to put this, that’s huge.

We need to hear from people across the project. And I just really encourage anyone, even if you don’t have the complete information or you’re not a hundred percent sure you’re afraid it’s been reported 10 times before, like, please still report it because we need those reports and also if 10 people reported it and it’s still not fixed, that also means we need to iterate.

[Anne McCarthy 00:09:40] 

And so that’s one of the things, especially with the Full Site Editing outreach program, I feel people will message me saying, hey, I’m sure you’ve heard this a bunch, but… And sometimes I’ve never heard it at all. And I shudder to think of all the people who have not reached out or have not posted in GitHub or Trac or wherever.

So yeah, share, and write blog posts. I think that another great way that people can give feedback is if you don’t know how to get into the depths of WordPress, writing a post and talking about it and sharing it on social media is also a great way to get attention. I read a lot of those. But as much as possible, getting to, if you can, if you’re comfortable, getting to the source where we’re able to see it in Github or Trac goes a really long way.

And share as much as you can. And don’t worry if you can’t spend hours writing the perfect bug report, we still wanna hear from you.

[Brian Alexander 00:10:21] 

Yeah. Building off of what Anne said, just the fact that you’re speaking out and raising an issue is a huge step for many, many people. And once, once you’ve actually done that, as Anne said, it doesn’t need to be perfect. There are a lot of other people who are going to be looking at these bugs, trying to figure out the replication steps used.

So even if you can’t provide all this detail up front, someone will help. On the back end, they’ll help kind of fill in those gaps. If you do have the time to actually get deep into providing a very detailed bug report, then there are some key aspects of the bug report that make it very helpful for contributors, not only testers, who should be able to reproduce the issue to validate and make sure that this isn’t something that’s unique, unique to a plugin, to a custom theme or snippet that you dropped into your functions PHP. 

But, also for the actual Core contributors, who then need to be able to understand what is happening so that they can fix the right thing. And some of those items are the information about your testing environment.

[Brian Alexander 00:11:34] 

So that could be your browser, your server, the type, whether it’s Apache, Nginx, et cetera, the operating system you’re running, what version of PHP you’re running, the version of WordPress, very critical, and… 

[Josepha Haden Chomphosy 00:11:49] 

Super important.

[Brian Alexander 00:11:51] 

Any themes and plugins that you’re using. And that kind of information helps set the stage, and then other people will be able to set up their environment similarly if they’re going to try to test it.

After you have provided the environmental information, the steps required to reproduce the issue should be as detailed as possible. You may not have realized that clicking this caused such and such to happen, so just try to remember, or maybe even walk through if it’s something you can repeat multiple times, walk through a couple of times and write down everything that you’re doing.

[Brian Alexander 00:12:30] 

So that you’re sure, hey, this is the way that I can reproduce this bug. And then those steps will be very helpful for other contributors when they’re reviewing it. And then it’s also very helpful if you have video, screenshots, debug logs, any of those other kinds of resources that you could refer to because not all bugs are easy to explain.

And we tend to… Trac and GitHub issues for the Gutenberg project, everybody’s writing in English. And maybe your main language is not English, and it might be a little bit challenging to do that. So providing a video, it’s worth a thousand words in any language. So, if you can provide those types of assets, that’s also very important.

[Josepha Haden Chomphosy 00:13:22] 

Yeah, and I’ll share a little bit of a you’re-not-alone-in-it sort of anecdotes from the first few bugs that I ever filed for WordPress. I sort of had this feeling that if I were to file a bug, everyone would know that I wasn’t a developer. And like everyone knows, I’m not a developer, but a little bit I was like, they’ll know now. And so if that’s where you are also,  Anne said it, and Brian said it as well, like, we can’t fix things that we don’t realize are broken. And just because you’ve run into it 15 times, which obviously should never happen, you should run into it once, and then we know, but it happens.

If you run into it 15 times, probably other people have as well. And if it’s still not fixed, it might be because no one has thought to themselves I should tell someone that’s broken. And so if that’s your primary hurdle, folks out there in our listening space, I was once there too. And honestly, knowing that it’s a problem is as valuable as knowing the solution to it most of the time. 

[Brian Alexander 00:14:23] 

Yeah, and those are, I wanted to add, there is a lot to that to remember. That’s a lot to remember in terms of what you should be submitting, what, or I should say, what would be ideal in what you’re submitting. But luckily, in the test handbook, there’s a test report section, and it includes a description, it goes everything from, it starts with why we do bug reports to examples of the types of testing, whether it be for bugs or enhancements, which also need testing, and it has templates in there that you can copy and paste directly into Trac. And that’s very helpful.

[Josepha Haden Chomphosy 00:15:03]

Yeah,, we will have links to those in the show notes as well. Since we’re right there at that moment, what do you think we could do as WordPress to make reporting problems easier?

[Brian Alexander 00:15:15] 

I know that this has been something that’s come up during our weekly meetings, discussions on the Core test channel, as well as in contributor day test table discussions. And the test documentation that’s on the website is a little bit fragmented. I believe that the current test handbook was originally written for a type of flow analysis and feedback testing that is not the norm today. So it’s a little bit confusing. The terminology is a little dated, and the most recent updates that have been provided on there relate solely to Gutenberg, which is very important that that also be represented, but, in order to find information about testing and Trac or PHP unit tests, you have to go over to the core handbook.

So we could definitely make things improved by consolidating, bringing everything into one area so that if you are interested in testing, you’ll have everything in one place and not be split between that and not have outdated methodologies that are asking you to submit videos that nobody’s going to really look at because we’re not doing the flow tests anymore. So I think that that would be a benefit to future testers.

[Josepha Haden Chomphosy 00:16:41] 

Anne, any thoughts?

[Anne McCarthy 00:16:43]

Yeah. I’ll also add that I think there are like two things we can do. One is, there’s so much happening in the WordPress project in such a cool way that I think the more we can write targeted tests and talk to people about, like, hey, here’s this new thing coming. This is a high-impact area to test. It’s under active iteration. You’re gonna get a lot of engagement. People are really thinking about this and pulling people into that where you kind of get the momentum of getting the feedback in right when someone needs it. 

I think we could do that a bit more to make reporting problems easier because it’s kind of like you’re in the thick of it with a lot of people rather than maybe exploring an area where someone hasn’t looked at it in a minute.

So that’s the thing that comes to mind is just the more we can take the time. I think this release cycle has been really good with that, where there’s been a call for testing for fluid typography. There’s also been one for using block template parts and classic themes. And there’s a ton of stuff that’s been happening where we can kind of make these both developer and more end user testing experiences easier and better.

And Brian has done a great job continuing the tradition of, you know, helping test this latest release cycle. And he’s taken those posts and done an amazing job of helping, having specific testing as well. Tied to this, I think just this has always been a thing but better, easier testing environments for developers and for quickly setting up more WordPress sites to test things for end users.

[Brian Alexander 00:17:56]

Yeah. Another thing that we have been discussing in Slack in the Core and Core Test channels is the possibility of pre-populating the Trac tickets. With a template based on what it is that you’re reporting. So similar to copying a template for a test report out of the handbook. Instead, you would hit a button to say the type of bug you are submitting, and then it would pre-populate that, and then you could fill in the gaps for that. This already happens over in Gutenberg. There, there are templates, and I find that that is very helpful. And so being able to do that in Trac would be useful. 

And then for reporting problems on the user side, I thought that it would be interesting to have like you have for any other modern app, a button that says Report Bug in WordPress that could capture some intelligence data for your installation, the page that you’re on and have a simple text box where you could provide a little description and then submit that.

[Brian Alexander 00:19:08] 

Now, these wouldn’t be the types of things that would just go straight into Trac, most likely. However, it would be an opportunity to allow end users to just send something in, and start having it looked at, rather than looking and saying, okay, I found a bug in WordPress. Now, what do I do?  And then not reporting. 

So that would be the worst case is that the bug just doesn’t get reported. So that would be information that is already harvested if you go to your site health screen and your WordPress installation. A lot of that information that would be useful is there. In this type of bug report, we would want to anonymize and strip a lot of that information out.

There’s a lot of private stuff you don’t wanna share, but there is that data there that’s available that could potentially help in doing a bug report.

[Josepha Haden Chomphosy 00:19:57]

Brilliant. All right. Question for everyone in the room: what opportunities are there currently to help with testing? Anne, I know, and you already mentioned a few, we can just bombard everybody with links to the tests if we want. But yeah, what opportunities are currently out there?

[Anne McCarthy 00:20:13]

Yeah, I’ll mention the Full Site Editing outreach program. I’m very biased, but we’re always looking for new folks. We just crossed, I think, 600 people, which was unbelievable. So even if you’re not necessarily always able to help join the calls for testing, you can always pop into the FSE outreach experiment channel, which we’ll also add a link to.

And that’s just a great way when you have time to join because I flag stuff all the time, whether it’s about the outreach program or just in general across the project. Brian does really good weekly round-up testing posts as well. So make.wordpress.org/test is also a great place to get started.

And then right now, I think when this comes out, will be a great time to be helping test WordPress 6.1. So check out that post. I kind of wanna just shove everyone in that direction currently cause I think that’s the most high-impact thing to get involved with and one of the great ways to give back to the next version of WordPress to make it really delightful and easy to use.

Yeah, I’m just gonna leave it there, even though there are so many ways you can help.

[Josepha Haden Chomphosy 00:21:11]

WordPress 6.1 coming out on November 1st if you haven’t yet heard about it. Brian, what else have you got out there?

[Brian Alexander 00:21:16] 

In terms of the online stuff, Anne covered that pretty well. I would say if you have a local WordCamp, sign up for their contributor day or if there are any local WordPress meetups. When Covid ended up hitting and lockdowns were rolled out, a lot of this stuff started to really slow down. So I think now is a good time to maybe introduce the idea for, hey, let’s have a local meetup, and for a couple of hours, we’ll just do some testing, and look at some stuff in WordPress. 

So it might be a good way of getting people re-engaged. It’s a little bit lighter weight if you’re doing testing versus trying to actually provide a patch to fix an issue. So, might be a good way of bringing in some new faces and re-engaging people who we lost over the lockdown.

[Josepha Haden Chomphosy 00:22:09] 

Yeah, and if you all have never done a testing party for WordPress before, and it sounds like it’s maybe a really boring thing, it’s actually not, she said with strong authority and opinions. But also, I have never had a more successful learning experience with the WordPress CMS than when I was trying to figure it out with other people.

They see things that you don’t see, they know things you don’t know, and it really covers a lot of the bases for unknown unknowns when you’re trying to learn something. And then also you have all these people that like, we’re really in it with you, and everyone’s really pulling for each other, and it’s actually a bit more fun than it sounds like when you’re just like, a testing party. It turns into just like jointly solving a puzzle together, which I think sounds like a lot of fun.

It’s like a party, but for technology, I would feel this way. I am a mad extrovert, and we all know it, but. Now you two know it as well.

[Josepha Haden Chomphosy 00:23:08] 

I have a final, just like a fun question for you both, and if you have an answer, great. And if you don’t have an answer, I would be surprised.

So here we go. Last question of the day. If five more volunteers suddenly appeared to help on the Test Team, what would they do? Just, I waved a magic wand. I guess that’s what made it fun. I don’t know why. I was like, fun question and then I’m, like, assigned tasks that, Yeah, I waved a magic wand. That’s what made it fun.

[Brian Alexander 00:23:38] 

Yeah, I would say I would probably point them to FSE outreach program posts because…

[Josepha Haden Chomphosy 00:23:45] 

Woot woot. 

[Brian Alexander 00:23:47]

…the outreach program does a great job of outlining steps. You’re isolating testing in one particular area. It’s got a lot of tests. There’s examples of the types of feedback that you’re looking for, et cetera.

That’s a really good introduction to it, and most FSE testing does not require a local dev environment. Which is probably the biggest hurdle for a new tester coming in. If you do have developers with more experience, then they could start–and they wanted to look into Trac tickets or GitHub issues– then it does take a little bit of setup and you may spend the next few hours configuring your development environment.

So instead, I would recommend that you start with something like FSE outreach program posts.

[Anne McCarthy 00:24:37]

I did not pay Brian to say that. 

[Josepha Haden Chomphosy 00:24:42] 

We’re just all partial to it here. That’s all.

[Anne McCarthy 00:24:45] 

No, we really are. Yeah, no, this is, I love this question, and I actually find it really fun cause I think about it a lot. And we’ve talked about some of this stuff too, and it’s something that when I think about five more people suddenly appearing, makes me giddy.

Because we have folks, who have helped with like, I think I’ve mentioned like translations and group testing and even responding to questions that come from the channel and like, I just wish if we had five folks full time dedicated to that, I could see way more hallway hangouts where we casually talk about stuff and actually go on a call and talk live.

I could see folks, someone dedicated to helping translations and translating even more places. We have an Italian contributor who does it regularly, and a couple of Japanese contributors every once in awhile we get Spanish translation. But I’d love to see more translations to bring more people in, more facilitating group testing, more types of testing, helping me be more creative because sometimes I get a creative wall.

But more than anything, if I really think long term about the project and thinking about this outreach program model, which I don’t think I fully appreciated how new it was, Josepha, when you introduced the idea, I think it would be so neat to bring in more folks to actually create new outreach programs.

[Anne McCarthy 00:25:52] 

So maybe there’s an outreach program for theme authors or block theme authors, or maybe there’s an outreach program around collaborative editing. Like what does this look like, and how can we expand this to bring more people in? And I think a lot of that will prove the resiliency and lessons we’ve learned from Covid in the WordPress community. 

We can’t necessarily always rely on the meetup groups, so how can we meet people where they are? And I think there’s something really interesting and almost serendipitous that the outreach program started literally, I think it was like May 2020, like a couple of months into the pandemic.

And I, like, I want to see it in a position of strength where we both have the in-person community alongside this outreach program model that can intertwine work. And I’d love to see the model expand to different types. And right now, maybe part of that is we use the outreach program model, the full site editing outreach program group itself, to experiment more and to keep that level of experimentation.

That’s something I feel really strongly about is continuing to find what works and what doesn’t. And so if we had five more people, I could just, I’d probably go wild and have all sorts of cool, cool things and spinoffs, but I’m more introverted than Josepha, so there’s limitations to this.

[Josepha Haden Chomphosy 00:26:56] 

Well, you heard it here first. If you’re one of my 6,000 listeners. I only need five of one of you. Five of the ones of you to come and make Anne’s whole life an exciting joy for the next 12 months. So, I only need five of you and I know that you’re out there. There are 2000 or something, 6,000. I have no idea.

I’ve got more than 1000 of you listening, and I know that you wanna come and help Anne cuz she’s a delight. I know you wanna come help Brian cuz he’s a delight. Both of you. This was such a fun conversation. Thank you for joining me today.

[Brian Alexander 00:27:29] 

Thank you, Josepha. Thank you, Anne.

[Anne McCarthy 00:27:31] 

Yeah. Thank you.

[Josepha Haden Chomphosy 00:27:33]

And there it is a bit of a deep dive on the Test Team and how to get started on it. Like I mentioned, we’ll have a ton of links in the show notes over on wordpress.org/news. And I wanna remind folks that if you have questions or thoughts that you’d like to hear from me about, you can always email us at WPbriefing@wordpress.org.

[Josepha Haden Chomphosy 00:27:58] 

That brings us now to our small list of big things. First and foremost, we are counting down the days to the WordPress 6.1 release. We are within a month of the target release date. So if you have not tested the latest version with your plugins or themes, now is the time. 

Secondly, we are seeing translated tutorials being submitted on learn.wordpress.org. I’m delighted to see that happening, and I encourage any polyglots out there who feel called to consider translating one into your language and help other people feel empowered to use WordPress. 

And then the third thing is that the WordPress Speaker Workshop for Women Voices in India just concluded, so to celebrate, we’ve opened registrations for the WordPress Speaker Workshop for Women Voices in Latin America. Unlike the last one, this event takes place in person on October 29th. And so I’ll include a link to registrations for that in the show notes as well. 

And that, my friends, is your small list of big things. Thanks for tuning in today for the WordPress Briefing. I’m your host, Josepha Haden Chomphosey, and I’ll see you again in a couple of weeks.

View the full article