The first meeting: Sometimes in testing, knowing nothing is knowing everything

Yesterday, the Danish part of House of Test met up and discussed life and testing over tiramisu (Which in my opinion is an ideal starting point for discussing anything). We ended up talking about mentoring and coaching, and how one becomes a better tester. That led us to discuss a statement that you sometimes meet when faced with a system that you simply do not understand:

But I don’t know how to test it” or “I don’t know the system, so I can’t test if it work correctly

The value in meeting a system for the first time

Last month I did a one-week project where I put an app through usability testing. The company that hired me was very careful not to tell me a lot about the app and their system, since they wanted me to see it for the first time. That was a very wise decision on their part. While I performed the first testing, and got to know the app, I was struck with the value of this very first meeting with the system. There’s a very important and sometimes overlooked lesson in looking at a system with new and fresh eyes. It’s a fleeting moment, but the very first time you set your eyes upon a system, is a test in itself.

So what can you do, even if you have very little idea about how the system you’re going to test, will work?

first meeting testing

1. We can all make assumptions

We assume things all the time. We take in clues and symbols, and hold that up against our own experiences and frame of reference. From that, we assume things. About the world, about people, about things, about systems. When faced with a system you’ve never met before, use your assumptions as a starting point for testing. If it’s a well designed system, you will know instinctively what the system is about. There will be clues and helpful text that can guide you through the system. If you don’t understand it, it’s not because you’re stupid. It’s because it’s badly designed. There’s a lot of badly designed systems out there.

So write down those assumptions you have before the first meeting, and allow them to change as you grow wiser on the system.

first meeting testing

2. Prepare for the first meeting and set some test missions

Reserve some time for the first test, and make sure you have ways to document it (More about that in point 3). Be comfortable and prepare yourself for meeting the system for the first time. Be aware of your assumptions about the system. Get into the mindset that you are meeting the system for the very first time, and that it probably will be confusing. It’s perfectly okay.

To help you on your way make one or several small test missions that you presume the system’s user would have. It could be anything from “I want to sign up and log in” or “I can search for a trip to Russia”, to “In the overview I can see relevant information on any customer who sent in a health declaration”. You might not know if, what you think is relevant information, is what is actually relevant. But it’s a starting point, and you might just notice that some data is missing from the list.

first meeting testing

3. Note down everything

Everything.Turn on your recorder in your phone, and talk your way through the first meeting and your test missions.If you can’t fulfill your test

Note down every single thought you get, such as:

  • Assumptions about what and who the system is intended for
  • Assumptions about how the system is supposed to work
  • Assumptions about how the system actually work, and how these assumptions slowly change as learn more about it
  • Any fleeting thought that triggers a “this is weird/annoying” response in you

It takes some time to summon the courage to note down the “stupid” questions. Questions where you find yourself thinking “I probably don’t understand this because I don’t know how the system works”. Note those down as well. Even if you feel like it’s a silly thought. If you have it, it’s valid, and others will probably have that thought as well. You can always remove thoughts from your notes later, but you cannot remember all your thoughts that you didn’t note down on the first meeting. It would be such a shame if that weird thought that actually held a crucial detail, was discarded.

first meeting testing

First meeting testing – An addition to your test tool library

I must point out that I don’t think this should be the only way to test a system. There are some systems that require expert knowledge, and are only used by expert users. I’m working with large and complex financial systems, where I need to learn a lot of things before I can fully understand what needs testing and how. I rush through examples, guides, business rules and requirements to fully grasp when a health declaration can and should match a request for life insurance. I have to understand 5 different programs. Not know how to use 5 different programs, understand what takes place behind the user interface of each one of them. If I didn’t know all the business rules, and I didn’t do any other testing than the above mentioned method, the systems would fail miserably.

I will however say that when a new program was developed in my project, and I was asked to test it, I did a “first meeting” test. I did that because I knew that it was a completely new program, that none of the experts had any experience with. That led to several bugs being found, followed by a lot of adjustments in the user interface. And that in turn led to a happier end-user. So there might just be room for a first meeting test in your complicated project as well.

30 Days Of Mobile Testing: Share a mobile testing blog post

Today is “Share a mobile testing blog post” day!

30 Days Of Mobile Testing, day 4: Share a mobile testing blog post with a mobile developer or designer.

I don’t follow a lot of blogs. My digital method of obtaining test knowledge goes through Google, and through whatever articles my colleagues or connections share on social media. I don’t enjoy traversing through endless mails or RSS feeds with notifications of new blogs that I might find useful.. So I just don’t. So for this post, I had to go into the great Internet. The biggest problem with that is that the first 10 pages of results are from larger companies who know their SEO. So the smaller, and interesting articles, are harder to come by.

I’m sharing a blog post on usability testing for mobile devices by Hannah from UserTesting. The article is from 2014 so it’s a little outdated, but it’s a library of brilliant articles and blog posts to get started on usability testing on mobile devices. There’s everything from the basics of usability, what’s special about usability testing on mobile devices, to presentations, books and courses. It’s a very nice start if you’re curious about the subject.

I love usability testing. I love the different processes, methods and the way you really need to think and empathize with the users. I’m working on small app usability test projects for a Swedish company, and it’s so much fun. More about that in later posts!

Go read Hannah’s article here: https://www.usertesting.com/blog/2014/10/10/mobile-app-usability-resources/

P.S. If you happen to know a great way to keep yourself updated with relevant and great blog post in a non-obtrusive and time-efficient way, please let me know!

The 30 Days Of Mobile Testing is the creative effort of Ministry of Testing, and anyone can join in. See more at https://dojo.ministryoftesting.com/lessons/30-days-of-mobile-testing

30 Days Of Mobile Testing: Mobile security testing

The theme of today is mobile security testing:

30 Days Of Mobile Testing, day 3: Perform a mobile security test.

I have no idea how to perform mobile security tests. Luckily, even though I have no idea, I am well traversed in the delicate art of Google, and thus the first result sent me to the OWASP Mobile security project.

Not surprisingly, mobile security testing is a very large area, so I’ve decided to do a micro-security test run. Since I’m starting from scratch, I’ve randomly selected one “check” from OWASP’s neat checklist for doing mobile security testing. I’ll gather information on thischecks, and then apply them to an app on my own smart phone.

I’ve ended up with the following check: “Android Backup Vulnerability”

Android Backup Vulnerability

Infosec Institute has a very informative article about this Android vulnerability. In short, it’s possible to alter the way Android backups/restores an app. This allows an app to introduce a malicious BackupAgent to feed the backup process with custom files and data (Such as new apps) without your consent. To get around this vulnerability, Infosec Institute suggests the following:

If your app holds sensitive data, you can stop allowing users from making a backup of the app. This is done by placing the following line in the AndroidManifest.xml file: android:allowBackup=”false”

Okay. That was simple.

Every Android application must have an AndroidManifest.xml file in its root directory. To get access to the directory, you need a “rooted” Android device, otherwise you’ll see an empty folder. Rooting a device is pretty tricky, so consider thoroughly if you want to put your private phone through the ordeal. However, most security mobile testing requires a rooted device.

After some discussion back and forth, I’ve decided not to root my everyday phone. Instead, I found an example of how the file with the line would look like:

mobile security testing

Image borrowed from the excellent blog post on this exact line at https://www.niih.de/android-allowbackup-false/

Done. That was one check. It looks so easy when I read it now, but I must admit that this post has been underway for some time. I needed to do an extensive amount of research on security testing and the check, simply because I’ve never worked with these things before. The same probably applies to future posts on mobile testing (And the new #30daysofsecuritytesting challenge).

What I’ve found is that the information I need is quite easy to come by, and that I learn a lot from these small challenges. I’d recommend anyone to try it out!

The 30 Days Of Mobile Testing is the creative effort of Ministry of Testing, and anyone can join in. See more at https://dojo.ministryoftesting.com/lessons/30-days-of-mobile-testing

30 Days Of Mobile Testing: Mobile proxy tools

The theme of day 2 in the #30daysoftesting is mobile proxy tools

30 Days Of Mobile Testing, day 2: Use a proxy tool, for example Charles Proxy, to intercept the traffic between the app and the back-end. For example, check the calls for encryption and describe your findings.

So… until this day, I don’t think I ever thought about what a proxy tool actually is. I remember reading about using proxy tools for watching online television in other countries.. But that’s it. I never worked with proxy tools before, and I honestly have no idea what .

What is a proxy tool?

If you look up Charles Proxy, and read about what the program does, you’ll get the following details:

Charles is a web proxy (HTTP Proxy / HTTP Monitor) that runs on your own computer. Your web browser (or any other Internet application) is then configured to get access to the Internet through Charles, and Charles is then able to record and display for you all the data sent and received.

So far so good.

I downloaded and installed Charles Proxy on my laptop. Then I opened Charles and after a few seconds, the program started reading and displaying the traffic between my laptop and the network. In my case, the first 40 entries was from an add-on to a java application that I use to play the online browser-based game Kingdom Of Loathing:

proxy test tool

Charles tells me that the plugin “relay_Guide.ash” sends information quite often over my network. Interesting. Since the add-on changes the GUI in my browser, it makes sense that it needs to update every time I do something in the game.

Now for the network traffic of my smart phone. I had some problems figuring this part out, but ended up finding a small guide that helped me set up the phone’s wireless connection to go through Charles. In short you need to set your phone’s wi-fi proxy setting to “manual” and enter your computer’s IP. I opened two different apps on my phone, the Pinterest and Asos apps.

mobile proxy test

Asos sends a lot more information compared to Pinterest. I couldn’t find any non-encrypted traffic, so everything that went back and forth between the apps and Charles basically looked like this:mobile proxy request

All in all I now have an idea about what proxy tools do. I’m sure that there are apps out there that display more information than necessary in the traffic, and if that traffic is intercepted, you could obtain a lot of vulnerable data.

The 30 Days Of Mobile Testing is the creative effort of Ministry of Testing, and anyone can join in. See more at https://dojo.ministryoftesting.com/lessons/30-days-of-mobile-testing

30 Days Of Mobile Testing: My mobile test lab

Did the thought of a mobile test lab and the problems with updating it and running it ever keep you from doing mobile testing? Don’t be discouraged, there are still so many possibilities.

I decided to join the mobile #30daysoftesting challenge from Ministry of Testing since it’s about mobile testing, and I love mobile testing. I did in-the-wild mobile testing on a project last year, and had a blast. The mobile test format is an interesting format to work with, especially if you’re used to working with mostly computers.

30 Days Of Mobile Testing, day 1: Take a photo of your mobile test lab

I don’t own a place or work anywhere with access to a fancy and up-to-date mobile test lab. But that doesn’t mean that I can’t do a lot of testing! My current lab consists of the following:

  • Samsung 6 smart phone (My own phone)
  • Samsung 4s (My partner’s phone)
  • iPod (that I got in a contest several years ago)
  • Samsung Galaxy tablet
  • My computer

That’s it.

Obviously this leaves a large gap for the Apple segment. Or any other segment other than Samsung and Android, really. So what do you do when your mobile test lab is not enough?

Emulate the devices

It’s possible to make a mobile test lab out of your laptop. There are countless of websites that let’s you paste in a web-address, and then displays how that website looks like on a certain device. There are also developer tools in your preferred browser that let’s you view a website in “Device Mode”.

Be aware that emulation is not a direct substitute for testing on an actual device. It can take you a long way, but it’s not the same experience. Emulation cannot mimic the way a device is used. It cannot emulate how different shapes of rain-soaked fingers taps on the screen. Or how an app is activated when rummaging around in a pocket.

Invite your friends

If the mobile test lab won’t come to me, then I must go to the mobile test lab. People in your network own many different kinds of devices. Dependent on your target group, they can be a good sample group for the devices used in the real world. Sometimes it’s even possible to let them do most of the testing. The times I’ve asked friends for help with testing, I’ve given them a small introduction to the project and their role in it, asked them to think out loud, and then sat back taking notes and observing their interaction with the devices.

mobile test lab

Good friends gathered to enact a test scenario where they were to order items from a web shop on their own devices after having a glass of wine. Or three.

…but plan it beforehand

You can give a lot, a little or no information about the app they are testing. It depends on how biased you want your testers. Make sure to choose your words carefully when introducing the task for your testers. If you tell them that their task is to trash the app and find as many problems as possible, they will do that. And more. To such a degree that their opinions are heavily exaggerated in a negative direction. If you tell them that you and your team spent several months working on this app, they might hold back on the negative feedback because you are their friend – and they don’t want to hurt you, knowing how hard you worked the last couple of months.

When inviting people to help you out with your testing, your most important task is to make them feel like you appreciate their effort. After all, the do take several hours out of their calendar to help you out. I usually either invite people over for dinner, invite people out for dinner, or visit them in their home bringing a little something.

Look for used devices

If you must test how a certain website or app functions on a very specific device, go out there and get your hands on it. I recommend asking if anyone in your social network (LinkedIn, Facebook, Twitter.. whatever) has one lying around. You’ll be surprised at what people keep in their drawers.

You can also look for used devices on sales pages (In Denmark dba.dk and degulesider.dk are good places to go hunting).

mobile test lab

The 30 Days Of Mobile Testing is the creative effort of Ministry of Testing, and anyone can join in. See more at https://dojo.ministryoftesting.com/lessons/30-days-of-mobile-testing