In cyber security, we adopted the military model of red teams for penetration testing and blue teams for vulnerability analysis. I say it’s time for a new gray team – people charged with thinking through the misuse of systems. These are not attackers, but hostile users. This would be an uphill battle.
Everyone has heard the phase: “Move fast and break things.” However, there’s another mantra that is just as binding in software engineering: “You don’t need a solution until there’s a problem.” A gray team would take on this mindset. It would not wait until the problems occurred, it would prevent them.
Here’s example of a problem that should have been anticipated, but wasn’t: A woman using a P2P payment app kept receiving $1 payments accompanied by evil, threatening messages from a stalking douchebag ex, but there was no way to block someone sending a payment.1 It’s tempting to think: “Why would you ever want to block someone sending you money?” But the problem was the hostile use of the messaging aspect of the system. A gray team would know that almost all forms of human communication have been misused for abuse.
Another horrific example has happened to women who tragically have miscarriages, but their browsers and social media feeds remain flooded with ads for new baby things for weeks and weeks.2 Why isn’t there a way to report these ads for being in situationally poor taste? Organizations have been funding algorithm development to spot expecting parents, but a gray team would have identified that we needed a user-controlled “flush” command on this profiling. (If you’ve ever tried to report an ad for being misleading or in poor taste, you’ll soon find that those are not considered legitimate reasons to report ads.)
Some other examples of problems that could have been prevented:
- Gay men have been hunted and beaten by homophobic thugs using the leaked location data from a popular gay dating app.3
- A person set up a small business account with a credit reporting bureau. He then downloaded hundreds of thousands of credit reports. 4
- Realtors are reporting that vacant properties are receiving multiple letters addressed to different people from State unemployment benefits departments.5
- End-users have clicked on ads that contain links to malware sites.6
These are all examples of how a huge bling spot is created by the mindset: “You don’t need a solution until there is a problem.” When I first entered the work force, there was a humorous saying: “When you’re up to your ass in alligators it’s difficult to remember your original objective was to drain the swamp.” The main benefit of a gray team is that preventing problems will always be cheaper and easier than fixing them afterwards. Also, anticipating problems will prevent “technical debt” from piling up.
We need gray teams because it’s not realistic to expect individual software engineers to deal with this. It’s not about individuals, it’s about organizations. Organizations need to step up their game to protect end-users and conserve cash by avoiding costly rework.
Back in olden days (the 1980s), a wise mentor of mine observed: “Any system humans can create, humans can subvert.” Digital transformation will be a lot more productive and less abused if organizations methodically searched for and prevented ways to subvert the systems we create. Gray teams would be one way to do that.
References:
- P2P Payment Stalker: https://twitter.com/TweetAnnaMarie/status/766774995057987585
- Miscarriages then targeted with new baby ads: https://www.huffingtonpost.co.uk/entry/women-affected-by-miscarriage-and-infertility-are-being-targeted-with-baby-ads-on-facebook_uk_5d7f7c42e4b00d69059bd88a
- Grindr location data leaking: https://www.newsweek.com/grindr-location-leak-1453697
- Credit Bureau Misuse: https://www.businessinsider.co.za/the-personal-details-of-millions-of-south-africans-have-just-been-hacked-2020-8
- Unemployment fraud https://www.ksby.com/news/local-news/central-coast-residents-flooded-with-fraudulent-edd-letters
- Malware in ads: https://www.imperva.com/learn/application-security/malvertising/
Much respect and appreciation to Jonathan Rothwell and Steve Freeman for their excellent presentation “So You Can Sleep at Night” https://youtu.be/A5umy4lUOOY They approach this as an inquiry into software engineering ethics. My approach is to build upon this from an organization systems perspective.