The social media policy ratchet | Open Rights Group
Blog The social media policy ratchet
Digital Privacy
The social media policy ratchet
For more than a decade, UK governments have introduced successive child safety measures, responding to public concern about the availability of content that is either unsuitable or harmful to children, or due to harmful interactions ranging from bullying. Yet at each stage, the measures have delivered far less than was expected. Meanwhile, as online problems appear to escalate, the measures imposed seem more and more extreme, and even politicians admit that they’re likely to be less than fully effective. What is going on?
This blog aims to be an in-depth analysis, using some analytical tools that you may not be familiar with. They should be reasonably intuitive, but take your time if you can, as I try to bring some new ways of looking at the Online Harms policy space, that government itself says it uses when dealing with complex policy problems.
How it started
To recap what measures have been imposed so far: This began with the UK persuading ISPs and mobile providers to have Internet filters on by default in 2013 The expectation was that these would largely restrict access to adult and harmful content for children and teens. In practice, users in households did not want filters installed across all adults and children. For mobile users, providers found it impractical to block much more than pornography. Most importantly, over-blocking has caused a lot of legitimate and entirely harmless material to be blocked, including advice for people at risk, as highlighted by ORG’s Blocked tool.
The Online Safety debate and then Act was a response to the limited impact of filters, with the policy process starting in 2017. The measures announced in 2019 centred around a ‘duty of care’ and attempted to impose restrictions on harmful content for adults as well as children. The Online Safety Act, passed in 2023, has fuelled the use of age verification to allow adults to access content that might be unsuitable for children. Age Verification remains improperly regulated and largely US processed. Even the poster child of ‘doing it right’, Yoti, has been fined by the Spanish data protection authorities and criticised by privacy researchers.
The OSA is too complicated for us to summarise everything about it. But clearly, segregating categories of content like ‘violence’ (including UK police or Israeli military violence), ‘sexuality’ (including LGBTQ+ advice and social discussions), ‘drugs’ (including discussion of managing harms and avoiding risks), and so on, is problematic. Firstly, because the aim is to prevent teenagers from accessing this content, and secondly because adults are required to age verify and sacrifice privacy to access what can include political and socially important content. Future measures will include automated takedowns, and could even include breaking encryption to scan content.
Finally, the OSA has ended up with small but incredibly safe sites from shutting down or blocking UK access, due to compliance problems. The OSA is in this dimension preventing solutions to the problem of online safety.
Now there are proposals to remove children from social media and games’ chat systems, and for phones to scan for user created sexual images by default. These proposals also mean age verification for adults to access social media or to stop phones from scanning what they take photos of.
What the policy process is doing
Let’s take a step back and look at the way the policy process is designed, and why the measures seem less than effective. Firstly, I would like to introduce something about how government says it develops policy. It provides policy development guidance that is designed to ensure that unintended consequences are avoided, and that policy aims will deliver the intended results. To ensure this is done properly, the Government Office for Science provides an Introductory systems thinking toolkit for civil servants. Patrick Vallance explains:
The design of effective policies for citizens, communities and the UK requires policy makers to understand interconnected systems. For example, the role of transport in accessing healthcare. The most effective policy professionals across the UK Civil Service develop and deploy systems approaches in their work and I encourage all my colleagues to make full use of this toolkit.
What Vallance is pointing to is the problem of repeated policy failures like the War on Drugs, or the way that new roads generate new traffic and fail to stop traffic jams, or how the criminal justice system and longer sentences generate new crime. Highly complex systems and simplistic interventions really do not mix. Genuine solutions require careful thought to ensure that systems and policy goals actually align.
Online Safety is truly an “interconnected system” with a great deal of complexity. Following the government’s policy guidance, then, we can use some of their tools to describe at least some of their interconnections.
This is a basic flow diagram using Systems Dynamics, as the GOS advises. Systems Dynamics is a tool made famous by Limits to Growth, which presented a model of world resource usage that lead to the environmental sustainability debates of the 1970s, and by Peter Senge in the business world, with his book The Fifth Discipline.
The graph describes the social media and online harms dynamics. In the diagram, each measurable part of the system, or “stock”, causes changes in other parts, or “stocks”. Where the relationship is positive, ie causing an increase, such as users creating more content, the arrow is marked with a plus (+). Where negative, a minus (-) shows a negative causal reduction.
This allows us to see the pattern of interactions. While the graphs are my view of the system, most of the relations are generally accepted by people in the debate, including safety campaigners.
The graph shows on the top right the “attention economy”, with attention and user interaction being reinforced by algorithms, and these also reinforcing profits. These are reinforcing loops, or “virtuous cycles” in the view of social media companies. However, from the perspective of safety campaigners and free expression organisations like ORG, Amnesty and others, these cycles also cause increases in harmful content, and illegal content. These are shown in the centre of the diagram.
On the left of the diagram, we see the policy results. Harmful content generates calls for age verification; this generates Evasion by users; this generates calls for tougher age verification, such as application to VPNs, in a reinforcing cycle. As the pipeline of harmful content is not reduced, calls for age restrictions are unlikely to reduce.
For illegal content, we see pressure for more and more content to be moved from Other illegal content, which is anything illegal that a platform must remove if reported, to the Priority illegal content bucket, starting with terrorism and child abuse material, where platforms must remove the content through scanning and algorithmic identification. Here, there may be a bit more policy resistance where too much content is removed.
What is also apparent is that the measures are mostly burdens on users, rather than platforms. They cause privacy and free expression costs, introduce age verification and other barriers that act on users.
The graph illustrates a policy system where the measures are not really acting on the production of problematic content. The engine of online harms, ie algorithmic prioritisation, remains essentially untouched. The attention-seeking goal of the social media system is not changed, and the policy interventions seem to work against rather than with the overall system. This ought to call into question whether the online safety policy approach is really trying to act systemically and effectively. Could other interventions make more sense than simply trying to restrict content and access to content?
In this graph, I have mapped out the basics of the user attention economy that we saw above in a bit more detail. Again, none of the relationships depicted are unknown or controversial. The main loops we see are that the Total active userbase increases Content production, which leads to content prioritisation due to the glut of content. Content production also increases User dependency, which increases the Total active userbase. Content production also increases User satisfaction, also increasing the Total active userbase. So far, all great for the social media company, which also uses the loop of Content prioritisation by algorithms to increase Ad revenue (and thereby Profits).
As before, we note that Illegal and lawful but awful content is also produced by the algorithms system, while Regulatory counter-measures seek to reduce this.
What is illustrated in the graph but appears forgotten in the policy debate however, is that the presence of lawful but awful content, and poor interactions, ought to lead to a decrease in User satisfaction, (illustrated by the long arrow at the foot of the graph) which would then break the reinforcement loop holding up the Total user base. Yet it doesn’t, presumably because of User dependency.
As is noted in many guides to Systems Dynamics, these kinds of graphs help show you where the intervention points really are. What we are looking for is places where there are lots of arrows flowing in. Here the obvious ones are Total active userbase and Content prioritisation by algorithms, and perhaps Content production. These seem central to the system we are illustrating.
In other words, if we could find ways to change how the social media companies retain their Total active userbase, by reducing User dependency, then social media companies might have to re-evaluate what was causing customers to leave. Or perhaps Content production could be less of a driver of User dependency, if people could interact with the content through third parties.
Similarly, if government introduced interventions that acted on the algorithms, like letting users choose any prioritisation engine they liked, then users might act to reduce the harmful content they receive.
Lastly, we should note that in these graphs I have narrowed the policy space that we are looking at. Social media is embedded in society, and wider economics. We have not looked at the role of social media in children’s lives, or their own offline social experiences. There are questions around the way that educators and parents are supported.
However, the analysis above does show that there are a few immediately obvious interventions government could take to change the way the whole system works. They are not a silver bullet, and we do not claim they are the entire answer, nor necessarily politically easy; nor are they a replacement for content duties, but they could help better align the social media content system with the policy goal of safety. They are:
- User switching rights: so users can leave Facebook, Instagram, X, Threads, etc, while retaining their contact networks, as you can with your phone number or your email account. This was identified by the Competition and Markets Authority, for example, as a potential intervention to create competition in social media.
- Freeing user feeds: ensuring that users can control content on platforms, directly or by third parties, so that users can decide what makes them safer and what gives them value.
- Creating competition by breaking up platforms: it is unclear why platforms should be able to own multiple social networks, or why tech giants should be able to use social media platforms as part of a wider business strategy.
- Socially-driven social media: would these dynamics be wholly different if social media was socially owned and controlled? That seems to be the case for Mastodon and BlueSky, where user switching rights are central, and their communities power small environments, which appear to lead to much safer online spaces
That is plenty to think about for now. The UK can stick to a policy ratchet where failed policies lead to more of the same, without really causing anyone at a social media company to blink. Or it can engage in more effective policy design, by looking at the social media system in its fuller context, and then shifting the measures from costs on users, to user empowerment, directly tackling the market power of social media which is generating online harms.
Free expression online
Fix the Online Safety Act
Read more about the Fix the Online Safety Act
campaign