An Overview of Facebook Moderation — Part 1

7 min readApr 9, 2020

Aside from the fun Happy birthday photos and endless pictures of cats, Facebook is regularly used for more sinister reasons such as child nudity, firearms sales, self-harm, grooming, and beastiality, to name a few.

In this article, I am focussing on the process Facebook employs to identify and remove child abuse images and videos. We are not making a distinction between content shared by children or adults.

Community Standards

Having to balance freedom of speech with illegal and nefarious activity, Facebook has devised a set of guidelines in an attempt clean up its platform. What is referred to as their ‘Community Standards’, these guidelines outline what is and isn’t allowed on Facebook, segmented into 6 categories: Violence and criminal behaviour, Safety, Objectionable content, Integrity and authenticity, Respecting intellectual property, and Content-related requests[1]. Specifically with regards to child safety, Facebook also seeks advice from their Facebook Safety Advisory board made up of independent online safety organisations and experts[2].

To briefly summarise, Facebook bans the following content in relation to minors:

Content that depicts participation in or advocates for the sexual exploitation of children
Content that constitutes or facilitates inappropriate interactions with children
Content (including photos, videos, real-world art, digital content and verbal depictions) that shows minors in a sexualised context[3]

Whilst these guidelines appear to be comprehensive, both the content removal process and the interpretation of the above guidelines are still failing to protect children and allowing this content to remain on the site.

Content Removal

There are three main methods for content analysis and removal: PhotoDNA, AI, and Human Moderation. We believe the order of the process is depicted in our diagram below, though this has been devised based on multiple reports so is subject to change.

Speculative Facebook Moderation Process by DragonflAI

PhotoDNA

Developed in 2009, Microsoft partnered with Dartmouth College to develop technology to aid in the detection and removal of child exploitation images, and more recently videos[4], which it subsequently donated to the National Centre for Missing and Exploited Children (NCMEC) in the USA.

For the 200 organisations employing PhotoDNA, they can scan an uploaded image by converting it into greyscale, applying a grid, and then assigning a numerical value to each grid square, which is cross referenced with already known child abuse images[5].

PhotoDNA process from Microsoft. Credit: https://news.microsoft.com/on-the-issues/2018/09/12/how-photodna-for-video-is-being-used-to-fight-online-child-exploitation/

PhotoDNA is one of the most progressive leaps in protecting children online by identifying content, removing the requirement for humans to view these images and more recently, it can identify content sent within E2E encrypted messages[6].

However, it only works if an image is one of the 300,000 images or videos already in the database[7], and it is merely voluntary resulting in lack of adoption.

There were claims that PhotoDNA struggles to match identical images that have had filters applied, such as cropping, colour changes, and text additions, though this has been proven untrue[8]. It has also been thought that many companies don’t employ PhotoDNA as they don’t want NCMEC to know how much child abuse content is on their platform, though this has not been proven.

Facebook has been rightly credited for employing PhotoDNA on their platform, but it is unknown how much content was removed as a result of applying this technology.

Algorithms

If PhotoDNA doesn’t find a match, the next step in the moderation process is proactive analysis using their own algorithms.

In Q3 last year alone, Facebook removed 11.6 million pieces of child nudity and associated content according to its bi-annual transparency report, with 99% of it removed proactively. This is a rise from Q1 2019 where they removed about 5.8 million pieces of content[9].

In a positive move, Facebook claimed to have expanded their moderation efforts to remove identical indecent child nudity content that is found on both Instagram and Facebook. However, with only 754,000 pieces of content removed on Instagram, this attempt appears to be somewhat futile.

Human Moderators

The final line of defence is human moderation; where any content that is detected by artificial intelligence or by user reporting is sent for human review. Facebook has approximately 15,000 content moderators in the US, who are hired by third-party contracting companies, mainly Cognizant.

This chart shows the overview of Facebook content moderator pay and jobs. Credit:https://www.statista.com/chart/17302/facebook-content-moderator/

Moderators will review a reported post roughly every 30 seconds and decide whether it remains or is removed, meaning a moderator sees roughly 400 items a day. Other sources dispute this number claiming moderators see up to 8,000 posts a day, or every 10 seconds[10].

Asuming 400 posts a day are viewed, across a week, a moderator will review around 1500 posts, of which 50–60 are audited by another employee to assess the accuracy of the moderator’s decision[11]. Moderators are expected to enforce Facebooks community policies and guidelines with a 98% accuracy, and if not, there are consequences which include being fired[12][13]. There are other articles stating that Facebook’s accuracy target is in fact 95%[14].

Whilst moderators are usually forced to sign 10+ page NDA’s[15], and more recently, post-traumatic stress disorder (PTSD) disclosure forms[16], many have broken these agreements to bring lawsuits against Facebook and their associated companies managing moderation for psychological trauma and developing PTSD from seeing horrific images day-in day out[17]. In more severe cases, human moderators are also becoming addicted to graphic and extreme content[18], and one employee even died at his desk[19].

As stated previously, the community guidelines may appear to be clear cut but that is simply not the case for moderators as it has been claimed that the guidelines internally change daily, are highly subjective based on context, and the volume of content is too high to make accurate decisions.

On top of the above issues, Covid-19 has forced Facebook to alter it’s moderation practises and shift content moderation for the most severe types of content, including child exploitation, to full-time employees because it can’t provide the same mental health support to contractors working remotely — though it has not disclosed figures[20]. Furthermore, due to privacy concerns it meant that some data could not be shared with the those working away from base, and as a result Facebook is unable to rely on the full extent of human moderation and is having to increase AI deployment[21].

Whilst human moderation currently plays an important role in removing child abuse content, people should not have to be subjected to such horrific and despicable content, making this method unsustainable.

Has it lost control of child abuse imagery?

With a rise in the number of child abuse content removals, as put by BBC’s Angus Crawford “either Facebook is doing an even better job of detection than before — or it has lost control of the problem”[22].

“It’s not something we are completely on top of”
- Nick Clegg, Head of Global Affairs, Facebook

A 2017 BBC investigation tested Facebook and reported 100 images which appeared to break its guidelines relating to child exploitation or nudity.

They included:

pages explicitly for men with a sexual interest in children
images of under-16s in highly sexualised poses, with obscene comments posted beside them
groups with names such as “hot xxxx schoolgirls” containing stolen images of real children
an image that appeared to be a still from a video of child abuse, with a request below it to share “child pornography”

Of the 100 images only 18 were removed — these figures potentially indicate that the problem is atleast 5x bigger than Facebook’s biannual report and they have likely have lost control of the problem. On another interesting note, when BBC brought the images to Facebook’s attention at their request, Facebook then reported the BBC to the UK’s Nation Crime Agency[23].

The BBC are not the only ones sceptical of Facebook’s attempts to curb child abuse. An Independent UK-held Inquiry into Child Sex Abuse shared this sentiment and found that Industry leaders, including Facebook, have all struggled to ascertain “the scale of the problem on their platforms and services”, and should “do more to identify the true scale of the different types of offending”[24]. Even Facebook’s Head of Global Affairs, the infamous Nick Clegg has freely admitted that “It’s not something we are completely on top of”[25].

To be continued.