Overview

The process for calculating the bias of each media source involves the following seven steps:

1. Ingest news articles: collect articles from New Zealand media sources.
2. Filter news articles: automatically select articles that discuss New Zealand politics.
3. Entity & Sentiment Analysis: extract references to members of parliament (MPs) and political parties from article text and analyse the sentiment of each sentence.
4. Party Bias Calculation: party bias is measured by taking the mean bias error (MBE) between the given party and all other parties, for a particular media source.
5. Media Bias Calculation: for a particular media source, the mean of the absolute MBE score for each party is taken.
6. Coverage Calculation: the sum of all times that an MP of a particular party or the party itself are mentioned in articles published by a particular media source.
7. Political Leaning Calculation: MBE between all left-wing parties and all right-wing parties.

Each step of the process is explained in more detail below.

1. Ingest news articles

Every day, we collect news articles from the most popular New Zealand political media sources, including the mainstream media and political blogs. The full text of each article is extracted with a content extraction algorithm and indexed with a full text search database for further filtering.

2. Filter news articles

Once the articles for a given day are collected, we determine whether to analyse an article based on whether the article mentions a current member of parliament (MP) or political party (current on the day of publication). We maintain a database of all elected MPs and political parties.

3. Entity & Sentiment Analysis

To analyse the bias of each media source, for each filtered article, we analyse the sentiment of each sentence in the article and extract references to MPs and political parties (entities) from the text. The sentiment score of sentences that mention MPs or political parties, contribute towards the calculation of each media source's bias.

The entity extraction and sentence sentiment analysis is performed with the Google Cloud Natural Language engine, which uses neural-network based entity extraction and sentiment analysis models.

3.1 Extract entities

The entity extraction model extracts references to MPs and political parties from the article text, including when people or organisations are mentioned with proper nouns (e.g. "Jacinda Ardern", Judith Collins"), mononyms (e.g. "Jacinda", "Judith"), nick names (e.g. "the Greens", "the Nats"), titles (e.g. "Prime Minister", "Leader of the Opposition") and pronouns (e.g. "he", "she", "they").

The model assigns a known identifier to each extracted mention of a person or organisation, enabling us to collect statistics about each MP and political party mentioned in the text.

3.2 Sentence sentiment

The sentence sentiment model calculates the sentiment of each sentence in the article. Sentences that mention MPs and political parties (determined by the entity extraction model) contribute towards the media source's bias score.

4. Party Bias Calculation

To calculate the bias a media source has towards or against a particular political party, a timeseries dataset is created which contains the mean sentiment score for each week, grouped into two party groups: the party being measured and all other parties. Weeks with empty sentiment scores are filled with the mean sentiment for the party group. The mean bias error (MBE) is calculated on the weekly sentiment data for each party group.

A positive MBE score indicates that the media source reports on the party in a more positive light than all other parties combined and a negative value indicates that media source reports on the party in a more negative light than all other parties combined.

The greater the magnitude of the MBE score, the greater the bias towards or against a particular political party.

More details about sentence inclusion and exclusion criteria are given below.

4.1 Sentence Inclusion & Exclusion Criteria

A sentence is identified as belonging to a party if the sentence contains at least one reference to a current MP or party (on the day of publication).

When a sentence contains references to multiple parties or MPs from different parties, it becomes ambiguous as to which party should be assigned the sentiment score. Is the sentence a quote of party A criticising party B or vice versa? Perhaps the reporter is praising or criticising all parties in the sentence, or maybe the reporter is criticising party A and not party B? For this reason we exclude these sentences.

There are two other conditions for inclusion & exclusion:

• For Government parties: if the other entity mentioned in the sentence is the Government, the sentence is included. Parties inside government are generally not critical of their own Government, so it is more likely that the party is saying something positive about their own Government or the reporter is saying something positive or negative about the party and the government.
• For non-Government parties: the sentence is excluded if the Government is also mentioned in the sentence. It is ambiguous as to whether the party is criticising the Government, the Government is criticising the party or if the reporter is saying different things about the party and the Government. It is more likely that the party is criticising the Government, however, in this case the sentence should be excluded anyway as the sentiment score would be better attributed to the Government.

5. Media Bias Calculation

The summary bias calculation for each media source is calculated by taking the mean of the absolute party MBE scores. The purpose of this metric is to gain an understanding of the magnitude of the bias of a particular media source, regardless of political leaning.

6. Coverage Calculation

Political party coverage for each media source is calculated by counting the number of times a media source mentions the MPs from a given party and the number of times the party itself is mentioned. This calculation counts all sentences where MPs or parties are mentioned, even if there is more than one entity from a different party mentioned in a sentence.

7. Political Leaning Calculation

The political leaning of each media source is calculated with a similar methodology to the party bias calculation, however, the MBE is calculated on two different groups: left and right-wing parties.

A political leaning less than zero indicates that a media source favours left wing parties, whilst a political leaning greater than zero indicates favour for right wing parties. The reason for left-wing being less than zero and right-wing being greater than zero is that on a number line, numbers less than zero are represented on the left side of the X axis and numbers greater than zero are on the right side.

The parties in each category, for each government term are given below.

8.1 52nd New Zealand Government: 2017/10/26 to present

Left-wing:

• Green Party
• Labour Party

Right-wing:

• ACT Party
• National Party

Excluded:

• NZ First: excluded because they market themselves as being a centre party.