Names

• Mia Jerphagnon
• Ari Juljulian
• Nicholas Jumaoas

Abstract

At a time when TV entertainment is almost exclusively dominated by streaming platforms, consumers rely on companies like Netflix, Amazon Prime, and Hulu to watch their favorite shows. However, online discourse has emerged that suggests streaming has led to many shows being canceled prematurely, even if the ratings are high. This accusation is particularly aimed at Netflix for canceling well-received shows such as Julie and the Phantoms and Shadow and Bone. In addition to the distress caused by canceled shows—or shows that are only greenlit for a couple seasons despite potential—for consumers, the trend of short lifespans also deeply impacts those employed in the entertainment industry. Cast, crew, and everyone else involved in a show are unable to have long-term and reliable unemployment. This impact is especially important within the context of the recent SAG-AFTRA strike of 2023.

Netflix’s actions have raised concerns over its business and content strategy, leading many to debate whether they should cancel their Netflix subscriptions. Thus, our project aims to analyze the longevity of Netflix shows in comparison to other shows, as well as in relation to other factors such as ratings, genre, and more.

Exploratory Data Analysis

Data Overview

For the purposes of this project, we will be using a collection of datasets compiled by Kaggle user Diego Enrique, constructed using data from site JustWatch in March of 2023. These datasets contain basic metadata for shows on streaming platforms, with separate, standardized datasets for Netflix, Amazon Prime, Max (formerly known as HBO Max), Paramount, and Apple TV.

This breadth of coverage was one of our main motivations for choosing this collection, since it allows for multiple comparisons under similar conditions. Other points of note include its recency, which is especially relevant considering that Netflix’s chronic cancellations are a somewhat newer phenomenon, as well as its specific inclusion of the number of seasons and release year of each show, which were not present on several of the alternatives.

Pre-Processing

Although the original datasets are conveniently already fairly clean and standardized, we performed some additional pre-processing tasks in order to adapt the data to best suit our purposes. Given that we are primarily concerned with show cancellations, we filtered out movie entries from the datasets, discarding irrelevant information in order to streamline both the testing process and code runtime. For the same reason, we also pruned unnecessary features such as listing description, age certification, and production countries, since they are not pertinent to our analysis. Some of our desired attributes required further transformation in order to be processed correctly or efficiently. After conducting this data wrangling, we arrived at the set of attributes detailed in Table 1.

## Main Pre-Processing Tasks

# Import necessary modules
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import pearsonr
import seaborn as sns

# Read in raw data
netflix = pd.read_csv('raw_data/netflix.csv')
amazon = pd.read_csv('raw_data/amazon.csv')
max = pd.read_csv('raw_data/max.csv')
apple = pd.read_csv('raw_data/apple.csv')
paramount = pd.read_csv('raw_data/paramount.csv')

# Combine datasets (for comparison between services)
data = pd.concat([netflix, amazon, paramount, max, apple], keys=['Netflix', 'Amazon', 'Paramount', 'Max', 'Apple'], names=['platform'])
data.reset_index(level='platform', inplace=True)

# Filter out movies and keep shows
data = data[data['type']=='SHOW']

# Keep relevant columns
data = data[['platform',
             'id',
             'release_year', 
             'runtime', 
             'genres', 
             'seasons', 
             'imdb_score', 
             'imdb_votes']]

# Switch out NaN values in imdb_score/votes
data = data.replace({np.nan: 0})

# Change seasons and imbd_votes to integers 
data[['seasons', 'imdb_votes']] = data[['seasons', 'imdb_votes']].astype(int)

# Save cleaned dataframe as an csv 
data.to_csv(f"all_clean.csv", index = False)

For convenience, an excerpt of the dataset is also provided. Scores of 0 in the ‘imdb_score’ and ‘imdb_votes’ columns were converted from their original NaN values to indicate that the corresponding show did not have a listing on IMDb; no show in the dataset had an initial value of 0 for either of these features.

## Generates excerpt of processed data

data.head()

	platform	id	release_year	runtime	genres	seasons	imdb_score	imdb_votes
0	Netflix	ts300399	1945	51	['documentation']	1	0.0	0
7	Netflix	ts22164	1969	30	['comedy', 'european']	4	8.8	75654
17	Netflix	ts45948	1972	43	['comedy']	1	8.1	2199
35	Netflix	ts20681	1989	24	['comedy']	9	8.9	326487
47	Netflix	ts21715	1984	10	['animation', 'family', 'fantasy', 'music', 'd...	24	6.5	5528

Visualizations

Visualizing Number of Seasons vs. Release Date

Since Netflix's show cancellations are supposedly a relatively recent phenomenon, one would expect that Netflix's shows would be significantly shorter than those of its counterparts in recent years. However, while Netflix shows are historically shorter than those featured on other streaming services, with the exception of the newer Apple TV, it's clear that show length across all platforms has become shorter in recent years. Although some of this can also be attributed to newer shows simply not having aired the entirety of their seasons yet, it would seem that the gap between Netflix and its counterparts is actually noticeably smaller when filtering the data to only include recent years.

## Graphing Average Show Length by Streaming Service over Times

platform_names = ['Netflix','Amazon', 'Paramount', 'Max', 'Apple']
colors = ['red', 'deepskyblue', 'mediumseagreen', 'darkblue', 'black']
palette = dict(zip(platform_names, colors))

def plot_avg_seasons_in_time():
    mean_seasons_all = data.groupby('platform').agg({'seasons': 'mean'}).rename(columns={'seasons': 'all_time'})
    mean_seasons_2016 = data[data['release_year'] > 2016].groupby('platform').agg({'seasons': 'mean'}).rename(columns={'seasons': 'since_2016'})
    mean_seasons_2020 = data[data['release_year'] > 2020].groupby('platform').agg({'seasons': 'mean'}).rename(columns={'seasons': 'since_2020'})

    mean_seasons = mean_seasons_all.join(mean_seasons_2016).join(mean_seasons_2020).reset_index()
    mean_seasons = mean_seasons.melt(id_vars='platform', var_name='period', value_name='seasons')

    plt.figure(figsize=(10, 6))
    sns.barplot(x='period', y='seasons', hue='platform', data=mean_seasons, palette=palette)

    plt.xlabel('Period', fontsize=12, labelpad=5)
    plt.ylabel('Seasons', fontsize=12, labelpad=5)
    plt.title('Average Show Length by Streaming Service', fontsize=12, pad=15)

    plt.xticks(fontsize=12)
    plt.yticks(fontsize=12)

    plt.grid(axis='y', linestyle='--', alpha=0.7)
    sns.despine()

    plt.tight_layout()
    plt.show()

plot_avg_seasons_in_time()

## Graphs Number of Seasons vs. Release Date

data['seasons_smoothed'] = data['seasons'].rolling(window=5, min_periods=1).mean()

platform_names = ['Netflix','Amazon', 'Paramount', 'Max', 'Apple']
colors = ['red', 'deepskyblue', 'mediumseagreen', 'darkblue', 'black']

plt.figure(figsize=(14, 7))
for platform_name, line_color in zip(platform_names, colors): 
    sns.lineplot(data=data[data['platform']==platform_name], x='release_year', y='seasons_smoothed', marker='o', ci=None, color=line_color)
plt.title('Number of Seasons vs. Release Date')
plt.xlabel('Release Year')
plt.ylabel('Number of Seasons')
plt.legend(platform_names, loc='upper left')
plt.show()

Furthermore, Netflix is particularly notorious for cancelling highly rated shows, much to the chagrin of its viewers. However, when graphing IMDb score against show length, there isn't really a strong correlation; if anything, there would be a positive correlation between number of seasons and show rating, which would run contrary to widely held belief. Given these initial discoveries, it would seem that there might be reason to believe that Netflix might simply be a victim of mob mentality, and its frequent cancellations might just be the product of confirmation bias and emotional viewerbases.

Visualizing Number of Seasons vs. IMDb Score

## Graphing Number of Seasons vs. IMDb Score

plt.figure(figsize=(10, 6))
for platform_name, line_color in zip(platform_names, colors): 
    sns.scatterplot(data=data[data['platform']==platform_name], x='imdb_score', y='seasons', color=line_color, alpha=0.3)
plt.title('Number of Seasons and IMDb Score')
plt.xlabel('IMDb Score')
plt.ylabel('Number of Seasons')
plt.legend(platform_names, loc='upper left')
plt.show()

Prior Analyses

Although Netflix's tendency towards show cancellations is popularly known, formal analyses on the subject are few and far between. While media outlets such as Forbes and Wired have published articles discussing these chronic cancellations, usually in attempts to capitalize on the buzz surrounding the cancellations of popular shows such as Shadow and Bone, they lean more on Netflix's press releases and, at most, vague references to data, rather than concrete statistical investigation.

The closest thing we found to an adjacent analysis was a blog post that focused on whether a show's gender and race diversity, both in cast and crew, was related to cancellation. While the author did find that cast diversity, especially in the lead, tended to have an inverse relationship with show renewal, the data that was analyzed was, by the author's own admission, vulnerable to bias by subjective interpretations of race and ethnicity, as well as suffering from small sample sizes.

As for analyses conducted on the dataset we intend to use, and those similar, the vast majority of existing work centers around recommender systems, likely inspired by Netflix's competition to produce the most effective algorithm for predicting user ratings. Diego Enrique, the original compiler of our dataset, also used this dataset for this purpose, which is the only significant analysis conducted on this dataset.

Research Questions

As such, there is a niche that we can fill: a robust statistical analysis of Netflix's shows and cancellations, taking other streaming services as comparisons, using a large, well-curated dataset. Our goal is to uncover the truth behind Netflix's show cancellations, answering two primary questions:

What characteristics lead to shows being cancelled?

Does Netflix really cancel more shows than other streaming services?

Statistical Analyses

Permutation Test 1: Do higher-rated shows have more seasons than lower-rated shows?

Hypotheses

$H_0$: There is no difference in the number of seasons between higher-rated and lower-rated shows. $$\mu_H - \mu_L = 0$$

$H_a$: Higher-rated shows have more seasons than lower-rated shows. $$\mu_H - \mu_L > 0$$

Test Statistic

Difference in means

Significance Value

$\alpha$ = 0.05

# Helper function to calculate the difference in means
def diff_means(df, group, col='seasons'):
    
    # Create a table with the means for each group 
    means_df = df.groupby(group)['seasons'].mean()

    # Calculate the difference in means
    return means_df.iloc[1] - means_df.iloc[0]


def perm_test_1(platform_name):
    
    # Add new boolean column indicating whether a show has a high rating (>7) or a low rating (<=7)
    df = data.copy()
    df = df[df['platform']==platform_name]
    df['is_high_rating'] = df['imdb_score'] > 7

    # Calculate the difference in means
    observed_stat = diff_means(df, 'is_high_rating')

    # Run 10,000 simulations
    n = 10000
    test_stats = []
    
    for i in range(n):
        shuffled_ratings = np.random.permutation(df['is_high_rating'])
        shuffled_df = df.assign(is_high_rating = shuffled_ratings)
        test_stats.append(diff_means(shuffled_df, 'is_high_rating'))

    p_value = np.sum(test_stats >= observed_stat) / n

    print(f'{platform_name} results')
    print(f'Observed test statistic: {observed_stat}')
    print(f'P-value: {p_value}\n')

for platform_name in platform_names:
    perm_test_1(platform_name)

Netflix results
Observed test statistic: 0.5950883583707138
P-value: 0.0

Amazon results
Observed test statistic: 0.6077180096926642
P-value: 0.0007

Paramount results
Observed test statistic: 0.14604515923856276
P-value: 0.3699

Max results
Observed test statistic: 1.4384394341290894
P-value: 0.0

Apple results
Observed test statistic: 0.31404644080700406
P-value: 0.0503

Paramount's p-value is larger than the significance level of 0.05, so, for Paramount, we fail to reject the null hypothesis. There is not statistically significant evidence suggesting that higher-rated shows have more seasons than lower-rated shows.

However, for Netflix, HBO, Amazon, and Apple, the p-values are smaller than the significance level of 0.05, so, for these streaming services, we reject the null hypothesis. There is statistically significant evidence suggesting that higher-rated shows have more seasons than lower-rated shows.

It is important to note that Paramount and Apple had a much higher p-value than the other platforms, so if we had chosen a signifance level of 0.04, for example, we would have failed to reject the null hypothesis for Apple as well.

Permutation Test 2: Do shows of different genres have different season lengths?

For example, let's take a look shows within the Drama genre.

data['is_drama'] = data['genres'].apply(lambda genres: 'drama' in genres)

avg_seasons_drama = data[data['is_drama']]['seasons'].mean()
avg_seasons_non_drama = data[~data['is_drama']]['seasons'].mean()

print(f'Average number of seasons for drama shows: {np.round(avg_seasons_drama, 2)}')
print(f'Average number of seasons for non-drama shows: {np.round(avg_seasons_non_drama, 2)}')

Average number of seasons for drama shows: 2.4
Average number of seasons for non-drama shows: 2.79

At first glance, it seems like Drama shows run for fewer seasons that non-Drama shows.

Let's a run a permutation test to further investigate.

Hypotheses

$H_0$: There is no difference in the number of seasons between Drama and non-Drama shows. $$\mu_{Drama} - \mu_{other} = 0$$

$H_a$: Drama shows have fewer seasons than non-Drama shows. $$\mu_{Drama} - \mu_{other} < 0$$

Test Statistic

Diff means

Significance Value

$\alpha$ = 0.05

def perm_test_2(platform_name):
    
    df = data.copy()
    df = df[df['platform']==platform_name]

    df['is_drama'] = df['genres'].apply(lambda genres: 'drama' in genres)

    # Calculate the difference in means
    observed_stat = diff_means(df, 'is_drama')

    # Run 10,000 simulations
    n = 10000
    test_stats = []
    
    for i in range(n):
        shuffled_ratings = np.random.permutation(df['is_drama'])
        shuffled_df = df.assign(is_high_rating = shuffled_ratings)
        test_stats.append(diff_means(shuffled_df, 'is_drama'))

    p_value = np.sum(test_stats <= observed_stat) / n

    print(f'{platform_name} results')
    print(f'Observed test statistic: {observed_stat}')
    print(f'P-value: {p_value}\n')

for platform_name in platform_names:
    perm_test_2(platform_name)

Netflix results
Observed test statistic: -0.2681166624489353
P-value: 1.0

Amazon results
Observed test statistic: -0.6405737704918031
P-value: 1.0

Paramount results
Observed test statistic: 1.6638592095035905
P-value: 1.0

Max results
Observed test statistic: -0.38415392960847505
P-value: 1.0

Apple results
Observed test statistic: -0.001046389954656446
P-value: 1.0

For all streaming platforms, because the p-value is larger than the signicance level of 0.05, we fail to reject the null hypothesis. There is no statistically significant evidence to suggest that Drama shows have fewer seasons that non-Drama shows.

Correlation Matrix: Netflix vs. Non-Netflix Platforms

df_without_netflix = data[data['platform'] != 'Netflix'] # retaining all platforms except for Netflix

corr_matrix_all = df_without_netflix[['seasons', 'imdb_score', 'runtime', 'release_year']].corr()
print(corr_matrix_all)

               seasons  imdb_score   runtime  release_year
seasons       1.000000    0.117377 -0.081997     -0.458402
imdb_score    0.117377    1.000000  0.070391     -0.155998
runtime      -0.081997    0.070391  1.000000      0.127239
release_year -0.458402   -0.155998  0.127239      1.000000

As we can see, release_year and seasons has a moderate negative association for all platforms.

netflix = data[data['platform']=='Netflix']
corr_matrix_netflix = netflix[['seasons', 'imdb_score', 'runtime', 'release_year']].corr()
print(corr_matrix_netflix)

               seasons  imdb_score   runtime  release_year
seasons       1.000000    0.101226 -0.126456     -0.517217
imdb_score    0.101226    1.000000  0.072713     -0.103113
runtime      -0.126456    0.072713  1.000000      0.143329
release_year -0.517217   -0.103113  0.143329      1.000000

The negative assocation between release_year and seasons is stronger for the Netflix data, but the difference is quite small.

Permutation Tests 3-4: Do Netflix shows have fewer seasons than Non-Netflix shows?

In order to investigate this question, we will run two permutation tests, one using total variation distance, and the other using difference in means. Total variation distance will help us understand how different the distributions are, while the difference in means will help us understand the direction of the difference. A positive difference in means indicates Netflix more seasons, while a negative difference in means indicates Netflix has fewer seasons. We also choose to only look at shows released in 2003 or later. In the last 20 years, there has been a fall in the number of seasons, and it is within this period that we would like to examine claims that shows are being cancelled more often.

Permutation Test 3

Hypotheses

$H_0$: There is no difference in the distribution of the number of seasons between Netflix shows and non-Netflix shows. Any observed difference in the distribution of the number of seasons is due to random chance.

$H_a$: There is a difference in the distribution of the number of seasons between Netflix TV shows and non-Netflix TV shows. The observed difference in the distribution of the number of seasons is not due to random chance.

Test Statistic

Total variation distance

Significance Value

$\alpha$ = 0.05

Permutation Test 4

Hypotheses

$H_0$: There is no difference in the number of seasons between Netflix and non-Netflix shows. $$\mu_{Netflix} - \mu_{other} = 0$$

$H_a$: Netflix shows have fewer seasons than non-Netflix shows. $$\mu_{Netflix} - \mu_{other} < 0$$

Test Statistic

Difference in means

Significance Value

$\alpha$ = 0.05

# Add new boolean column indicating whether a show is from Netflix or another platform
df = data.copy()
df['is_netflix'] = df['platform'] == 'Netflix'
df = df[df['release_year'] >= 2005]

# Calculate the observed difference in means
observed_mean = diff_means(df, 'is_netflix')

# Helper function for calculating TVD

def calc_tvd(df, group, col='seasons'):
    group1 = df[df[group]==True][col]
    group2 = df[df[group]==False][col]
    proportions1, _ = np.histogram(group1, bins=bin_edges, density=True)
    proportions2, _ = np.histogram(group2, bins=bin_edges, density=True)
    return 0.5 * np.sum(np.abs(proportions1 - proportions2))

# Make sure bin edges are consistent throughout groups and permutations
bin_edges = np.histogram_bin_edges(data['seasons'], bins='auto')

# Calculate the observed TVD
observed_tvd = calc_tvd(df, 'is_netflix')

# Run 10,000 simulations
n = 10000
tvds = []
means = []

for i in range(n):
    shuffled_netflix = np.random.permutation(df['is_netflix'])
    shuffled_df = df.assign(is_netflix = shuffled_netflix)

    # Find TVD
    tvd = calc_tvd(shuffled_df, 'is_netflix')
    tvds.append(tvd)
    
    # Find difference in means
    mean = diff_means(shuffled_df, 'is_netflix')
    means.append(mean)

tvd_p_value = np.sum(tvds >= observed_tvd) / n
mean_p_value = np.sum(means <= observed_mean) / n

print(f'TVD results')
print(f'Observed TVD: {observed_tvd}')
print(f'P-value: {tvd_p_value}\n\n')
print(f'Difference in means results')
print(f'Observed diff. in means: {observed_mean}')
print(f'P-value: {mean_p_value}\n\n')

TVD results
Observed TVD: 0.4282658501076648
P-value: 0.0


Difference in means results
Observed diff. in means: -0.41076545614593685
P-value: 0.0

For permutation test 3, the p-value is approximately 0 and smaller than the significance level of 0.05, so we reject the null hypothesis. There is statistically significant evidence to suggest that the distriubtion of the number of seasons of Netflix shows is different than the the distribution for non-Netlix shows.

For permutation test 3, the p-value is approximately 0 and smaller than the significance level of 0.05, so we reject the null hypothesis. There is statistically significant evidence to suggest that Netflix shows have fewer seasons than non-Netflix shows.

Netflix Heatmap

plt.figure(figsize=(12, 6))
corr_matrix = netflix[['runtime', 'seasons', 'imdb_score', 'imdb_votes']].corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Netflix Correlation Heatmap')
plt.show()

data['is_drama'] = data['genres'].apply(lambda genres: 'drama' in genres)

avg_seasons_drama = data[data['is_drama']]['seasons'].mean()
avg_seasons_non_drama = data[~data['is_drama']]['seasons'].mean()

print(f'Average number of seasons for drama shows: {np.round(avg_seasons_drama, 2)}')
print(f'Average number of seasons for non-drama shows: {np.round(avg_seasons_non_drama, 2)}')

Average number of seasons for drama shows: 2.4
Average number of seasons for non-drama shows: 2.79

We hypothesized that drama shows would, on average, have more seasons as it takes a longer time for the plot and character arcs to develop. However, the opposite is proven, as non-drama shows, on average, have more seasons.

from scipy.stats import ttest_ind

drama_seasons = data[data['is_drama']]['seasons']
non_drama_seasons = data[~data['is_drama']]['seasons']

t_stat, p_value = ttest_ind(drama_seasons, non_drama_seasons)

print(f'T-statistic: {t_stat}')
print(f'P-value: {p_value}')

T-statistic: -3.8234832934074454
P-value: 0.00013310411269057816

We conducted a t-test to confirm that the difference between average seasons for non-drama and drama shows is statistically significant. A t-test, however, makes the assumption that the data comes from a normal distribution. This is an issue when the sample size is small. Since our sample size is >5000, we follow the Central Limit Theorem which suggests that the

plt.figure(figsize=(12, 6))
corr_matrix = netflix[['runtime', 'seasons', 'imdb_score', 'imdb_votes']].corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Netflix Correlation Heatmap')
plt.show()