COGS 108 - Is Human-Generated or AI-Generated Art More Successful?

Permissions

Place an X in the appropriate bracket below to specify if you would like your group's project to be made available to the public. (Note that student names will be included (but PIDs will be scraped from any groups who include their PIDs).

• [ X ] YES - make available
• [ ] NO - keep private

Names

• Rachel Chau
• Jennifer Hung
• Mia Jerphagnon
• Maansi Narain
• Haneen Qarana

Abstract

This study examines the reception of AI-generated and human-generated art on social media platforms, specifically focusing on the number of likes and comments as indicators of audience engagement. Our investigation was prompted by concerns surrounding the increasing presence of AI in creative domains and the challenges in discerning between AI-generated and human-generated art. Leveraging datasets sourced from Kaggle, we analyzed posts on platforms like Reddit and Twitter, comparing metrics of engagement for AI-generated and human-generated art.

Contrary to our initial hypothesis, our findings revealed no significant difference in the number of likes or comments between posts of AI-generated art and human-generated art. This aligns with prior research indicating that non-art experts struggle to differentiate between the two forms of art. We speculate that the lack of expertise among social media users, combined with the limitations of our dataset in documenting user expertise, may contribute to the absence of discernible differences in engagement metrics. Additionally, we attempted to construct a predictive model to understand the factors influencing the reception of AI-generated art posts on Twitter. However, our dataset lacked sufficient attributes to provide comprehensive insights. While our predictions primarily relied on available numeric values such as retweets, comments, and sentiment, we acknowledge the potential influence of additional variables, such as the category of art generated by AI. Notably, AI-generated art often exhibits flaws in depicting humans accurately, which may impact engagement levels. Despite these limitations, our study tentatively explores the role of various factors in shaping the online reception of AI-generated art.

Overall, our research underscores the complexity of assessing audience engagement with AI-generated art in the digital landscape and highlights the need for further investigation into the nuanced factors influencing perceptions and interactions with this emerging form of creativity.

Research Question

What are the quantitative differences in the online social media reception of AI-generated and human-generated art? Specifically, how do metrics such as likes and comments compare between these two art forms on platforms such as Reddit and Twitter?

Background and Prior Work

With the increasing implementation of AI in various fields and aspects of technology, many concerns have been raised regarding AI’s overtaking of some creative fields. Those concerns from the digital art community include the AI's lack of originality, theft of real human-designed artwork, ownership and copyright issues, and most importantly, the potential loss of jobs in the art industry due to AI use. A survey published in March of 2023 discovered that only 27% of 2000 surveyed U.S. adults have seen AI imagery, but that is not to eliminate that the rest of the survey pool have not.¹ It is very likely, with the extent of AI improvements, that they have encountered AI-generated images and lacked awareness of that fact. The inability to discern between AI-generated images and human-generated art can generate success for artificial tools rather real people who make a livelihood of their craft. Only 29% of those who have seen AI imagery said that it was not to the same standards as human art, while 31% said it was, and 34% believed it was even better than human art.¹ The widespread use of AI images and the general inability to discern between it and human art jeopardizes the careers of real online artists, which is why we would like to assess, through measurable data collected from social media platforms such as Twitter and Reddit, the success metric of AI imagery compared to human-generated art.

Along with discerning between art that is generated by AI or created by humans, the existence of AI art also impacts how favorably people tend to view both types of images. A study conducted with art experts and non-experts alike on their opinions on AI-generated vs. artist made paintings found that art experts were more inclined to view AI paintings more negatively and were less likely to want to purchase them. However, it also found that non-experts had no difference of opinion between the two types, suggesting that art expertise was related to willingness to enjoy and buy AI-generated art.² The results of this research further indicate that the careers of real artists are at stake if non-art experts, which make up a majority of an artist’s audience, cannot differentiate between AI and real art and do not necessarily prefer real art over AI.

In this endeavor to make sense of where AI fits into the realm of art and creativity, additional research has examined the overall attitudes on AI and art among people. One longitudinal study found that positive attitudes towards AI and art seemed to be connected to having a sense of autonomy over AI, in that it still requires human input to create. Thus, it may be important to emphasize AI as a tool for artists instead of a complete replacement to find more general positive attitudes towards AI imagery ³. These differing attitudes on artificial intelligence in art give us all the more reason to further explore how AI imagery is viewed relative to human made art, as well as what potential factors contribute to these perspectives.

1. ^ Half of Americans think AI art will be used to spread fake news
2. ^ Who made the paintings: Artists or artificial intelligence? The effects of identity on liking and purchase intention
3. ^ AI as an Artist? A Two-Wave Survey Study on Attitudes Toward Using Artificial Intelligence in Art

Hypothesis

We hypothesize that human-generated art will have a greater online reception than AI-generated art, as measured by the number of likes, comments, and reposts on social media platforms such as Reddit and Twitter.

We predict people will perceive AI art as unnatural, unoriginal, and lacking of human touch. For example, perhaps they notice a backdrop with elements that do not seem correctly placed in three-dimensional space. Instances like these may lead to less engageement and interaction online. Whereas with human art, we predict people will notice the elements that indicate the artist's human influence—such as a facial expression that conveys a specific bittersweet emotion—and therefore engage with the art more.

Data

Data overview

• Dataset #1
  • Dataset Name: Reddit: /r/Art
  • Link to the dataset: Reddit: /r/Art
  • Number of observations: 1710
  • Number of variables: 8
• Dataset #2
  • Dataset Name: AI Text-2-Image Tweets
  • Link to the dataset: Twitter
  • Number of observations: 17972
  • Number of variables: 11

/r/Art dataset

The Reddit: /r/Art dataset from Kaggle analyzes the success of human-generated art on the r/Art subreddit. We can verify that the art on r/Art should be non-AI as there is a rule (11. "No AI Art or NFT: You will be banned for attempting to post AI Art or NFT.") They rely on online detectors to verify what images are AI-generated. This dataset also includes signigicant variables for popularity metrics, including the number of upvotes and comments that the posts have. We plan on using those variables to conduct our hypothesis testing in comparison with the AI-generated dataset, which includes similar popularity metrics. Most of the data is in clean format, so our main focus would be renaming the columns so that the two datasets match. However, we converted some columns into a datetime format, and we clarified the meaning of created and timestamp. The title column was also not in a clean format. Some titles had to be cleaned manually, as they did not fit into any particular format and could not be wrangled with functions. We added new columns such as medium and art_title to make it easier for analysis later on. We plan to check for outliers and odd data later during our EDA process.

AI Text-2-Image Tweets dataset

The Twitter: AI Text 2 Image dataset from Kaggle examines the online attention received from AI-generated artworks in the form of comments, retweets, quotes, and hearts. The observations included in this dataset are images generated by specific text prompts (which are also included) as well as a sentiment analysis. For the purposes of our research, we are primarily focusing on popularity statistics, such as the amount of comments an image has, to analyze how AI images are received by the public compared to art created by humans.

The dataset on its own already appeared fairly clean. Each variable was in a single column and every observation of a variable was in a different row. There were not any units specified in cells. Each observation in the dataset represented a different tweet. There were not any null values in the dataset that were not explainable. Because of this, there was nothing we had to do to get the data into a usable format. When we looked at the dtypes of the columns of the datatypes, all the variables that were expected to be numerical were an int64 datatype. We then checked for any null values in the columns. There were only null values in the attachment_type and body columns. This is explainable because it is possible to have a tweet without an attachment, or an empty body. To handle these null values in the attchement_type, we only included the observations with image as an attachment_type since we are only insterested in comparing images anyways to answer our question.

## Imports
import pandas as pd
import numpy as np 
import matplotlib.pyplot as plt
import seaborn as sns
import string
import scipy.stats as stats
from scipy.stats import ttest_ind_from_stats
from scipy.stats import t
import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.sentiment.vader import SentimentIntensityAnalyzer 
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, FunctionTransformer
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, r2_score

/r/Art Dataset

Link: Reddit: /r/Art

## Load data
reddit = pd.read_csv('datasets/reddit.csv')
reddit.head()

	title	score	id	url	comms_num	created	body	timestamp
0	Look closer, me, mixed media,2022	2	zny4p9	https://i.redd.it/hqbnhvd2hf6a1.jpg	0	1.671252e+09	NaN	2022-12-17 06:33:23
1	Paused Movie Frame, Me, Watercolour, 2022	1	zny3sp	https://i.redd.it/go1tdcw0zd6a1.jpg	0	1.671252e+09	NaN	2022-12-17 06:31:52
2	Look, me, mixed media, 2022	1	zny3pi	https://i.redd.it/xy0coo2rgf6a1.jpg	0	1.671252e+09	NaN	2022-12-17 06:31:41
3	moody water,me,oil/acrylic,2021	3	zny1xz	https://i.redd.it/9rfxha39gf6a1.jpg	1	1.671251e+09	NaN	2022-12-17 06:28:51
4	A shark by me, acrylic, 2022	6	znxwcp	https://i.redd.it/zig3qselef6a1.jpg	0	1.671251e+09	NaN	2022-12-17 06:19:34

## Check the data types for each column
reddit.dtypes

title         object
score          int64
id            object
url           object
comms_num      int64
created      float64
body          object
timestamp     object
dtype: object

To make the timestamp and created columns readable, we will change them into datetime format.

## Clean up data types
reddit['created'] = pd.to_datetime(reddit['created'], unit='s')

## Make sure all timestamps are Datetime format
reddit['timestamp'] = pd.to_datetime(reddit['timestamp'])

## Rename the Columns
reddit.columns = ['title', 'likes', 'id', 'url', 'num_comments', 
                  'time_created', 'body', 'timestamp']
reddit

	title	likes	id	url	num_comments	time_created	body	timestamp
0	Look closer, me, mixed media,2022	2	zny4p9	https://i.redd.it/hqbnhvd2hf6a1.jpg	0	2022-12-17 04:33:23	NaN	2022-12-17 06:33:23
1	Paused Movie Frame, Me, Watercolour, 2022	1	zny3sp	https://i.redd.it/go1tdcw0zd6a1.jpg	0	2022-12-17 04:31:52	NaN	2022-12-17 06:31:52
2	Look, me, mixed media, 2022	1	zny3pi	https://i.redd.it/xy0coo2rgf6a1.jpg	0	2022-12-17 04:31:41	NaN	2022-12-17 06:31:41
3	moody water,me,oil/acrylic,2021	3	zny1xz	https://i.redd.it/9rfxha39gf6a1.jpg	1	2022-12-17 04:28:51	NaN	2022-12-17 06:28:51
4	A shark by me, acrylic, 2022	6	znxwcp	https://i.redd.it/zig3qselef6a1.jpg	0	2022-12-17 04:19:34	NaN	2022-12-17 06:19:34
...	...	...	...	...	...	...	...	...
1705	Comment	1	j0h0yx2	NaN	0	2022-12-16 16:10:07	yeah, exactly I what I did.	2022-12-16 18:10:07
1706	Comment	4	j0h0p95	NaN	0	2022-12-16 16:08:34	vaporware x Edward Hopper	2022-12-16 18:08:34
1707	Comment	10	j0h0oze	NaN	0	2022-12-16 16:08:31	I like this. So refreshing!	2022-12-16 18:08:31
1708	Comment	1	j0h0mtd	NaN	0	2022-12-16 16:08:10	This feels right at home in a Failbetter game.	2022-12-16 18:08:10
1709	Comment	1	j0h0lkw	NaN	0	2022-12-16 16:07:56	Reminds me of Fuschia from Titus Groan	2022-12-16 18:07:56

1710 rows × 8 columns

It looks like there are comments included at the very end of the dataset. While these may be helpful because some of them may be linked to the original art/post, more work needs to be done for them to be used appropriately. Because of this, we will drop all rows that contain comment in the title, and create a separate dataset for them. First, we will change object types of title and body to string type, and check where the comments start in the dataset.

## Change type from object to string
reddit['title'] = reddit['title'].astype('string')
reddit['body'] = reddit['body'].astype('string')

## Change all title and body inputs to lowercase
reddit['title'] = reddit['title'].str.lower()
reddit['body'] = reddit['body'].str.lower()

## Check where the comments start
reddit[reddit['title'].str.contains('comment')].head()

	title	likes	id	url	num_comments	time_created	body	timestamp
994	comment	1	j0ju0yz	NaN	0	2022-12-17 04:35:46	the border is moody af! it is a perfect descri...	2022-12-17 06:35:46
995	comment	1	j0jtzn6	NaN	0	2022-12-17 04:35:25	ah yes, my favorite fine china cheese burger	2022-12-17 06:35:25
996	comment	1	j0jtywg	NaN	0	2022-12-17 04:35:14	your submission title most likely does not com...	2022-12-17 06:35:14
997	comment	1	j0jtxq9	NaN	0	2022-12-17 04:34:54	extra bones? and how! i enjoy this.	2022-12-17 06:34:54
998	comment	1	j0jtrxu	NaN	0	2022-12-17 04:33:18	i've seen recommendations to keep coffe inside...	2022-12-17 06:33:18

We will add the comments to a separate dataframe.

## Add all comments to a new dataframe
reddit_comments = reddit[reddit['title'].str.contains('comment')]

Let's drop all the columns that are unnecessary for the comments DataFrame...

## Drop title, url, num_comments
# Keep time_created to join back to art piece dataset
reddit_comments = reddit_comments.drop(columns=['title', 'url', 'num_comments'])
reddit_comments.head()

	likes	id	time_created	body	timestamp
994	1	j0ju0yz	2022-12-17 04:35:46	the border is moody af! it is a perfect descri...	2022-12-17 06:35:46
995	1	j0jtzn6	2022-12-17 04:35:25	ah yes, my favorite fine china cheese burger	2022-12-17 06:35:25
996	1	j0jtywg	2022-12-17 04:35:14	your submission title most likely does not com...	2022-12-17 06:35:14
997	1	j0jtxq9	2022-12-17 04:34:54	extra bones? and how! i enjoy this.	2022-12-17 06:34:54
998	1	j0jtrxu	2022-12-17 04:33:18	i've seen recommendations to keep coffe inside...	2022-12-17 06:33:18

...and then, we will drop all comments from the original dataset.

## Keep only the art pieces
reddit = reddit[~reddit['title'].str.contains('comment')]

## Drop any unnecessary columns, such as body or comment_timestamp
reddit = reddit.drop(columns=['body', 'timestamp'])

reddit.head()

	title	likes	id	url	num_comments	time_created
0	look closer, me, mixed media,2022	2	zny4p9	https://i.redd.it/hqbnhvd2hf6a1.jpg	0	2022-12-17 04:33:23
1	paused movie frame, me, watercolour, 2022	1	zny3sp	https://i.redd.it/go1tdcw0zd6a1.jpg	0	2022-12-17 04:31:52
2	look, me, mixed media, 2022	1	zny3pi	https://i.redd.it/xy0coo2rgf6a1.jpg	0	2022-12-17 04:31:41
3	moody water,me,oil/acrylic,2021	3	zny1xz	https://i.redd.it/9rfxha39gf6a1.jpg	1	2022-12-17 04:28:51
4	a shark by me, acrylic, 2022	6	znxwcp	https://i.redd.it/zig3qselef6a1.jpg	0	2022-12-17 04:19:34

The title of the art pieces seem untidy. We can look into this further and clean it up.

From the /r/Art subreddit, there are strict guidelines on how the title should be formatted: Artwork - Title, Artist/Your username, Medium, Year. We can make these into separate columns.

Seeing as we already have the timestamp of when the art was created and we want to keep the artists anonymous, we can drop Year and Artist/Your username.

## There are 59 art pieces with incorrect formatting
reddit[reddit['title'].str.split(',').apply(len) != 4]['title'].head()

4                          a shark by me, acrylic, 2022
12    collection of collections, perfume me, gouache...
66                          hold on - me, digital, 2022
89                   behind the fence, mixedmedia, 2022
99    the beatles illustrated lyrics book, alan aldr...
Name: title, dtype: string

## Look through all 59 cases to find the special cases that cannot be regulated
special_titles = reddit.iloc[[99, 110, 177, 494, 504, 506, 570, 830, 873, 914]]['title']
special_titles.head()

99     the beatles illustrated lyrics book, alan aldr...
110    self (dreams + fears), tristen trivett, wood, ...
177    unnamed, my sister, canvas board, air dry clay...
494       through a lone winding road, tran nguyen, 2019
504    nightmare, me, markers,/nail polish/watercolor...
Name: title, dtype: string

## Define a function to clean up title
def clean_title(title):
    ## First, split the string by commas to extract data
    title = title.split(',')
    art_title = np.nan
    if len(title) == 4:
        ## Turn Title and Medium into lowercase
        art_title = title[0].strip('"').lower()
       
    elif len(title) > 4:
        if title[1] != 'me':
            art_title = f'{title[0]}, {title[1]}'.lower()
        else: 
            art_title = title[0].lower()
    else: 
        if 'by' in title[0]:
            art_title = title[0].split('by')[0].strip()
        elif '- me' in title[0]:
            art_title = title[0].split('-')[0].strip()
        elif 'me' in title[0]:
            art_title = title[0].split('me')[0].strip()
        elif '.' in title[0]:
            art_title = title[0].split('.')[0].strip()
        elif 'me' not in title:
            art_title = title[0].strip('"').lower()
    return art_title

def clean_medium(title):
    ## First, split the string by commas to extract data
    title = title.split(',')
    medium = np.nan
    if len(title) == 4:
        ## Turn Title and Medium into lowercase
        medium = [title[2].strip().lower()]
       
    elif len(title) > 4:
        if title[1] != 'me':
            medium = [title[3].strip().lower()]
        else: 
            medium = [title[i].strip().lower() for i in range(2,len(title))]
    else: 
        medium = [title[1].strip().lower()]
    return medium

## Clean up all the titles except for the 10 special cases; we'll manually adjust those!
formated_titles = reddit[~reddit.index.isin([99, 110, 177, 494, 504, 506, 570, 830, 873, 914])]

reddit['art_title'] = \
    formated_titles['title'].apply(clean_title)

reddit['medium'] = formated_titles['title'].apply(clean_medium)

## Now, we manually adjust the 10 titles
reddit.loc[99, 'art_title'] = reddit.iloc[110]['title'].split(',')[0].lower()
reddit.loc[99, 'medium']  = np.nan

reddit.loc[110,'art_title'] = reddit.iloc[110]['title'].split(',')[0].lower()
reddit.loc[110, 'medium']  = str(['wood', 'silicon', 'nails', 'ink', 
                            'mannequin hands', 'fabric', 'candles'])

reddit.loc[177, 'art_title'] = reddit.iloc[110]['title'].split(',')[0].lower()
reddit.loc[177, 'medium']  = str(['canvas board', 'air dry clay', 'acrylic paint'])

reddit.loc[494, 'art_title'] = reddit.iloc[494]['title'].split(',')[0].lower()
reddit.loc[494, 'medium']  = np.nan

reddit.loc[504, 'art_title'] = reddit.iloc[504]['title'].split(',')[0].lower()
reddit.loc[504, 'medium'] = str(['markers', 'nail polish', 'watercolor',
                            'multimedia paper'])

reddit.loc[506, 'art_title'] = reddit.iloc[506]['title'].split(',')[0].lower()
reddit.loc[506, 'medium']  = np.nan

reddit.loc[570, 'art_title'] = np.nan
reddit.loc[570, 'medium']  = str(['inks'])

reddit.loc[830, 'art_title'] = 'a strange guy, 19973'
reddit.loc[830, 'medium'] = str(['3d'])

reddit.loc[873, 'art_title'] = 'aurora kidnapping'
reddit.loc[873, 'medium']  = str(['canvas', 'tempera & gold leaf'])

reddit.loc[914, 'art_title'] = reddit.iloc[914]['title'].split(',')[0].lower()
reddit.loc[914, 'medium']  = str(['acrylic', 'canvas board'])

## Save as string of lists; now, clean and save the special cleaned medium as lists
def to_list(medium):
    return medium[1:-1].strip().split(',')
reddit.loc[[110, 177, 504, 570, 830, 873, 914]]['medium'] = \
    reddit.loc[[110, 177, 504, 570, 830, 873, 914]]['medium'].apply(to_list)

reddit.shape

(994, 8)

reddit = reddit.drop(columns=['title'])

## Reorder the Columns
reddit = reddit.iloc[:, [1, 5, 6, 4, 0, 3, 2]]
reddit.head()

	id	art_title	medium	time_created	likes	num_comments	url
0	zny4p9	look closer	[mixed media]	2022-12-17 04:33:23	2	0	https://i.redd.it/hqbnhvd2hf6a1.jpg
1	zny3sp	paused movie frame	[watercolour]	2022-12-17 04:31:52	1	0	https://i.redd.it/go1tdcw0zd6a1.jpg
2	zny3pi	look	[mixed media]	2022-12-17 04:31:41	1	0	https://i.redd.it/xy0coo2rgf6a1.jpg
3	zny1xz	moody water	[oil/acrylic]	2022-12-17 04:28:51	3	1	https://i.redd.it/9rfxha39gf6a1.jpg
4	znxwcp	a shark	[acrylic]	2022-12-17 04:19:34	6	0	https://i.redd.it/zig3qselef6a1.jpg

Now that we have our final, tidy dataset for human-generated art on Reddit, we can develop it further and understand its contents more in the EDA checkpoint. For example, to understand the success of these art pieces more, we may consider splitting it into traditionally-made and digitally-made dataframes, and how those compare with the success of AI-generated art. Because we now have separate columns for the title and medium of the art piece, we can better visualize the success of certain types of works through their like and comment counts. To finalize the data cleaning, here is a summary of the finalized dataframe.

reddit.describe()

	time_created	likes	num_comments
count	994	994.000000	994.000000
mean	2022-12-14 23:58:43.024144896	214.795775	7.455734
min	2022-12-12 16:14:29	0.000000	0.000000
25%	2022-12-13 20:37:50	8.000000	0.000000
50%	2022-12-14 22:29:47.500000	16.000000	1.000000
75%	2022-12-16 03:05:11.500000	39.750000	2.000000
max	2022-12-17 04:33:23	40095.000000	3525.000000
std	NaN	1662.514604	113.013677

Because the original dataset also contained comments left on the reddit posts, we can conduct sentiment analysis on them to have an idea of how human-generated sentiment compares to ai-generated sentiment, which already exists in the ai-art dataset. We will tokenize and remove all stop words from the body text of the comments in the reddit_comments dataset so we can look at the most commonly used words during EDA.

nltk.download('stopwords')
nltk.download('punkt')
nltk.download('vader_lexicon') 

[nltk_data] Downloading package stopwords to
[nltk_data]     C:\Users\rache\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\rache\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package vader_lexicon to
[nltk_data]     C:\Users\rache\AppData\Roaming\nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!

True

tokenizer = RegexpTokenizer(r'\w+')
reddit_comments['body_token'] = reddit_comments['body'].apply(tokenizer.tokenize) 

stop_words = set(stopwords.words('english'))
reddit_comments['body_stop'] = reddit_comments['body_token'].apply(lambda x: [item for item in x if item not in stop_words])

reddit_comments.head()

	likes	id	time_created	body	timestamp	body_token	body_stop
994	1	j0ju0yz	2022-12-17 04:35:46	the border is moody af! it is a perfect descri...	2022-12-17 06:35:46	[the, border, is, moody, af, it, is, a, perfec...	[border, moody, af, perfect, description, mood...
995	1	j0jtzn6	2022-12-17 04:35:25	ah yes, my favorite fine china cheese burger	2022-12-17 06:35:25	[ah, yes, my, favorite, fine, china, cheese, b...	[ah, yes, favorite, fine, china, cheese, burger]
996	1	j0jtywg	2022-12-17 04:35:14	your submission title most likely does not com...	2022-12-17 06:35:14	[your, submission, title, most, likely, does, ...	[submission, title, likely, comply, rule, 1, t...
997	1	j0jtxq9	2022-12-17 04:34:54	extra bones? and how! i enjoy this.	2022-12-17 06:34:54	[extra, bones, and, how, i, enjoy, this]	[extra, bones, enjoy]
998	1	j0jtrxu	2022-12-17 04:33:18	i've seen recommendations to keep coffe inside...	2022-12-17 06:33:18	[i, ve, seen, recommendations, to, keep, coffe...	[seen, recommendations, keep, coffe, inside, a...

AI Text 2 Image Dataset

Link: AI Text-2-Image Tweets (+ Sentiment labels)

## Load in the datset and check it out
ai_df = pd.read_csv('datasets/text2image_twitter.csv')
ai_df.head()

	Unnamed: 0	ai_service	attachment_type	body	comments	fullname	heart	quote	retweet	sentiment	username
0	0	dalle2	image	3D digital art created to explain deep learnin...	0.0	Yusuke Hayashi（林祐輔）	1.0	0.0	0.0	neutral	@hayashiyus
1	1	dalle2	image	#dalle2 selfie generated using # #stablediffus...	0.0	Surreal Googling	0.0	0.0	0.0	neutral	@SurrealGoogling
2	2	dalle2	image	PyPiPy gonna let it fly! #digitalart #aiart #d...	0.0	aiGrampy	1.0	0.0	0.0	positive	@AiGrampy
3	3	dalle2	image	Primeras pruebas con DALL·E 2: People travelin...	0.0	Cento Avaria	0.0	0.0	2.0	neutral	@AvariaCento
4	4	dalle2	image	The Hand Of Fear (1976) #DoctorWho #DrWho #Dalle2	1.0	Celestial Colour Maker	14.0	0.0	2.0	neutral	@Mrunknownfc1

## Look at the shape of the data
ai_df.shape

(17972, 11)

## Check if there is any null data in the columns
ai_df.isnull().sum()

Unnamed: 0            0
ai_service            0
attachment_type    2550
body                 17
comments              0
fullname              0
heart                 0
quote                 0
retweet               0
sentiment             0
username              0
dtype: int64

It makes sense that some bodies of tweets may be empty, but we since we are interested in AI art tweets, we should only look at tweets that have images as an attachment type.

##Check what different kinds of attachment types are in the data
ai_df['attachment_type'].value_counts()

attachment_type
image              15141
video-container      281
Name: count, dtype: int64

Looks like we have some videos in the dataset. These aren't comparable to human generated art, so we'll drop these rows.

## Drop null data
ai_df = ai_df.dropna(subset=['attachment_type'])

## Only keep rows with image attachment type
ai_df = ai_df[ai_df['attachment_type'] == 'image']

## Look at the datatypes of each column
ai_df.dtypes

Unnamed: 0           int64
ai_service          object
attachment_type     object
body                object
comments           float64
fullname            object
heart              float64
quote              float64
retweet            float64
sentiment           object
username            object
dtype: object

We only want the numerical variables for comparing metrics, so let's update the dataset accordingly.

# Mapping for sentiment labels
sentiment_mapping = {'neutral': 0, 'negative': -1, 'positive': 1}

# Replace sentiment labels with numerical values
ai_df['sentiment'] = ai_df['sentiment'].replace(sentiment_mapping)

# Display the first few rows of the updated dataframe
print(ai_df['sentiment'].head())

columnnames = ['comments','heart','quote','retweet', 'sentiment']
ai_df_metrics = ai_df[columnnames]
ai_df_metrics

0    0
1    0
2    1
3    0
4    0
Name: sentiment, dtype: int64

C:\Users\rache\AppData\Local\Temp\ipykernel_50584\3655983795.py:5: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
  ai_df['sentiment'] = ai_df['sentiment'].replace(sentiment_mapping)

	comments	heart	quote	retweet	sentiment
0	0.0	1.0	0.0	0.0	0
1	0.0	0.0	0.0	0.0	0
2	0.0	1.0	0.0	0.0	1
3	0.0	0.0	0.0	2.0	0
4	1.0	14.0	0.0	2.0	0
...	...	...	...	...	...
17967	1.0	3.0	0.0	0.0	0
17968	0.0	11.0	0.0	1.0	0
17969	0.0	11.0	0.0	1.0	0
17970	1.0	37.0	0.0	4.0	0
17971	1.0	22.0	0.0	2.0	0

15141 rows × 5 columns

ai_df_metrics.describe()

	comments	heart	quote	retweet	sentiment
count	15141.000000	15141.000000	15141.000000	15141.000000	15141.000000
mean	1.085463	12.928803	0.423222	2.593884	0.113995
std	10.818508	48.467146	6.606360	20.964554	0.550204
min	0.000000	0.000000	0.000000	0.000000	-1.000000
25%	0.000000	1.000000	0.000000	0.000000	0.000000
50%	0.000000	3.000000	0.000000	0.000000	0.000000
75%	1.000000	9.000000	0.000000	1.000000	0.000000
max	859.000000	998.000000	459.000000	760.000000	1.000000

Now we want to check for and remove any potential outliers that can skew our data.

## Create boxplots for columns to check for any outliers
plt.figure(figsize=(5, 8))
sns.boxplot(
    data=[ai_df_metrics["comments"], ai_df_metrics["heart"], 
          ai_df_metrics["quote"], ai_df_metrics["retweet"]],
    showmeans=True,
)
plt.title(f'Boxplot of Comments, Hearts, Quotes, and Retweets on AI Art')
plt.show()

For consistency, we may want to remove the more potent outliers that we observe from the comments and quotes plots.

Examining the comments plot, most of the observations are under 400. There are a couple of outliers that are over 800, so we should try to remove these.

## Create a maximum threshold for number of comments
threshold = 400

## Remove the observations that surpass this number
ai_df = ai_df[ai_df['comments'] < threshold]
ai_df_metrics = ai_df_metrics[ai_df_metrics['comments'] < threshold]

## Check to see the new maximum value for comments
ai_df_metrics['comments'].max()

289.0

Now looking at the quotes plot, there seems to be an outlier of 459 comments that deviates from the rest of the observations. We will remove this using the same process as before.

## Create a maximum threshold for number of quotes
threshold = 400

## Remove the observations that surpass this number
ai_df = ai_df[ai_df['quote'] < threshold]
ai_df_metrics = ai_df_metrics[ai_df_metrics['quote'] < threshold]

## Check to see the new maximum value for quotes
ai_df_metrics['quote'].max()

311.0

Standardization of Likes

## Use standard scaler for comparability
scaler = StandardScaler()
ai_df_metrics[['standardized_comments', 'standardized_heart', 'standardized_quote', 'standardized_retweet', 'standardized_sentiment']] = pd.DataFrame(scaler.fit_transform(ai_df_metrics))
ai_df_metrics.head()

	comments	heart	quote	retweet	sentiment	standardized_comments	standardized_heart	standardized_quote	standardized_retweet	standardized_sentiment
0	0.0	1.0	0.0	0.0	0	-0.211099	-0.246160	-0.071264	-0.123744	-0.207261
1	0.0	0.0	0.0	0.0	0	-0.211099	-0.266791	-0.071264	-0.123744	-0.207261
2	0.0	1.0	0.0	0.0	1	-0.211099	-0.246160	-0.071264	-0.123744	1.610532
3	0.0	0.0	0.0	2.0	0	-0.211099	-0.266791	-0.071264	-0.028351	-0.207261
4	1.0	14.0	0.0	2.0	0	0.006898	0.022047	-0.071264	-0.028351	-0.207261

reddit[['standardized_likes', 'standardized_num_comments']] = pd.DataFrame(scaler.fit_transform(reddit[['likes', 'num_comments']]))
reddit.head()

	id	art_title	medium	time_created	likes	num_comments	url	standardized_likes	standardized_num_comments
0	zny4p9	look closer	[mixed media]	2022-12-17 04:33:23	2	0	https://i.redd.it/hqbnhvd2hf6a1.jpg	-0.128061	-0.066005
1	zny3sp	paused movie frame	[watercolour]	2022-12-17 04:31:52	1	0	https://i.redd.it/go1tdcw0zd6a1.jpg	-0.128663	-0.066005
2	zny3pi	look	[mixed media]	2022-12-17 04:31:41	1	0	https://i.redd.it/xy0coo2rgf6a1.jpg	-0.128663	-0.066005
3	zny1xz	moody water	[oil/acrylic]	2022-12-17 04:28:51	3	1	https://i.redd.it/9rfxha39gf6a1.jpg	-0.127459	-0.057152
4	znxwcp	a shark	[acrylic]	2022-12-17 04:19:34	6	0	https://i.redd.it/zig3qselef6a1.jpg	-0.125654	-0.066005

Results

Exploratory Data Analysis

Human: /r/ Art EDA

In the following section, we explored our Reddit data set with human-generated art. We'll start off with checking for any outliers that we would want to potentially remove later.

## Outlier when plotting the number of likes 
sns.boxplot(
    data=[reddit['likes'], reddit['num_comments']],
    palette=[sns.xkcd_rgb["pale red"], sns.xkcd_rgb["medium green"]],
    showmeans=True,
)
plt.ylabel('Values')
plt.title('Boxplot of Distributions in Likes and Comments of Reddit Dataset')
plt.show()

Now we want to look at the amount of likes and the amount of comments compared to the time the post was made. This is to check whether time may be a confound in the amount of attention a post gets (with amounts of likes and comments). Since we are curious about the relationship between how the art was created (AI or human-generated) and its online popularity, we'll have to make sure that the time a post was made is not a significant influence on this.

## Distribution of likes vs time
plt.figure(figsize=(20, 4))
plt.plot(reddit['time_created'], reddit['likes'], color='deeppink')
plt.xlabel('Timestamp')
plt.ylabel('Number of Likes')
plt.title('Distribution of Likes vs Time in Human-Generated Art')
plt.show()

## Distribution of comments vs time
plt.figure(figsize=(20, 4))
plt.plot(reddit['time_created'], reddit['num_comments'], color='deeppink')
plt.xlabel('Timestamp')
plt.ylabel('Number of Comments')
plt.title('Distribution of Comments vs Time in Human-Generated Art')
plt.show()

It looks like there isn't a particular time that generates more likes (except for outlier days), so we can be sure that time isn't a big confounding factor in the online success of an art post. Now we will check to see if there is any relationship between the amount of likes and comments.

plt.scatter(reddit['likes'], reddit['num_comments'], color='deeppink')
plt.xlabel('Number of Likes')
plt.ylabel('Number of Comments')
plt.title('Correlation Between Likes and Comments in Human-Generated Art')
plt.show()

It seems like there is a slight positive correlation between likes and comments, though this may be due to some outliers skewing the graph.

Let's filter the Likes and Comments so we can see a better picture. We removed Likes over 5000, and comments under 2.

## Calculate the line of best fit
x = reddit[(reddit['likes'] < 5000) & (reddit['num_comments'] > 2)]['likes']
y = reddit[(reddit['likes'] < 5000) & (reddit['num_comments'] > 2)]['num_comments']
slope, intercept = np.polyfit(x, y, 1)
line = slope * x + intercept
plt.plot(x, line, color='deepskyblue', label='Line of Best Fit')

## Plot the scatter plot with the line of best fit
plt.scatter(reddit[(reddit['likes'] < 5000) & (reddit['num_comments'] > 2)]['likes'], 
            reddit[(reddit['likes'] < 5000) & (reddit['num_comments'] > 2)]['num_comments'], color='deeppink')
plt.xlabel('Number of Likes')
plt.ylabel('Number of Comments')
plt.suptitle('Correlation Between Likes and Comments in Human-Generated Art, Filtered')
plt.title('All likes over 5000 and number of comments under 2 have been removed', fontsize = 9)
plt.show()

redd_correlation = reddit['likes'].corr(reddit['num_comments'])
print(f"Correlation between Likes and Comments: {redd_correlation}")

Correlation between Likes and Comments: 0.8311502599851622

This seems to confirm that there is a high positive correlation between likes and comments, meaning that posts with a higher amount of likes also have more comments relative to ones with lower amounts of likes. This helps us understand how these social media tools can be represented as "success" metrics of posts, meaning, more popular posts generally have a high number of likes and comments.

Now we will explore the various mediums used in this human-generated art dataset by examining count. We want to know how often these mediums are used to understand what is most popular among human artists and consumers of art.

## See the count of some mediums

num_acrylic = reddit['medium'].explode().str.contains('acrylic').sum()
num_oil = reddit['medium'].explode().str.contains('oil').sum()
num_digital = reddit['medium'].explode().str.contains('digital').sum()
num_watercolor = reddit['medium'].explode().str.contains('water').sum()
num_ink = reddit['medium'].explode().str.contains('ink').sum()
num_pen = reddit['medium'].explode().str.contains('pen').sum()
num_pencil = reddit['medium'].explode().str.contains('pencil').sum()
num_pastel = reddit['medium'].explode().str.contains('pastel').sum()
num_3d = reddit['medium'].explode().str.contains('3d').sum()
num_marker = reddit['medium'].explode().str.contains('marker').sum()

all_dict = {'acrylic': num_acrylic,
                'oil': num_oil,
                'digital': num_digital,
                'watercolor': num_watercolor,
                'ink': num_ink,
                'pen': num_pen,
                'pencil': num_pencil,
                'pastel': num_pastel,
                '3d': num_3d,
                'marker': num_marker
            }

some_medium = pd.DataFrame({'medium':all_dict.keys() ,'count':all_dict.values()}).sort_values(by='count', ascending=False)

plt.barh(some_medium['medium'], some_medium['count'], color='deeppink')
plt.ylabel('Type of Medium')
plt.xlabel('Count')
plt.title('Count of Some Mediums in Human Generated Art')
plt.show()

It looks like a lot of the art in the dataset is digital art, which can be alluded to its still novel and ever-evolving platforms and mechanisms. However, there is still a good mix of other mediums included like acrylic paint, pen, and oil paint.

Now let's check whether the type of medium and likes have a relationship or not, since we'll want to also make sure this isn't a potential confound before proceeding.

## Correlation between likes and type of medium

## Create number of likes per medium type
exploded_reddit = reddit.explode('medium')
mediums = ['acrylic', 'oil', 'digital', 'water', 'ink', 'pen', 'pencil', 'pastel', '3d', 'marker']
mediums_likes = [exploded_reddit[exploded_reddit['medium'].str.contains(medium, na=False)]['likes'].mean() for medium in mediums]
likes_df = pd.DataFrame({'medium': mediums, 'likes': mediums_likes})

## Create a bar plot
sns.barplot(data=likes_df, x='medium', y='likes',order=likes_df.sort_values('likes', ascending=False).medium, color='deeppink')
plt.title('Average Number of Likes Per Medium Type in Human Generated Art')
plt.xlabel('Medium Type')
plt.ylabel('Average Number of Likes')
plt.show();

From this plot, we can see that the pastel medium seems to yield the most likes on average, with mediums like watercolor, digital, and ink also having higher averages relative to other mediums. This suggests variability in the perception and success of different art forms compared to the amount of content they yield, something that can be based on artistic preferences. This may be something we'd want to keep in mind when comparing this dataset to our AI-generated art dataset.

Now that we've examined likes more in-depth, we'll shift our focus to the comments. First, we'll look at what words show up most frequently in the comments to see if this can tell us anything about its potential as a success metric and how human art is perceived in general.

comments = reddit_comments['body_stop'].apply(pd.Series).stack()

## Calculate word frequency
fdist_comments = comments.value_counts()[0:20]

## Remove punctuation
translator = str.maketrans('', '', string.punctuation)
fdist_comments.index = fdist_comments.index.map(lambda x: x.translate(translator))

plt.barh(fdist_comments.index, fdist_comments.values, color='deeppink')
plt.title('r/Art Comments Word Count Frequency')
plt.xlabel('Comments')
plt.ylabel('Count')
plt.show()

It looks like 'title' and 'artist' are the most frequent, which is what we would expect. There are also frequent words such as "guidelines" and "please" that tell us that a lot of comments may be specific to following Reddit guidelines and not about the art itself. We may tentatively assume that the comments may not be a good indicator of Art success (if people are violating guidelines under the post.)

Finally, for this dataset's comments, we want to do a sentiment analysis of the comments to see how it compares to the word frequency visualization and further explore comments as a means of success.

analyser = SentimentIntensityAnalyzer()

## Create a list of the sentences in the comments
comments_list = list(reddit_comments['body'].values)

# From 07_14_text Lecture:
def get_sentiments(input_list):
    
    output = pd.DataFrame()

    for sentence in input_list:
        ss = analyser.polarity_scores(sentence)
        ss['sentence'] = sentence
        # Note use of pd.concat
        output = pd.concat([output, pd.DataFrame([ss])], ignore_index=True)


    return output

comments_sentiment = get_sentiments(comments_list)

comments_sentiment['compound'].plot.density(color='deeppink')
plt.xlabel('Compound Sentiment Scores')
plt.title('Density of Compound Sentiment Scores')
plt.xlim(-1,1)

(-1.0, 1.0)

There seems to be a bimodal distribution of Compound Sentiment score with a left-skew. Many Compound Sentiment scores fall around Neutral (0.0), but also somewhere between 0.5 and 0.75, indicating positive correlation. This may indicate that Human Generated Art tends to have a positive feedback in terms of comments, which goes against our initial assumption that words regarding guidelines may indicate negativity. Because of this, we will still be using comments as a potential success metric when conducting our hypothesis testing.

AI: Text-2-Image EDA

Let's take a look at the relationship between the likes and comments of tweets of AI art.

sns.scatterplot(x='heart', y='comments', data=ai_df_metrics, color='orangered')
plt.title('Relationship between Likes and Comments in AI Generated Art')
plt.xlabel('Number of Likes')
plt.ylabel('Number of Comments')
plt.show()

correlation = ai_df_metrics['heart'].corr(ai_df_metrics['comments'])
print(f"Correlation between Likes and Comments: {correlation}")

Correlation between Likes and Comments: 0.3272789174212475

There seems to be a weak positive correlation between likes and comments.

Let's also take a look at the relationship between likes and retweets.

sns.scatterplot(x='heart', y='retweet', data=ai_df_metrics, color='orangered')
plt.title('Relationship between Likes and Retweets in AI-Generated Art')
plt.xlabel('Likes')
plt.ylabel('Retweets')
plt.show()

We can note that perhaps these posts about AI art generally have a logarithmic correlation between likes and retweets, which also seems weak.

correlation = ai_df_metrics['heart'].corr(ai_df_metrics['retweet'])
print(f"Correlation between Likes and Retweets: {correlation}")

Correlation between Likes and Retweets: 0.3312195933919181

There is a similar correlation as with likes and comments. There is a weak positive correlation between likes and retweets.

The above weak correlations are good for our model building, as we don't want many dependent variables. It is also worth to note that having more comments the more likes a post receives is not reflected here with AI art, neither in the comments nor the retweets. We must keep in mind that this difference could potentially be due to the different social media platforms and the different ways in which users interact with them.

Let's take a look at the distribution of sentiment (-1 for negative, 0 for neutral, 1 for positive).

sentiment_counts = ai_df_metrics['sentiment'].value_counts()

# plotting the distribution of sentiment labels
sns.countplot(x='sentiment', data=ai_df_metrics, order=[-1, 0, 1], color='orangered')
plt.title('Distribution of Sentiment Labels in AI Generated Art')
plt.xlabel('Sentiment')
plt.ylabel('Count')
plt.show()

We see that the AI generated art tends to have more neutral comments. Surprisingly, there were quite fewer negative comments (<2000), contrary to our assumption. However, it seems generally less positive than our Human-generated art comments, which still give us the idea that Human generated art may be more favored than AI art.

To look at the correlation of all variables to each other, we can create a correlation matrix.

## Select the variables of interest
variables_of_interest = ['comments', 'heart', 'quote', 'retweet']
data = ai_df_metrics[variables_of_interest]

## Calculate the correlation matrix
correlation_matrix = data.corr()

## Create the heatmap
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', vmin=0, vmax=1, fmt=".2f")
plt.title('Correlation Matrix of Success Metrics')
plt.show()

Again, this correlation matrix only further confirms that our variables of interest (comments, likes, quotes, and retweets) seem to have a very weak correlation to one another.

However, these visualizations do not inform us of what the relationship is like between Human and AI generated art.

EDA Between Datasets

We decided to plot the standardized likes and comments between the two datasets. We could only use likes (or in terms of the AI generated dataset, hearts) and comments as those were the only mutual variables between the datasets.

standard_reddit = reddit[['standardized_likes', 'standardized_num_comments']]
standard_twitter = ai_df_metrics[['standardized_heart', 'standardized_comments']]

We compare the distribution of likes with AI and human-generated Art. We separated the two because there is significantly less data for the human-generated one. We also filtered by the standardized likes that fall around -1 to 2, as the outliers made it hard to visualize.

plt.hist(standard_reddit['standardized_likes'],
        label='Human Generated',color='deeppink', bins=100, alpha=1)
plt.xlim(-1, 2)
plt.xlabel('Standardized Likes')
plt.ylabel('Counts')
plt.title('Distribution of Likes For Human Generated Art')
plt.show()

plt.hist(standard_twitter['standardized_heart'],
        label='AI Generated', color='orangered',bins=100)
plt.xlim(-1, 2)
plt.xlabel('Standardized Likes')
plt.ylabel('Counts')
plt.title('Distribution of Likes For AI-Generated Art')
plt.show()

We did not overlay the plots as the AI dataset had significantly more observations than the human dataset. However, judging from the graph above, we can see that both datasets had a similar distribution of standardized likes. As our likes are standardized to their respective population, it also seems like AI generated art has a lot of likes doing worse than the average (falling more on the negative values). But compared to human generated art, there also seems to be more distributed on the right side, indicating likes being higher than average.

Let's observe the distribution of our comments, not in sentiment analysis. We filtered by the standardized comments that fall around -1 to 2 standard deviations away from mean, as the outliers made it hard to visualize.

plt.hist(standard_reddit['standardized_num_comments'],
        label='Human Generated',color='deeppink', bins=100, alpha=1)
plt.xlim(-0.5, 2)
plt.xlabel('Standardized Comments')
plt.ylabel('Counts')
plt.title('Distribution of Comments For Human-Generated Art')
plt.show()

plt.hist(standard_twitter['standardized_comments'],
        label='AI Generated',color='orangered', bins=100, alpha=1)
plt.xlim(-0.5, 2)
plt.xlabel('Standardized Comments')
plt.ylabel('Counts')
plt.title('Distribution of Comments For AIGenerated Art')
plt.show()

It is harder to compare the distribution of comments with the likelihood of getting more likes, as we know that while human art comments have a correlation to likes, AI art comments have a weaker correlation.

Let's perform some hypothesis testing to truly understand whether Human generated art truly recieve more likes than AI art.

Testing our Hypothesis

Why we use standardized likes: The number of upvotes in Reddit (and specific subreddits) and the number of hearts in a tweet would not be the same on average, because what counts as popular is different across different social media platforms.

Let's run some t-tests!

Null hypothesis: The population mean of the number of likes are the same across both human and AI-generated art.

Altnerative hypothesis: The population mean of the number of likes is higher in human art than in AI-generated art.

## Calculate the mean, standard deviation, and the number of observations for each dataset.
stand_hum_mean = standard_reddit['standardized_likes'].mean()
stand_hum_std = standard_reddit['standardized_likes'].std(ddof=1)
stand_hum_no = standard_reddit['standardized_likes'].size

stand_ai_mean = standard_twitter['standardized_heart'].mean()
stand_ai_std = standard_twitter['standardized_heart'].std(ddof=1)
stand_ai_no = standard_twitter['standardized_heart'].size

tstat, pval = ttest_ind_from_stats(stand_hum_mean, stand_hum_std, stand_hum_no, stand_ai_mean, 
                                    stand_ai_std, stand_ai_no, alternative='greater')

X = np.linspace(-4, 4, 200)
Y = t.pdf(X, df=len(standard_reddit)-1)
plt.plot(X, Y, color='deepskyblue');
X_fill = np.linspace(tstat, 4, 100)
Y_fill = stats.t.pdf(X_fill, df=len(standard_reddit)-1)
plt.fill_between(X_fill, Y_fill, alpha=0.3, color='deepskyblue')
plt.axvline(x=tstat, color='r', linestyle='--', label=f"t-statistic: {tstat:.2f}")
plt.text(tstat + 0.1, 0.1, f"p-value: {pval:.4f}", fontsize=12, color='r')
plt.suptitle('Hypothesis Testing')
plt.title('Visualization of p-value for probability of mean likes being equal (AI vs Human Generated Art)', fontsize=10)
plt.legend()
plt.show();

The p-value is approximately 0.4247, which is far greater than our significance level of 0.05. Thus, we fail to reject the null. There does not appear to be a difference between the mean of likes in human and AI-generated art, thus failing to indicate human art is more successful than AI art.

Seeing as comments have some kind of correlation to likes, we also will perform a t-test on the number comments.

Null hypothesis: The population mean of the number of comments are the same across both human and AI-generated art.

Altnerative hypothesis: The population mean of the number comments is higher in human art than in AI-generated art.

hum_mean = standard_reddit['standardized_num_comments'].mean()
hum_std = standard_reddit['standardized_num_comments'].std(ddof=1)
hum_no = standard_reddit['standardized_num_comments'].size

ai_mean = standard_twitter['standardized_comments'].mean()
ai_std = standard_twitter['standardized_comments'].std(ddof=1)
ai_no = standard_twitter['standardized_comments'].size

tstat, pval = ttest_ind_from_stats(hum_mean, hum_std, hum_no, ai_mean, 
                     ai_std, ai_no, alternative='greater')

X = np.linspace(-4, 4, 200)
Y = t.pdf(X, df=len(standard_reddit)-1)
plt.plot(X, Y, color='deepskyblue');
X_fill = np.linspace(tstat, 4, 100)
Y_fill = stats.t.pdf(X_fill, df=len(standard_reddit)-1)
plt.fill_between(X_fill, Y_fill, alpha=0.3, color='deepskyblue')
plt.axvline(x=tstat, color='r', linestyle='--', label=f"t-statistic: {tstat:.2f}")
plt.text(tstat + 0.1, 0.1, f"p-value: {pval:.4f}", fontsize=12, color='r')
plt.legend()
plt.suptitle('Hypothesis Testing')
plt.title('Visualization of p-value for probability of mean comments being equal (AI vs Human Generated Art)', fontsize=10)
plt.show();

Similarly, we see that the p-value is approximately 0.4630, which is far greater than our significance level of 0.05. Thus, we fail to reject the null. There does not appear to be a difference between the mean of the number of comments in human and AI-generated art, thus failing to indicate Human art is more successful than AI art.

One thing to consider is that this might shift based on the medium type used by the artists of the r/Art dataset. Many AI-generated art use elements akin to those found in digital, human-generated art. Considering this varying factor and the fact that digital art is the most commonly posted medium on reddit compared to other traditional mediums, we can take a second look at the difference in like and comment counts between the two art forms, while specifying the human art medium as "digital".

Null hypothesis: The population means of the number of likes are the same across both digital human-generated and AI-generated art.

Altnerative hypothesis: The population mean of the number of likes is higher in digital human-generated art than in AI generated art.

hum_mean_dig = exploded_reddit.loc[exploded_reddit['medium'] == 'digital', 'standardized_likes'].mean()
hum_std_dig = exploded_reddit['standardized_likes'].std(ddof=1)
hum_no_dig = exploded_reddit['standardized_likes'].size

ai_mean_dig = standard_twitter['standardized_heart'].mean()
ai_std_dig = standard_twitter['standardized_heart'].std(ddof=1)
ai_no_dig = standard_twitter['standardized_heart'].size

tstat, pval = ttest_ind_from_stats(hum_mean_dig, hum_std_dig, hum_no_dig, ai_mean_dig, 
                     ai_std_dig, ai_no_dig, alternative='greater')

X = np.linspace(-4, 4, 200)
Y = t.pdf(X, df=len(standard_reddit)-1)
plt.plot(X, Y, color='deepskyblue');
X_fill = np.linspace(tstat, 4, 100)
Y_fill = stats.t.pdf(X_fill, df=len(standard_reddit)-1)
plt.fill_between(X_fill, Y_fill, alpha=0.3, color='deepskyblue')
plt.axvline(x=tstat, color='r', linestyle='--', label=f"t-statistic: {tstat:.2f}")
plt.text(tstat + 0.1, 0.1, f"p-value: {pval:.4f}", fontsize=12, color='r')
plt.suptitle('Hypothesis Testing')
plt.title('Visualization of p-value for probability of mean likes being equal (AI vs Human Generated Digital Art)', fontsize=10)
plt.legend()
plt.show();

Contrasting with the previous p-values, this one is approximately 0.0542, which is much closer, yet still more than our significance level of 0.05. Thus, we fail to reject the null. There is not a significant enough difference between the mean of likes of human-generated digital art and AI art.

We now can look at the relationship between comments in the r/Art and AI twitter datasets, still only looking at digital art.

Null hypothesis: The population mean of the number of comments are the same across both digital human-generated and AI generated art.

Altnerative hypothesis: The population mean of the number of comments is higher in digital human-generated art than in AI generated art.

stand_hum_mean_dig = exploded_reddit.loc[exploded_reddit['medium'] == 'digital', 'standardized_num_comments'].mean()
stand_hum_std_dig = exploded_reddit['standardized_num_comments'].std(ddof=1)
stand_hum_no_dig = exploded_reddit['standardized_num_comments'].size

stand_ai_mean_dig = standard_twitter['standardized_comments'].mean()
stand_ai_std_dig = standard_twitter['standardized_comments'].std(ddof=1)
stand_ai_no_dig = standard_twitter['standardized_comments'].size

tstat, pval = ttest_ind_from_stats(stand_hum_mean_dig, stand_hum_std_dig, stand_hum_no_dig, stand_ai_mean_dig, 
                     stand_ai_std_dig, stand_ai_no_dig, alternative='greater')

X = np.linspace(-4, 4, 200)
Y = t.pdf(X, df=len(standard_reddit)-1)
plt.plot(X, Y, color='deepskyblue');
X_fill = np.linspace(tstat, 4, 100)
Y_fill = stats.t.pdf(X_fill, df=len(standard_reddit)-1)
plt.fill_between(X_fill, Y_fill, alpha=0.3, color='deepskyblue')
plt.axvline(x=tstat, color='r', linestyle='--', label=f"t-statistic: {tstat:.2f}")
plt.text(tstat + 0.1, 0.1, f"p-value: {pval:.4f}", fontsize=12, color='r')
plt.suptitle('Hypothesis Testing')
plt.title('Visualization of p-value for probability of mean comments being equal (AI vs Human-Generated Digital Art)', fontsize=10)
plt.legend()
plt.show();

Finally, the p-value (0.0012) is less than the significan level of 0.05. In fact, it is almost 0. Therefore, we reject the null because there the mean number of likes of digital Human art is significanly greater than that of AI art. This evidence suggests that human art, when only looking at pieces created digitally, is more successful than AI art. We can attribute this success to the still new and challenging medium of digital art. People may contribute more to feedback for the artist, tips on using various digital art tools, and sharing their own experiences with digital art. As opposed to AI art, which does not foster that same human-to-human feedback and connection over a new and popular medium such as digital art.

This finding is interesting since our r/Art comments word frequency visualization suggested no valuable significance of comments as a success metric for art. We can revisit this by making a new visualization looking at the word frequency in comments only on digital art; however, this is currently out of our scope since the comments in the dataset are not attached to any specific post. We can attribute this success to the still new and challenging medium of digital art. People may contribute more to feedback for the artist, tips on using various digital art tools, and sharing their own experiences with digital art. As opposed to AI art, which does not foster that same human-to-human feedback and connection over a new and popular medium such as digital art.

Building our Model

Let's predict the success of AI generated art given different features. This is a regression problem, so let's try out with a simple Decision Tree Regressor Model

target = 'heart'
features = ['ai_service', 'retweet', 'sentiment']

X = ai_df[features]
y = ai_df[target]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

ct = ColumnTransformer(
    transformers=[("ohe", OneHotEncoder(handle_unknown='ignore'), ['ai_service']),
     ("std", StandardScaler(), ['retweet'])],
     remainder='passthrough'
)

pipe = Pipeline([('preprocess', ct), ('model', DecisionTreeRegressor(random_state=42))])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'MSE of the Decision Tree Regressor is: {mse}\n')
print(f'R-squared of the Decision Tree Regressor is: {r2}')

MSE of the Decision Tree Regressor is: 830.4938302740446

R-squared of the Decision Tree Regressor is: 0.657775291762335

While our MSE is still fairly high, our $R^2$ score is around 0.65. A good model may have an $R^2$ score of around 0.7-0.9.

After adding different features such as quotes, comments etc. and trying different ways to preprocess our features (such as applying logarithmic transformations), we found that the different AI platforms, the number of retweets a post has, and the sentiment of the comments (but not the number of comments) were some of the bigger determining factors in how many hearts an AI generated art post will receive.

Ethics & Privacy

As we are comparing whether art made by both AI or real people is more favored, there is a problem of artists’ privacy when using their art. It is important to anonymize any information we may collect about the artists by omitting personally identifiable information as it may leave a bad impression to the public if the real art is ranked lower than AI-generated art. It would also be difficult to understand whether these individual artists consented to having their art and discussion on their art scraped from the internet into a database. As most of the personally identifiable information seems to be removed from the databases we found, we will assume informed consent on their part.

Furthermore, as we currently found art from Reddit and Twitter, there is clearly bias in the kind of people that would post on these social media platforms. This audience is likely to be young adults, and thus our data is mostly skewed towards how younger people feel about art versus AI art. Furthermore, how successful art is (in this case, we coin “success” as proportion of upvotes) sometimes relies on the algorithm and whether it recommends it to users. We have no insight on how the algorithm works for Twitter or Reddit, so there could be inherent biases that we are unaware of. To mitigate these biases, we should make sure we understand how our datasets are collected, and address the biases that may be present. We should put out disclaimers before publishing our work on all the potential biases that our data has. We should also make sure that if any artists do want their art to be removed from our data analysis, we can easily drop it from the dataset. We will also only keep the dataset for as long as we need it, and delete it after we have finalized and published our findings by the end of this quarter.

Another ethical concern is our questions themselves. We also want to be wary of how our findings could negatively impact the art community. Our conclusion should in no way be suggestive that one type of art performs significantly better than another, and it should never discourage production of authentic art. We strive to be transparent on how our data is collected, how we clean and explore the data, and how our question uses our data meaningfully. We must present visualizations that are not deceptive and earnestly convey our findings. Moreover, our team of 5 will try to cross-check to ensure that we do not have any biases when conducting the hypothesis, ex: selecting certain observations instead of using the entire dataset or drawing conclusions based on biased opinions. These are all steps that could help ensure the reproducibility/ replicability of our data analysis. We should also make sure to do fairness testing across groups to make sure that our data analysis is not biased towards a certain attribute or group.

Discussion and Conclusion

Our results revealed that there is no significant difference in number of likes or comments between posts of AI generated art vs human generated. This corresponds to our earlier background research where it was discovered that non art experts cannot differentiate between AI generated and human generated art. This may be the reason why there is not a significant difference in reception of these art pieces, whether it was like count or comment count. As our datasets were sourced from Kaggle, we are uncertain of the expertise of the users that are leaving comments. Furthermore, it is impossible to document the expertise of every single user on these social media platforms. Thus, we may tentatively assume that they are likely NOT experts, which explains why we cannot find a difference in likes or comments for human vs AI generated art.

While we did try to run a model to understand what attributes shape the number of hearts an AI generated art post will receive on Twitter, we realize that our dataset has far too little columns to truly understand. Our predictions mainly use the numeric values available in the dataset (retweets, comments, sentiment of comments, quotes etc.), there may be other variables such as what category of art the AI is generating (people/ scenery/ objects) that determine how many hearts the post will get. Currently, AI art is often bad at creating accurate images of humans, sometimes giving them an extra finger or weird limbs. These uncanny valley type images may also result in lower hearts. However, as we are limited in our variables, we could only tentatively predict using retweets, AI platforms, and sentiment of comments.

Future work includes directly scraping comments, likes etc. from social media platforms and extending our research to other social media platforms that regularly engage in posting art, such as Instagram. We should strive to find a platform where we have both AI and human generated art for a fairer comparison. Our AI art dataset did not include whether viewers were aware of whether the art was AI-generated and what the viewer’s general perception of AI art is, however, these are important factors to consider when determining the online reception of AI vs. human art. This can be expanded on by sending out surveys to analyze people's attitude towards AI vs. human generated art due to the limitations of how much data we can scrape from social media platforms alone. Additionally, having more complex datasets that collect and provide qualitative information on people’s perspectives on AI art compared to human generated art can allow us to have a more nuanced understanding on how these posts are perceived beyond just examining differences in amounts of likes and comments. We would also strive to create models that accurately reflect what influences the number of likes AI art gets. What would be even better is if we were able to find features that are the same for human and AI generated art, and predict the number of likes a human generated art piece would get if it was run under the model trained by AI generated art. We can then use that to figure out if human generated art would have more success than AI art, even if it was "assumed" to be AI art.

Our research aimed to provide a comprehensive look on the differences between online reception of AI vs human-generated art by comparing amounts of likes and comments, examining sentiments of comments, and more. Further research should be conducted to understand what factors contribute to an online art post’s “success” and how directly a person’s perspective on AI vs human-generated art impacts their online behavior and response in conjunction to the post. More elaborate datasets including data on people’s personal beliefs towards AI art, people’s art expertise level, and people’s knowledge of whether a post is AI-generated can allow this research to be more founded and thus give us a better insight into the implications of the rise of AI-generated art on the online art community as a whole.

Team Contributions

• Rachel: Helped with research, wrote the data section for AI dataset, did the data cleaning and EDA for AI dataset, wrote the abstract.
• Jennifer: Organized team meetings, wrote the ethics and privacy section, did the data cleaning and EDA for the Reddit dataset, ran the initial t-tests and built the initial decision tree model, wrote part of the discussion and conclusion.
• Mia: Research question, hypothesis, hypothesis testing, introduction and background presentation slides, programming, visualizations, editing for code and text consistency throughout.
• Maansi: Wrote part of background, data cleaning for Twitter data set, wrote part of EDA analysis, wrote part of discussion and conclusion.
• Haneen: Wrote part of background and prior works, wrote the data section for r/Art dataset, helped with the data cleaning and EDA for r/Art dataset, and ran the t-tests for digital art.