Using machine learning to predict and analyze high and low reader engagement for New York Times articles posted to Facebook.
Author: Jessica Miles - jess.c.miles@gmail.com
In this repository, I used machine learning to understand the characteristics of Facebook content posted by The New York Times that lead to high user engagement. The analysis includes Natural Language Processing (NLP) of the post and article metadata as well as categorical features from both the posts and original articles.
My model was ultimately not accurate enough for me to recommend using it as a “black box” deciding which articles to post to Facebook. Instead, I used its coefficients to determine the most important keyword and topics among high versus low engagement posts, then grouped together similar topics and keywords together into themes to form recommendations. I believe this approach resulted in generalized results which are more likely to be useful over time, as compared to focusing on specific high engagement topics which only occur once, such as “the 2016 Presidential Election.”
Modern Americans consume news in multiple formats: in print, browsing and searching websites online, and on social media. To remain relevant in modern times, news organizations need to be able to engage users on social media platforms such as Facebook as well as using traditional print and web methods.
However, news outlets may produce far more content than can reasonably be posted to such platforms, so they need a methodology to decide what content formats and topics will be most successful. The type of content Facebook users engage with most may differ from what is run on the front page of the printed paper, so it’s prudent to analyze user engagement with Facebook content as a standalone exercise.
One criterion Facebook’s News Feed algorithm uses to prioritize content’s visibility to users is the amount of initial engagement (shares, comments, and likes) on a given post. Higher prioritization in News Feed may help content be disseminated to a wider audience, some of whom may decide to become subscribers.
It’s important to note that while this analysis focuses on increasing engagement, I do not advocate for engagement level to be the only consideration in deciding what to post. Just because people enjoy sharing recipes and reading pieces about animals doesn’t mean The Times should unequivocally prioritize those topics over pieces related to general politics or the economy. However, understanding the patterns behind high Facebook engagement could be included as one factor of many in the ultimate editorial decision.
I started with a found dataset of about 48,000 Facebook posts from The New York Times’ account covering the time period from late 2012 to late 2016. Data used in the analysis included the text in the post, when it was posted, and post type (link, video, or photo).
I also used the NYT API to pull all article metadata from this time period, and went through several steps to match the content in the Facebook posts to their original articles and multimedia features. The data I retrieved from the NYT API included article headline and abstract as well as metadata such as the news desk the article came from, topical subjects and other entities mentioned in the articles, word count, and format (written versus multimedia).
Of the original 48,000 Facebook posts, I was only able to match about 43,000 of them to articles. There were some challlenges in performing the matching due to differences between FB post text and links and the current article abstract and links; the links in the Facebook posts were often shortened, and even once expanded did not always match directly to a current article. Therefore, I modeled the features derived from all Facebook posts as a separate data set from the features derived from matched articles.
Engagement metrics included number of comments, shares, and likes/loves. Rather than focus on each of these separately, I created a single engagement metric.
Below are the distributions for each engagement metric. Note: Histograms don’t include outliers for visibility, but percentiles are calculated with outliers included.
The distributions of all Facebook posts and the smaller subset of posts matched to articles were quite similar. Although I modeled them separately, I used results from both sets of models in my final recommendations.
There appeared to be a slight uptick in engagement when posts were made on weekends, so I engineered a categorical variable for that.
I also engineered a categorical variable for time of day the post was made, as posts made in the morning and evening appeared to get more engagement.
When I visualized the most frequent words for high versus low engagement, I saw that words related to 2016 presidential candidates were more common in high engagement than in average. Otherwise though, they looked quite similar.
I also noticed that some high engagement posts posed questions to users, and asked their opinions. I created custom stopwords lists without permutations of the word “you” and without the quotation mark, to test performance compared to full stop words and punctuation list.
I used sklearn’s GridSearchCV combined with pipelines and other transformer classes to iterate over several different NLP preprocessing approaches, as well as Logistic Regression and Nauve Bayes model hyperparameters. The preprocessing steps I tested included:
max_features to be used in modelingI modeled both a binary and multi-class problem.
The binary performed slightly better on High Engagement, but the multi-class was interesting for what it showed about how the model tended to get confused.
The best binary classification model was able to identify about 62% of high engagement posts correctly (score is cross-validated, and on unseen test data).
Preprocessing and model parameters for the best model were as follows:
Distributions of all three engagement metrics were very right-skewed: they had many outliers on the high end and tapered off very smoothly. I ultimately chose the 75th percentile as the cutoff for “high engagement”, but there truly was no obvious cutoff point. Earlier in my analysis, I initially onsidered only outliers (using IQR * 1.5) to be high engagement, but these models did not perform as well. I believe it’s natural that the model would be confused about posts towards the middle of the distribution regardless of where the cutoff point is drawn. The multi-class model confirms this, as it performed most poorly on the Moderate Engagement middle class which represents the 25th through 75th percentiles.
To select features to factor into recommendations, I examined the predictors that had greatest odds ratios of High and Low engagement. After using a held-out test set to evaluate performance on unseen data, I trained the model on 10 random splits (without replacement), each consisting of 90% the entire data set, and calculated the mean odds ratio for each predictor. TF-IDF scores and top 2000 bi- and un-grams were re-calculated in each split. Feature importances represented as odds ratios were selected based on the aggregated mean across all 10 splits, and the standard error is shown on all graphs of odds ratios.
To group features into categories, I reviewed the top 300 predictors of both High and Low for Facebook post uni-grams and bi-grams as well as original article subjects, for a total of about 1,200 words and topics that I categorized. I made several passes through the list of features, assigning logical categories to words and subjects that seemed sufficiently unambiguous in their meaning. Several passes allowed me to refine the categories. Not all words and subjects were categorized; only those I recognized as common. Categories that tended more towards either high or low engagement were considered in the running for final recommendations.
See the Appendix in my presentation for additional charts showing the odds ratios for high engagement on these topics.
Please review the narrative of my analysis in my introductory jupyter notebook, modeling and analysis notebook, and my presentation.
For any additional questions, please contact **jess.c.miles@gmail.com
├── README.md <- The top-level README for reviewers of this project.├── data_gathering.ipynb <- 1. Notebook used to gather data from NYT API and match it to posts├── intro_eda.ipynb <- 2. Project introduction and data cleaning and exploration├── model_analysis.ipynb <- 3. Modeling and analysis of model results to form recommendations├── presentation.pdf <- PDF version of project presentation└── images└── images <- images of visualizations└── data└── data <- found and generated during analysis└── models└── models <- exported copies of best model pipelines, as well as notebook used to model in Google colab