Student-Teacher Evaluation Analysis¶
This document provides an overview of the methods used to produce the figures and values in the blog post shown here.
It utilizes predominantly the pandas and seaborn libraries for Python. Note that, by convention, these libraries are imported as pd and sns, respectively, so that any reference to these namespaces will indicate a call to the respective library.
Background¶
Tidy Data¶
Each of these libraries, and especially seaborn, relies upon the data being represented in a tidy format. Briefly, this tidy standard (also known as data matrix) insists that each observation occupies its own row, and that each feature/variable of an observation occupies its own column. This simple format, when enforced, allows graphing libraries to make the programming interfaces more user-friendly and terse based on assumptions about the arrangement of the incoming data. Many excellent examples of tidy (and messy) datasets are available in a pioneering paper by Hadley Wickham, developer of the renowned ggplot library in R (H.Wickham, 2014). Typically, there are multiple measures that could be considered an "observation" and used to denote a distinct row; the appropriate dimension can often be found by close inspection of the dataset and the relationships you want to inspect.
As an example, my company uses imaging techniques to capture oxygenation biomarkers in patients' feet. When we look at this data, should we consider a patient to be a single observation, such that each patient is in their own row? Seems logical. However, if a patient occupies a single row, our features will have to be associated with the left and right foot of that patient, e.g. 'Left Foot Biomarker 1', 'Right Foot Biomarker 1', 'Left Foot Biomarker 2', etc. This organization is useful if we want to look at properties for the Left foot compared to the Right foot for each patient. For most applications, it makes more sense to consider a single limb as a base unit of observation, such that the features need not be associated with a direction. With this organization, we denote a new feature - 'Direction' - which takes the values 'Left' or 'Right', and the other biomarkers can drop their explicit association with direction. In practice, it sometimes make sense to keep multiple copies of the dataframe in memory, with each organized according to a different base unit of measurement.
We will be working with predominantly tidy data throughout this tutorial, where each observation/row is a section of a course; That is, one instance of a course taught by one instructor. Our data will come in this tidy format, but know that pandas library comes with many tools for converting messy data to a tidy format (search pd.wide_to_long, pd.melt, pd.DataFrame.unstack functions for more info).
The first step is to read in the datasets. Both datasets will be read in as pandas DataFrame objects, which you can think of as analogous to spreadsheets, large tables - the main student reviews dataset read as df and the gender information (obtained from Gender API based on instructor first names) as df_gender). Instantiating these DataFrame objects will allow us to use the methods (functions tied to classes) associated with the DataFrame class to operate on our data.
A Note on Method-Chaining¶
Although dataframes (the chief abstraction provided by Pandas) are inherently mutable (by passing in the inplace=True operator), this approach is slated for deprecation and negates much of the beauty of pandas. The example below displays two equivalent operations, one using inplace operators and the other using a functional approach called method-chaining.
| In-place operations | Method Chaining |
|---|---|
| df.reset_index(inplace=True) df.join(df2, by='', inplace=True) df.drop_duplicates(inplace=True) df.round(2, inplace=True) |
df=df.reset_index().join(df2, by='Name').drop_duplicates().round(2) |
By chaining commands together, arguments are kept close to their respective function calls. Furthermore, this approach implicitly informs anyone reading the code that the intermediate state of the df object within the method chain is irrelevant. In this way, these custom user-defined method chains can be more easily understood as serial functions which perform some task. In pandas, method chaining allows you to write code that is more terse and clearer. A better example is shown below, borrowed generously from here.
| Wrapped Functions | Imperative, treating objects as immutable | Method-Chained |
|---|---|---|
tumble_after(broke(fell_down(fetch(went_up(jack_jill, "hill"),"water"),"jack"),"crown"),"jill") |
on_hill = went_up(jack_jill, 'hill') with_water = fetch(on_hill, 'water') fallen = fell_down(with_water, 'jack') broken = broke(fallen, 'jack') after = tumble_after(broken, 'jill') |
jack_jill.went_up("hill").fetch("water").fell_down("jack").broke("crown").tumble_after("jill") |
As we work through these analyses, many examples will include method chaining with pandas.
A Note on Imports¶
In Python, it's idiomatic to place all of your import statements at the top of the script. In this notebook, I've neglected to do that, such that I can keep the import statement close to the code which calls the imported module. This approach was taken to increase the clarity and readability of the examples.
import pandas as pd
import seaborn as sns
from matplotlib import pyplot as plt # matplotlib will be used to generate figures that seaborn fills with content
# IPython.display will let us nicely render code as markdown
from IPython.display import display, Markdown
# Assign the function dm to render markdown to standard output
dm = lambda x: display(Markdown(x))
# Read in the Aggregated dataset
df_reviews = pd.read_csv('2019-10-06-Aggregated-Reviews.csv', index_col=False)
# Read in the instructor gender information
df_gender = pd.read_csv('instructors_with_gender.csv', index_col=False)
Now we come to an issue. We need to join the two dataframes in order to enforce our tidy data standards and to use tidy tools (like Seaborn) to explore the dataset. We can accomplish this easily via the pd.DataFrame.merge method. To better represent this join, we'll first set the index of each array to the columns along which they will be joined, via the pd.DataFrame.set_index method. To display the reindexed DataFrame, we'll use the pd.DataFrame.head function, which displays the first n elements. Well also print out the pd.DataFrame.columns attribute, which contains the names of the columns that can be operated upon in the DataFrame.
df_gender = df_gender.sort_values(by='Last Name')
dm('##### df_gender') # Recall that dm was assigned to the function display markdown, so it is basically a print statement now
display(df_gender[['Last Name', 'First Name', 'ga_gender']].head(5)) # Print the first 5 rows
print(f'Columns: {df_gender.columns.values}') # .values returns an np.array from the .columns attribute
df_reviews = df_reviews.sort_values(by='Instructor Last Name')
dm('<br />') # Recall that dm was assigned to the function display markdown, so it is basically a print statement now
dm('##### df_reviews')
display(df_reviews[['Instructor Last Name', 'Instructor First Name', 'Instructor ID']].drop_duplicates().head(5)) # Slice columns and print the first 5 rows
print(f'Columns: {df_reviews.columns.values}') # .values returns an np.array from the .columns attribute
Take a moment to note all of the columns of each dataframe, and to recognize the breadth of analysis opportunities this affords.
To create the Dataframe we'll use for analysis, we can merge df_gender to df_reviews via the First Name and Last Name columns. But, as we can see the equivalent columns in df_reviews are named Instructor First Name and Instructor Last Name, so we can pass in some additional arguments to the .merge() method to account for this distinction.
df = df_reviews.merge(df_gender, left_on=['Instructor First Name', 'Instructor Last Name'], right_on=['First Name', 'Last Name'], how='inner')
display(df[['Instructor Last Name', 'Instructor First Name', 'Instructor ID', 'ga_gender']]\
.sort_values(by='Instructor Last Name')\
.drop_duplicates().head(5)) # Slice columns then use df.head() method to output first 5 columns
You can see from the above that we have successfully merged our two dataframes into the df variable - Note the gender and ID equivalence to the individual tables above.
Now, what can we do with this tidy and merged dataset? That's where seaborn comes in.
Gender Differences In Student Ratings¶
Seaborn knows how to fetch and display data from a tidy dataframe, like the one(s) we have loaded in and the df we created from the .merge() method. To show how this accelerates the visualization process, consider the example below.
print(f'Before filtering to ensure gender accuracy, the dataset had {len(df)} entries.\n')
df = df[df['ga_accuracy']>90] # filter the df by ga_accuracy to ensure that we are confident in the gender of all of our professors
print(f'After filtering to ensure gender accuracy, the dataset has {len(df)} entries.\n\n')
sns.reset_defaults() # See other seaborn styles here: https://seaborn.pydata.org/tutorial/aesthetics.html
# Create a larger axes for the plot
fig, ax = plt.subplots(2,1, figsize=(10,8))
fullscale = sns.boxplot(x='Avg Instructor Rating In Section', y='ga_gender', data=df, ax=ax[0])
zoomed = sns.boxplot(x='Avg Instructor Rating In Section', y='ga_gender', data=df, ax=ax[1])
ax[1].set_xlim([4.2,4.5])
plt.show()
It looks like, for our dataset at the very least, male professors are rated more highly than female. But how much higher are they rated? And whats our sample size like? And how did these biases vary over time? And is the difference significant? To answer these questions that any statistician/data scientist would pose, we can use a combination of pandas, seaborn, and another common Python package regularly used in scientific computing known as scipy.
What is the magnitude of the difference and whats our sample size?¶
To answer this, we'll use the pd.DataFrame.Groupby method to separate the two groups, then the pd.DataFrame.aggregate method to compute some metrics. With the magic of pandas, we can answer both of these questions (and more) in one line of code.
display(df.groupby('ga_gender').aggregate({'Avg Instructor Rating In Section': ['count', 'mean', 'median', 'std']}))
We note that men teach ~62% of the course sections at OU, and that the mean scores are actually extremely similar ($\Delta\approx0.006$) between male and female profesors.
How much have these gender effects changed over time?¶
To answer this, lets pull out another column using the pd.Series.apply method. Note that a pd.Series is simply a column in a pd.DataFrame, s.t. df['Term Code'] returns a pd.Series object. Think of our pd.Series.apply method as analogous to an excel operation where you write out a formula in a top cell and drag the formula down to apply to all of your rows. To pull out this Year Series/column, note that the df column Term Code takes format YYYYSS, where YYYY is the year and SS the semester (10: Fall, 20:Spring, 30:Summer). We'll pull out these first 4 digits and assign them to the Year column.
# Note that our lambda function converts the int to string, then slices off the first 4 characters and converts these back to int type
df['Year'] = df['Term Code'].apply(lambda x: int(str(x)[:4]))
Now, we can plot the average ratings of male and female professors by year.
df_by_year = df.groupby(['Year', 'ga_gender']).aggregate({'Avg Instructor Rating In Section': 'mean'}).reset_index()
display(df_by_year.set_index('Year').round(2).T) # .T transposes the table
fig, ax = plt.subplots(1, figsize=(10,4))
bar = sns.barplot(x='Year', y='Avg Instructor Rating In Section', hue='ga_gender', data=df, ax=ax)
ax.set_ylim([4,4.5])
plt.show()
Things are less apparent from this viewpoint. Although men have higher mean ratings in the majority of years - 2010, 2012, 2013, 2014, 2017, and 2018 - Men are by no means consistently or exclusively rated higher than women.
Are the gender differences significant?¶
We can use the python package scipy to conduct a two-sample students t-test to compare the means of the two groups.
from scipy import stats
male_ratings = df[df['ga_gender']=='male']['Avg Instructor Rating In Section']
female_ratings = df[df['ga_gender']=='female']['Avg Instructor Rating In Section']
dm('Based on a Individual Means t-test for all tested instructor ratings, the p-value for the test is '\
f'{round(stats.ttest_ind(male_ratings, female_ratings)[1], 2)}, '\
'meaning we cannot reject the null hypothesis that the male and female group ratings are equivalent.')
There are many more comparisons between variables we could conduct, but lets move on to futher analaysis and demonstrate some other functionality in seaborn and pandas.
Differences by College¶
We'll take a chance to assess how the ratings for each college have varied over time. However, the colleges are stored by codes, rather than the full name of the department. To back out the full department name, we can use the pd.DataFrame.map function to apply the following hash map to convert college codes to college names:
from pprint import pprint # pprint stands for pretty print; Just outputs nicely formatted text
college_code_mapper = {'College of Architecture': 'CoA',
'College of Arts and Sciences': 'CoAaS',
'College of Atmospheric & Geographic Sciences': 'CoA&GS',
'College of Continuing Education - Department of Aviation': 'CoCE-DoA',
'Michael F. Price College of Business': 'MFPCoB',
'Melbourne College of Earth and Energy': 'MCoEaE',
'Jeannine Rainbolt College of Education': 'JRCoE',
'Gallogly College of Engineering': 'GCoE',
'Weitzenhoffer Family College of Fine Arts': 'WFCoFA',
'Honors College': 'HC', 'College of International Studies': 'CoIS',
'Gaylord College of Journalism and Mass Communication': 'GCoJaMC',
'College of Professional and Continuing Studies': 'CoPaCS',
'University College': 'UC', 'Center for Independent and Distance Learning': 'CfIaDL',
'Expository Writing Program': 'EWP', 'ROTC - Air Force': 'R-AF'}
# Invert the mapper
code_college_mapper = {v:k for k,v in college_code_mapper.items()} # dict comprehensions are great
pprint(code_college_mapper)
df['College Name'] = df['College Code'].map(code_college_mapper)
fig, ax = plt.subplots(1, figsize = (12, 4))
# Plot the lines
lines = sns.lineplot(x='Year', y='Avg Department Rating', markers=True, hue='College Name', ci=None, data=df, ax=ax)
# Plot the regression lines
reg = sns.regplot(x='Year', y='Avg Department Rating', color='k', label='Linear Regression', ci=95, data=df, scatter=False, ax=ax)
lg = plt.legend(bbox_to_anchor=(1.05, 1), loc=2)
plt.show()
# Compute the linear regression information via stats package
x = df['Year'].values
y = df['Avg Department Rating'].values
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
# Display our results
dm('With this plot, we can get a good indication of '
'the overall temporal changes in ratings by department via the black line.'
f'We note that the slope of the change (in rating points per year) is {round(slope, 3)}, and that,'
f'although we are confident in this trend ($p<10^{{-30}}$), it represents only about $R^2=${round(100*r_value**2, 2)}%'
' of the variation observed in the dataset. Visually, it appears that much more of the variance can be attributed'
' to differences between the colleges. However, with this busy plot its tough to distinguish these colleges.'
' With the next graph, we will break up these curves to visually distinguish the colleges.')
Do the number of courses offered by a college impact its ratings?¶
# We've used these functions before; See if you can follow the operations.
collegecounts = df.drop_duplicates(subset=['course_uuid']).groupby('College Name')\
.aggregate({'course_uuid':'count', 'Avg Department Rating': 'mean'})\
.rename(columns={'course_uuid':'NumberofCoursesInCollege', 'Avg Department Rating': 'Avg College Rating'})
display(collegecounts.sort_values(by='NumberofCoursesInCollege').round(2))
df = df.merge(collegecounts, left_on='College Name', right_index=True).reset_index()
dm('Now, we can plot the smaller departments and the larger departments in their respective graphs. Well '\
'use the `pd.DataFrame.query` method to separate the df into three groups for plotting.')
fig, ax = plt.subplots(3,1, figsize = (12, 10), sharex=True)
for index, query in enumerate(['NumberofCoursesInCollege < 200', '500 > NumberofCoursesInCollege > 200', 'NumberofCoursesInCollege > 500']):
lp = sns.lineplot(x='Year', y='Avg Department Rating', hue='College Name', size='NumberofCoursesInCollege', data=df.query(query), ax=ax[index]) # Add width
ax[index].set_title(f'Colleges with {query}')
ax[index].legend(bbox_to_anchor=(1.02, 1), loc=2)
ax[index].set_ylim([3.4, 4.6])
plt.show()
From first glance, it looks like the departments in smaller colleges don't necessarily have higher ratings - A noteworthy example of this is in the 3rd row, where courses in the College of Fine Arts (the second largest course) has traditionally received the highest rating of any college.
However, we should be careful not to confuse this finding with the effect of course enrollment on student ratings. We will investigate this in the following section.
Effect of Course Level and Course Enrollment on Ratings¶
At OU, the course level is indicated via the first digit of the Course Number column. Courses beginning with a 1 (e.g. 1114, 1415, etc.) are first-year courses courses, whereas courses beginning with a 2 (e.g. 2124, 2510) are second-year courses, ... Additionally, any course beginning with a 5, 6, or 7 is a graduate course. We can extract a Course Level feature based on this explicit mapping, then plot this feature.
course_level_map = {'1':'First-Year Course',
'2': 'Second-Year Course',
'3': 'Third-Year Course',
'4': 'Fourth-Year Course',
'5': 'Graduate Course',
'6': 'Graduate Course',
'7': 'Graduate Course'}
df['CourseLevel'] = df['Course Number'].apply(lambda x: str(x)[0]).map(course_level_map)
df = df.sort_values(by='Course Number')
g = sns.lmplot(x = 'Course Enrollment', y='Avg Course Rating', hue='CourseLevel', aspect=2.,
markers = ['o', '^', 's', 'x', '.'], data=df)
g.set(ylim=(0, 5), xlim=(0, 2000))
plt.show()
The above regression lines, fit via a least-squares bootstrapping method, show a loose negative correlation between the size of the course and the number of students enrolled. However, in the upper-level (namely fourth-year and graduate), we see very few courses with >250 students enrolled, which makes it difficult to view the patterns indicated by these trendlines.
Despite the rich information it provides about the course enrollment trends across the different course levels, the above curve displays relatively little about the differences in course ratings across the different course levels. To assess these differences, we can use the seaborn violinplot function, along with the pandas groupby().aggregate procedure from before. Note that we now define custom functions to represent Q1 and Q3, via the numerical computing library numpy.
import numpy as np
fig, ax = plt.subplots(1, figsize=(14, 6))
sns.violinplot(x='Avg Course Rating', y='CourseLevel', data = df, ax=ax)
plt.show()
# Define functions to compute q1 and q3
def Q1(col):
return np.percentile(col, 25)
def Q3(col):
return np.percentile(col, 75)
df_courselevel = df.sort_values(by='Course Number').groupby('CourseLevel', sort=False)\
.aggregate({'Avg Course Rating': ['mean', Q1, 'median', Q3], 'Course Enrollment': ['mean', 'median']})\
.round(2)
display(df_courselevel)
This plot and accompanying graph certainly show an increase in course ratings with increased course level, but let's see what happens if we run a linear regression on the dataset. For brevity, we'll not show a plot, but the regression will assess correlations between Avg Course Rating (y) and Course Level (x).
# Compute linear regressian for group medians and enrollments
# Step 1: map the course number to the type of course as an int
course_value_map = {'1':1,
'2': 2,
'3': 3,
'4': 4,
'5': 5,
'6': 6,
'7': 7}
x = df['Course Number'].apply(lambda x: str(x)[0]).map(course_value_map)
y = df['Avg Course Rating']
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y) # Linreg stats for the avg course ratings
y = df['Course Enrollment']
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y) # Linreg stats for the avg course ratings
Without exception, the Q1, median, and Q3 ratings all increased as the level of the course increased. A linear regression applied shows that the Average Course Rating increases on average 0.077 for each course level increased, and that the course level explains $R^2=4.56$% of the variation observed in the Average Course Ratings ($p\approx0$). Furthermore, the courses got smaller as the Course Level increased, changing from an average course enrollment of 101 for First-Year Courses to an average enrollment of 11 for Graduate Courses. Through similar linear regression methods, we found an average drop of 43 students per course level increased, and that the course level accounts for $R^2=19.4$% of the variance observed in course enrollments ($p\approx0$).
Conclusions¶
Gender Differences¶
Although male instructors have a higher mean rating, this observed gender difference ($\mu^{male}_{avg} - \mu^{female}_{avg}=0.006$) was not temporally consistent nor statistically significant (p=0.32).
Changes over Time¶
Average course ratings have increased by 0.01 per year over the previous 10 years ($p\approx0$, $R^2=0.0048$) - It looks like grade inflation has been a quid pro quo in this respect.
Differences by College¶
Colleges have unique ratings which exhibited minimal year-to-year variation, especially in larger colleges, and were not qualitatively correlated with the number of courses offered in the college (used as a proxy for the size of the college).
Effects of Course Level¶
Course ratings increase with increased course level ($p\approx 0$, $R^2=0.0456$), to the extent that the median ratings for graduate courses are 0.43, or 10.56%, higher than the median ratings for first-year courses. Conversely, the course enrollments decreased with the increased course level ($p\approx 0$, $R^2=0.1940$), such that the median student enrollment for graduate courses was 89% lower than the median student enrollment in first-level courses (11 for graduate vs. 101 for first-year).
Given these analyses (interpreted further in the article here), we feel we have presented an honest assessment and exploration of the STEV data presented in our web app at ou.evals.info. Concurrently, if you read through this notebook hoping to learn more about the pandas and seaborn packages, I hope you appreciate the way in which these tools can facilitate rapid and robust exploration of a dataset.
Thanks for reading,
Sam Jett and Zach Schuermann