We have used various state-of-the-art Natural Language Processing Techniques to determine the strength of an argument.
Writing is crucial for success. In particular, argumentative writing fosters critical thinking and civic engagement skills, and can be strengthened by practice. However, only 13 percent of eighth-grade teachers ask their students to write persuasively each week. Additionally, resource constraints disproportionately impact Black and Hispanic students, so they are more likely to write at the “below basic” level as compared to their white peers. An automated feedback tool is one way to make it easier for teachers to grade writing tasks assigned to their students that will also improve their writing skills.
This was intended as our solution to a compeition posted on Kaggle.
The Dataset can be found here: https://www.kaggle.com/competitions/feedback-prize-effectiveness
The dataset presented there contained argumentative essays written by U.S students in grades 6-12. These essays were annotated by expert raters for discourse elements commonly found in argumentative writing:
- Lead - an introduction that begins with a statistic, a quotation, a description, or some other device to grab the reader’s attention and point toward the thesis
- Position - an opinion or conclusion on the main question
- Claim - a claim that supports the position
- Counterclaim - a claim that refutes another claim or gives an opposing reason to the position
- Rebuttal - a claim that refutes a counterclaim
- Evidence - ideas or examples that support claims, counterclaims, or rebuttals.
- Concluding Statement - a concluding statement that restates the claims
Our task was to predict the quality rating of each discourse element. Human readers rated each rhetorical or argumentative element, in order of increasing quality, as one of:
- Ineffective
- Adequate
- Effective
There are Two Notebooks:
- NLP_Project: Here We used Transformer model BERT and added a simple NN on top to classify the arguments
- NLP_Embeds_and_Classifiers: Here we extracted all the embeddings of input using BERT and then added custom classifiers on top.
The Neural Network and Logistic Regression gave the same overall accuracy but Neural Network could not classify ineffective arguments but Logistic Regression had a better F1 Score.
