Smart Industry Operations 2024-2025 Individual Assignment Classification, Bias, and Fairness Context of the Assignment The context of this assignment is the use of predictive AI tools to support decision making inoperationalsettings. The specific operational setting is in the criminal justice system and the case of
interest is the COMPAS system, referred to in the lectures:COMPAS: Correctional Offender Management Profiling for Alternative Sanction
https://en.wikipedia.org/wiki/COMPAS_%28software%29https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithmThe purpose of the assignment is to understand by applying in practice- classification methods work and how they can be appropriately assessed- Understand aspects of bias in AI outcomes- Make suggestions about how such bias can be mitigated.The COMPAS system was operationalised in the United States and was provided to the states’justice systems by private companies. The motivation for the justice system was to have a moreefficient system in place, which would also potentially avoid human bias in making a decision. The
decision is about whether to deny or grant an offender release from prison on the basis of a riskassessment regarding whether the specific person was at risk of offence recidivism. The riskassessment is based on a form such as:www.documentcloud.org/documents/2702103-Sample-Risk-Assessment-COMPAS-COREThe COMPAS system was designed to use collected data from such risk assessments to make aprediction about whether an offender under consideration was predicted to be at an risk or not ofrecidivism.Case Data To perform the tasks in this assignment, you are provided with real data from the operational useof the COMPAS system. The data set is structured afollows:age: A numerical variable, marking the age of the person
juv_fel_count: A numerical variable, marking the number of juvenile feloniesjuv_misd_count: A numerical variable, counting the number of juvenilemisdemeanoursjuv_other_count: A numerical variable, indicating the number of juvenile convictions that wereneither felonies nor misdemeanors
priors_count: A numerical variable, providing the number of prior crimes committedis_recid: Binary variable, stating whether the person recidivate within 2 years (1:yes, 0:no)?sex: Categorical, the gender of the defendant (broken down in 2 dummy variables)race: Categorical, the race of the defendant, broken down in dummy variablesc_charge_degree: Categorical, the degree by which the person is currently being charged with. Thecategories are: felonies, misdemeanors, and infractions, ordered from most serious to least. Theseare further broken down to charge types (felonies or misdemeanours), and in each category fromthe 1st (most serious offenses) to the least severe (F1-3, M1-MO3). Finally there is an attribute forother offences and this is a binary attribute (0 or 1).compas_score: This is the final prediction that the COMPAS system has made for each person,being at risk or no of offence recidivism. 1: high/medium risk, 0: low risk.The provided datasets are:
- A. recidivism-risk.csvA complete dataset, populated with all above information. You will use this for your analysis and forclassification modelling training purposes.
- . recidivism-risk-predict.csv
This is a similar dataset to A, but on this one you have no 代 寫 Smart Industry Operationsaccess to the compass_score. You will usethis to predict the compass score (classify unknown cases).
Assignment Questions
A.1. Exploratory Data Analysis (10% of Assignment 1 mark) In this part you are expected to:A1.1. Explore the variables, their types, and their basic statistics.A1.2. Analyse further the data regarding data distributions, range of values, existence of outliersand correlations between attributes, as well as between input attributes and compass_score. Towhat extent is the dataset balanced regarding the different categories of sex and race and acrossthe age ranges?
A.2. Classification (30% of Assignment 1 mark) In this part you are expected to develop classifier models. You will have to consider how best to useyour training data (recidivism-risk.csv) and you are asked to apply the developed models to the
“recidivism-risk-predict.csv” data.A2.1. Apply a decision tree classifier, choosing different tree depths on the recidivism-risk.csv data.Motivate your solution analysis in relation to overfit and generalization.Report and analyseperformance using different performance metrics. Analyse your findings. Finally, choose adeveloped model and apply it to the recidivism-risk-predict.csv data to produce your predictions.
A2.2. Apply a random forest classifier. random forest regression choosing different number ofestimators and tree depths. Motivate your solution and analysis in relation to overfit andgeneralization. Report and analyse performance using different performance metrics. Analyse yourfindings. Finally, choose a developed model and apply it to the recidivism-risk-predict.csv data toproduce your predictions.A2.3. Apply a Naïve Bayes classifier. Report andanalyse performance using different performancemetrics. Analyse your findings. Finally, choose a developed model and apply it to the recidivismrisk-predict.csv data to produce your predictionsA2.4. Apply a support vector classifier. Report and analyse performance using different
oduce your predictions.A2.5. Make a comparative analysis across all classifier experiments. Make a reasoned choice of alassifier to select and motivate the choice referring to the evidence obtained from performancemetrics.A.3. Bias Analysis and Management (50% of Assignment 2 mark) In this part you are expected to further analyse the data regarding potential bias. Specificallyconsider the characteristics ‘age’, and sex (female, male), and race (all categories).A3.1.1-A3.1.4. Perform exactly the same experiments as in part A.2. but without taking into accountthe ‘race’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.1.1 is fordecision tree, A3.1.2 for random forest, A3.1.3 for Naïve Bayes, A3.1.4 for support vector classifier)Consider also your observations from part A.1. regarding the distribution of data in the recidivism
risk.csv dataset with regard to the race attributes in your analysis.
A3.2.1-A3.2.4. Perform exactly the same experiments as in part A.2. but without taking into accountthe ‘sex’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.2.1 is fordecision tree, A3.2.2 for random forest, A3.2.3 for Naïve Bayes, A3.2.4 for support vector classifier).Consider also your observations from part A.1. regarding the distribution of data in the recidivismrisk.csv dataset with regard to the sex attributes in your analysis.A3.3.1A3.3.4. Perform exactly the same experiments as in part A.2. but without taking into accountthe ‘sex’ and ‘race’ attributes. Analyse the difference of the obtained results compared to A.2.
A3.3.1 is for decision tree, A3.3.2 for random forest, A3.3.3 for Naïve Bayes, A3.3.4 for supportvector classifier). Consider also your observations from part A.1. regarding the distribution of datain the recidivism-risk.csv dataset with regard to the sex attributes in your analysis.
A.4. Overall comparisons and analysis (10% of Assignment 2 mark)) In this part you are expected to:
Discuss comparatively the obtained results highlight only what you see as most interestingregarding the obtained performance and/or aspects of data unbalance, and fairness, motivatingyour analysis on the basis of the obtained evidence. What would be your concludingrecommendations?.
A.5. Bias Management and Mitigation (up to 15% extra of Assignment 2 mark, capped to maximum assignment mark)
This is a bonus part of the assignment for the teams that aim to ensure a very high mark. In thispart you are free to work creatively on the basis of what you have seen in the lectures andpracticals. The aim in this part is to apply bias management and mitigationmethods as applicable toeither the race or the sex attributes in the dataset. You are not expected to do this exhaustively butto work on limited experiments of your choice (for example regarding one of the two sensitive attributes, sex or race). The aim of this extra ‘question’ is to trigger you into creative thinking andaction, rather than instruct you to perform a very specific task.A general remark on grading: Note that the quality of your analysis in each step will be taken intoaccount. Make sure your code is well-documented and your report is readable, such that what isdone is clear and motivated. We will evaluate your contribution based on what is explained anddocumented in your Jupyter notebook report.