F.A.C.T SCORE FOR RESPONSIBLE AI

WHAT IS THE F.A.C.T QUESTIONNAIRE?

F.A.C.T stand for fairness, accountability, confidentiality and transparency. The F.A.C.T. questionnaire linked here is a 510 initiative to guide data scientists and other decision-makers in designing and using AI in a responsible way. It comprises a list of 44 questions one must be aware of when making decisions related to data, models and deployment for any data science. Based on the answers provided, a % score is generated to reflect the level of fairness, accountability, confidentiality and transparency at each of these 3 stages. An average F.A.C.T. score is then calculated based on these individual scores to indicate the overall AI responsibility level of the project.

While designing the questionnaire, the following aspects were taken into consideration:

  • Representative scoring: Questions were weighted differently to account for their varying degree of importance regarding responsibility concerns;
  • Time management: The questionnaire was elaborated to be answered in a timely manner (a maximum of 1h for each specific project);
  • Applicability: Questions were made as generalizable as possible to apply to a broad range of projects.

 

WHO HAS BEEN INVOLVED?

The F.A.C.T. questionnaire has been elaborated by a wide range of professionals at 510. This includes data scientists Gulfaraz Rahman, Kamal Ahmed, Jacopo Margutti, Tinka Valentijn and data responsibility policy officer Joachim Ramakers. Further input was given outside of 510 by AI researchers from the University of Amsterdam, such as Samarth Bhargav, Kishan Parshotam and Akash Gupta.

WHY IS IT NEEDED?

This questionnaire is a starting point to bridge the gap between theory and practical knowledge concerning responsible AI. As of today, academic research in the field is widespread but practical guidelines are missing. It is essential to provide decision-makers with concrete guidance on the ethical choices they make when designing and using AI. In that regard, the F.A.C.T. questionnaire provides a good starting point to make informed decisions.

WHAT ARE THE MAIN CHALLENGES?

While the questionnaire should be a good starting point for making more informed decisions, there is still room for improvement. Foremost, AI is a constantly evolving field. The questionnaire must hence be regularly updated to address responsibility concerns implied by these changes.

In addition, one must be aware that the team who designed the set of questions was informed by its past experience and knowledge. Other professionals might come up with a different set of questions and might attribute them different degrees of importance. The list is therefore non-exhaustive and will always benefit from further collaboration for refinement.

WHAT COMES NEXT?

The questionnaire would benefit from feedback. You can find the app here and provide suggestions for improvement via this email.

grahman@rodekruis.nl

Thank you for your contribution!

If you would like more information about the ethical implications of data science in the humanitarian sector then please read DSEG’s Linked piece A Framework for the Ethical Use of Advanced Data Science in the Humanitarian Sector.

Comments are closed