AlphaGo[vernment]: Takeaways for designing machine-assisted decision-making systems for the public sector

What prompted the Ombudsman Report?

The Ombudsman Report was prompted by an investigation undertaken by the NSW Ombudsman into Revenue NSW, following complaints regarding Revenue NSW’s use of garnishee orders. Under statute, the Commissioner of Fines Administration (Commissioner) has the discretion to issue garnishee orders, if satisfied that certain circumstances exist. Garnishee orders allow a debt collector (the Commissioner) to recover debts directly from a financial institution (such as a bank).

Revenue NSW used automated technology to analyse and profile customers to enforce fines.

Between 2010 and 2019, the number of garnishee orders issued by Revenue NSW increased from 6,905 to 1.6 million. Its reported impact included completely depleting people’s accounts, which at times held the complainant’s only source of income from Centrelink. Prior notice was not given to account holders.

Although Revenue NSW cooperated with the NSW Ombudsman and made a number of modifications to its process (such as excluding particularly vulnerable persons from the process), external advice sought by the NSW Ombudsman confirmed that the automated process was not lawful because:

• no authorised person engaged in a mental process of reasoning to reach the state of satisfaction prescribed by statute in order to issue a garnishee order; and
• discretionary power was not being exercised by the authorised person.

Revenue NSW’s process is only one of a number of recent examples where insufficient attention has been paid to administrative law principles in designing automated systems.

The public sector context

The Ombudsman Report emphasises that the powers and actions of government agencies and officials are “constitutionally unique” from that of the private sector. As such, although the use of machine technology raises a range of ethical, legal and privacy concerns, its use in the public sector must be assessed primarily from an administrative law perspective.

It is imperative to design systems which meet the essential requirements of good administrative decision-making. Those requirements, as summarised by the Ombudsman, are:

• Proper authorisation: the decision-maker has legal authority and their decision is within scope of decision-making power;
• Appropriate procedures: the process of decision-making is fair, legal and ethical, and reasons for the decision are given to affected persons;
• Appropriate assessment: the decision answers the right question, is based on a proper analysis of relevant material and is defensible; and
• Adequate documentation: the circumstances surrounding the making of the decision are adequately recorded.

Takeaways for government

• Seek advice: Engage a multidisciplinary team to meet the challenge of translating law into code. Be aware of the limitations of your legislative framework and whether legislative amendment is required. Consider how an affected person would participate in the decision-making process and the practical implications of challenging decisions.
• Buy smart: Observe responsible procurement practices that emphasise transparency over decision-making processes, especially regarding copyright and intellectual property rights of suppliers.
• Test, test, test: prior to implementation and when the system is established. Test to ensure the correctness of machine process and legal interpretation, and to ensure that machines maintain an audit trail that satisfies principles of transparency and accountability.
• Be transparent: Government should consider releasing source codes and technical information of machine processes to the public. Greater transparency has the benefit of engaging external expertise and providing greater oversight through both public and parliamentary scrutiny.

Evolving law and policy: Resources for government

The legal and policy frameworks governing the implementation and use of machine-assisted decision-making systems are continually evolving. The challenge facing government is to establish legal frameworks that are future-focused and drafted in a manner that facilitates technological iterations.

While attention must be paid to the particular legal constraints that apply to each public sector function or program in designing systems, the following resources serve as helpful guidance:

• Ethics Policies:

• Australian Government Artificial Intelligence (AI) Ethics Framework: includes 8 AI Ethics Principles, guidance on applying the principles and case studies of companies trialling the principles.
• NSW Government Artificial Intelligence (AI) Ethics Policy: includes the Ethical Policy Statement, Mandatory Ethical Principles for the use of AI and the AI User Guide.

• AI Strategies:

• NSW Government Artificial Intelligence (AI) Strategy: focuses on five key areas developed through multi-stakeholder consultation (1) building public trust; (2) digital uplift; (3) building data capability; (4) procurement; (5) innovation and collaboration;
• Australian Government Department of Industry, Science and Energy Resources’ Artificial Intelligence Policy and Initiatives: refers to the Australian Government’s AI Action Plan, AI Ethics Framework and other initiatives related to artificial intelligence;
• Australian Government Artificial Intelligence (AI) Action Plan: sets out the initiatives the government is taking to achieve its aim of Australia becoming a global leader in developing and adopting trusted and responsible artificial intelligence.

• Practice guides

• Commonwealth Ombudsman’s Automated Decision-Making Better Practice Guide: intended to be a practical tool for agencies during the design and implementation of automated systems;
• NSW Government Artificial Intelligence (AI) User Guide: intended to be a practical guide for NSW Government agencies to utilise AI in enhancing customer service delivery in a way that builds and maintains public trust;
• NSW Government Artificial Intelligence (AI) Assurance Framework: consists of advice to assist agencies to design, build and use AI-enabled solutions.