Ensuring Human Oversight
Why must your startup have a human-in-the-loop (HITL) process?
Regulatory compliance
Key legislation like the EU AI Act, the GDPR and the DUAA mandate a certain amount of meaningful human review and intervention in AI systems, particularly where decisions are automated.
For example, the EU AI Act stipulates that high-risk AI systems must be ‘designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use'.
Requirements for human oversight are particularly strict in the case of high-risk AI systems.
The financial penalties for not complying with human oversight requirements can be very hefty.
Building trust
Even if your startup isn’t directly subject to these laws currently, aligning early with regulatory requirements elsewhere can demonstrate trustworthiness to stakeholders and simplify compliance as you scale.
How can your business ensure sufficient human oversight?
Your business must implement mechanisms to ensure meaningful human oversight of its AI systems. Although your user may perform the human oversight step (often by consenting to the use of the AI system), your business is responsible for providing that human oversight mechanism and documenting its successful functioning.
Your business is responsible for:
Providing a human oversight mechanism
While AI can perform assessments and plan next steps, a human must oversee its use.
Often, the oversight step is performed by your user. For example, your business will provide a consent mechanism whereby your user consents to the continued use of the AI system.
Documenting this human oversight mechanism
Your startup is responsible for documenting what the intervention mechanism is and how it functions day-to-day.
Maintain detailed records of these workflows. Log:
when a human intervention mechanism is presented to the user;
the user’s response;
that these mechanisms are applied consistently across the business and across versions of the tech you use.
Keep these logs for at least ten years (required for high-risk systems under the EU AI Act). These paper trails are critical for both external audits and internal reflection.
Allowing humans to functionally override all high-risk AI decisions
This is mandated under Article 14 of the EU AI Act.
Make sure that human operators are able to override AI decisions when necessary.
These overrides should also be logged in the HITL workflow records to ensure transparency and aid development.
Facilitating users to challenge automated decisions
Article 22 of GDPR grants individuals the right to obtain human intervention in any automated decisions that affect them (note that this has been tweaked in the UK following the Data (Use and Access) Act).
This is a right your users can invoke when they please.
Therefore, your business must have an established mechanism via which users can request a human review of the automated decision.
Additional technical safeguards
There are also some key technical tests that can be carried out on AI output to ensure it isn’t straying too far from the model. Often, this involves using benchmark comparisons to regularly validate model outputs. These may include fairness metrics, accuracy metrics, and domain-specific standards.
There are established systems for monitoring dashboards and drift detection which can help you practically implement these safeguards.
Last updated