Flaws in ex-post enforcement in the AI Act

European Commission DG CNECT A2

15 February 2022 

Flaws in ex-post enforcement in the AI Act

Dear Mr. Gross,

1. We write on behalf of Irish Council for Civil Liberties (ICCL), Ireland’s oldest independent human rights monitoring organisation. We suggest four enhancements to protect people and their rights in the draft EU regulation, Artificial Intelligence (AI) Act.[1]

2. We propose amendments to do the following:

   i. strengthen ex-post enforcement;
   ii. enhance assessments by moving beyond self-assessment by providers;
   iii. enhance redress and safeguards by giving complainants legal standing; and
   iv. empower Market Surveillance Authorities to act.

      Below, we elaborate on each.

3. Before we do so, we should make clear that ICCL has reservations on the question of whether product liability is the correct framework to protect fundamental rights under the Act.

Strengthen ex-post enforcement

4. The Commission’s proposal assumes (emphasis added):

“A comprehensive ex-ante conformity assessment through internal checks, combined with a strong ex-post enforcement, could be an effective and reasonable solution for those systems, given the early phase of the regulatory intervention and the fact the AI sector is very innovative and expertise for auditing is only now being accumulated.”[2]

5. However, “strong ex-post enforcement” is the proposal. The proposal establishes a governance system with five tiers,[3] within which “market surveillance authorities” (MSAs) have the proactive investigatory function. Regulation (EU) 2019/1020 provides that MSAs independently decide when and what to investigate based on their assessment of risk.[4] 
6. There are four problems:

   i. The Commission itself noted in its Communication of April 2021 that enforcement will be challenging because AI is not transparent.[5] It is therefore unclear how MSAs can effectively investigate complex AI matters such as fake content, manipulation, or bias.[6]
   ii. The Commission’s assessment of the number of staff required at each Member State’s MSA is far too small,[7] as should be evident from our experience of the GDPR.
   iii. The pressures on particular MSAs may be further exacerbated by forum-shopping providers from outside the Union opting for comparatively lax or under-resourced MSAs.[8]
   iv. MSAs will also have difficulty recruiting experts with adequate competence.

7. Many of these problems repeat those of GDPR enforcement.[9] Therefore, we recommend that the Commission reconvene the High-Level Expert Group to reflect on what is now known about enforcement in the parallel domain of data protection. The Expert Group may also wish to consider the value of an expert enforcement support capacity at EU level, and whether this should be attached to the European Artificial Intelligence Board.

Enhance assessments by moving beyond self-assessment by providers

8. It is no longer controversial to suggest that relying on self-regulation in the technology sector has led to significant harms that could otherwise have been avoided. Indeed, ICCLs recent experience of the self-regulatory provisions in the GDPR has again proven this.[10] 
9. Despite this, the Act as proposed relies on providers to i) declare whether their systems are high-risk, ii) voluntarily provide information and manage risk,[11] and iii) inform authorities responsible for post-market monitoring.
10. Moreover, Article 62 (1) says that providers must report serious problems to MSAs only after they have established “a causal link” between their AI systems and the incidents, or a reasonable likelihood of one. This allows providers to evade their responsibility by finding explanations that do not include their own AI systems, especially when these are part of a larger system.
11. We make two recommendations:

    i. The Act should require all providers of AI systems, not only those that claim to be providers of high-risk AI systems, to register in the public EU database[12] so that the uses and the users of the AI systems can be scrutinised by the public and by independent authorities such as notified bodies.
    ii. Article 62 should require that operators report an incident or malfunction whenever an AI system is a part of the system concerned, and not only for serious incidents. This should include near-misses[13] so that other operators can learn from these incidents.

Enhance redress and safeguards by giving complainants legal standing

12. Although MSAs can receive information from “consumers” under Article 11 (3) (e) of Regulation (EU) 2019/1020, the Commission’s proposed AI Act does not provide a mechanism for the public or organisations representing them to lodge formal complaints, or to enjoy attendant rights.[14] 
13. Nor is there a right to judicial remedy against a provider or user for AI-specific harms or infringements of the Regulation.
14. The proposal therefore puts the primary burden of ex-post enforcement on MSAs, despite the high risk that MSA resources and powers envisaged by the Commission are unlikely to play the necessary role.
15. We make two recommendations:

    i. Individuals and relevant representative organisations should be enabled to lodge complaints against AI operators.[15] Competent authorities should be obliged to act on these complaints within set timeframes. Complainants should have standing in that process.[16]
    ii. The proposal should provide individuals and representative organisations with an effective judicial remedy against “operators”.[17]

Empower Market Surveillance Authorities to act

16. Chapter 3 of Title VIII, especially Article 64 (1) and (2), of the AI Act set out MSAs enforcement powers. These powers are much weaker than the powers conferred on MSAs in Article 14 (4) of Regulation (EU) 2019/1020.
17. Article 14 (4) (d, e, j) of Regulation (EU) 2019/1020 have not been adapted to the AI Act. The MSAs should be empowered “to enter any premises”,[18] “to reverse-engineer … to identify non-compliance and to obtain evidence”,[19] and “to carry out unannounced on-site inspections”[20] of physical premises such as data centres. These powers should be adapted for AI systems so that inspections can be carried out remotely and unannounced.
18. Therefore, we recommend the following:

    i. In addition to “unannounced on-site inspections and physical checks of products”,[21] remote inspections should be explicitly and unambiguous provided for, since physical access may be unnecessary for certain AI systems.
    ii. The AI Act should empower MSAs to do so without notice, as they are empowered to do in other sectors.[22] Currently the proposal only provides that MSAs shall request access from While using providers’ Application Programming Interfaces (‘API’) [23] by arrangement with them may yield useful information, it is important that MSAs retain their powers to investigate by independent means too, and without prior notice.

    This is necessary to assess the resilience of AI systems “as regards attempts by unauthorised third parties to alter their use or performance by exploiting the system vulnerabilities”[24] and to check whether “measures to prevent and control for attacks”[25] have been taken by the operators.

19. We would be happy to meet with you and your team to discuss this.

On behalf of ICCL,