温馨提示：本站仅提供公开网络链接索引服务，不存储、不篡改任何第三方内容，所有内容版权归原作者所有
AI智能索引来源：http://www.icf.com/insights/analytics/ai-ethical-collaborations-evaluation
点击访问原文链接

Humans and machines: Ethical collaborations in evaluation | ICF

进入原网站 导航首页
Humans and machines: Ethical collaborations in evaluation | ICF Skip to main content Back Industries Powerful solutions that drive innovation and deliver results. Our Expertise Utilities Disaster management Transportation Aviation Federal health Energy Environmental services and infrastructure Climate resilience Community and economic development U.S. federal State and local government UK government European government Client stories See how we help fast-changing industries succeed. Back Capabilities Applying technology and expertise to deliver bold, lasting impact. Technology Digital modernization Artificial intelligence Data and analytics Experience and design Cloud Cybersecurity Partner ecosystem Solutions Program implementation Strategy and innovation Workforce and change management Policy and regulatory Grants management Strategic communications ICF Fathom Explore our suite of AI solutions and services Strategic agency for engagement and transformation. Back Insights Stay ahead with research, insights, and success stories developed by industry experts. Trending topics Using AI against fraud, waste, and abuse AI innovation Grid modernization Workforce development Extreme weather resilience Latest thinking Articles Client stories News Reports Webinars Smart charts Top federal AI trends New findings for 2025 The impact of rapid demand growth Key recommendations for utilities Back About A leading global company turning data into insights and innovation into impact. About ICF News Leadership History Corporate Citizenship Ethics and compliance Data privacy Contracts Locations Europe and U.K. Learn about our work with governments, businesses, and organizations in Europe and the U.K. Need help or want to learn more?  Contact us Careers Investors Popular Searches Federal IT modernization Artificial intelligence Disaster mitigation Energy efficiency Federal health Featured Learn more Federal IT modernization services Subscribe to modernization insights Subscribe

Don't miss out

Don't miss out

Don't miss out

Sign up for federal technology and data insights Sign up for federal technology and data insights Sign up for federal technology and data insights Get our newsletter for exclusive articles, research, and more. Get our newsletter for exclusive articles, research, and more. Get our newsletter for exclusive articles, research, and more. Subscribe now Insights / Data and analytics Article Humans and machines: Ethical collaborations in evaluation Humans and machines: Ethical collaborations in evaluation Oct 8, 2024 7 MIN. READ Ethical collaborations between humans and AI in EU policy evaluations at ICF enhance data collection, analysis, and dissemination, while addressing challenges through robust governance and transparency.

As public policy evaluators at ICF, predominantly working for the European Commission and other EU and international institutions, we have been eagerly following the rise of AI. Our collaboration with data scientists has yielded some very promising results, yet also brought some challenges and dilemmas to the fore, which we share in this article.

The unique features of EU evaluations EU evaluations usually require collecting data in all Member States of the EU (and beyond, depending on the evaluation object), which means covering 27 different national contexts and nearly as many different languages. Also, while EU evaluations are usually extremely data rich (think, for example, of large databases compiling details on all projects for EU programmes like Horizon or Erasmus+), there are significant data comparability and quality issues. Evaluations of EU policies, programmes, and legislation also always require a mixed-method, multi-source approach. Evaluating EU policies, programmes, and legislation thus comes with some rather unique challenges that, at least in part, AI can help tackle successfully. Also, EU evaluations often address pressing social, economic, and environmental challenges, and their timely completion is crucial to serve the public interest. Incorporating AI can help meet public interest more efficiently.

Applying AI in evaluations Although AI is not a new phenomenon, it gained significant public attention with the advent of technologies like ChatGPT. The term “machine learning” was coined as early as in 1959 by Arthur Samuel, a computer gaming and AI pioneer at IBM, and the first natural language processing (NLP) techniques date back to the late 1950s as well.

At ICF, we have been doing evaluations and research for many years and have been using artificial intelligence technologies in our work since the early 2000s. Starting with text mining software that involved some degree of machine learning, and progressing to the use of sentiment analysis to collect and analyse data in evaluations by the mid-2000s, we know the strengths and weaknesses of these technologies—and how to employ them advantageously.

The most recent AI technologies offer a raft of new and exciting possibilities for conducting evaluations, as not only can AI process vast amounts of data within a very short time period; it can also be used to support data collection, assist in more complex triangulation and analysis, and even help with dissemination. On the other hand, its use also raises some new ethical dilemmas and comes with new risks.

From everyday to transformative AI solutions To date, at ICF we have mostly been applying AI tools and technologies in evaluations that we would categorise as “everyday AI use” to support, in particular, certain types of data collection and analysis, including surveys, interview write-ups, social media analysis, etc, using off-the-shelf large language models (LLM). These yield important benefits in terms of analysing large volumes of data and delivering quick insights that would not have been achievable otherwise, within the available amount of time and resources.

Increasingly we have also been rolling out more “transformative AI uses” which offer important advantages as they can be trained with vast factual information on the wider context, so that LLM understands the evaluation object better. This deeper understanding of the evaluation object is achieved through the use of tailormade AI software, as opposed to the more generic, off-the-shelf models used in everyday applications. Tailormade AI systems are specifically trained and fine-tuned to address particular evaluation questions, thereby enhancing their relevance and accuracy. By tweaking the algorithms to suit the unique context and objectives of each evaluation, these bespoke AI solutions can offer more precise and insightful analyses compared to their off-the-shelf counterparts. This helps the model focus on retrieving information that is highly relevant, avoid so-called “hallucinations,” and provide much better and more complete responses to queries. This process has allowed us, for example, to obtain a highly reliable analysis per evaluation question of hundreds of programme reports. It has also enabled us to formulate specific follow-up queries to obtain more in-depth information on trends and developments that seemed of interest from the initially generated analysis.

AI technologies offer tremendous opportunities as not only can they support data collection and initial analysis; if trained well, these tools can also help with more sophisticated analytical techniques such as trend analysis, predictive modelling and foresight, and real-time evaluation. In the U.S. we have even been developing—but not as part of an evaluation— a chatbot able to interact with users and answer queries on HIV by retrieving documentation relevant to their query in a fully private and anonymous way. Possibly, such technology could in the future also be considered for surveying people about sensitive subjects.

Ethical considerations and risk mitigation strategies Using AI, whether for professional or personal purposes, is accompanied by well-documented risks, including data privacy and security breaches, and they are also extremely relevant in evaluations.

Based on our experience, here are some risks of using AI in evaluations and our strategies for mitigating them.

1. Ensuring human oversight to mitigate hallucination risks One of the most pressing issues is the risk of unbalanced and unfair judgements in evaluations, and to analyses which do not reflect contextual subtleties, e.g., in terms of political and cultural understanding. Another related risk to avoid is AI producing inaccurate findings (also called ‘hallucinations’).

It is crucial to have deep technical understanding of AI capabilities and limitations, to make sure that the data used to train AI systems is representative, and monitored / audited continuously if used over a longer time period. It is equally important to ensure human oversight and involvement in all stages of data collection, analysis, triangulation, and synthesis. While the efficiency gains of AI are significant, it does not have the subtle understanding and context that only humans can provide. For example, analyses produced by AI should be double-checked and where possible human test analyses should be undertaken to compare results.

A collaborative approach between evaluators, data scientists, policy specialists, and ethical experts throughout the evaluation cycle works best to mitigate hallucination risks.

2. Establishing a robust governance framework for data privacy and security Any evaluation that includes stakeholder consultation, whether using AI or not, should make sure that all those interviewed, surveyed, participating in workshops, etc. are providing informed consent in relation to their participation. However, AI has brought to light a few new challenges, such as whether consent is needed from social media users, when using AI for sentiment analysis on platforms like X, Facebook, or LinkedIn, or whether evaluation stakeholders should be made aware that the information they provide will be analysed by an AI tool.

At ICF, we rely on responsible use principles that set out privacy information notices and consent forms to ensure that data used comply with licence terms of social media/web platforms or intellectual property. This governance framework is overseen by an Internal AI Review Board within the organisation that reviews and approves the use of AI in evaluation projects (or parts of the projects) that pose a high risk (e.g., AI chatbots that interact with vulnerable groups).

3. Fostering trust through transparency and accountability In evaluations, it is essential that the evaluator can explain the link between the judgement and the evidence base, but this can be a challenge with off-the-shelf tools that are designed for everyday AI use.

This is why the choice of AI tool is very important, as there are models that offer interfaces that allow tracing the interpretive process. Increasingly, however, we prefer working with our own fine-tuned models, so that our data scientists, when developing AI solutions, make sure that all algorithms and analytical steps are well documented and explained also to the non-expert reader.

An ethical approach to AI In summary, our comprehensive approach integrates ethical considerations, rigorous technical oversight, and a strong commitment to transparency. By fostering continuous learning and improvement, we ensure that our AI systems enhance human judgment while maintaining ethical integrity and inclusivity.

File Under Artificial intelligence Data and analytics EU European and UK policy Regulatory support and public policy Your mission, modernized.

Subscribe for insights, research, and more on topics like AI-powered government, unlocking the full potential of your data, improving core business processes, and accelerating mission impact.

Related insights Article Making data AI-ready: How federal agencies must rethink integration E863CFDC-E4F9-44F8-A33B-EC91C168E6EA Created with sketchtool. Article How to establish a load management program in 8 steps E863CFDC-E4F9-44F8-A33B-EC91C168E6EA Created with sketchtool. Webinar Modernizing data systems and harnessing AI in the public health system E863CFDC-E4F9-44F8-A33B-EC91C168E6EA Created with sketchtool. Our client stories Discover how we help clients achieve success. Learn more Insights in your inbox Receive perspectives on the industries and issues that matter. Subscribe Industries and capabilities Energy and utilities Federal health Disaster management Transportation Environmental services Climate resilience Aviation U.S. federal Social programs Digital modernization Artificial intelligence Data and analytics Experience and design Cloud Cybersecurity Program implementation Strategy and innovation Change management Policy and regulatory Grants management Strategic communications ICF Next Digital agency Partnerships Partnership ecosystem ICF suppliers Client stories Insights All topics Marketing insights By ICF Next About ICF News Leadership History Corporate Citizenship Ethics and compliance Data privacy Contracts Locations ICF Europe ICF UK Careers Investors Contact Terms of Use Privacy Statement Cookie Policy Do Not Sell or Share My Personal Information Limit the Use of My Sensitive Personal Information Accessibility Statement Sitemap Copyright 2026, ICF International Inc.
All Rights Reserved

智能索引记录