Desk Research and Interviews

The project began in August 2023 with desk research into available literature on government AI procurement, public-private partnerships, and responsible AI practices. This phase included engaging with organizations working in AI procurement to gather their insights on developing indicators for responsible AI procurement, as well as advice on index structure and format.

We gathered insights from civil society experts in AI procurement including the Center for Democracy & Technology (CDT), IEEE SA Procurement Working Group, and the International Center for Not-for-Profit Law (ICNL). Based on these insights, we developed a preliminary list of indicators for our index, which underwent continuous refinement throughout the project. This iterative process focused on assessing each indicator’s viability and feasibility to ensure they could generate meaningful data. Ultimately, we finalized 22 indicators (see the full list of indicators here).

Data Collection, Cleaning, and Analysis

Structured Data

For structured data, we utilized USAspending.gov, an open data platform that provides federal procurement data across U.S. government agencies. We applied the keywords “artificial intelligence” and “machine learning” to capture procurement records that mentioned these terms in the award/transaction descriptions. The search was limited to fiscal years 2023 and 2024 to align with the researcher’s fellowship timeline at UVA and to capture recent data relevant to Executive Order Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (2023), which emphasizes the acquisition of trustworthy AI technologies and underscores the importance of transparency and accountability in AI procurement within public-private partnerships.

To focus on relevant contracts, we filtered our search to include only “Contract” and “IDV Contract” award types. After narrowing the results by agency, we downloaded the data in CSV format for further processing.

Screen Recording 2024-11-07 at 1.48.03 PM (2).mov

Our data cleaning process used a Python script to extract only the necessary columns from the dataset, including contract details such as:

• Basic Contract Info: ‘contract_transaction_unique_key,’ ‘awarding_agency_name,’ ‘recipient_name,’ ‘transaction_description,’ ‘award_type’
• Financial and Timeline Information: ‘current_total_value_of_award,’ ‘period_of_performance_start_date,’ ‘period_of_performance_potential_end_date’
• Compliance and Standards: ‘clinger_cohen_act_planning_code,’ ‘labor_standards,’ ‘performance_based_service_acquisition’
• Diversity Indicators: ‘minority_owned_business,’ ‘woman_owned_business,’ and specific diversity attributes.
• Competitiveness: ‘number_of_offers_received,’ ‘extent_competed,’ ‘solicitation_procedures,’ and ‘fair_opportunity_limited_sources,’ to assess the level of competition and exclusivity in AI contract procurement
• Technology Type (NAICS Codes): ‘naics_description’ and ‘product_or_service_code_description,’ to identify the specific type of technology and services.

With the cleaned data, we developed additional Python scripts to answer each project indicator. These scripts facilitated the analysis of procurement practices, compliance with standards, and transparency across agencies, providing critical insights for each indicator. See the scripts here.

Unstructured Data

AI.gov Use Case Inventory (2023 Data)