Higher Education Research and Development (HERD) Survey 2023

Sampling error . Because the FY 2023 survey was distributed to all institutions in the universe, there was no sampling error.

Coverage error . Coverage error of large research institutions is minimal because of comprehensive lists. These institutions are easily identified using the NCSES Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions. However, institutions with smaller amounts of R&D expenditures have been more difficult to identify because they often do not receive federal funding for science and engineering (S&E) R&D.

Non response error . Forty universities and colleges did not respond in FY 2023, out of a total of 914 eligible institutions, for a nonresponse rate of 4.4%. Table A-4 displays a detailed breakdown of response rates by survey population and highest degree granted.

Nonresponse rates were less than 5.0% for all but three questions. Question 6, R&D expenditures by type of R&D (basic research, applied research, and experimental development); Question 15, R&D personnel headcount; and Question 16, R&D full-time equivalents (FTEs) had nonresponse rates of 6.2%, 10.1%, and 22.8%, respectively. Table A-5 displays a breakdown of response rates for each question in each of the two surveys. See section "Imputation" for mitigation of item nonresponses. Table A-6 through table A-20 present imputed amounts for each applicable survey variable.

Measurement error . The most likely source of measurement error is institutional records containing categories different from those on the survey. For example, institutions were asked to report all R&D expenditures by field. The NCSES-designed fields do not always translate to an institution's departmental structure, and adjustments must be made by the institution to complete the survey. Fields were revised for the FY 2016 survey to better reflect the R&D currently being conducted at universities and colleges and make HERD Survey fields more consistent with those used by other NCSES surveys as well as with the National Center for Education Statistics Classification of Instructional Programs (CIP) codes. Details of this change are included in the methodology report and technical notes for the FY 2016 survey. Minor revisions were also made in FY 2020.

Another source of error is the survey's category of institutionally funded research. The survey requests that institutions report discretionary internal funds used for research. NCSES discovered through debriefings conducted at the conclusion of the FY 2010 survey that there were varying definitions of what should be included on the HERD Survey as institutionally funded research. Some institutions included all expenditures from separate accounts designated for research; others included only internal R&D projects that are competitively awarded and have detailed budgets. A workshop was held in summer 2012 to discuss these differences in definitional interpretation. Based on the findings from the workshop, the FY 2012 survey was modified to clarify that all expenditures designated for research can be included in this category. This includes expenditures for organized research and expenditures of other funds designated for research but not categorized as organized research. A checklist question (Question 1.1) was also added to encourage the inclusion of all eligible expenditures and to determine the full extent of the variations in reporting across institutions. This question has been on the survey since FY 2012. An analysis of Question 1.1 responses from FY 2023 indicated that some institutions still could not report institutionally funded research that was not organized research (6.7% could not report startup packages, bridge funding, or seed funding and 7.0% were unable to identify other departmental funds designated for research). Therefore, survey totals are missing expenditures for R&D that come from multipurpose accounts, and as such, they represent an undercount of the total amount of internal discretionary funding that institutions make available to conduct R&D.

The reporting of unrecovered indirect costs is another known source of error. The survey requests that the total amount of indirect costs associated with a research grant or contract be calculated and reported, including costs that were not reimbursed by the external funding source. The unrecovered indirect cost is calculated by multiplying the institution's negotiated indirect cost rate by the corresponding base and then subtracting the actual indirect cost recovery, preferably on a project-by-project basis. In FY 2023, 5.0% of respondents reported unrecovered indirect costs as unavailable. Respondents who were unable to provide values were asked to provide information on their nonresponse. Based on the collected information, survey guidance is revised to encourage response.

The reporting of expenditures from projects that are not R&D is another possible source of error. The R&D in the HERD Survey excludes public service grants, outreach programs, curriculum development, or training grants that do not support work on research projects. As part of a federal government effort to reduce administrative burdens associated with research grants and contracts, agencies began adopting Research Terms and Conditions (RTC) to be consistent with Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards issued by the U.S. Office of Management and Budget (OMB) in the Federal Register [80 FR 61849, October 14, 2015]. In these RTCs, agencies employed a broader definition of research, resulting in many institutions reporting projects as research that do not match HERD Survey definitions. Additionally, in recent years R&D expenditure data has been used more frequently in university and college benchmarking, which may encourage institutions to employ broader definitions of research than those provided on the survey.

It should also be noted that because institutions were asked to include funds passed through to higher education institutions as well as subrecipient funding from higher education institutions, there is double counting included in national and group totals. For example, Institution A's survey included the $2 million passed through to Institution B, and Institution B's survey also included the $2 million in subrecipient funding that it received from Institution A. Overall, institutions reported $5.3 billion in expenditures from subrecipient funding received from other universities in FY 2023 and $5.2 billion in funds passed through to higher education subrecipients in FY 2023. Adjustments are made to R&D totals presented in the NCSES National Patterns of R&D Resources publications.