Marginal Returns and Levels of Research Grant Support among Scientists Supported by the National Institutes of Health
=====================================================================================================================

* Michael Lauer
* Deepshikha Roychowdhury
* Katie Patel
* Rachael Walsh
* Katrina Pearson

## Abstract

The current era of worsening hypercompetition in biomedical research has drawn attention to the possibility of decreasing marginal returns from research funding. Recent work has described decreasing marginal returns as a function of annual dollars granted to individual scientists. However, different fields of research incur varying cost structures. Therefore, we developed a Grant Support Index (GSI) that focuses on grant activity code, as opposed to field of study or cost. In a cohort of over 71,000 unique scientists funded by NIH between 1996 and 2014 we analyzed the association of grant support (as measured by annual GSI) with 3 bibliometric outcomes, maximum Relative Citation Ratio (which arguably reflects a scientist’s most influential work), median Relative Citation Ratio, and annual weighted Relative Citation Ratio (which is more dependent on publication counts). We found that for all 3 measures marginal returns decline as annual GSI increases. Thus, we confirm prior findings of decreasing marginal returns with higher levels of research funding support.

## Introduction

In a recent essay on the “systemic flaws” besetting biomedical research, Alberts et al called on funding agencies to “be sensitive to the total numbers of dollars granted to individual laboratories, recognizing—although different research activities have different costs — at some point, returns per dollar diminish.”1 A growing body of literature supports the claim that “at some point, returns per dollar diminish,” or more precisely, that as funding levels or personnel support2 for any given laboratory increases, *marginal* (that is incremental) returns decline.3 However, as Alberts et al recognized, it may be problematic to focus solely on dollars precisely because different types of research entail varying cost structures. We therefore developed a “Grant Support Index” (GSI) that focuses on grant activity code, as opposed to field of study or cost, and using this measure attempted to confirm other reports of decreasing marginal returns with increased research funding.3-6,7,8

## Methods

### Study sample and description of grant support

We identified NIH Research Project Grants (“RPGs”) that use currently available grant activity codes and were funded between 1996 and 2014 approximately 2,500 grantee organizations, including domestic and private institutions of higher education, research institutions and hospitals. The data were primarily extracted from the Information for Management, Planning, Analysis, and Coordination (IMPAC II) database for grants and Research and Development contracts, and public interface is known as eRA Commons. Over the period analyzed, 71,936 unique scientists served as Principal Investigators (PIs) or Multiple Principal Investigators (MPIs) on NIH RPG grants. For each grant-PI/MPI– Fiscal Year combination, we assigned GSI points using a scheme that NIH staff derived without knowledge of research outcomes. A trans-NIH Extramural Activities Working Group (EAWG) subcommittee, under the leadership of Jon Lorsch (Director, NIGMS), held several dedicated meetings that involved dozens of NIH program and grants management staff to develop the GSI (originally designed the “Research Commitment Index” or RCI). We began with assigning a standard R01 grant 7 points, and then, based on that benchmark, assigned more or fewer points to other mechanisms. The agreed-upon point scheme is shown in Table 1; only after the scheme was agreed to did we begin analyzing outcomes.

View this table:
[Table 1:](http://biorxiv.org/content/early/2017/05/29/142554/T1)

Table 1: 
Grant Support Index (GSI) point values for NIH Research Project Grants. These were values assigned in the summer of 2016 and reported out in earlier in 2017.

We gathered data on Howard Hughes Medical Institution (HHMI) funding support from the HHMI web-site and manually matched HHMI awardees to NIH awardees by name and institution.

### Outcomes

The “Relative Citation Ratio” (RCR) is a bibliometric measure developed by the NIH Office of Portfolio Analysis that compares citation influence of articles while accounting for field of study and time of publication.9 Briefly, RCR compares actual and expected citation rates by assembling and inspecting a “co-citation network,” papers cited alongside the paper of interest. The RCR has been shown to correlate well with expert opinion.9 We used the NIH Scientific Publication Information Retrieval and Evaluation Systems10 (“SPIRES”) to link NIH PI’s to publications; we obtained RCR values for each publication from the NIH Office of Portfolio Analysis.

We limited our analyses to papers published during the study period of 1996-2014 (N=1,027,462). We focused on 3 outcomes:

*   Maximum RCR: the highest RCR for NIH RPG-supported paper linked to a scientist’s RPG grants; in communications with the authors, some have endorsed the importance of this measure as a reflection of scientist’s most influential work. Other recent efforts have estimated individual scientists’ impacts by considering their most highly cited papers;11

*   Median RCR; and

*   Annual weighted RCR: we summed the RCR values for all papers linked to a scientist’s grants. This measure is most sensitive to publication counts, which vary by discipline.12In cases where more than one scientist was linked to a paper (e.g. papers that acknowledge support from multiple grants, from multi-PI grants, and from multicomponent grants), we divided and distributed the RCR values among all involved PI’s to avoid double-counting papers.

### Analyses

Our approach mirrors those of prior reports3,4 that specifically addressed the question of how research impact scales with funding. To examine the association between grant support and research output, we examined the association of annual GSI with the 3 RCR outcomes. We focused on annual GSI as the independent variable because all NIH budgeting and grant decisions are made on a year-to-year basis. As annual GSI and RCR values followed skewed distributions, we used natural log-transformed values throughout.

With only one independent variable, GSI, we used a Cobb-Douglas function3 with the form *ln(Q) ~ ln(α)* + *β(ln(K)),* where Q is output as assessed by maximum RCR, median RCR, or annual weighted RCR, *K* is input (in this case annual GSI), and β is a slope coefficient reflecting marginal returns that links the natural logarithm of input (annual GSI) with the natural logarithm of output. A value of β > 1 corresponds to increasing returns (e.g. a 10% increase in input is associated with > 10% increase in output); a value of β = 1 corresponds to constant returns (e.g. a 10% increase in input corresponds to a 10% increase in output); a value of b between 0 and 1 corresponds to decreasing returns (e.g. a 10% increase in input corresponds to < 10% increase in output); and a value of β < 0 corresponds to negative returns (e.g. as input increases, output decreases). We allowed for non-linear associations between *Q* and annual GSI by testing natural cubic spline terms with 3, 4, or 5 knots. For all models tested, all terms (linear and non-linear spline) were highly significant (with all *P* < 2 * 10-16). We used Akaike Information Criterions (AIC) to compare models.

Because of community concerns regarding the use of natural log-log transformations, we also derived loess regression plots of a random subsample of 20,000 scientists to show associations with annual GSI of maximum RCR and annual RCR.

All analyses were performed in R ([https://www.r-project.org/](https://www.r-project.org/)).

## Results

### Sample

We identified 71,936 unique scientists who served as Principal Investigators (PI’s) on NIH Research Project Grants (including sub-projects) between 1996 and 2014. We assigned “Grant Support Index” points according to the scheme shown in Table 1; points varied by mechanism and by whether there were more than one PI for grants in which multi-PI applications were allowed. A GSI value of 7 corresponds to 1 single-PI R01 while a GSI of 21 corresponds to 3 single-PI ROls.

Table 2 shows characteristics and bibliometric outcomes of PI’s according to four categories of Grant Support Index (GSI) that correspond to ≤ 1 R01 equivalent, > 1 to ≤ 2 R01 equivalents, > 2 to ≤ 3 R01 equivalents, and > 3 R01 equivalents. Among all PI’s, 46,365 (64%) had an annual GSI of 7 or less, while 21,182 (29%) had an annual GSI between 7 and 14. Only 6% of PI’s had an annual GSI of 14 or higher.

Figures S1 – S6 show box plots of the distributions of maximum RCR, median RCR, and annual weighted RCR. These box plots demonstrate the skewed distributions.

### Grant Support and Productivity

Figure 1A shows the association of maximum RCR with annual GSI (with, as previously noted, both axes natural log-transformed to reflect the skewed distributions and the underlying Cobb-Douglas function). The rug plots along the axis demonstrate the log-linear distributions of annual GSI and maximum RCR. Increasing levels of annual GSI were associated with increasing levels of maximum RCR, but the *rate* of increase decreases with increasing annual GSI. Figure 2A illustrates this point by showing the instantaneous slope (first derivative, or instantaneous value of β) across values of annual GSI; this curve effectively shows marginal returns. Marginal returns fall as annual GSI increases; once GSI exceeds 21, marginal returns decrease to the point that any increase in input was associated with a proportionally lower increase in output.

View this table:
[Table 2:](http://biorxiv.org/content/early/2017/05/29/142554/T2)

Table 2: 
Characteristics and outputs of 71,936 NIH-supported scientists according to their annual number of RPG Grant Support Index points.

![Figure 1A:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F1.medium.gif)

[Figure 1A:](http://biorxiv.org/content/early/2017/05/29/142554/F1)

Figure 1A: Maximum Relative Citation Ratios by level of Grant Support.
Association, as assessed by spline regression, of the maximum Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). Due to skewed distributions, both RCR and GSI values are natural log-transformed; the numbers inside the axes represent the raw, non-transformed values. An annual GSI value of 7 corresponds to ~ 1 R01 grant, while annual GSI values of 14 and 21 correspond to 2 and 3 R01 grants. The rugplots by the axes illustrate the log-normal distributions. Note the decreasing slope of the regression curve as annual GSI increases.

![Figure 2A:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F2.medium.gif)

[Figure 2A:](http://biorxiv.org/content/early/2017/05/29/142554/F2)

Figure 2A: Marginal returns of maximum RCR by level of Grant Support.
Association of the marginal returns of maximum Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). The curve shows the slope of the curve shown in Figure 1A and corresponds to the first derivative *d(ln(Q))/d(ln(K))* of the Cobb-Douglas function *ln(Q) ~ ln(α)* + *β(ln(K))*, where *Q* is output as assessed by maximum RCR and p is a coefficient linking the natural logarithm of input (K or *annual GSI)* with the natural logarithm of output, here maximum RCR. For convenience, vertical dashed lines are shown corresponding to GSI values of 7 (equivalent to 1 single-PI R01) and 21 (equivalent to 3 single-PI R01s). Marginal productivity falls as annual Grant Support Index increases.

We tested a non-transformed model using spline regression and found substantially worse fit (by AIC) than with a natural log-log model. Figure 1B is based on loess regression of a randomly chosen subsample of 20,000 scientists. Figure 2B shows that the instantaneous slope (or first derivative) of the loess regression curve becomes shallower with increasing annual GSI, reflecting a decline in marginal returns.

![Figure 1B:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F3.medium.gif)

[Figure 1B:](http://biorxiv.org/content/early/2017/05/29/142554/F3)

Figure 1B: Maximum Relative Citation Ratios and Grant Support – loess regression of random subsample
Association, as assessed by loess regression in a random subsample of 20,000, of the maximum Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). Here values are not log-transformed. Loess fit values and changes are shown along the curve and in the upper left-hand corner to illustrate decreasing incremental returns with increasing levels of grant support. The decreasing instantaneous slope (by first derivative) is shown in Figure 2B.

![Figure 2B:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F4.medium.gif)

[Figure 2B:](http://biorxiv.org/content/early/2017/05/29/142554/F4)

Figure 2B: Marginal returns of maximum RCR by level of Grant Support – loess regression of random subsample.
Association of the marginal returns of maximum Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). The curve shows the slope of the loess curve shown in Figure 1B and corresponds to the first derivative *dQ/dK* of the where *Q* is output as assessed by annual weighted RCR and *K* is input, here maximum RCR. Here *Q* and *K* are not log-transformed. Marginal returns fall as annual Grant Support Index increases.

Figure 3 and 4 are corresponding plots of the association of median RCR with annual GSI. As seen in Figure 4, marginal returns fall as annual GSI increases; negative returns are seen at annual GSI levels exceeding 21. Between annual GSI levels of 10 and 21, a proportional increase of input is associated with a lesser increase in output; beyond an annual GSI of 21, absolute output falls as annual GSI increases

![Figure 3:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F5.medium.gif)

[Figure 3:](http://biorxiv.org/content/early/2017/05/29/142554/F5)

Figure 3: Median Relative Citation Ratio by Level of Grant Support.
Association, as assessed by spline regression, of the median Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). Due to skewed distributions, both RCR and GSI values are natural log-transformed; the numbers inside the axes represent the raw, non-transformed values. An annual GSI value of 7 corresponds to ~ 1 R01 grant, while annual GSI values of 14 and 21 correspond to 2 and 3 R01 grants. The rugplots by the axes illustrate the log-normal distributions. Note the decreasing slope of the regression curve as annual GSI increases.

![Figure 4:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F6.medium.gif)

[Figure 4:](http://biorxiv.org/content/early/2017/05/29/142554/F6)

Figure 4: Marginal returns of median RCR by level of Grant Support.
Association of the marginal returns of median Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). The curve shows the slope of the curve shown in Figure 3 and corresponds to the first derivative *d(ln(Q))/d(ln(K))* of the Cobb-Douglas function *ln(Q) ~ ln(α)* + *β(ln(K)),* where *Q* is output as assessed by median RCR and p is a coefficient linking the natural logarithm of input (K or *annual GSI)* with the natural logarithm of output, here median RCR. For convenience, vertical dashed lines are shown corresponding to GSI values of 7 (equivalent to 1 single-PI R01) and 21 (equivalent to 3 single–PI R01s). Marginal productivity falls as annual Grant Support Index increases. Values below zero, seen above annual GSI levels of 21, indicate negative returns.

Figures 5A, 5B, and 6A and 6B are corresponding spline and loess regression plots of the association of annual weighted RCR with annual GSI (along with first derivatives). Marginal returns fall with increasing annual GSI.

### Consideration of “elite” scientists

To assess these association among “elite” scientists, particularly those successful in securing additional funding, we stratified our sample by receipt of Howard Hughes Medical Institute (HHMI) funding between 1996 and 2014. Table 3 shows characteristics and outcomes as listed in Table 2; as might be expected, HHMI scientists received more NIH support for a longer time. They also published more papers that had higher RCR values. Figure 7 shows the associations of annual GSI with maximum (panel A), median (panel B), and annual weighted (panel C) RCR; decreasing marginal returns with higher annual GSI are evident for both HHMI and non-HHMI scientists.

View this table:
[Table 3:](http://biorxiv.org/content/early/2017/05/29/142554/T3)

Table 3: 
Characteristics and outputs of 71,936 NIH-supported scientists according to whether they received Howard Hughes Medical Institute (HHMI) funding between 1996 and 2014.

![Figure 5A:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F7.medium.gif)

[Figure 5A:](http://biorxiv.org/content/early/2017/05/29/142554/F7)

Figure 5A: Annual Weighted Relative Citation Ratio by Level of Grant Support.
Association, as assessed by spline regression, of the annual weighted Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). Due to skewed distributions, both RCR and GSI values are natural log-transformed; the numbers inside the axes represent the raw, non-transformed values. An annual GSI value of 7 corresponds to ~ 1 R01 grant, while annual GSI values of 14 and 21 correspond to 2 and 3 R01 grants. The rug-plots by the axes illustrate the log-normal distributions. Note the decreasing slope of the regression curve as annual GSI increases.

![Figure 5B:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F8.medium.gif)

[Figure 5B:](http://biorxiv.org/content/early/2017/05/29/142554/F8)

Figure 5B: Annual Weighted Relative Citation Ratios and Grant Support – loess regression of random subsample
Association, as assessed by loess regression in a random subsample of 20,000, of the annual weighted Relative Citation Ratio (wRCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). Here values are not log-transformed. Loess fit values and changes are shown along the curve and in the upper left-hand corner to illustrate decreasing incremental returns with increasing levels of grant support. The decreasing instantaneous slope (by first derivative) is shown in Figure 6B.

![Figure 6A:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F9.medium.gif)

[Figure 6A:](http://biorxiv.org/content/early/2017/05/29/142554/F9)

Figure 6A: Marginal returns of annual weighted RCR by level of Grant Support.
Association of the marginal returns of annual weighted Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). The curve shows the slope of the curve shown in Figure 5A and corresponds to the first derivative *d(ln(Q))/d(ln(K))* of the Cobb-Douglas function *ln(Q) ~ ln(α)* + *β(ln(k)),* where *Q* is output as assessed by annual weighted RCR and p is a coefficient linking the natural logarithm of input (K or *annual GSI)* with the natural logarithm of output, here annual weighted RCR. For convenience, vertical dashed lines are shown corresponding to GSI values of 7 (equivalent to 1 single-PI R01) and 21 (equivalent to 3 single-PI R01s). Marginal productivity falls as annual Grant Support Index increases.

![Figure 6B:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F10.medium.gif)

[Figure 6B:](http://biorxiv.org/content/early/2017/05/29/142554/F10)

Figure 6B: Marginal returns of annual weighted RCR by level of Grant Support – loess regression of random subsample.
Association of the marginal returns of annual weighted Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). The curve shows the slope of the loess curve shown in Figure 5B and corresponds to the first derivative *dQ/dK* of the where *Q* is output as assessed by annual weighted RCR and *K* is input, here annual weighted RCR. Here *Q* and *K* are not log-transformed. Marginal returns fall as annual Grant Support Index increases.

![Figure 7:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F11.medium.gif)

[Figure 7:](http://biorxiv.org/content/early/2017/05/29/142554/F11)

Figure 7: 
Associations of maximum (A), median (B), and annual weighted (C) Relative Citation Ratio according to annual GSI and HHMI funding status.

As another assessment of the impact of elite scientists, we identified the “top 50” who received at least 4 years of NIH funding by the 3 different RCR measures. Among the top 50 by maximum RCR, only 3 (6%) were scientists with annual GSI levels exceeding 21. Among the top 50 by median RCR, none had annual GSI levels exceeding 21. Among the top 50 by annual weighted RCR, only 7 (10%) had annual GSI levels exceeding 21.

### Secondary Analyses

We performed several other secondary analyses, including focusing on human–and non-human grants as well as grants from specific NIH Institutes (among them the National Cancer Institute, the National Institute of Allergy and Infectious Disease, the National Institute of General Medical Sciences, and the National Institute of Neurological Diseases and Stroke). In all cases, we saw similar patterns of decreasing marginal returns with increasing annual GSI.

Because of the complexity of multi-component awards and cooperative agreements, and because of misclassification errors in assigning scientists to sub-projects of those awards, we performed a secondary analysis focusing on scientists’ support from single-component R awards only (in other words, excluding P and U awards and excluding all subprojects, N=65,574 scientists). Figure 8 shows similar patterns of falling marginal returns for maximum (panel A), median (panel B), and annual weighted (panel C) RCR as annual GSI increases.

![Figure 8:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F12.medium.gif)

[Figure 8:](http://biorxiv.org/content/early/2017/05/29/142554/F12)

Figure 8: Secondary analyses limited to grant support on single-component “R” grants.
Association of the marginal returns of maximum (panel A), median (panel B), and annual weighted (panel C) Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual Grant Support Index (GSI). These analyses were based on R grants only. The curves shows the slope of the curve corresponding to the first derivative of the Cobb-Douglas function *ln(Q) ~ ln(α)* + *β(ln(annual GSI)),* where Q is output as assessed by maximum, median, or annual weighted RCR and b is a coefficient linking the natural logarithm of input (annual GSI) with the natural logarithm of output. For convenience, vertical dashed lines are shown corresponding to GSI values of 7 (equivalent to 1 single-PI R01) and 21 (equivalent to 3 single–PI R01s).

Finally, we looked at marginal returns as a function of dollars of funding (as done in previously published reports) among all scientists and among all RPG grants. Figure 9 shows rising and falling marginal returns for maximum (panel A), median (panel B), and annual weighted (panel C), with peaks at ~$400,000 per year.

![Figure 9:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F13.medium.gif)

[Figure 9:](http://biorxiv.org/content/early/2017/05/29/142554/F13)

Figure 9: Marginal returns by levels of financial support.
Association of the marginal returns of maximum (panel A), median (panel B), and annual weighted (panel C) Relative Citation Ratio (RCR) among all papers linked to a scientists’ grants with annual funding in BRDPI-adjusted dollars. The curves shows the slope of the curve corresponding to the first derivative of the Cobb-Douglas function *ln(Q) ~ ln(α)* + *β(ln(annual funding)),* where Q is output as assessed by maximum, median, or annual weighted RCR and b is a coefficient linking the natural logarithm of input (annual funding) with the natural logarithm of output. For convenience, vertical dashed lines are shown corresponding to funding values of $250,000, $1 million, and $2 million.

## Discussion

Despite using a different approach to measuring levels of laboratory support, we confirm the findings of others who have claimed that research output does not necessarily scale to input. Previous reports have primarily focused on financial or personnel measures2–8, and have described decreasing marginal returns – that is output that does not scale proportionally to input. One group used a regression discontinuity design, and consistent with our results, found that receipt of an R01 grant (usually on top of existing funding) led to a modest 7% increase in productivity.13 In that study, the increment in productivity was greater in younger scientists (age < 45 years), perhaps because they had lesser baseline funding.

Because different fields of science entail different cost structures, we developed a “Research Commitment Index” or “Grant Support Index” to describe research support that is relatively independent of field or inherent costing challenges. Even with this measure, we find that by 3 measures – Relative Citation Ratio, median Relative Ratio, and annual weighted Relative Citation Ratio9 – marginal returns decrease with higher levels of grant support. Thus, we confirm with data Alberts’ et al contention that “at some point returns per dollar diminish.”1

There are important limitations in our analyses. We could not measure additional non–NIH resources, such as “hard money” that might come with endowed chairs or university–supported tenured faculty positions. Nonetheless, we found similar patterns of decreasing marginal returns among HHMI–supported faculty. We could not be particularly granular across fields of study, though the Relative Citation Ratio seeks to account for differences across disciplines. We did not have data on the make–up of scientific teams supporting each scientists; more recent work from our group14 suggests that prospective assessments of team size and make–up may be feasible. Nonetheless, prior work has found evidence of decreasing marginal returns with larger team size.2 NIH grant mechanisms and funding structures are complex, and, as an example, we couldn’t tease the effects of infrastructure and training support. We only focused on citation metrics, and will leave to future work to assess the associations of NIH grant support on training, development of technologies, translation, and sharing of resources, these and other measures have been advocated as legitimate measures of research success.15 The GSI point system was developed by NIH staff using a semi–quantitative approach, and of note was written down before any of these outcomes analyses were performed. We were aware of questions about how best to classify work on parent and subprojects for multicomponent grants; nonetheless, secondary analyses focused on singlecomponent R grants yielded similar findings (Figure 8). Since we publicized a proposed policy on grant limits to enable funding more new investigators, we have heard much feedback and are considering several changes which may affect the findings presented here. We present the findings here based on the older point scale since that point scale was agreed upon before undertaking any outcomes analyses.

Despite these limitations, we found that we could reproduce others’ findings of decreasing marginal returns even by using a measure of research support distinct from money or personnel. Some have argued that the phenomena of decreasing marginal returns offers funders an opportunity to enhance the likely productivity of their research portfolio by seeking to fund as many scientists as possible.16 Furthermore, as others have argued3, science is often unpredictable11,17, and therefore funders increase the likelihood of supporting transformative discoveries by seeking ways to fund more scientists across the spectrum of their careers.11 Funders’ ability to support as many meritorious scientists as possible is arguably more important than ever given rapidly improving technologies and worsening hypercompetition.18

## Supplementary materials

All data and annotated R statistical code used to generate the Tables and Figures in this report are available at [Bioarxiv.org](http://Bioarxiv.org).

### Supplemental Figures

![Figure S1:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F14.medium.gif)

[Figure S1:](http://biorxiv.org/content/early/2017/05/29/142554/F14)

Figure S1: 
Boxplots of maximum RCR according to the annual GSI categories shown in Table 2.

![Figure S2:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F15.medium.gif)

[Figure S2:](http://biorxiv.org/content/early/2017/05/29/142554/F15)

Figure S2: 
Same data as in Figure S1, but natural log-transformed

![Figure S3:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F16.medium.gif)

[Figure S3:](http://biorxiv.org/content/early/2017/05/29/142554/F16)

Figure S3: 
Boxplots of median RCR according to the annual GSI categories shown in Table 2.

![Figure S4:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F17.medium.gif)

[Figure S4:](http://biorxiv.org/content/early/2017/05/29/142554/F17)

Figure S4: 
Same data as in Figure S3, but natural-log transformed.

![Figure S5:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F18.medium.gif)

[Figure S5:](http://biorxiv.org/content/early/2017/05/29/142554/F18)

Figure S5: 
Boxplots of annual weighted RCR according to the annual GSI categories shown in Table 2.

![Figure S6:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2017/05/29/142554/F19.medium.gif)

[Figure S6:](http://biorxiv.org/content/early/2017/05/29/142554/F19)

Figure S6: 
Same data as in Figure S5, but natural log-transformed

*   Received May 26, 2017.
*   Accepted May 29, 2017.


*   © 2017, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)

## References

1.  1.Alberts, B., Kirschner, M. W., Tilghman, S. & Varmus, H. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci U S A 111, 5773–5777 (2014).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTExLzE2LzU3NzMiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAxNy8wNS8yOS8xNDI1NTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  2.Cook, I., Grange, S. & Eyre-Walker, A. Research groups: How big should they be? PeerJ 3, e989 (2015).
    
    

3.  3.Mongeon, P., Brodeur, C., Beaudry, C. & Larivière, V. Concentration of research funding leads to decreasing marginal returns. Res. Eval. 25, 396–404 (2016).
    
    

4.  4.Fortin, J. M. & Currie, D. J. Big Science vs. Little Science: How Scientific Impact Scales with Funding. PLoS One 8, e65263 (2013).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0065263&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23840323&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 

5.  5.Berg, J. M. Science policy: Well-funded investigators should receive extra scrutiny. Nature 489, 203 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/489203a&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=22972279&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 

6.  6.Nag, S., Yang, H., Buccola, S. & Ervin, D. Productivity and financial support in academic bioscience. Appl. Econ. 45, 2817–2826 (2013).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1080/00036846.2012.676737&link_type=DOI) 

7.  7.Doyle, J. M. et al. Association of percentile ranking with citation impact and productivity in a large cohort of de novo NIMH-funded R01 grants. Mol. Psychiatry 20, 1030–6 (2015).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/mp.2015.71&link_type=DOI) 

8.  8.Lauer, M. S., Danthi, N. S., Kaltman, J. R. & Wu, C. O. Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute. Circ. Res. 117, CIRCRESAHA.115.306830-(2015).
    
    

9.  9.Hutchins, B. I., Yuan, X., Anderson, J. M. & Santangelo, G. M. Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level. PLoS Biol. 14, (2016).
    
    

10. 10.Boyack, K. W. & Jordan, P. Metrics associated with NIH funding: a high-level view. J. Am. Med. Informatics Assoc. 18, 423–431 (2011).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1136/amiajnl-2011-000213&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=21527408&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 

11. 11.Sinatra, R., Wang, D., Deville, P. & Song, C. Quantifying patterns of individual scientific impact. Science (80-.). 354, aaf5239–1 (2016).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjE2OiIzNTQvNjMxMi9hYWY1MjM5IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMTcvMDUvMjkvMTQyNTU0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. 12.Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. & Rafols, I. Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–31 (2015).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/520429a&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25903611&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 

13. 13.Jacob, B. A. & Lefgren, L. The impact of research grant funding on scientific productivity. J Pub Econ 95, 1168–1177 (2011).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.jpubeco.2011.05.005&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=21857758&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000294512000013&link_type=ISI) 

14. 14.Pool, L. R. et al. Size and characteristics of the biomedical research workforce associated with U.S. National Institutes of Health extramural grants. FASEB J. 30, 1023–1036 (2016).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1096/fj.14-264358&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=26625903&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 

15. 15.Ioannidis, J. P. & Khoury, M. J. Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA 312, 483–484 (2014).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1001/jama.2014.6932&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=24911291&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2017%2F05%2F29%2F142554.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000339808600013&link_type=ISI) 

16. 16.Lorsch, J. R. Maximizing the return on taxpayers’ investments in fundamental biomedical research. Mol. Biol. Cell 26, 1578–82 (2015).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTE6Im1vbGJpb2xjZWxsIjtzOjU6InJlc2lkIjtzOjk6IjI2LzkvMTU3OCI7czo0OiJhdG9tIjtzOjM3OiIvYmlvcnhpdi9lYXJseS8yMDE3LzA1LzI5LzE0MjU1NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

17. 17.Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven prediction in the science of science. Science (80-.). 355, 477–80 (2017).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEyOiIzNTUvNjMyNC80NzciO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAxNy8wNS8yOS8xNDI1NTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

18. 18.Kimble, J. et al. Strategies from UW-Madison for rescuing biomedical research in the US. Elife 4, (2015).