Neal D. Goldstein, PhD, MBI

Final thoughts on obtaining a K01 career development award

Last year I posted a three-part series on submitting a career development award (part 1, part 2, part 3). These three posts covered the process of submitting a proposal for an NIH K01 career development award with a focus on logistical and practice concerns not directly discussed in the NIH application instructions. Mostly my information was obtained through discussions with colleagues and first-hand experience of filling out every item in the ASSIST application. A lot has transpired since I submitted that application over a year ago, and this fourth and final post is to share some of my experiences since then.

When I submitted the grant the first time, I felt as if it just disappeared into the black hole of NIH. Aside from an automated acknowledgement email, I had no further communication from NIH. A quick email to the project officer (PO) was able to confirm that she saw my application and flagged it for her portfolio. After a short period of time, some information started appearing on ERA Commons (if you don't know what this is, you are probably reading the wrong article!). I could see the scheduled study section date, the roster, and the scientific review officer (SRO; an NIH scientist who oversees the study section). Based on this, I could learn about the expertise and backgrounds of the reviewers and ensure that my grant is being handled by reviewers with relevant expertise. As it turns out, and unbeknownst to me at the time, my institute (NIAID) had a special emphasis panel that focused exclusively on career development awards. This, as it turns out, is a very good thing. I had heard from several other folks including the PO that when this is not the case, occasionally K01s get evaluated alongside R01s and the reviewers may evaluate things unfairly (one comment I heard was a reviewer dinged a K01 because of lack of pilot data - the SRO promptly intervened).

Once the study section date came and went, an update was pretty quickly available on Commons: Not Discussed. A decision of "not discussed" means the application was read and evaluated by three reviewers, but not discussed at the study section meeting. This happens to about half of all applications for a given study section. The applicant will still receive the reviewer's comments, called pink sheets by the old-timers, and possibly SRO comments, but it is highly unlikely it will be funded. If the application were Discussed, it also means the application was read and evaluated by three reviewers but importantly it was also orally discussed at the study section meeting. A "discussed" application is also assigned what's known as an impact score. The impact score largely, but not wholly, determines which applications get funded, based a threshold (percentile or score) called the payline. Confusingly, this threshold may change year to year and is not consistent between NIH institutes and centers. Further, some NIH entities do not even publicly disclose what their payline is. For those outside the threshold, the application is not exactly unfundable, but it does enter a more ambiguous status (more on that later) and is less likely to receive an award.

Reflecting on this decision now a "not discussed" was disappointing, yes, but not devastating. I really just wanted to see if this project was possible or not: that is, could I ultimately get it funded. In other words, did the reviewers and SRO think that I would be a good candidate for an award at some point. I think of this as testing the waters of your ideas with NIH. A week or so after the study section met, the reviewers' comments were available on Commons. In the summary statement, each reviewer dissected the application's various sections (e.g., Candidate, Environment, Research, etc.) and highlighted proposal strengths and weaknesses; I also had additional SRO comments at the top. The SRO provided a high-level overview of the reviewer comments, and for my application, was encouraging a resubmission. I found the majority of the reviewer critiques to be thoughtful and insightful, and crucially, actionable. Buoyed by this, I decided to resubmit immediately for the next deadline and I then scheduled a few phone calls to strategize. My first call was with my PO to discuss the critiques. This was extremely helpful as we were able to prioritize weaknesses I should address (or omit from the proposal) and strengths to further highlight. This was followed by a call among my primary mentors. Of course, each application will be different, but for me, the overarching concerns were related to the mentorship team (too many people to coordinate), the training courses (too elementary in material), and evaluating the outcomes of the science (too brief). As I had been told before, and was echoed by my reviewers, the weight of the application was on the career development and training plan, although the science still received rigorous scrutiny. I was able to rebalance the mentorship team, including just two co-mentors, and find higher level classes to take. By trimming some words from the research strategy, and even just reformatting some text, I was able to obtain sufficient space in the proposal to detail the evaluation plan in the research strategy section. For the resubmission, I focused on a few things in particular:

• The response to reviewers is only one page in length, therefore you need to identify themes among the three reviewers and group them together. Try to be responsive to as many critiques as possible. Inevitably there will be some throw away comments; these do not need to be dealt with. To make the reviewers' lives easier upon resubmission, most people use a trick like highlighting or bolding the changes in the modified proposal documents. Given how much time reviewers spend evaluating all of the proposals for the study section, you want to make their lives as easy as possible to find the important points in your proposal.
• Update all mentor and collaborator letters. Even if it is just changing the date, the reviewers' want to see this. I even included wording in the letters from my mentors indicating we read, strategized, and addressed the reviewer's comments. This shows you have taken the comments seriously and did not do this in isolation. Also, have your three independent referees update their letters.
• Update the budget. Things have probably changed. Maybe you are including or excluding classes, got a salary bump, or just are adjusting for other factors.

Presently, NIH only allows a single resubmission for K01 applications. This is the only time you will be assured of the same reviewers, so they will want to see you were responsive to their suggestions. As before, I submitted the revised application, and waited for the study section date to come and go.

A few days after the resubmission the first indication of the outcome became available in Commons: Discussed. As mentioned earlier a discussed application also receives a corresponding impact score, and, possibly funding. It was certainly a welcome sign of the reviewers' enthusiasm towards this resubmission. The impact score and summary sheets became available a short time later. The score, an average of all of the reviewer scores, was a 22. Recall in NIH-land the lower the score the better, with the best possible score being 10. With a score in hand, now began the most confusing part of the award process. Here's my takeaway points based on what has transpired since the application was scored.

• Contact the PO immediately to find out what the payline is for your institute. At the time my grant was reviewed, the interim payline was 18. The PO indicated that sometimes an interim payline is issued while they grapple with their budget, evaluate current and past awards, and so on, before the end of the fiscal year. The payline is the first hurdle to be cleared for funding; BUT, and here is the confusing part, there is no guarantee that a score at or below this threshold will receive funding, and there is no guarantee that a score above this threshold will not receive funding. The final payline for my institute moved to a 20 at the end of the fiscal year, so I was still outside of the comfort zone.
• I want to interject a brief comment on the impact score. While it is tempting to want to compare your score against others, that serves no utility other than to frustrate yourself. You will hear of people getting funded in the upper 20s while some in the lower 20s not getting funded. Each study section and each institute is different. You are only evaluated against the grants in your particular study section, and then based on research priorities in your institute. It is luck of the draw, and in some sense this kind of ensures the process works on average.
• If you are requested to provide just-in-time (JIT) documentation, do this immediately and without delay. At NIAID a score of <30 is encouraged to submit the JIT. Note that receiving a request to provide JIT documents is no guarantee of funding. JIT materials typically include current and pending support and IRB approval, plus other materials specifically requested by the institute and grant type. I needed to additionally provide an updated budget justification and training in human subjects research documentation for my key collaborators.
• Side note on the IRB requirement. My understanding is that NIH used to accept IRB under review prior to awarding a grant. This is no longer the case. IRB approval is required now before a grant can be awarded. This can put you in a difficult position. After all, IRBs do not want to take time to review proposals that may not get funded (why waste everyone's time if the project will never happen?). On the other hand, receiving a favorable impact score is a good sign that sooner or later you will get funded, therefore you can move ahead with the IRB, although you are now working against tight deadlines. I was very fortunate that the timing of my IRB worked out so that I could obtain a full IRB approval within a matter of weeks from when I received the NIH review. In hindsight, I would have started much, much sooner rather than wait for a last-minute scramble to get IRB approval. The IRB process caused the most stress out of all other parts of the awarding process because of the urgent timing. By contacting your IRB administrator, you may be able to receive a provisional/conditional acceptance, and while this can move some things along for the NIH process, they still require a full unconditional approval prior to an award.
• If you have not received funding right away based on your score, there is one other opportunity for an award at the end of the NIH fiscal year. Grant funds are allocated year to year for each institute, and these funds are expended during the course of the fiscal year as grants are awarded. NIH tends err on the conservative side: they are likely to underspend. Towards the end of the fiscal year (August-September), they will check how much money has been spent and how much is still available to spend on grant awards. At this point, they can usually though not always fund a few additional grants that were not awarded during the year. Based on my experience of having this happen, here is how it seemed to play out.
• The grants not funded but with the lowest impact score will receive priority for year-end funding. I cannot say exactly how close you need to be to the payline, but it sounded like a few points is all (i.e. don't expect one if you are 10 points removed).
• Those who have completed the JIT move to the front of this line.
• The PO is your advocate for getting your grant funded; you want to be in constant contact with them and demonstrate your competence to manage a grant.
• Prepare a response to reviewers as an informal document. There is no requirement to do this, but I received consistent advice to proactively do this. Prepare it just like a normal response to reviewers document but keep it concise (no more than a couple of pages in my opinion). Send this to the PO so they can use it when the time comes to advocate for your proposal. At the very least, this will also help you strategize when you submit your grant again if it remains unfunded.
• Despite how process driven and detailed a government entity like NIH is, the year-end funding is extremely nebulous to the applicant. You will never receive a clear-cut answer that you will or will not be funded. Therefore, be prepared to submit again while waiting for the decision. Ultimately, I was awarded the grant on my first resubmission through the year end funding mechanism. I had also submitted the grant a third time (as a new submission, since only a single resubmission is allowed) - therefore I needed to withdraw this new submission. But, better safe than sorry, and I didn't want to let much time pass before submitting again, as I wanted to return to the same study section since they were familiar with my work at this point.

And this concludes my first foray into the NIH grant funded world. I was extremely fortunate to receive funding on my second shot and feel like I have a much better understanding of the grant funding process. I certainly have immense appreciation and gratitude for all of the scientists and NIH officials who take their time to participate. Overall, I found the entire process to be both interesting and frustrating (more so the former fortunately). I hope the information I have documented in these four blog posts (part 1, part 2, part 3) prove useful to others, especially if you are going after a career development award from NIH, such as the K01.

Cite: Goldstein ND. Final thoughts on obtaining a K01 career development award. Sep 12, 2019. DOI: 10.17918/goldsteinepi.