Common Metrics Initiative Frequently Asked Questions

Using this feature is helpful when handling a large amount of data on a frequently repeating schedule. Given the small amount of data required for this project, the economies of scale that usually make importing a good idea may not apply. Also, narrative text can’t be imported from Excel since it is rich formatted text. You would still need to add narrative to the software.

If you want to use additional Scorecards, please put them in a section other than Common Metrics.

No, you can have multiple programs on a single Scorecard.

The forecast is not required, but you can include it. If you want to include a forecast in Scorecard, place it in the Current Target Value box when you enter your current data value.

Please do not edit tags. Information for the data uploads is based on specific tag names. If you enter you data under a different tag, your data will not be reported accruately.

There is no limit to the number of strategies that you can include in Scorecard. Enter the strategies that you believe you can reasonably implement - you can always enter more strategies in Scorecard at a later time.

Please review the TTC Planning tools on the CLIC website for additional suggestions. You can use these tools to keep notes of your entire TTC Plan development process.

Each hub is allocated two Scorecard user licenses - these are the only individuals in your hub that can see your data. The CLIC, as an administrator, can see what the hubs enter into the Scorecard software.

Hubs can purchase additional licenses from Clear Impact.

Common metric data and the Turn-the-Curve plans are entered annually. All metric data is downloaded for analysis on or near August 31st of each year. The CLIC communicates due dates to the hubs.

The CLIC provides two licenses for each hub. To request a license, contact common_metrics@clic-ctsa.org.

Those questions and suggestions can be emailed to the CLIC team (common_metrics@clic-ctsa.org) or submitted through the CLIC website. CLIC will forward them to NCATS.

TTC planning is important for the improvement of these metrics at the hub level, but the hope is also that we can share those best practices that are very successful in the hubs across the CTSA Consortium. We ask you to develop a TTC Plan and provide in your Story Behind the Curve some of the underlying factors that lead you to have such great success with a particular metric, as well the strategies that you used to sustain that performance and who the partners are involved to sustain that performance.  We ask you to document what you believe are the underlying factors and strategies that you use to achieve your performance. Hubs may find that there are few changes to the TTC plan year to year - this is acceptable.

The Scorecard data for the common metrics is downloaded and sent to the CLIC. CLIC de-identifies and aggregates the data. NCATS receives a report of the de-identified aggregated data. Hubs receive individual reports. Whether hubs choose to include these data in their RPPR is their choice.

Only enter data related to what happened after the award. If your hub has historical data that reflects information for earlier time periods, it should be considered in your work on developing your Story Behind the Curve.

The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics.

The deadline is the August 31st of each year.

If the data are available and you have the resources to go back and calculate them for previous years, this could be helpful in determining your forecast; however, per the Operational Guidelines, this is not required.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into the three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

It is not possible to have a KL2 Program without a U54 CTSA Program award.

Yes.

Please include only CTSA program-funded KL2 and TL1 scholars.

NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.

Graduates should be added to the denominator of the metric starting in the calendar year that they finish their program. In your example, you would add the T scholar in the denominator for 2014. You should then also assess their eligibility for the numerator of the metric (i.e., are they involved in CTS research) starting in 2014.

If your program was still "running" and you had bridge funding and none were awarded, please enter 0 (zero). If you "froze" the program then leave the field blank - and please put a note in your TTC plan.

(8/7/2019)

Remove them from the numerator and denominator of the metric.

Yes, remove them from the numerator and denominator of the metric.

Yes, this is considered to still be engaged in research.

See Career Metric Operational Guiedelines. “If primary role is as a clinician: some effort (e.g., 20%) in research or as a site PI for industry-sponsored clinical trials.”

Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.

If medical students or residents do not have dedicated time for research, they are not considered to be “engaged in research.”

Yes. Include this information in your Story Behind the Curve and develop strategies that you may want to implement (e.g., adding an exit interview to your process for departing graduates) even before you are able to start collecting data for the metric.

The following is a Notice of NIH’s Interest in Diversity: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-122.html
As presented in the notice: 
C. Individuals from disadvantaged backgrounds, defined as: 
1. Individuals who come from a family with an annual income below established low-income thresholds. These thresholds are based on family size, published by the U.S. Bureau of the Census; adjusted annually for changes in the Consumer Price Index; and adjusted by the Secretary for use in all health professions programs. The Secretary periodically publishes these income levels at http://aspe.hhs.gov/poverty/index.shtml.  
2. Individuals who come from an educational environment such as that found in certain rural or inner-city environments that have demonstrably and directly inhibited the individual from obtaining the knowledge, skills, and abilities necessary to develop and participate in a research career. 

The disadvantaged background category (C1 and C2) is applicable to programs focused on high school and undergraduate candidates. Thus, as the career metric is focused on TL1 trainees and KL2 scholars the disadvantaged category does not apply to this metric. 

Graduates are to be assessed for their engagement in clinical and translational research annually.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

The list of activities that indicate engagement in research are examples only and not criteria.

If they are not directly funded by NCATS then they are excluded from the metric.

We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.

This is the link to the most current NIH definition: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total to make an updated denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.

eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.

Yes. If you do some thinking about this now, you may identify some additional performance measures or strategies that you may want to implement (e.g., adding an exit interview to your process for departing graduates) even before you are able to start collecting data for the metric.

Exclude the pre-docs who have only attended the 12-week summer experience from your Careers Common Metric counts. These don’t sound like they are comparable to TL1 students who you might expect to pursue research careers.

NCATS expects that hubs collect this data on an annual basis from each graduate for each year after completion of the program (for graduates since January 1, 2012). If a graduate does not respond in a given year, they are not included in either the numerator or denominator for that year. However, an attempt should be made to follow up with them again the next year to determine their status. A graduate should only be permanently removed from the denominator if they are deemed “lost to follow-up”, (e.g., no address or email to send a survey to, persistent refusal to respond).

There is no limit on the number of years currently specified.

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric). For KL2s, following completion of CTSA-funded training as a KL2 scholar, add them to the denominator of the metric.

It is the approval year that determines which cohort of data the protocol is included with. If it is submitted in 2014 and approved in 2015, count it in 2015.

For this metric, you report the median duration from the IRB(s) from your CTSA hub's primary institution. If your primary institution has three separate IRBs, then the data from all three should be combined together to compute the median. But if you have more than one institution, one of which is primary and the others which are not, you only need to report the data from the IRB(s) of the primary hub institution. 

Protocols include all clinical research (including multisite studies). There is no exclusion for industry-related projects. So, if they are industry initiated or investigator initiated, they are both included in the metric.

Institutional-wide, not just CTSA-related studies.

Just include the protocols that have received IRB approval during the timeframe.

Please use this link (it's also available in the Operational Guidelines) - https://grants.nih.gov/grants/glossary.htm#ClinicalResearch.