There are line graphs available for the data. This feature will evolve over time as the metric values reported increase.
Common Metrics Initiative Frequently Asked Questions
Can hubs view graphs of their data?
How many years of data are required for the Informatics metric?
Informatics scripts are written to collect data for one year and the five previous years.
Implementation
Is there a specific mechanism for submitting language to revise the guidelines and/or communicate any additions (e.g., definitions of engaged in research)?
Please submit your questions and suggestions to the CLIC team (common_metrics@clic-ctsa.org) or submit them through the CLIC website. CLIC will forward these items to NCATS.
Metrics - General
Is there an expectation about how the Common Metrics will be used in annual reports?
The CM-PRISM data is de-identified and aggregated and provided to NCATS in a report. Since this data is de-identified, NCATS cannot use this report to make funding decisions.
Hubs receive a report that shows consortium aggregated data and their own individual data.
What definition of “annually” should we be using? Calendar year, grant year, fiscal year, or academic year?
The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics (January 1 - December 31).
What is the deadline for completing data entry for the Common Metrics?
Data and Program Summaries are due annually on the last business day of August.
Metrics - Careers
All of our TL1 grantees are pre-doctoral scholars, after they complete the award they have years of medical school, etc. before starting their research careers. Should they all be excluded because they are still considered "in training programs"?
If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).
If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).
If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.
Are we collecting data on the number and percentage of each of the three underrepresented categories (racial and ethnic groups, individuals with disabilities and those from disadvantaged backgrounds) or just the overall number and percentage?
You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html
Can a CTSA hub have a KL2 program that is only institutionally funded?
It is not possible to have a KL2 Program without a U54 CTSA Program award.
Did you say that women are considered underrepresented persons?
At this time, women are not considered underrepresented persons. See the NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
Do they need to collect information regarding an individual’s primary role is in research? (e.g., clinician, academia, gov’t, industry, etc.)?
No, these are examples of what it means to be engaged in research, given these different roles. Hubs may want information regarding an individual’s primary role in research for their own evaluation, but it is not reported for the common metrics.
Do we include institutionally funded KL2 and TL1 scholars or only those whose training was paid for by the grant?
Please include only CTSA program-funded KL2 Scholars and TL1 Trainees.
Do we include KL2 and TL1 scholars that are institutionally funded or just NIH/NCATS funded?
NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.
For gender, if there are individuals who prefer not to identify, would you add another category, gender not provided?
If any demographic characteristic is missing, then the data for the individual should be removed from the numerator and denominator.
For T scholars, do we count them for the year ending data in the same year they finish their program or not until the following year? Ex: T scholar finished program in June 2014. Do we count their responses in the 2014 data or wait until 2015?
Graduates should be added to the denominator of the metric starting in the calendar year that they finish their program. In your example, you would add the T scholar in the denominator for 2014. You should then also assess their eligibility for the numerator of the metric (i.e., are they involved in CTS research) starting in 2014.
How closely do hubs need to follow the definitions of engaged in research by primary role?
These are examples of what people in those roles would be doing to assist hubs in capturing “engaged” data.
How should we define females at the senior faculty level? Is this the “associate” level and higher? Is there an equivalent level outside of academia?
This information is not currently required for the Common Metrics.
How would you suggest we map our data to the current OG which only has these three options: Female, Male, and Gender Non-binary?
The gender categories have been changed to Male, Female, and Other. This issue will continued to be addressed as the metric evolves.
If a graduate is lost to follow up, should they be removed from the numerator and denominator of the metric? If a graduate does not respond to a survey, should they be included in the denominator?
Lost to follow up designation is for any Scholars/ Trainees Trainee for whom their current research status cannot be determined. Hubs should continue to try to follow up with this Scholar/Trainee for 5 years before dropping them from the dataset. Those who are lost to follow up after 5 years are not included in the denominator, for common metrics reporting but hubs should continue to try to track them, as they may re-enter clinical and translational research.
If a graduate is lost to follow-up (e.g., no address or email to send a survey), should they be removed from the numerator and denominator of the metric?
Yes, remove them from the numerator and denominator of the metric.
If a PhD student does not get compensated for time for research, but that research is a required part of the degree and they receive tuition assistance, should this be considered "engaged in CTR"?
Yes, this is considered to still be engaged in research.
Is there a minimum % for someone involved in research? If someone reports they are only involved in research 5% does that count or is there a minimum of for instance 20%?
There is a specified percentage of time defined for "engaged in research" - please see the Technical Description section of the Operational Guideline.
Isn't the metric those who are engaged specifically in Clinical and Translational research, not just "research" overall?
This metric reflects all research.
Many of our participants exit the KL2 program after getting into another “K” program. We view this as a success, but it seems that the definition does not view this as a success. Please clarify.
Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.
Many of our short-term TL1 Trainees return to medical school and go on to residency after completing the program. Would that be considered “engaged in research”?
If medical students or residents do not have dedicated time for research, they are not considered to be “engaged in research.”
Occasionally the URP or gender status of a Scholar might change over time. How should we handle that?
Use the scholar’s most recent classification.
Please provide some guidance on how to define individuals from "disadvantaged backgrounds".
Please use the definition provided by NCATS: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html
Regarding "engaged in clincial research", if a scholar takes a break in order to start a family or care for parents, do we not include him in CM-PRISM for that particular year or so, even though he will resume his research?
Graduates are to be assessed for their engagement in clinical and translational research annually.
Several of the TL1 trainees at our institution are in the MD/PhD program. Once they have completed their dual degrees, they continue with a residency for several additional years as well as subsequent fellowships. Is this considered “still in training”?
If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).
If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).
If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.
Should hubs apply the "Engaged in research" bullet points as a closed set of criteria that defines engagement or as examples that hubs should use to assess engagement in research?
The list of activities that indicate engagement in research are examples only and not criteria.
Should we clarify people engaged specifically in clinical and translational research, or just research overall?
This metric reflects all research.
Should we include Ks and Ts who fully participate in the CTSA Career Development Program that are funded by institutional funds, not the CTSA grants?
If they are not directly funded by NCATS then they are excluded when reporting the metric.
Since gender is fluid, it's possible that individuals have shifted identities over time. Is the expectation that programs will survey training program alums to update their current status on gender?
At this time, use gender identification at the time of application.
This is a cumulative metric. If a participant is engaged in research 1 year after program completion but no longer is 3 years after completion, how should this be tracked?
The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total and updating the denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.
To what extent can eRA extract help with this data collection?
eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.
What are examples of things that people may do that would not be considered engaged in research?
Refer to the definitions of being engaged in research.
What is the definition of underrepresented persons?
We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.
This is the link to the most current NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
What is the definition of URM in the new Common Metrics?
Use the NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
What is the expectation for the number of years that participants are tracked upon completion of a program?
Scholars/Trainees should be followed for at least five years.
What’s the purpose of the narrative?
CLIC conducts qualitative analysis on the narratives. This information then serves as the foundation for the next year’s Insights to Inspire program. This program fosters innovation and ideas, and encourages collaboration across the consortium. Click here to learn more.
When is a TL1 Trainee or KL2 Scholar eligible to be counted for the careers metric?
If the Trainee has completed the TL1 training program and is no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric). For KL2 Scholars, following completion of CTSA-funded training as a KL2 scholar, add them to the denominator of the metric.
Who is still considered to be in training?
Scenario: So for our pre-doctoral trainees, they'll finish medical school, do an internship in residency so they are in training for all that time, let's say they come on as faculty and they do a career development award from the NIH. Technically, if they have any kind of K award they're considered in training. So we can't count them.
Response: Any individual who receives a new career development award is counted as being engaged in research.
Exclusion Criteria: Trainees and Scholars who are still in training or who have left the program without completing the full training program requirements. Program alumni who are still in residency or other degree-seeking programs are also excluded.
Metrics – Informatics
Can you enter qualitative data for more than one topic in the Program Summary?
Yes, hubs can enter qualitative data for up to three topics. Only one is required.
Can you tell me if the metrics will be looking at all patients (inpatient and outpatient) or are there specific guidelines for this?
The metric includes all patients in the research data warehouse – inpatient, outpatient, ER, etc.
CM-PRISM - Are there tools available to assist hubs with developing the Program Summary sections?
Tools are available on the CLIC website: https://clic-ctsa.org/common-metrics-initiative
Denominators - What are the denominators for the informatics metrics?
Total # of unique patients in the data warehouse
Total # of unique encounters in the data warehouse
Total # of unique patients with age >12 in the data warehouse
Do the scripts provide the min/max dates to use for your data inclusion table?
The scripts are updated each year to "pull" the correct dates. For the Informatics metric, data is collected for one- and five-year periods.
Does free text include imaging notes (narratives/impressions) or just clinical progress notes and discharge summaries etc.?
No, only notes: admission, progress, discharge, procedure, etc.
Does the last 12 months mean January 1 through December 31?
Yes, and the scripts will be updated each year to reflect the timeframe.
How are the Program Summaries used?
CLIC conducts qualitative analysis on the Program Summaries to select the content that will be shared in the Insights to Inspire program. https://clic-ctsa.org/common-metrics-initiative/i2i