There are line graphs available for the data. From the CM-PRISM form, enter and save your data, then click the "View Chart" button - the graph will display in a separate browser window. This feature will evolve over time.
Common Metrics Initiative Frequently Asked Questions
Can hubs view graphs of their data?
How many years of data are required for the Informatics metric?
Starting with 2020 data, the informatics scripts were written to collect data for one year and the five previous years.
Implementation
Is there a specific mechanism for submitting language to revise the guidelines and/or communicate any additions (e.g., definitions of engaged in research)?
Please submit your questions and suggestions to the CLIC team (common_metrics@clic-ctsa.org) or submit them through the CLIC website. CLIC will forward these items to NCATS.
Metrics - General
Is there an expectation about how the Common Metrics will be used in annual reports?
The CM-PRISM data is de-identified and aggregated and provided to NCATS in a report. Since this data is de-identified, NCATS cannot use this report to make funding decisions.
Hubs receive a report that shows consortium aggregated data and their own individual data.
What definition of “annually” should we be using? Calendar year, grant year, fiscal year, or academic year?
The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics (January 1 - December 31).
What is the deadline for completing data entry for the Common Metrics?
Data and Program Summaries are due annually on the last business day of August.
Metrics - Careers
Are we collecting data on the number and percentage of each of the three underrepresented categories (racial and ethnic groups, individuals with disabilities and those from disadvantaged backgrounds) or just the overall number and percentage?
You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html
Are women considered underrepresented persons?
At this time, women are not considered underrepresented persons. See the NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
Can a CTSA hub have a KL2 program that is only institutionally funded?
It is not possible to have a KL2 Program without a U54 CTSA Program award.
Do they need to collect information regarding an individual’s primary role is in research? (e.g., clinician, academia, gov’t, industry, etc.)?
No, these are examples of what it means to be engaged in research, given these different roles. Hubs may want information regarding an individual’s primary role in research for their own evaluation, but it is not reported for the common metrics.
Do we include institutionally funded KL2 and TL1 scholars or only those whose training was paid for by the grant?
Please include only CTSA program-funded KL2 Scholars and TL1 Trainees when reporting Common Metric data.
Do we include KL2 and TL1 scholars that are institutionally funded or just NIH/NCATS funded?
NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.
For gender, if there are individuals who prefer not to identify, would you add another category, gender not provided?
If any demographic characteristic is missing, then the data for the individual should be removed from the numerator and denominator.
For T scholars who begin their program in the middle of the calendar year, do we count them in that year or the next year when they finish the program?
Graduates should be added to the denominator for the calendar year that they finish their program.
For TL1 pre- and post-docs, what do we use for "year started"?
Use the year that they started the grant.
How closely do hubs need to follow the definitions of engaged in research by primary role?
These are examples of what people in those roles would be doing to assist hubs in capturing “engaged” data.
How do I use the Careers Template in Excel?
The template, which is optional, is designed to assist hubs in collecting data for KL2/TL1 scholars and trainees. Hubs are not required to collect data from previous years - use the template to collect data only for the year that you are reporting/entering data in CM-PRISM.
How should we define females at the senior faculty level? Is this the “associate” level and higher? Is there an equivalent level outside of academia?
This information is not currently required for the Common Metrics.
How would you suggest we map our data to the current OG which only has these three options: Female, Male, and Other?
The gender categories have been changed to Male, Female, and Other - at this time these are the only options. This issue will continue to be addressed as the metric evolves.
If a graduate is lost to follow up, should they be removed from the numerator and denominator of the metric? If a graduate does not respond to a survey, should they be included in the denominator?
Lost to follow-up designation is for any Scholars/Trainees for whom their current research status cannot be determined. Hubs should continue to try to follow up with these Scholars/Trainees for 5 years before dropping them from the dataset. Those who are lost to follow-up after 5 years are not included in the denominator, for common metrics reporting but hubs should continue to try to track them, as they may re-enter clinical and translational research.
If a graduate is lost to follow-up (e.g., no address or email to send a survey), should they be removed from the numerator and denominator of the metric?
Yes, remove them from the numerator and denominator of the metric.
If a PhD student does not get compensated for time for research, but that research is a required part of the degree and they receive tuition assistance, should this be considered "engaged in CTR"?
Yes, this is considered to still be engaged in research.
Is there a minimum % for someone involved in research? If someone reports they are only involved in research 5% does that count or is there a minimum of for instance 20%?
There is a specified percentage of time defined for "engaged in research" - please see the Technical Description section of the Operational Guideline.
Isn't the metric those who are engaged specifically in Clinical and Translational research, not just "research" overall?
This metric reflects all research.
Many of our participants exit the KL2 program after getting into another “K” program. We view this as a success, but it seems that the definition does not view this as a success. Please clarify.
Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.
Occasionally the URP or gender status of a Scholar might change over time. How should we handle that?
Use the scholar’s most recent classification.
Please provide some guidance on how to define individuals from "disadvantaged backgrounds".
Please use the definition provided by NCATS: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-20-031.html
Regarding "engaged in clincial research", if a scholar takes a break in order to start a family or care for parents, do we not include him in CM-PRISM for that particular year or so, even though he will resume his research?
Graduates are to be assessed for their engagement in clinical and translational research annually.
Should hubs apply the "Engaged in research" bullet points as a closed set of criteria that defines engagement or as examples that hubs should use to assess engagement in research?
The list of activities that indicate engagement in research are examples only and not criteria.
Should students who were funded by the TL1 in their first two years of Med school, and are now in the Grad school phase be included in the numerator/denominator?
No, they should not be counted until they finish grad school.
Should students who were funded by the TL1 in their first two years of Med school, and are now in their residency be included in the numerator/denominator?
No, they should not be included until they finish their residency.
Should we clarify people engaged specifically in clinical and translational research, or just research overall?
This metric reflects all research.
Should we count trainees who leave the program to accept a research position in the numerator/denominator?
Yes, they should be counted. The goal of the program is to place scholars and trainees in research positions.
Should we exclude individuals who have completed the TL1 or KL2 and have returned to residency or other degree programs from being counted as currently engaged??
Yes, if they are going back to medical school (or another clinical degree program like PharmD, DPT, etc.). These individuals are currently in "training" and are not engaged in research. Counting them in the currently engaged numerator would affect the hubs' percentages in a negative way.
Should we include Ks and Ts who fully participate in the CTSA Career Development Program that are funded by institutional funds, not the CTSA grants?
If they are not directly funded by NCATS then they are excluded when reporting the metric.
Since gender is fluid, it's possible that individuals have shifted identities over time. Is the expectation that programs will survey training program alums to update their current status on gender?
At this time, use gender identification at the time of application.
This is a cumulative metric. If a participant is engaged in research 1 year after program completion but no longer is 3 years after completion, how should this be tracked?
The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total and updating the denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.
To what extent can eRA extract help with this data collection?
eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.
What are examples of things that people may do that would not be considered engaged in research?
Refer to the definitions of being engaged in research.
What is the definition of underrepresented persons?
We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.
This is the link to the most current NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
What is the definition of URM in the new Common Metrics?
Use the NIH definition: https://diversity.nih.gov/about-us/population-underrepresented
What is the expectation for the number of years that participants are tracked upon completion of a program?
Scholars/Trainees should be followed for at least five years.
What’s the purpose of the narrative?
CLIC conducts qualitative analysis on the narratives. This information then serves as the foundation for the next year’s Insights to Inspire program. This program fosters innovation and ideas, and encourages collaboration across the consortium. Click here to learn more.
Who is still considered to be in training?
Scenario: So for our pre-doctoral trainees, they'll finish medical school, do an internship in residency so they are in training for all that time, let's say they come on as faculty and they do a career development award from the NIH. Technically, if they have any kind of K award they're considered in training. So we can't count them.
Response: Any individual who receives a new career development award is counted as being engaged in research.
Exclusion Criteria: Trainees and Scholars who are still in training or who have left the program without completing the full training program requirements. Program alumni who are still in residency or other degree-seeking programs are also excluded.
Why was the denominator "number of currently engaged" removed from the Careers metric?
Using the denominator of "cumulative number of graduates" more accurately reflects the percentage of women and underrepresented persons (URPs) engaged in research.
Metrics – Informatics
Can you enter qualitative data for more than one topic in the Program Summary?
Yes, hubs can enter qualitative data for up to three topics.
While only one Program Summary is required for the Informatics and Career metrics, hubs are encouraged to enter additional topics. This information is shared by the Insights to Inspire program with the consortium as a method of sharing actionable intelligence.
Can you tell me if the metrics will be looking at all patients (inpatient and outpatient) or are there specific guidelines for this?
The metric includes all patients in the research data warehouse – inpatient, outpatient, ER, etc.
CM-PRISM - Are there tools available to assist hubs with developing the Program Summary sections?
Tools are available on the CLIC website: https://clic-ctsa.org/common-metrics-initiative
Denominators - What are the denominators for the informatics metrics?
Total # of unique patients in the data warehouse
Total # of unique encounters in the data warehouse
Total # of unique patients with age >12 in the data warehouse
Do the scripts provide the min/max dates to use for your data inclusion table?
The scripts are updated each year to "pull" the correct dates. Starting with 2020 data, the scripts were written to collect data for one- and five-year periods.