Implementation

Is there a specific mechanism for submitting language to revise the guidelines and/or communicate any additions (e.g., definitions of engaged in research)?

Those questions and suggestions can be emailed to the CLIC team (common_metrics@clic-ctsa.org) or submitted through the CLIC website. CLIC will forward them to NCATS.

Metrics - General

Is there an expectation about how the Common Metrics will be used in annual reports?

The Scorecard data for the common metrics is downloaded and sent to the CLIC. CLIC de-identifies and aggregates the data. NCATS receives a report of the de-identified aggregated data. Hubs receive individual reports. Whether hubs choose to include these data in their RPPR is their choice.

Our CTSA first received funding last August, so currently there is very little data for what has happened since then. Should we look back at data from before we received our CTSA in order to have more of a trend, or should it be based solely on data after

Only enter data related to what happened after the award. If your hub has historical data that reflects information for earlier time periods, it should be considered in your work on developing your Story Behind the Curve.

What definition of “annually” should we be using? Calendar year, grant year, fiscal year, or academic year?

The Operational Guidelines were updated to indicate CTSA program hubs should use the calendar year for each of the Common Metrics.

What is the deadline for completing data entry for the Common Metrics?

The deadline is the August 31st of each year.

Metrics - Careers

All of our TL1 grantees are pre-doctoral scholars, after they complete the award they have years of medical school, etc. before starting their research careers. Should they all be excluded because they are still considered "in training programs"?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

Are we collecting data on the number and percentage of each of the three underrepresented categories (racial and ethnic groups, individuals with disabilities and those from disadvantaged backgrounds) or just the overall number and percentage?

You should be collecting data on the number and percentage of underrepresented persons. You do not need to break it down into the three categories. Please see NIH's Interest in Diversity statement at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

Can a CTSA hub have a KL2 program that is only institutionally funded?

It is not possible to have a KL2 Program without a U54 CTSA Program award.

Do we combine post-doc and pre-doc trainees in the common metrics for the TL1 program?

Yes.

Do we include institutionally funded KL2 and TL1 scholars or only those whose training was paid for by the grant?

Please include only CTSA program-funded KL2 and TL1 scholars.

Do we include KL2 and TL1 scholars that are institutionally funded or just NIH/NCATS funded?

NCATS has clarified that non-CTSA grant funded scholars and trainees who participate in your KL2 or TL1 program should not be included.

For T scholars, do we count them for the year ending data in the same year they finish their program or not until the following year? Ex: T scholar finished program in June 2014. Do we count their responses in the 2014 data or wait until 2015?

Graduates should be added to the denominator of the metric starting in the calendar year that they finish their program. In your example, you would add the T scholar in the denominator for 2014. You should then also assess their eligibility for the numerator of the metric (i.e., are they involved in CTS research) starting in 2014.

How should we report KL2/TL1 data if we were in a no-cost extension from 6/2017-6/2018? Some of the Ks were active before 6/2017 so we reported on them for CY2017, but for CY 2018, there were no active K awards. Our funding was renewed 6/2018.

If your program was still "running" and you had bridge funding and none were awarded, please enter 0 (zero). If you "froze" the program then leave the field blank - and please put a note in your TTC plan.

(8/7/2019)

If a graduate is lost to follow up, should they be removed from the numerator and denominator of the metric? If a graduate does not respond to a survey, should they be included in the denominator?

Remove them from the numerator and denominator of the metric.

If a graduate is lost to follow-up (e.g., no address or email to send a survey), should they be removed from the numerator and denominator of the metric?

Yes, remove them from the numerator and denominator of the metric.

If a PhD student does not get compensated for time for research, but that research is a required part of the degree and they receive tuition assistance, should this be considered "engaged in CTR"?

Yes, this is considered to still be engaged in research.

Is there a minimum % for someone involved in research? If someone reports they are only involved in research 5% does that count or is there a minimum of for instance 20%?

See Career Metric Operational Guiedelines. “If primary role is as a clinician: some effort (e.g., 20%) in research or as a site PI for industry-sponsored clinical trials.”

Many of our participants exit the KL2 program after getting into another “K” program. We view this as a success, but it seems that the definition does not view this as a success. Please clarify.

Following completion of CTSA-funded training as a KL2 scholar, they are eligible to be counted for the metric. If they are engaged in further training by another K award, they are considered engaged in research. However, if a scholar leaves the KL2 program without completing the full training program requirements, they are excluded.

Many of our short-term TL1 trainees return to medical school and go on to residency after completing the program. Would that be considered “engaged in research”?

If medical students or residents do not have dedicated time for research, they are not considered to be “engaged in research.”

Please provide some guidance on how to define individuals from "disadvantaged backgrounds".

The following is a Notice of NIH’s Interest in Diversity: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-122.html
As presented in the notice: 
C. Individuals from disadvantaged backgrounds, defined as: 
1. Individuals who come from a family with an annual income below established low-income thresholds. These thresholds are based on family size, published by the U.S. Bureau of the Census; adjusted annually for changes in the Consumer Price Index; and adjusted by the Secretary for use in all health professions programs. The Secretary periodically publishes these income levels at http://aspe.hhs.gov/poverty/index.shtml.  
2. Individuals who come from an educational environment such as that found in certain rural or inner-city environments that have demonstrably and directly inhibited the individual from obtaining the knowledge, skills, and abilities necessary to develop and participate in a research career. 

The disadvantaged background category (C1 and C2) is applicable to programs focused on high school and undergraduate candidates. Thus, as the career metric is focused on TL1 trainees and KL2 scholars the disadvantaged category does not apply to this metric. 

Regarding "engaged in clincial research", if a scholar takes a break in order to start a family or care for parents, do we not include him in our Scorecard for that particular year or so, even though he will resume his research?

Graduates are to be assessed for their engagement in clinical and translational research annually.

Several of the TL1 trainees at our institution are in the MD/PhD program. Once they have completed their dual degrees, they continue with a residency for several additional years as well as subsequent fellowships. Is this considered “still in training”?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric).

If they’ve completed the TL1 training program and are participating in additional training that has a research component, they are considered “engaged in research” (add them to the numerator of the metric).

If the residency or PhD program includes dedicated time for research, they are engaged in research and added to the numerator.

Should hubs apply the "Engaged in research" bullet points as a closed set of criteria that defines engagement or as examples that hubs should use to assess engagement in research?

The list of activities that indicate engagement in research are examples only and not criteria.

Should we include Ks and Ts who fully participate in the CTSA Career Development Program that are funded by institutional funds, not the CTSA grants?

If they are not directly funded by NCATS then they are excluded from the metric.

There are multiple definitions for underrepresented minorities. Which one are we using?

We are using the NIH definitions. The Operational Guideline has a link to more information about these definitions. The most up to date versions of the Operational Guidelines can be found on the Established Common Metrics page.

This is the link to the most current NIH definition: https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-210.html

This is a cumulative metric. If a participant is engaged in research 1 year after program completion but no longer is 3 years after completion, how should this be tracked?

The total number of program graduates over time is cumulative. It gets updated once a year by adding the new number of graduates to the previous total to make an updated denominator for the metric. The numerator, the number and percent of graduates from the denominator who are currently engaged in clinical and translational research is assessed each year.

To what extent can eRA extract help with this data collection?

eRA will be a source of information but it won’t be a complete source, because they are not all inclusive; there are activities that are outside of the system.

We funded short-term pre-docs (12-wk summer experience) 2006- 2014 and then discontinued it and have just supported year-long pre-docs for the last 2 years. Should these short-term awardees be excluded from the common metric counts?

Exclude the pre-docs who have only attended the 12-week summer experience from your Careers Common Metric counts. These don’t sound like they are comparable to TL1 students who you might expect to pursue research careers.

What does "current" mean in terms of research engagement and reporting on it?

NCATS expects that hubs collect this data on an annual basis from each graduate for each year after completion of the program (for graduates since January 1, 2012). If a graduate does not respond in a given year, they are not included in either the numerator or denominator for that year. However, an attempt should be made to follow up with them again the next year to determine their status. A graduate should only be permanently removed from the denominator if they are deemed “lost to follow-up”, (e.g., no address or email to send a survey to, persistent refusal to respond).

What is the expectation for the number of years that participants are tracked upon completion of a program?

There is no limit on the number of years currently specified.

When is a TL1 or KL2 student eligible to be counted for the careers metric?

If they have completed the TL1 training program and are no longer on the TL1 grant, they can be assessed for whether they are engaged in research (add them to the denominator of the metric). For KL2s, following completion of CTSA-funded training as a KL2 scholar, add them to the denominator of the metric.

Metrics – Informatics

Can you add dates in the script? Start Date will be beginning and End Date = 12/31/2018?

They will be updated for the 2019 reporting period.

Can you tell me if the metrics will be looking at all patients (inpatient and outpatient) or are there specific guidelines for this?

The metric includes all patients in the research data warehouse– inpatient, outpatient, ER, etc.

Do the PCORNET scripts work with the version 4.1?

Yes. If you experience any problems with the script, please contact common_metrics@clic-ctsa.org.

Do the scripts provide the min/max dates to use for your data inclusion table?

For the launch of the metric the script did not include min/max dates. The scripts will be updated throughout the life of the metric. Check the Github site for the most current script.

Does free text include imaging notes (narratives/impressions) or just clinical progress notes and discharge summaries etc.?

No, only notes: admission, progress, discharge, procedure, etc.

How does a hub access a script?

The scripts are available on Github: https://github.com/ncats/CTSA-Metrics.

How much of the long-term plans should be included in the TTC plan?

Typically the TTC plan is for one year. If you are entering longer-term goals, please identify that the strategies will be addressed over the next ## years.

ICD 9/10 is listed for two domains, why is that?

ICD 9/10 is used in healthcare to code both diagnoses and procedures – be sure to enter data under the appropriate performance measure.

If our numbers are near 99% for each of these measures, what would you like to see us include in our TTC since there isn't much room to improve?

For hubs who have achieved, 100% or near 100%, please document the strategies that have helped your hub to reach that target.
Also, you may want to begin to identify additional metrics that would be helpful consortium-wide.

Incomplete data in the EHR can be outside the control of our CTSA. How can we remedy this issue without having the necessary authority?

Please enter this information in the Story Behind the Curve section of your TTC plan.

Is Postgres under consideration as a supported DBMS?

Hubs are welcome to use Postgres (or another data model). Hubs will need to create their own query and make sure that they match the measures of the existing script. If a hub creates a new query, please considering putting it in Github so that other hubs can use it. [SQLrender (https://github.com/OHDSI/SqlRender) allows OMOP sites to run against several other back end DBs beyond MS SQLserver and ORACLE: PostgresSql, Microsoft Parallel Data Warehouse (a.k.a. Analytics Platform System), Amazon Redshift, Apache Impala, Google BigQuery, BM Netezza.] At this time, scripts are available for these data models: OMOP, PCORnet, and i2b2/ACT. Hubs that use TriNetX, can contact TriNetX to get data.

Is the Observations domain a simple nominal level count of everything in the warehouse for that domain?

Regarding observations, the scripts are looking for a general information on if observations are being recorded at all (present/absent) in hoped that future research data warehouse characterizations could include vital signs.

Our research Informatics team has little control over what data points are filtered to the warehouse – we are at the mercy of the health system’s IT efforts.

That is not an uncommon situation. Please collect the data that you can and then list this issue under the Negative: section of your Story Behind the Curve (in your TTC plan). For your TTC plan, consider partnering with your IT team and determine if there are strategies that could be implemented to improve your access to the data.

Regarding gender, is patient reported sex counted as administrative sex?

Administrative gender is a person’s gender (from HL7 and OMB), which describes their sex. It’s a standard dataset (M or F). The domain reflects if it is recorded as a count.

Regarding the unique patients that are included in the denominator: Would the script address patients who have died or are no longer our patients? Should we be considering that for this metric at all?

For the Informatics metric launch, it’s the percentage of people who had those values at the time they were being seen.

The TTC plan entered for August 30, 2019, is a plan for which year? When do we report on progress afterwards?

The TTC plan entered for the August 30th deadline, represents the hub’s strategies for addressing how they will turn the curve in the coming year. TTC plans are entered annually, so progress is represented by comparing progress since the previous year.

We are a multi-site CTSA, and we can only access one site. Is there a way to indicate that?

Collect the data that you have available. If you can collect data from more sites next year it will make your data look even better.

We don’t have direct influence over the data domains in the Informatics CM. Are you looking for something different in this CM than in the other Common Metrics?

The focus of the Informatics metric is the presence or absence of data in the research data warehouse (RDW). The TTC plan should be focused on how to get absent data, or how to improve accessing data.

We filter our patient table to only include patients that have visit data in our location because we have billing data from several hospitals where our faculty work, but no actual visit data for those patients – no other information is available.

Include information that you think would be most helpful for investigators across the consortium.