An comparison of a leading fully-automated ECG waveform component algorithm vs. computer-assisted expert analysis. Barbey, JT, White, KV, Pezzullo, JC, Affrime, M. Man vs. Machine: Are Cardiac Core Labs still Relevant? (2011). Journal of Clinical Pharmacology, 51:1343.
American College of Clinical Pharmacology Annual Meeting. J Clin Pharmacol, 51:1343.
MAN vs. MACHINE: ARE CARDIAC CORE LABS STILL RELEVANT?
J T Barbey,1 K V White,1 J C Pezzullo,1 M Affrime,2
1Social & Scientific Systems, Inc., Silver Spring, MD
2Icon Development Solutions, Ellicott City, MD
Background: Most ECGs are recorded using digital equipment that can provide an
automated assessment of heart‐rate corrected (QTc) interval. Several
manufacturers have developed new enhanced QTc algorithms but core lab‐read
ECGs are still preferred in the context of thorough QT studies. The objective of this
study was to compare the unedited Fridericia‐corrected QT (QTcF) values generated
by Mortara’s VERITAS algorithms with those generated by a core lab‐based,
computer assisted, manual measurement method (HeartSignalsTM, Social & Scientific
Systems Inc) using a single over‐reading cardiologist.
Methods: ECGs were acquired using the Surveyor telemetry central system
(Mortara Instruments) during a prospective, single‐blind, placebo‐controlled,
randomized, crossover study of moxifloxacin in 24 healthy volunteers. Maximum
mean placebo‐corrected change from baseline QTcF (∆∆QTcF) was estimated by
repeated measures analysis of averaged replicates for each method.
Results: ECGs from 23 completed subjects at 3 pre‐dose and 12 post‐dose time
points during the 2 treatment periods resulted in 2028 analyzable tracings, which
were assessed independently by each method. Adequacy of the study sensitivity
(lower bound of the 95% confidence interval of ∆∆QTcF >5 ms) was confirmed by
the manual method at 3/12 and by the VERITAS algorithms at 7/12 post‐dose time
points. The maximal mean ∆∆QTcF and 90% CIs were 12.3 (9.1‐15.5) and 18.1
(12.6‐23.7) ms respectively, occurring 3 hours post‐dose for both methods. Intra‐
triad QTcF variability, was 7.9 ms for the manual method and 9.7 ms for the
VERITAS algorithms (P<0.001) The increased variability of the VERITAS
measurements resulted from a number of excessively short QT measurements in
several different subjects.
Conclusion: In this population of normal volunteers, unedited automated QTcF
values generated by the VERITAS algorithms showed greater variability and yielded
a markedly larger moxifloxacin effect profile than those generated by a computer
assisted manual method using a single over‐reading cardiologist
The techniques currently in use for the measurement of ECG intervals can be classified
into three broad categories: fully manual, computer assisted with manual adjudication,
and fully automated. The ICH E14 guidance recommends either fully manual or
computer assisted manual adjudication approaches for clinical trials in which the
assessment of ECG safety is an important objective, such as the Thorough QT/QTc
study. While concern remains that fully automated reading methods can yield
misleading results in the presence of artifact, flat T waves or irregular rhythm, an
increasing number of investigators have reported that the newer improved automatic
algorithms can match and even surpass the precision of core lab manual or computer
assisted measurements, particularly in the context of studies conducted normal
Our core lab recently participated in a prospective, single‐center, single‐blind, placebo‐
controlled, 2‐period, crossover study to assess the effect of moxifloxacin 400 mg on the
QTc interval comparing the performance of our computer assisted manual approach
(HeartSignalsTM, Social & Scientific Systems, Inc.) with that of an automated system
(QTinno, NewCardio, Inc).2 Both the manual and automated approaches were successful
in demonstrating assay sensitivity for a positive control and each demonstrated a high
and comparable degree of precision. Since the Surveyor Telemetry Central System
(Mortara Instrument, Inc) had been used to collect the raw 12‐lead ECG waveforms
analyzed in the original study, we decided to compare our QTcF values to those
generated by the Mortara VERITAS algorithm.
Study Design: This was a single-center, single-blind, placebo-controlled, 2-period,
crossover study initially designed to assess the effect of moxifloxacin 400 mg on the QTc
interval using our computer-assisted, manual approach with 100% cardiologist
adjudication (HeartSignalsTM, Social & Scientific Systems, Inc.) and a fully automated
system (QTinno, NewCardio, Inc). In the current study, we compared the QTcF values
generated by HeartSignals with those generated by the VERITAS algorithm using the
matching ECGs from our original analysis.
Subjects participated in 2 treatment periods separated by a 1-week washout. Subjects
received moxifloxacin 400 mg or placebo after an 8-hr fast. Subjects had baseline ECGs
performed 5 times over 5 minutes at each of 3 pre-dose time points (0.75, 0.5, and 0.25
hr) and then 5 times at each of 12 time points from 0.5 hr through 24 hr post-dose.
QT Measurements and Quality Control Process (QC): The Surveyor Telemetry
Central System was used to collect the raw 12-lead ECG waveforms for analysis. To
produce the original manual data set, a single cardiologist blinded to the treatment
sequence and time order of the ECGs measured on a magnified screen the QT and RR of
the 2nd through 4th replicate from each time point. Based on objective criteria, the 1st or
5th replicate was used in place of another, if fewer than 3 optimal replicates were
reported after the initial read. In addition, the ECGs with the longest QTcF values, the
shortest QTcF values and a random sample of tracings with normal QTcF values were
reviewed and corrected if needed. To produce the machine data set, QT, RR and QTcF
values generated by the VERITAS algorithm were extracted from the exact tracings used
for the manual analysis. The original unedited machine ECGs then underwent a QC
process comparable to the one used for manually measured ECGs, generating a second
“edited” machine QTcF dataset. No corrections were made to the VERITAS QT
measurements but tracings with marking errors of >40 ms were excluded. The QTcF of
each replicate was calculated from QT and RR. Intervals of each method were
summarized by the arithmetic mean for data listings, summary, and analysis.
Statistical Methods: The following analyses were performed separately using the
unedited or edited, manual and machine generated QTcF data sets as indicated.
To evaluate assay sensitivity, placebo corrected QTcF changes from baseline (ΔQTcF)
were calculated using the mean arithmetic differences in QTcF. The arithmetic difference
approach was preferred over the mixed model analysis of variance approach to allow for
analysis of the unedited and edited datasets by a single approach. Assay sensitivity was
considered adequate if the lower bound of at least one 95% one-sided confidence interval
for the mean placebo-corrected ΔQTcF was greater than 5 msec and occurred at a
biologically plausible time.
An estimate of intra‐triad variability of single QTcF measurements was calculated
for each data set and used to characterize precision of the respective ECG
measurement approaches and data sets.
QT data from 23 completed subjects at 30 time points during the 2 treatment periods
resulted in 2028 valid, QC’d replicates by the manual approach. The initial unedited
machine data set including the same 2028 ECGs showed significantly greater intra-triad
variability than the manually measured data set (9.7 ms vs 7.9 ms, P<.001) This appeared
due, in part at least, to implausibly short QTcF values as shown in figure 1.
QC review of the machine read ECGs led to the deletion of 116 of 127 ECGs with a
QTcF<370 ms, 1 of 94 randomly chosen ECGs with a normal QTcF and 1 of 85 tracings
with a QTcF >450 ms resulting in a new “edited” machine dataset with 1910 tracings.
See figures 2a, 2b and 2c for examples of accepted and rejected measurements.
FIGURE 2a. Accepted machine QT measurement (accurate)
FIGURE 2b Rejected machine QT measurement (too short)
FIGURE 2c Rejected machine QT measurement (too long)
As shown on figure 3, the QC process eliminated many of the excessively short QTcF
values measured by the machine. Intra triad variability of the machine generated values
after QC decreased from 9.7 to 8.3 msec but remained significantly greater than that of
the matching manually read ECGs (8.3 msec vs. 7.8 msec, P<0.002).
Although applied to a set of ECGs successfully read using the manual approach, the
unedited machine read QTcF values produced a suboptimal QTcF data set, characterized
by unacceptably high intra-triad variability and several implausibly short QTcF
measurements. Adequacy of the study sensitivity for the unedited data set(lower bound of
the one-sided 95% confidence interval of ΔQTcF > 5 msec) was confirmed at only one
time point using the arithmetic differences approach. The maximal mean ΔQTcF was
15.3 (6.5-24.1) msec and occurred at 3 hrs. (Figure 4a) Assay sensitivity was
demonstrated at 7 time points using a mixed model of variance analysis, with positive
values persisting beyond the expected decay curve for moxifloxacin. (Data not shown)
The QC process produced an edited machine generated QTcF data set with fewer ECGs
and lower intra triad variability. Adequacy of the study sensitivity for this edited data set
(lower bound of the one-sided 95% confidence interval of ΔQTcF greater than 5 msec)
was confirmed at 6 time point using the arithmetic differences approach. The maximal
mean ΔQTcF was 16.7 (11.3-22.1)-24.1) msec and occurred at 3 hrs. (Figure 4b)
Assay sensitivity was demonstrated at a 3 time points by the manual approach whether by
arithmetic differences or mixed model of variance analysis. The maximal mean ∆QTcF
by manual measurements was 12.2 (9.1-15.3) msec and occurred at 3 hrs as well. (Figure
The summary of QTcF measured by machine (edited) vs. QTcF measured by man
differences suggests that intervals by the machine (edited) method were, on average,
about 12 msec shorter than the intervals measured by man, with a standard deviation of
+/- 15 msec.
Mean ΔQTcF following moxifloxacin was typically 3.5 msec higher for edited machine
reads than for man measured ECGs, while ΔQTcF following placebo was not
systematically different between man and edited machine reads.
DISCUSSION AND CONCLUSIONS:
This study was designed to evaluate the performance of a commercially available
automated ECG algorithm (VERITAS) when measuring QTcF on a set of good quality
ECGs acquired in a population of normal volunteers receiving moxifloxacin and placebo
in a crossover design. Compared with the values generated by a 100% manually
adjudicated computer assisted system, the unedited data generated by the automated
algorithm showed significantly greater intra triad QTcF variability due predominantly to
erroneously short QT measurements. After completion of a QC process involving review
of short and long QTcF outliers as well as a random sample of tracings with normal
QTcF, a second edited machine-generated data set excluding markedly incorrectly
measured tracings was produced. This second, smaller data set showed lower intra-triad
variability still however greater than the matching man-generated data set. Although
mean machine-measured QTcFs were shorter than those measured by the manual method,
the machine-generated moxifloxacin effect profile was more pronounced than the profile
based on manual reads, a difference that has been reported with other automated reading
In summary unedited automated QTcF values generated by the VERITAS algorithm
showed greater intra triad variability and yielded a suboptimal moxifloxacin effect profile
compared with the data generated from the same ECGs by a computer assisted, 100%
manually adjudicated approach. Extensive QC and deletion of markedly incorrectly
measured ECGs, increased the precision of the machine reads but still did not match the
precision of the computer assisted 100% manually adjudicated system. These findings
apply to the VERITAS automated algorithm and the HeartSignalsTM computer assisted
100% manually adjudicated approach and may not reflect the performance of other
automated and computer assisted systems.
1. Couderc J C, Garnett C, Li C, Handzel R, McNitt S, Xia X, Polonsky S, Zareba W.
Highly Automated QT Measurement Techniques in 7 Thorough QT Studies Implemented
under ICH E14 Guidelines Ann Noninvasive Electrocardiol 2011;16(1):13–24
2. Ruff D, Connolly M, Brueckner R, Bynum L, Beck D, Gussak I, Barbey J T, White K,
Krantz M J, Affrime JM A prospective, single-blind, placebo-controlled, randomized,
crossover study to assess the performance of automated and manual methodologies for
detecting QTCc interval prolongation. Clin Pharmacol Ther . 2011; 89:s15