Man vs. Machine

An comparison of a leading fully-automated ECG waveform component algorithm vs. computer-assisted expert analysis. Barbey, JT, White, KV, Pezzullo, JC, Affrime, M. Man vs. Machine: Are Cardiac Core Labs still Relevant? (2011). Journal of Clinical Pharmacology, 51:1343.

Transcript

1. Barbey,
JT,
White,
KV,
Pezzullo,
JC,
Affrime,
M.
(2011)
Man
vs.
Machine:
Are
Cardiac
Core
Labs
still
Relevant?.
 American
College
of
Clinical
Pharmacology
Annual
Meeting.
J
Clin
Pharmacol,
51:1343.
 
 MAN
vs.
MACHINE:
ARE
CARDIAC
CORE
LABS
STILL
RELEVANT?
 J
T
Barbey,1
K
V
White,1
J
C
Pezzullo,1
M
Affrime,2
 1Social
&
Scientific
Systems,
Inc.,
Silver
Spring,
MD
 2Icon
Development
Solutions,
Ellicott
City,
MD
 

 ABSTRACT:
 Background:
Most
ECGs
are
recorded
using
digital
equipment
that
can
provide
an
 automated
 assessment
 of
 heart‐rate
 corrected
 (QTc)
 interval.
 Several
 manufacturers
 have
 developed
 new
 enhanced
 QTc
 algorithms
 but
 core
 lab‐read
 ECGs
are
still
preferred
in
the
context
of
thorough
QT
studies.
The
objective
of
this
 study
was
to
compare
the
unedited
Fridericia‐corrected
QT
(QTcF)
values
generated
 by
 Mortara’s
 VERITAS
 algorithms
 with
 those
 generated
 by
 a
 core
 lab‐based,
 computer
assisted,
manual
measurement
method
(HeartSignalsTM,
Social
&
Scientific
 Systems
Inc)
using
a
single
over‐reading
cardiologist.
 Methods:
 ECGs
 were
 acquired
 using
 the
 Surveyor
 telemetry
 central
 system
 (Mortara
 Instruments)
 during
 a
 prospective,
 single‐blind,
 placebo‐controlled,
 randomized,
 crossover
 study
 of
 moxifloxacin
 in
 24
 healthy
 volunteers.
 Maximum
 mean
 placebo‐corrected
 change
 from
 baseline
 QTcF
 (∆∆QTcF)
 was
 estimated
 by
 repeated
measures
analysis
of
averaged
replicates
for
each
method.
 Results:
 ECGs
 from
 23
 completed
 subjects
 at
 3
 pre‐dose
 and
 12
 post‐dose
 time
 points
during
the
2
treatment
periods
resulted
in
2028
analyzable
tracings,
which
 were
 assessed
 independently
 by
 each
 method.
 Adequacy
 of
 the
 study
 sensitivity
 (lower
bound
of
the
95%
confidence
interval
of
∆∆QTcF
>5
ms)
was
confirmed
by
 the
manual
method
at
3/12
and
by
the
VERITAS
algorithms
at
7/12
post‐dose
time
 points.
 The
 maximal
 mean
 ∆∆QTcF
 and
 90%
 CIs
 were
 12.3
 (9.1‐15.5)
 and
 18.1
 (12.6‐23.7)
ms
respectively,
occurring
3
hours
post‐dose
for
both
methods.
Intra‐ triad
 QTcF
 variability,
 was
 7.9
 ms
 for
 the
 manual
 method
 and
 9.7
 ms
 for
 the
 VERITAS
 algorithms
 (P<0.001)
 The
 increased
 variability
 of
 the
 VERITAS
 measurements
 resulted
 from
 a
 number
 of
 excessively
 short
 QT
 measurements
 in
 several
different
subjects.
 Conclusion:
 In
 this
 population
 of
 normal
 volunteers,
 unedited
 automated
 QTcF
 values
generated
by
the
VERITAS
algorithms
showed
greater
variability
and
yielded
 a
markedly
larger
moxifloxacin
effect
profile
than
those
generated
by
a
computer
 assisted
manual
method
using
a
single
over‐reading
cardiologist

2. INTRODUCTION:
 The
techniques
currently
in
use
for
the
measurement
of
ECG
intervals
can
be
classified
 into
three
broad
categories:
fully
manual,
computer
assisted
with
manual
adjudication,
 and
 fully
 automated.
 The
 ICH
 E14
 guidance
 recommends
 either
 fully
 manual
 or
 computer
 assisted
 manual
 adjudication
 approaches
 for
 clinical
 trials
 in
 which
 the
 assessment
 of
 ECG
 safety
 is
 an
 important
 objective,
 such
 as
 the
 Thorough
 QT/QTc
 study.
 While
 concern
 remains
 that
 fully
 automated
 reading
 methods
 can
 yield
 misleading
 results
 in
 the
 presence
 of
 artifact,
 flat
 T
 waves
 or
 irregular
 rhythm,
 an
 increasing
number
of
investigators
have
reported
that
the
newer
improved
automatic
 algorithms
can
match
and
even
surpass
the
precision
of
core
lab
manual
or
computer
 assisted
 measurements,
 particularly
 in
 the
 context
 of
 studies
 conducted
 normal
 volunteers.1

 
 Our
core
lab
recently
participated
in
a
prospective,
single‐center,
single‐blind,
placebo‐ controlled,
2‐period,
crossover
study
to
assess
the
effect
of
moxifloxacin
400
mg
on
the
 QTc
 interval
 comparing
 the
 performance
 of
 our
 computer
 assisted
 manual
 approach
 (HeartSignalsTM,
 Social
 &
 Scientific
 Systems,
 Inc.)
 with
 that
 of
 an
 automated
 system
 (QTinno,
NewCardio,
Inc).2
Both
the
manual
and
automated
approaches
were
successful
 in
demonstrating
assay
sensitivity
for
a
positive
control
and
each
demonstrated
a
high
 and
 comparable
 degree
 of
 precision.
 Since
 the
 Surveyor
 Telemetry
 Central
 System
 (Mortara
 Instrument,
 Inc)
 had
 been
 used
 to
 collect
 the
 raw
 12‐lead
 ECG
 waveforms
 analyzed
 in
 the
 original
 study,
 we
 decided
 to
 compare
 our
 QTcF
 values
 to
 those
 generated
by
the
Mortara
VERITAS
algorithm.
 METHODS: Study Design: This was a single-center, single-blind, placebo-controlled, 2-period, crossover study initially designed to assess the effect of moxifloxacin 400 mg on the QTc interval using our computer-assisted, manual approach with 100% cardiologist adjudication (HeartSignalsTM, Social & Scientific Systems, Inc.) and a fully automated system (QTinno, NewCardio, Inc). In the current study, we compared the QTcF values generated by HeartSignals with those generated by the VERITAS algorithm using the matching ECGs from our original analysis. Subjects participated in 2 treatment periods separated by a 1-week washout. Subjects received moxifloxacin 400 mg or placebo after an 8-hr fast. Subjects had baseline ECGs performed 5 times over 5 minutes at each of 3 pre-dose time points (0.75, 0.5, and 0.25 hr) and then 5 times at each of 12 time points from 0.5 hr through 24 hr post-dose. QT Measurements and Quality Control Process (QC): The Surveyor Telemetry Central System was used to collect the raw 12-lead ECG waveforms for analysis. To produce the original manual data set, a single cardiologist blinded to the treatment sequence and time order of the ECGs measured on a magnified screen the QT and RR of the 2nd through 4th replicate from each time point. Based on objective criteria, the 1st or 5th replicate was used in place of another, if fewer than 3 optimal replicates were
3. reported after the initial read. In addition, the ECGs with the longest QTcF values, the shortest QTcF values and a random sample of tracings with normal QTcF values were reviewed and corrected if needed. To produce the machine data set, QT, RR and QTcF values generated by the VERITAS algorithm were extracted from the exact tracings used for the manual analysis. The original unedited machine ECGs then underwent a QC process comparable to the one used for manually measured ECGs, generating a second “edited” machine QTcF dataset. No corrections were made to the VERITAS QT measurements but tracings with marking errors of >40 ms were excluded. The QTcF of each replicate was calculated from QT and RR. Intervals of each method were summarized by the arithmetic mean for data listings, summary, and analysis. Statistical Methods: The following analyses were performed separately using the unedited or edited, manual and machine generated QTcF data sets as indicated. To evaluate assay sensitivity, placebo corrected QTcF changes from baseline (ΔQTcF) were calculated using the mean arithmetic differences in QTcF. The arithmetic difference approach was preferred over the mixed model analysis of variance approach to allow for analysis of the unedited and edited datasets by a single approach. Assay sensitivity was considered adequate if the lower bound of at least one 95% one-sided confidence interval for the mean placebo-corrected ΔQTcF was greater than 5 msec and occurred at a biologically plausible time. An
estimate
of
intra‐triad
variability
of
single
QTcF
measurements
was
calculated
 for
each
data
set
and
used
to
characterize
precision
of
the
respective
ECG
 measurement
approaches
and
data
sets.

 RESULTS: QT data from 23 completed subjects at 30 time points during the 2 treatment periods resulted in 2028 valid, QC’d replicates by the manual approach. The initial unedited machine data set including the same 2028 ECGs showed significantly greater intra-triad variability than the manually measured data set (9.7 ms vs 7.9 ms, P<.001) This appeared due, in part at least, to implausibly short QTcF values as shown in figure 1.
4. QC review of the machine read ECGs led to the deletion of 116 of 127 ECGs with a QTcF<370 ms, 1 of 94 randomly chosen ECGs with a normal QTcF and 1 of 85 tracings with a QTcF >450 ms resulting in a new “edited” machine dataset with 1910 tracings. See figures 2a, 2b and 2c for examples of accepted and rejected measurements. FIGURE 2a. Accepted machine QT measurement (accurate)
5. FIGURE 2b Rejected machine QT measurement (too short) FIGURE 2c Rejected machine QT measurement (too long) As shown on figure 3, the QC process eliminated many of the excessively short QTcF values measured by the machine. Intra triad variability of the machine generated values after QC decreased from 9.7 to 8.3 msec but remained significantly greater than that of the matching manually read ECGs (8.3 msec vs. 7.8 msec, P<0.002).
6. Although applied to a set of ECGs successfully read using the manual approach, the unedited machine read QTcF values produced a suboptimal QTcF data set, characterized by unacceptably high intra-triad variability and several implausibly short QTcF measurements. Adequacy of the study sensitivity for the unedited data set(lower bound of the one-sided 95% confidence interval of ΔQTcF > 5 msec) was confirmed at only one time point using the arithmetic differences approach. The maximal mean ΔQTcF was 15.3 (6.5-24.1) msec and occurred at 3 hrs. (Figure 4a) Assay sensitivity was demonstrated at 7 time points using a mixed model of variance analysis, with positive values persisting beyond the expected decay curve for moxifloxacin. (Data not shown) The QC process produced an edited machine generated QTcF data set with fewer ECGs and lower intra triad variability. Adequacy of the study sensitivity for this edited data set (lower bound of the one-sided 95% confidence interval of ΔQTcF greater than 5 msec) was confirmed at 6 time point using the arithmetic differences approach. The maximal mean ΔQTcF was 16.7 (11.3-22.1)-24.1) msec and occurred at 3 hrs. (Figure 4b) Assay sensitivity was demonstrated at a 3 time points by the manual approach whether by arithmetic differences or mixed model of variance analysis. The maximal mean ∆QTcF by manual measurements was 12.2 (9.1-15.3) msec and occurred at 3 hrs as well. (Figure 4b)
8. The summary of QTcF measured by machine (edited) vs. QTcF measured by man differences suggests that intervals by the machine (edited) method were, on average, about 12 msec shorter than the intervals measured by man, with a standard deviation of +/- 15 msec. Mean ΔQTcF following moxifloxacin was typically 3.5 msec higher for edited machine reads than for man measured ECGs, while ΔQTcF following placebo was not systematically different between man and edited machine reads. DISCUSSION AND CONCLUSIONS: This study was designed to evaluate the performance of a commercially available automated ECG algorithm (VERITAS) when measuring QTcF on a set of good quality ECGs acquired in a population of normal volunteers receiving moxifloxacin and placebo in a crossover design. Compared with the values generated by a 100% manually adjudicated computer assisted system, the unedited data generated by the automated algorithm showed significantly greater intra triad QTcF variability due predominantly to erroneously short QT measurements. After completion of a QC process involving review of short and long QTcF outliers as well as a random sample of tracings with normal QTcF, a second edited machine-generated data set excluding markedly incorrectly measured tracings was produced. This second, smaller data set showed lower intra-triad variability still however greater than the matching man-generated data set. Although mean machine-measured QTcFs were shorter than those measured by the manual method, the machine-generated moxifloxacin effect profile was more pronounced than the profile
9. based on manual reads, a difference that has been reported with other automated reading systems1,2. In summary unedited automated QTcF values generated by the VERITAS algorithm showed greater intra triad variability and yielded a suboptimal moxifloxacin effect profile compared with the data generated from the same ECGs by a computer assisted, 100% manually adjudicated approach. Extensive QC and deletion of markedly incorrectly measured ECGs, increased the precision of the machine reads but still did not match the precision of the computer assisted 100% manually adjudicated system. These findings apply to the VERITAS automated algorithm and the HeartSignalsTM computer assisted 100% manually adjudicated approach and may not reflect the performance of other automated and computer assisted systems. REFERENCES: 1. Couderc J C, Garnett C, Li C, Handzel R, McNitt S, Xia X, Polonsky S, Zareba W. Highly Automated QT Measurement Techniques in 7 Thorough QT Studies Implemented under ICH E14 Guidelines Ann Noninvasive Electrocardiol 2011;16(1):13–24 2. Ruff D, Connolly M, Brueckner R, Bynum L, Beck D, Gussak I, Barbey J T, White K, Krantz M J, Affrime JM A prospective, single-blind, placebo-controlled, randomized, crossover study to assess the performance of automated and manual methodologies for detecting QTCc interval prolongation. Clin Pharmacol Ther . 2011; 89:s15