Man vs. Machine

An comparison of a leading fully-automated ECG waveform component algorithm vs. computer-assisted expert analysis. Barbey, JT, White, KV, Pezzullo, JC, Affrime, M. Man vs. Machine: Are Cardiac Core Labs still Relevant? (2011). Journal of Clinical Pharmacology, 51:1343.

Transcript

1. Barbey,
JT,
White,
KV,
Pezzullo,
JC,
Affrime,
M.
(2011)
Man
vs.
Machine:
Are
Cardiac
Core
Labs
still
Relevant?.

   American
College
of
Clinical
Pharmacology
Annual
Meeting.
J
Clin
Pharmacol,
51:1343.

   

   MAN
vs.
MACHINE:
ARE
CARDIAC
CORE
LABS
STILL
RELEVANT?

   J
T
Barbey,1
K
V
White,1
J
C
Pezzullo,1
M
Affrime,2

   1Social
&
Scientific
Systems,
Inc.,
Silver
Spring,
MD

   2Icon
Development
Solutions,
Ellicott
City,
MD

   


   ABSTRACT:

   Background:
Most
ECGs
are
recorded
using
digital
equipment
that
can
provide
an

   automated
 assessment
 of
 heart‐rate
 corrected
 (QTc)
 interval.
 Several

   manufacturers
 have
 developed
 new
 enhanced
 QTc
 algorithms
 but
 core
 lab‐read

   ECGs
are
still
preferred
in
the
context
of
thorough
QT
studies.
The
objective
of
this

   study
was
to
compare
the
unedited
Fridericia‐corrected
QT
(QTcF)
values
generated

   by
 Mortara’s
 VERITAS
 algorithms
 with
 those
 generated
 by
 a
 core
 lab‐based,

   computer
assisted,
manual
measurement
method
(HeartSignalsTM,
Social
&
Scientific

   Systems
Inc)
using
a
single
over‐reading
cardiologist.

   Methods:
 ECGs
 were
 acquired
 using
 the
 Surveyor
 telemetry
 central
 system

   (Mortara
 Instruments)
 during
 a
 prospective,
 single‐blind,
 placebo‐controlled,

   randomized,
 crossover
 study
 of
 moxifloxacin
 in
 24
 healthy
 volunteers.
 Maximum

   mean
 placebo‐corrected
 change
 from
 baseline
 QTcF
 (∆∆QTcF)
 was
 estimated
 by

   repeated
measures
analysis
of
averaged
replicates
for
each
method.

   Results:
 ECGs
 from
 23
 completed
 subjects
 at
 3
 pre‐dose
 and
 12
 post‐dose
 time

   points
during
the
2
treatment
periods
resulted
in
2028
analyzable
tracings,
which

   were
 assessed
 independently
 by
 each
 method.
 Adequacy
 of
 the
 study
 sensitivity

   (lower
bound
of
the
95%
confidence
interval
of
∆∆QTcF
>5
ms)
was
confirmed
by

   the
manual
method
at
3/12
and
by
the
VERITAS
algorithms
at
7/12
post‐dose
time

   points.
 The
 maximal
 mean
 ∆∆QTcF
 and
 90%
 CIs
 were
 12.3
 (9.1‐15.5)
 and
 18.1

   (12.6‐23.7)
ms
respectively,
occurring
3
hours
post‐dose
for
both
methods.
Intra‐
   triad
 QTcF
 variability,
 was
 7.9
 ms
 for
 the
 manual
 method
 and
 9.7
 ms
 for
 the

   VERITAS
 algorithms
 (P<0.001)
 The
 increased
 variability
 of
 the
 VERITAS

   measurements
 resulted
 from
 a
 number
 of
 excessively
 short
 QT
 measurements
 in

   several
different
subjects.

   Conclusion:
 In
 this
 population
 of
 normal
 volunteers,
 unedited
 automated
 QTcF

   values
generated
by
the
VERITAS
algorithms
showed
greater
variability
and
yielded

   a
markedly
larger
moxifloxacin
effect
profile
than
those
generated
by
a
computer

   assisted
manual
method
using
a
single
over‐reading
cardiologist

   

   View Slide

2. INTRODUCTION:

   The
techniques
currently
in
use
for
the
measurement
of
ECG
intervals
can
be
classified

   into
three
broad
categories:
fully
manual,
computer
assisted
with
manual
adjudication,

   and
 fully
 automated.
 The
 ICH
 E14
 guidance
 recommends
 either
 fully
 manual
 or

   computer
 assisted
 manual
 adjudication
 approaches
 for
 clinical
 trials
 in
 which
 the

   assessment
 of
 ECG
 safety
 is
 an
 important
 objective,
 such
 as
 the
 Thorough
 QT/QTc

   study.
 While
 concern
 remains
 that
 fully
 automated
 reading
 methods
 can
 yield

   misleading
 results
 in
 the
 presence
 of
 artifact,
 flat
 T
 waves
 or
 irregular
 rhythm,
 an

   increasing
number
of
investigators
have
reported
that
the
newer
improved
automatic

   algorithms
can
match
and
even
surpass
the
precision
of
core
lab
manual
or
computer

   assisted
 measurements,
 particularly
 in
 the
 context
 of
 studies
 conducted
 normal

   volunteers.1


   

   Our
core
lab
recently
participated
in
a
prospective,
single‐center,
single‐blind,
placebo‐
   controlled,
2‐period,
crossover
study
to
assess
the
effect
of
moxifloxacin
400
mg
on
the

   QTc
 interval
 comparing
 the
 performance
 of
 our
 computer
 assisted
 manual
 approach

   (HeartSignalsTM,
 Social
 &
 Scientific
 Systems,
 Inc.)
 with
 that
 of
 an
 automated
 system

   (QTinno,
NewCardio,
Inc).2
Both
the
manual
and
automated
approaches
were
successful

   in
demonstrating
assay
sensitivity
for
a
positive
control
and
each
demonstrated
a
high

   and
 comparable
 degree
 of
 precision.
 Since
 the
 Surveyor
 Telemetry
 Central
 System

   (Mortara
 Instrument,
 Inc)
 had
 been
 used
 to
 collect
 the
 raw
 12‐lead
 ECG
 waveforms

   analyzed
 in
 the
 original
 study,
 we
 decided
 to
 compare
 our
 QTcF
 values
 to
 those

   generated
by
the
Mortara
VERITAS
algorithm.

   METHODS:
   Study Design: This was a single-center, single-blind, placebo-controlled, 2-period,
   crossover study initially designed to assess the effect of moxifloxacin 400 mg on the QTc
   interval using our computer-assisted, manual approach with 100% cardiologist
   adjudication (HeartSignalsTM, Social & Scientific Systems, Inc.) and a fully automated
   system (QTinno, NewCardio, Inc). In the current study, we compared the QTcF values
   generated by HeartSignals with those generated by the VERITAS algorithm using the
   matching ECGs from our original analysis.
   Subjects participated in 2 treatment periods separated by a 1-week washout. Subjects
   received moxifloxacin 400 mg or placebo after an 8-hr fast. Subjects had baseline ECGs
   performed 5 times over 5 minutes at each of 3 pre-dose time points (0.75, 0.5, and 0.25
   hr) and then 5 times at each of 12 time points from 0.5 hr through 24 hr post-dose.
   QT Measurements and Quality Control Process (QC): The Surveyor Telemetry
   Central System was used to collect the raw 12-lead ECG waveforms for analysis. To
   produce the original manual data set, a single cardiologist blinded to the treatment
   sequence and time order of the ECGs measured on a magnified screen the QT and RR of
   the 2nd through 4th replicate from each time point. Based on objective criteria, the 1st or
   5th replicate was used in place of another, if fewer than 3 optimal replicates were
   

   View Slide

3. reported after the initial read. In addition, the ECGs with the longest QTcF values, the
   shortest QTcF values and a random sample of tracings with normal QTcF values were
   reviewed and corrected if needed. To produce the machine data set, QT, RR and QTcF
   values generated by the VERITAS algorithm were extracted from the exact tracings used
   for the manual analysis. The original unedited machine ECGs then underwent a QC
   process comparable to the one used for manually measured ECGs, generating a second
   “edited” machine QTcF dataset. No corrections were made to the VERITAS QT
   measurements but tracings with marking errors of >40 ms were excluded. The QTcF of
   each replicate was calculated from QT and RR. Intervals of each method were
   summarized by the arithmetic mean for data listings, summary, and analysis.
   Statistical Methods: The following analyses were performed separately using the
   unedited or edited, manual and machine generated QTcF data sets as indicated.
   To evaluate assay sensitivity, placebo corrected QTcF changes from baseline (ΔQTcF)
   were calculated using the mean arithmetic differences in QTcF. The arithmetic difference
   approach was preferred over the mixed model analysis of variance approach to allow for
   analysis of the unedited and edited datasets by a single approach. Assay sensitivity was
   considered adequate if the lower bound of at least one 95% one-sided confidence interval
   for the mean placebo-corrected ΔQTcF was greater than 5 msec and occurred at a
   biologically plausible time.
   An
estimate
of
intra‐triad
variability
of
single
QTcF
measurements
was
calculated

   for
each
data
set
and
used
to
characterize
precision
of
the
respective
ECG

   measurement
approaches
and
data
sets.


   RESULTS:
   QT data from 23 completed subjects at 30 time points during the 2 treatment periods
   resulted in 2028 valid, QC’d replicates by the manual approach. The initial unedited
   machine data set including the same 2028 ECGs showed significantly greater intra-triad
   variability than the manually measured data set (9.7 ms vs 7.9 ms, P<.001) This appeared
   due, in part at least, to implausibly short QTcF values as shown in figure 1.
   

   View Slide

4. QC review of the machine read ECGs led to the deletion of 116 of 127 ECGs with a
   QTcF<370 ms, 1 of 94 randomly chosen ECGs with a normal QTcF and 1 of 85 tracings
   with a QTcF >450 ms resulting in a new “edited” machine dataset with 1910 tracings.
   See figures 2a, 2b and 2c for examples of accepted and rejected measurements.
   FIGURE 2a. Accepted machine QT measurement (accurate)
   

   View Slide

5. FIGURE 2b Rejected machine QT measurement (too short)
   FIGURE 2c Rejected machine QT measurement (too long)
   As shown on figure 3, the QC process eliminated many of the excessively short QTcF
   values measured by the machine. Intra triad variability of the machine generated values
   after QC decreased from 9.7 to 8.3 msec but remained significantly greater than that of
   the matching manually read ECGs (8.3 msec vs. 7.8 msec, P<0.002).
   

   View Slide

6. Although applied to a set of ECGs successfully read using the manual approach, the
   unedited machine read QTcF values produced a suboptimal QTcF data set, characterized
   by unacceptably high intra-triad variability and several implausibly short QTcF
   measurements. Adequacy of the study sensitivity for the unedited data set(lower bound of
   the one-sided 95% confidence interval of ΔQTcF > 5 msec) was confirmed at only one
   time point using the arithmetic differences approach. The maximal mean ΔQTcF was
   15.3 (6.5-24.1) msec and occurred at 3 hrs. (Figure 4a) Assay sensitivity was
   demonstrated at 7 time points using a mixed model of variance analysis, with positive
   values persisting beyond the expected decay curve for moxifloxacin. (Data not shown)
   The QC process produced an edited machine generated QTcF data set with fewer ECGs
   and lower intra triad variability. Adequacy of the study sensitivity for this edited data set
   (lower bound of the one-sided 95% confidence interval of ΔQTcF greater than 5 msec)
   was confirmed at 6 time point using the arithmetic differences approach. The maximal
   mean ΔQTcF was 16.7 (11.3-22.1)-24.1) msec and occurred at 3 hrs. (Figure 4b)
   Assay sensitivity was demonstrated at a 3 time points by the manual approach whether by
   arithmetic differences or mixed model of variance analysis. The maximal mean ∆QTcF
   by manual measurements was 12.2 (9.1-15.3) msec and occurred at 3 hrs as well. (Figure
   4b)
   

   View Slide

7. View Slide

8. The summary of QTcF measured by machine (edited) vs. QTcF measured by man
   differences suggests that intervals by the machine (edited) method were, on average,
   about 12 msec shorter than the intervals measured by man, with a standard deviation of
   +/- 15 msec.
   Mean ΔQTcF following moxifloxacin was typically 3.5 msec higher for edited machine
   reads than for man measured ECGs, while ΔQTcF following placebo was not
   systematically different between man and edited machine reads.
   DISCUSSION AND CONCLUSIONS:
   This study was designed to evaluate the performance of a commercially available
   automated ECG algorithm (VERITAS) when measuring QTcF on a set of good quality
   ECGs acquired in a population of normal volunteers receiving moxifloxacin and placebo
   in a crossover design. Compared with the values generated by a 100% manually
   adjudicated computer assisted system, the unedited data generated by the automated
   algorithm showed significantly greater intra triad QTcF variability due predominantly to
   erroneously short QT measurements. After completion of a QC process involving review
   of short and long QTcF outliers as well as a random sample of tracings with normal
   QTcF, a second edited machine-generated data set excluding markedly incorrectly
   measured tracings was produced. This second, smaller data set showed lower intra-triad
   variability still however greater than the matching man-generated data set. Although
   mean machine-measured QTcFs were shorter than those measured by the manual method,
   the machine-generated moxifloxacin effect profile was more pronounced than the profile
   

   View Slide

9. based on manual reads, a difference that has been reported with other automated reading
   systems1,2.
   In summary unedited automated QTcF values generated by the VERITAS algorithm
   showed greater intra triad variability and yielded a suboptimal moxifloxacin effect profile
   compared with the data generated from the same ECGs by a computer assisted, 100%
   manually adjudicated approach. Extensive QC and deletion of markedly incorrectly
   measured ECGs, increased the precision of the machine reads but still did not match the
   precision of the computer assisted 100% manually adjudicated system. These findings
   apply to the VERITAS automated algorithm and the HeartSignalsTM computer assisted
   100% manually adjudicated approach and may not reflect the performance of other
   automated and computer assisted systems.
   REFERENCES:
   1. Couderc J C, Garnett C, Li C, Handzel R, McNitt S, Xia X, Polonsky S, Zareba W.
   Highly Automated QT Measurement Techniques in 7 Thorough QT Studies Implemented
   under ICH E14 Guidelines Ann Noninvasive Electrocardiol 2011;16(1):13–24
   2. Ruff D, Connolly M, Brueckner R, Bynum L, Beck D, Gussak I, Barbey J T, White K,
   Krantz M J, Affrime JM A prospective, single-blind, placebo-controlled, randomized,
   crossover study to assess the performance of automated and manual methodologies for
   detecting QTCc interval prolongation. Clin Pharmacol Ther . 2011; 89:s15
   

   View Slide