Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Man vs. Machine

Kenn White
December 19, 2013

Man vs. Machine

An comparison of a leading fully-automated ECG waveform component algorithm vs. computer-assisted expert analysis. Barbey, JT, White, KV, Pezzullo, JC, Affrime, M. Man vs. Machine: Are Cardiac Core Labs still Relevant? (2011). Journal of Clinical Pharmacology, 51:1343.

Kenn White

December 19, 2013
Tweet

More Decks by Kenn White

Other Decks in Science

Transcript

  1. Barbey,
JT,
White,
KV,
Pezzullo,
JC,
Affrime,
M.
(2011)
Man
vs.
Machine:
Are
Cardiac
Core
Labs
still
Relevant?.

    American
College
of
Clinical
Pharmacology
Annual
Meeting.
J
Clin
Pharmacol,
51:1343.


    MAN
vs.
MACHINE:
ARE
CARDIAC
CORE
LABS
STILL
RELEVANT?

    J
T
Barbey,1
K
V
White,1
J
C
Pezzullo,1
M
Affrime,2

    1Social
&
Scientific
Systems,
Inc.,
Silver
Spring,
MD

    2Icon
Development
Solutions,
Ellicott
City,
MD

    


    ABSTRACT:

    Background:
Most
ECGs
are
recorded
using
digital
equipment
that
can
provide
an

    automated
 assessment
 of
 heart‐rate
 corrected
 (QTc)
 interval.
 Several

    manufacturers
 have
 developed
 new
 enhanced
 QTc
 algorithms
 but
 core
 lab‐read

    ECGs
are
still
preferred
in
the
context
of
thorough
QT
studies.
The
objective
of
this

    study
was
to
compare
the
unedited
Fridericia‐corrected
QT
(QTcF)
values
generated

    by
 Mortara’s
 VERITAS
 algorithms
 with
 those
 generated
 by
 a
 core
 lab‐based,

    computer
assisted,
manual
measurement
method
(HeartSignalsTM,
Social
&
Scientific

    Systems
Inc)
using
a
single
over‐reading
cardiologist.

    Methods:
 ECGs
 were
 acquired
 using
 the
 Surveyor
 telemetry
 central
 system

    (Mortara
 Instruments)
 during
 a
 prospective,
 single‐blind,
 placebo‐controlled,

    randomized,
 crossover
 study
 of
 moxifloxacin
 in
 24
 healthy
 volunteers.
 Maximum

    mean
 placebo‐corrected
 change
 from
 baseline
 QTcF
 (∆∆QTcF)
 was
 estimated
 by

    repeated
measures
analysis
of
averaged
replicates
for
each
method.

    Results:
 ECGs
 from
 23
 completed
 subjects
 at
 3
 pre‐dose
 and
 12
 post‐dose
 time

    points
during
the
2
treatment
periods
resulted
in
2028
analyzable
tracings,
which

    were
 assessed
 independently
 by
 each
 method.
 Adequacy
 of
 the
 study
 sensitivity

    (lower
bound
of
the
95%
confidence
interval
of
∆∆QTcF
>5
ms)
was
confirmed
by

    the
manual
method
at
3/12
and
by
the
VERITAS
algorithms
at
7/12
post‐dose
time

    points.
 The
 maximal
 mean
 ∆∆QTcF
 and
 90%
 CIs
 were
 12.3
 (9.1‐15.5)
 and
 18.1

    (12.6‐23.7)
ms
respectively,
occurring
3
hours
post‐dose
for
both
methods.
Intra‐
    triad
 QTcF
 variability,
 was
 7.9
 ms
 for
 the
 manual
 method
 and
 9.7
 ms
 for
 the

    VERITAS
 algorithms
 (P<0.001)
 The
 increased
 variability
 of
 the
 VERITAS

    measurements
 resulted
 from
 a
 number
 of
 excessively
 short
 QT
 measurements
 in

    several
different
subjects.

    Conclusion:
 In
 this
 population
 of
 normal
 volunteers,
 unedited
 automated
 QTcF

    values
generated
by
the
VERITAS
algorithms
showed
greater
variability
and
yielded

    a
markedly
larger
moxifloxacin
effect
profile
than
those
generated
by
a
computer

    assisted
manual
method
using
a
single
over‐reading
cardiologist


    View full-size slide

  2. INTRODUCTION:

    The
techniques
currently
in
use
for
the
measurement
of
ECG
intervals
can
be
classified

    into
three
broad
categories:
fully
manual,
computer
assisted
with
manual
adjudication,

    and
 fully
 automated.
 The
 ICH
 E14
 guidance
 recommends
 either
 fully
 manual
 or

    computer
 assisted
 manual
 adjudication
 approaches
 for
 clinical
 trials
 in
 which
 the

    assessment
 of
 ECG
 safety
 is
 an
 important
 objective,
 such
 as
 the
 Thorough
 QT/QTc

    study.
 While
 concern
 remains
 that
 fully
 automated
 reading
 methods
 can
 yield

    misleading
 results
 in
 the
 presence
 of
 artifact,
 flat
 T
 waves
 or
 irregular
 rhythm,
 an

    increasing
number
of
investigators
have
reported
that
the
newer
improved
automatic

    algorithms
can
match
and
even
surpass
the
precision
of
core
lab
manual
or
computer

    assisted
 measurements,
 particularly
 in
 the
 context
 of
 studies
 conducted
 normal

    volunteers.1



    Our
core
lab
recently
participated
in
a
prospective,
single‐center,
single‐blind,
placebo‐
    controlled,
2‐period,
crossover
study
to
assess
the
effect
of
moxifloxacin
400
mg
on
the

    QTc
 interval
 comparing
 the
 performance
 of
 our
 computer
 assisted
 manual
 approach

    (HeartSignalsTM,
 Social
 &
 Scientific
 Systems,
 Inc.)
 with
 that
 of
 an
 automated
 system

    (QTinno,
NewCardio,
Inc).2
Both
the
manual
and
automated
approaches
were
successful

    in
demonstrating
assay
sensitivity
for
a
positive
control
and
each
demonstrated
a
high

    and
 comparable
 degree
 of
 precision.
 Since
 the
 Surveyor
 Telemetry
 Central
 System

    (Mortara
 Instrument,
 Inc)
 had
 been
 used
 to
 collect
 the
 raw
 12‐lead
 ECG
 waveforms

    analyzed
 in
 the
 original
 study,
 we
 decided
 to
 compare
 our
 QTcF
 values
 to
 those

    generated
by
the
Mortara
VERITAS
algorithm.

    METHODS:
    Study Design: This was a single-center, single-blind, placebo-controlled, 2-period,
    crossover study initially designed to assess the effect of moxifloxacin 400 mg on the QTc
    interval using our computer-assisted, manual approach with 100% cardiologist
    adjudication (HeartSignalsTM, Social & Scientific Systems, Inc.) and a fully automated
    system (QTinno, NewCardio, Inc). In the current study, we compared the QTcF values
    generated by HeartSignals with those generated by the VERITAS algorithm using the
    matching ECGs from our original analysis.
    Subjects participated in 2 treatment periods separated by a 1-week washout. Subjects
    received moxifloxacin 400 mg or placebo after an 8-hr fast. Subjects had baseline ECGs
    performed 5 times over 5 minutes at each of 3 pre-dose time points (0.75, 0.5, and 0.25
    hr) and then 5 times at each of 12 time points from 0.5 hr through 24 hr post-dose.
    QT Measurements and Quality Control Process (QC): The Surveyor Telemetry
    Central System was used to collect the raw 12-lead ECG waveforms for analysis. To
    produce the original manual data set, a single cardiologist blinded to the treatment
    sequence and time order of the ECGs measured on a magnified screen the QT and RR of
    the 2nd through 4th replicate from each time point. Based on objective criteria, the 1st or
    5th replicate was used in place of another, if fewer than 3 optimal replicates were

    View full-size slide

  3. reported after the initial read. In addition, the ECGs with the longest QTcF values, the
    shortest QTcF values and a random sample of tracings with normal QTcF values were
    reviewed and corrected if needed. To produce the machine data set, QT, RR and QTcF
    values generated by the VERITAS algorithm were extracted from the exact tracings used
    for the manual analysis. The original unedited machine ECGs then underwent a QC
    process comparable to the one used for manually measured ECGs, generating a second
    “edited” machine QTcF dataset. No corrections were made to the VERITAS QT
    measurements but tracings with marking errors of >40 ms were excluded. The QTcF of
    each replicate was calculated from QT and RR. Intervals of each method were
    summarized by the arithmetic mean for data listings, summary, and analysis.
    Statistical Methods: The following analyses were performed separately using the
    unedited or edited, manual and machine generated QTcF data sets as indicated.
    To evaluate assay sensitivity, placebo corrected QTcF changes from baseline (ΔQTcF)
    were calculated using the mean arithmetic differences in QTcF. The arithmetic difference
    approach was preferred over the mixed model analysis of variance approach to allow for
    analysis of the unedited and edited datasets by a single approach. Assay sensitivity was
    considered adequate if the lower bound of at least one 95% one-sided confidence interval
    for the mean placebo-corrected ΔQTcF was greater than 5 msec and occurred at a
    biologically plausible time.
    An
estimate
of
intra‐triad
variability
of
single
QTcF
measurements
was
calculated

    for
each
data
set
and
used
to
characterize
precision
of
the
respective
ECG

    measurement
approaches
and
data
sets.


    RESULTS:
    QT data from 23 completed subjects at 30 time points during the 2 treatment periods
    resulted in 2028 valid, QC’d replicates by the manual approach. The initial unedited
    machine data set including the same 2028 ECGs showed significantly greater intra-triad
    variability than the manually measured data set (9.7 ms vs 7.9 ms, P<.001) This appeared
    due, in part at least, to implausibly short QTcF values as shown in figure 1.

    View full-size slide

  4. QC review of the machine read ECGs led to the deletion of 116 of 127 ECGs with a
    QTcF<370 ms, 1 of 94 randomly chosen ECGs with a normal QTcF and 1 of 85 tracings
    with a QTcF >450 ms resulting in a new “edited” machine dataset with 1910 tracings.
    See figures 2a, 2b and 2c for examples of accepted and rejected measurements.
    FIGURE 2a. Accepted machine QT measurement (accurate)

    View full-size slide

  5. FIGURE 2b Rejected machine QT measurement (too short)
    FIGURE 2c Rejected machine QT measurement (too long)
    As shown on figure 3, the QC process eliminated many of the excessively short QTcF
    values measured by the machine. Intra triad variability of the machine generated values
    after QC decreased from 9.7 to 8.3 msec but remained significantly greater than that of
    the matching manually read ECGs (8.3 msec vs. 7.8 msec, P<0.002).

    View full-size slide

  6. Although applied to a set of ECGs successfully read using the manual approach, the
    unedited machine read QTcF values produced a suboptimal QTcF data set, characterized
    by unacceptably high intra-triad variability and several implausibly short QTcF
    measurements. Adequacy of the study sensitivity for the unedited data set(lower bound of
    the one-sided 95% confidence interval of ΔQTcF > 5 msec) was confirmed at only one
    time point using the arithmetic differences approach. The maximal mean ΔQTcF was
    15.3 (6.5-24.1) msec and occurred at 3 hrs. (Figure 4a) Assay sensitivity was
    demonstrated at 7 time points using a mixed model of variance analysis, with positive
    values persisting beyond the expected decay curve for moxifloxacin. (Data not shown)
    The QC process produced an edited machine generated QTcF data set with fewer ECGs
    and lower intra triad variability. Adequacy of the study sensitivity for this edited data set
    (lower bound of the one-sided 95% confidence interval of ΔQTcF greater than 5 msec)
    was confirmed at 6 time point using the arithmetic differences approach. The maximal
    mean ΔQTcF was 16.7 (11.3-22.1)-24.1) msec and occurred at 3 hrs. (Figure 4b)
    Assay sensitivity was demonstrated at a 3 time points by the manual approach whether by
    arithmetic differences or mixed model of variance analysis. The maximal mean ∆QTcF
    by manual measurements was 12.2 (9.1-15.3) msec and occurred at 3 hrs as well. (Figure
    4b)

    View full-size slide

  7. The summary of QTcF measured by machine (edited) vs. QTcF measured by man
    differences suggests that intervals by the machine (edited) method were, on average,
    about 12 msec shorter than the intervals measured by man, with a standard deviation of
    +/- 15 msec.
    Mean ΔQTcF following moxifloxacin was typically 3.5 msec higher for edited machine
    reads than for man measured ECGs, while ΔQTcF following placebo was not
    systematically different between man and edited machine reads.
    DISCUSSION AND CONCLUSIONS:
    This study was designed to evaluate the performance of a commercially available
    automated ECG algorithm (VERITAS) when measuring QTcF on a set of good quality
    ECGs acquired in a population of normal volunteers receiving moxifloxacin and placebo
    in a crossover design. Compared with the values generated by a 100% manually
    adjudicated computer assisted system, the unedited data generated by the automated
    algorithm showed significantly greater intra triad QTcF variability due predominantly to
    erroneously short QT measurements. After completion of a QC process involving review
    of short and long QTcF outliers as well as a random sample of tracings with normal
    QTcF, a second edited machine-generated data set excluding markedly incorrectly
    measured tracings was produced. This second, smaller data set showed lower intra-triad
    variability still however greater than the matching man-generated data set. Although
    mean machine-measured QTcFs were shorter than those measured by the manual method,
    the machine-generated moxifloxacin effect profile was more pronounced than the profile

    View full-size slide

  8. based on manual reads, a difference that has been reported with other automated reading
    systems1,2.
    In summary unedited automated QTcF values generated by the VERITAS algorithm
    showed greater intra triad variability and yielded a suboptimal moxifloxacin effect profile
    compared with the data generated from the same ECGs by a computer assisted, 100%
    manually adjudicated approach. Extensive QC and deletion of markedly incorrectly
    measured ECGs, increased the precision of the machine reads but still did not match the
    precision of the computer assisted 100% manually adjudicated system. These findings
    apply to the VERITAS automated algorithm and the HeartSignalsTM computer assisted
    100% manually adjudicated approach and may not reflect the performance of other
    automated and computer assisted systems.
    REFERENCES:
    1. Couderc J C, Garnett C, Li C, Handzel R, McNitt S, Xia X, Polonsky S, Zareba W.
    Highly Automated QT Measurement Techniques in 7 Thorough QT Studies Implemented
    under ICH E14 Guidelines Ann Noninvasive Electrocardiol 2011;16(1):13–24
    2. Ruff D, Connolly M, Brueckner R, Bynum L, Beck D, Gussak I, Barbey J T, White K,
    Krantz M J, Affrime JM A prospective, single-blind, placebo-controlled, randomized,
    crossover study to assess the performance of automated and manual methodologies for
    detecting QTCc interval prolongation. Clin Pharmacol Ther . 2011; 89:s15

    View full-size slide