Slide 10
Slide 10 text
/21
ॱংྨͷࢦඪ ͕ຬ͖ͨ͢ੑ࣭
Eff
: system output, : ground truth, : data
• Ordinal Invariance
where : strictly increasing func.
• Ordinal Monotonicity
if
• Class Imbalance
where
s g d ∈ D
Eff(s, g) = Eff(f(s), f(g)) f
Eff(s′ , g) > Eff(s, g) ∃d . (s(d) ≠ s′ (d)) ∧ (∀d . ((s(d) > s′ (d) ≥ g(d)) ∨ (s(d) = s′ (d))))
Eff(gd1
→c2
, g) > Eff(gd3
→c2
, g) nc1
> nc3
10
ਤ https://www.aclweb.org/anthology/2020.acl-main.363/ ΑΓҾ༻
lly, at interval scale, CEMINT would be
lent to a logarithmic version of MAE when-
ems are uniformly distributed across classes.
eave a more detailed formal and empirical
s of CEM at other scales for future work, as
t the primary scope of this paper.
heoretical Evidence
ing a methodology previously applied for
fication (Sebastiani, 2015; Sokolova, 2006),
ing (Dom, 2001; Meila, 2003; Amigó et al.,
and document ranking tasks (Moffat, 2013;
et al., 2013b), here we define a formal
work for OC via desirable properties to be
d, which are illustrated in Figure 2 and in-
ed below.
Metric Properties
st property states that an effectiveness met-
f(s, g) should not assume predefined inter-
etween classes, i.e., it should be invariant
permissible transformation functions at ordi-
le.
Figure 2: Illustration of desirable formal properties for
Ordinal Classification. Each bin is a system output,
where columns represent ordered classes assigned by
the system, and colors represent the items’ true classes,
ordered from black to white. "=" means that both out-
puts should have the same quality, and ">" that the left
output should receive a higher metric value than the
right output.
strictly better, then the metric score of s0 must be
higher.
Finally, in order to manage the effect of im-
balanced data sets, another desirable property is
that an item classification error in a frequent class
should have less effect than a classification error
e, CEMINT would be
version of MAE when-
tributed across classes.
d formal and empirical
ales for future work, as
f this paper.
e
previously applied for
2015; Sokolova, 2006),
ila, 2003; Amigó et al.,
ng tasks (Moffat, 2013;
e we define a formal
irable properties to be
ted in Figure 2 and in-
Figure 2: Illustration of desirable formal properties for
Ordinal Classification. Each bin is a system output,
where columns represent ordered classes assigned by
the system, and colors represent the items’ true classes,
ordered from black to white. "=" means that both out-
puts should have the same quality, and ">" that the left
output should receive a higher metric value than the
right output.
cale, CEMINT would be
mic version of MAE when-
distributed across classes.
iled formal and empirical
r scales for future work, as
pe of this paper.
ence
gy previously applied for
ni, 2015; Sokolova, 2006),
Meila, 2003; Amigó et al.,
nking tasks (Moffat, 2013;
here we define a formal
desirable properties to be
Figure 2: Illustration of desirable formal properties for
Ordinal Classification. Each bin is a system output,
where columns represent ordered classes assigned by
the system, and colors represent the items’ true classes,
ordered from black to white. "=" means that both out-
puts should have the same quality, and ">" that the left