Slide 58
Slide 58 text
Key question: Can we develop a theory to explain
when transfer learning works?
58
Target (Learn)
Source (Given)
Data
Model
Transferable
Knowledge
II. INTUITION
rstanding the performance behavior of configurable
e systems can enable (i) performance debugging, (ii)
mance tuning, (iii) design-time evolution, or (iv) runtime
on [11]. We lack empirical understanding of how the
mance behavior of a system will vary when the environ-
the system changes. Such empirical understanding will
important insights to develop faster and more accurate
g techniques that allow us to make predictions and
ations of performance for highly configurable systems
ging environments [10]. For instance, we can learn
mance behavior of a system on a cheap hardware in a
ed lab environment and use that to understand the per-
ce behavior of the system on a production server before
g to the end user. More specifically, we would like to
what the relationship is between the performance of a
in a specific environment (characterized by software
ration, hardware, workload, and system version) to the
t we vary its environmental conditions.
is research, we aim for an empirical understanding of
mance behavior to improve learning via an informed
g process. In other words, we at learning a perfor-
model in a changed environment based on a well-suited
g set that has been determined by the knowledge we
in other environments. Therefore, the main research
A. Preliminary concepts
In this section, we provide formal definitions of four con-
cepts that we use throughout this study. The formal notations
enable us to concisely convey concept throughout the paper.
1) Configuration and environment space: Let Fi
indicate
the i-th feature of a configurable system A which is either
enabled or disabled and one of them holds by default. The
configuration space is mathematically a Cartesian product of
all the features C = Dom(F1) ⇥ · · · ⇥ Dom(Fd), where
Dom(Fi) = {0, 1}. A configuration of a system is then
a member of the configuration space (feature space) where
all the parameters are assigned to a specific value in their
range (i.e., complete instantiations of the system’s parameters).
We also describe an environment instance by 3 variables
e = [w, h, v] drawn from a given environment space E =
W ⇥H ⇥V , where they respectively represent sets of possible
values for workload, hardware and system version.
2) Performance model: Given a software system A with
configuration space F and environmental instances E, a per-
formance model is a black-box function f : F ⇥ E ! R
given some observations of the system performance for each
combination of system’s features x 2 F in an environment
e 2 E. To construct a performance model for a system A
with configuration space F, we run A in environment instance
e 2 E on various combinations of configurations xi
2 F, and
record the resulting performance values yi = f(xi) + ✏i, xi
2
ON
behavior of configurable
erformance debugging, (ii)
e evolution, or (iv) runtime
understanding of how the
will vary when the environ-
mpirical understanding will
op faster and more accurate
to make predictions and
ighly configurable systems
or instance, we can learn
on a cheap hardware in a
that to understand the per-
a production server before
cifically, we would like to
ween the performance of a
(characterized by software
and system version) to the
conditions.
empirical understanding of
learning via an informed
we at learning a perfor-
ment based on a well-suited
ned by the knowledge we
erefore, the main research
A. Preliminary concepts
In this section, we provide formal definitions of four con-
cepts that we use throughout this study. The formal notations
enable us to concisely convey concept throughout the paper.
1) Configuration and environment space: Let Fi
indicate
the i-th feature of a configurable system A which is either
enabled or disabled and one of them holds by default. The
configuration space is mathematically a Cartesian product of
all the features C = Dom(F1) ⇥ · · · ⇥ Dom(Fd), where
Dom(Fi) = {0, 1}. A configuration of a system is then
a member of the configuration space (feature space) where
all the parameters are assigned to a specific value in their
range (i.e., complete instantiations of the system’s parameters).
We also describe an environment instance by 3 variables
e = [w, h, v] drawn from a given environment space E =
W ⇥H ⇥V , where they respectively represent sets of possible
values for workload, hardware and system version.
2) Performance model: Given a software system A with
configuration space F and environmental instances E, a per-
formance model is a black-box function f : F ⇥ E ! R
given some observations of the system performance for each
combination of system’s features x 2 F in an environment
e 2 E. To construct a performance model for a system A
with configuration space F, we run A in environment instance
e 2 E on various combinations of configurations xi
2 F, and
record the resulting performance values yi = f(xi) + ✏i, xi
2
oad, hardware and system version.
e model: Given a software system A with
ce F and environmental instances E, a per-
is a black-box function f : F ⇥ E ! R
rvations of the system performance for each
ystem’s features x 2 F in an environment
ruct a performance model for a system A
n space F, we run A in environment instance
combinations of configurations xi
2 F, and
ng performance values yi = f(xi) + ✏i, xi
2
(0, i). The training data for our regression
mply Dtr = {(xi, yi)}n
i=1
. In other words, a
is simply a mapping from the input space to
ormance metric that produces interval-scaled
ume it produces real numbers).
e distribution: For the performance model,
associated the performance response to each
w let introduce another concept where we
ment and we measure the performance. An
mance distribution is a stochastic process,
that defines a probability distribution over
sures for each environmental conditions. To
ormance distribution for a system A with
ce F, similarly to the process of deriving
models, we run A on various combinations
2 F, for a specific environment instance
values for workload, hardware and system version.
2) Performance model: Given a software system A with
configuration space F and environmental instances E, a per-
formance model is a black-box function f : F ⇥ E ! R
given some observations of the system performance for each
combination of system’s features x 2 F in an environment
e 2 E. To construct a performance model for a system A
with configuration space F, we run A in environment instance
e 2 E on various combinations of configurations xi
2 F, and
record the resulting performance values yi = f(xi) + ✏i, xi
2
F where ✏i
⇠ N (0, i). The training data for our regression
models is then simply Dtr = {(xi, yi)}n
i=1
. In other words, a
response function is simply a mapping from the input space to
a measurable performance metric that produces interval-scaled
data (here we assume it produces real numbers).
3) Performance distribution: For the performance model,
we measured and associated the performance response to each
configuration, now let introduce another concept where we
vary the environment and we measure the performance. An
empirical performance distribution is a stochastic process,
pd : E ! (R), that defines a probability distribution over
performance measures for each environmental conditions. To
construct a performance distribution for a system A with
configuration space F, similarly to the process of deriving
the performance models, we run A on various combinations
configurations xi
2 F, for a specific environment instance
Extract Reuse
Learn Learn
Q1: How source and target
are “related”?
Q2: What characteristics
are preserved?
Q3: What are the actionable
insights?