Slide 1
Slide 1 text
Datastore͔ΒSpannerʹ
Ҡߦ్͍ͨͧ͠தฤ
GCPUG Tokyo Spanner Day August 2021
Θ͔Ί ·͞ͻΖ
Slide 2
Slide 2 text
Θ͔Ί ·͞ͻΖ @v vakame
Masahiro Wakame
GAE/Go
GraphQL
ٕज़ॻయ
Slide 3
Slide 3 text
ຊͷςʔϚ
•ٕज़ॻయWebΛҠߦ͍ͨ͠ʂ
•Datastore→Spanner ࡞ۀதͰ͢
•ݱঢ়ͷݟ
͍ͭྃ͢Δ͔Θ͔ΒΜ…
Slide 4
Slide 4 text
Ϟνϕʔγϣϯ
•UNION ALLͱ͔͔ͨͬͨ͠
•ձ͕ࣾओʹSpannerͳͷͰ৮Γ͔ͨͬͨ
•ίετ͕ 1/10 ʹͳͬͨͷͰݱ࣮తʹ
ΫΤϦͷॊೈ͞ʹظ…ʂ
Slide 5
Slide 5 text
ΞʔΩςΫνϟ1
•AppEngine/Go 1.1
5
•Datastore → Spanne
r
•GraphQL (gqlgen)
GraphQLͷԸܙ͕݁ߏσΧ͍
Slide 6
Slide 6 text
ΞʔΩςΫνϟ2
1. API (GraphQL
)
2. Service (domain
)
3. Repository (DB)
LayeredͬΆ͍͚ͲશવͦΜͳ͜ͱͳ͍
Slide 7
Slide 7 text
جຊํ
•Datastore༻RepositoryΛࠩ͠ସ͑
•ڥมͰΓସ͑
•Spanner friendlyͳมߋೖΕͳ͍
·ͣΓ͖Γ͍ͨ
Slide 8
Slide 8 text
ਐḿ
•શRepositoryͷSpanner൛ͷ࡞
•Datastore,SpannerͰಉ͡ςετ͕௨ͬͨ
•ڥඋσʔλҠߦݕ౼͜Ε͔Β
ંΓฦͨ͠ײ͋Δ
Slide 9
Slide 9 text
Tips
ຊͷຊͰ͢
Slide 10
Slide 10 text
ओு
•OpenCensusͷTraceେࣄʂʂʂ
Slide 11
Slide 11 text
Spanner Emulator
•ศར CIͰͬͯ·͢
•ReadWriteTransaction͕ෳషΕͳ͍
•͜ͷ੍͕݁ߏΩπ͍ʂ
•spanner.Client#Apply ྫ֎͡Όͳ͍
ApplyAtLeastOnceͩΊ
Slide 12
Slide 12 text
Ͳ͏ͳΔʁ
•ΤϥʔʹͳΔ
•Timeout·ͰϦτϥΠ͢Δ
•ΞϓϦ͔ΒΤϥʔݟ͑ͳ͍
ී௨ʹ༹͚࣌ؒ·͢
Slide 13
Slide 13 text
ରࡦ
•GoogleOpenCensusͬͯΔ
•https://bit.ly/37qymo
W
•Open CensusͷTraceϩάΛݟΔͱ݁ߏ
৭ʑग़ͯΔ…ʂʂ
Spanner ClientͷίʔυࣗମࢀߟʹͳΓ·͢
Slide 14
Slide 14 text
ͬͯΈͨ
Traceࠐͷ࣮ߦ݁Ռ
2021-08-05T10:42:25.1409165Z === RUN TestSpanne
r
2021-08-05T10:42:25.2019180Z 2021/08/05 10:42:25 Duration: 68.2807ms Name: google.spanner.admin.database.v1.DatabaseAdmin.CreateDatabase Code: 0 Message: Attributes: map[Client:true
FailFast:true] Annotations: [
]
2021-08-05T10:42:25.2030092Z ParentSpanID: 66d0f65276a786dc SpanID: 41a185b1034c5e57 ChildSpanCount:
0
2021-08-05T10:42:25.2034069Z INFO: create spanner client with projects/tbf-test/instances/tbf-test-instance/databases/for-ut-a76
3
2021-08-05T10:42:25.2035431Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_utils.go:7
0
2021-08-05T10:42:25.2037531Z 2021/08/05 10:42:25 Duration: 179.701µs Name: cloud.google.com/go/spanner.NewClient Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2038616Z ParentSpanID: 66d0f65276a786dc SpanID: 1c72141091f035d2 ChildSpanCount:
0
2021-08-05T10:42:25.2050051Z 2021/08/05 10:42:25 Duration: 2.842616ms Name: google.spanner.v1.Spanner.BatchCreateSessions Code: 0 Message: Attributes: map[Client:true FailFast:true]
Annotations: [
]
2021-08-05T10:42:25.2051837Z ParentSpanID: d21332cdab39e5c7 SpanID: ade4c02b39debc42 ChildSpanCount:
0
2021-08-05T10:42:25.2054728Z 2021/08/05 10:42:25 Duration: 2.925417ms Name: cloud.google.com/go/spanner.BatchCreateSessions Code: 0 Message: Attributes: map[] Annotations:
[***Time:2021-08-05 10:42:25.201796419 +0000 UTC m=+157.694579279 Message:Creating a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204703536 +0000 UTC m=+157.697486496
Message:Received a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204718536 +0000 UTC m=+157.697501396 Message:Finished creating 2 sessions Attributes:map[]***
]
2021-08-05T10:42:25.2056889Z ParentSpanID: 0000000000000000 SpanID: d21332cdab39e5c7 ChildSpanCount:
1
2021-08-05T10:42:25.2058866Z 2021/08/05 10:42:25 Duration: 410.303µs Name: google.spanner.v1.Spanner.BeginTransaction Code: 0 Message: Attributes: map[Client:true FailFast:true]
Annotations: [
]
2021-08-05T10:42:25.2060645Z ParentSpanID: f742a36e1e950d4d SpanID: 88b54f8ac68294bd ChildSpanCount:
0
2021-08-05T10:42:25.2061510Z INFO: before ReadRo
w
2021-08-05T10:42:25.2062577Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2
0
2021-08-05T10:42:25.2064464Z 2021/08/05 10:42:25 Duration: 9.8µs Name: cloud.google.com/go/spanner.Read Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2065690Z ParentSpanID: f742a36e1e950d4d SpanID: 6386dee853276c38 ChildSpanCount:
1
2021-08-05T10:42:25.2067399Z 2021/08/05 10:42:25 Duration: 555.803µs Name: google.spanner.v1.Spanner.StreamingRead Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations:
[
]
2021-08-05T10:42:25.2069446Z ParentSpanID: 3e576d47e1cb43b3 SpanID: 1928fca56e701b2e ChildSpanCount:
0
2021-08-05T10:42:25.2071376Z 2021/08/05 10:42:25 Duration: 748.605µs Name: cloud.google.com/go/spanner.RowIterator Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2072630Z ParentSpanID: 6386dee853276c38 SpanID: 3e576d47e1cb43b3 ChildSpanCount:
1
2021-08-05T10:42:25.2074888Z WARNING: err: spanner: code = "NotFound", desc = "row not found(Table: Organization, PrimaryKey: (\"111\"))
"
2021-08-05T10:42:25.2076611Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2
9
2021-08-05T10:42:25.2078426Z INFO: before Apply with ApplyAtLeastOnc
e
2021-08-05T10:42:25.2081087Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:4
9
2021-08-05T10:42:25.2084536Z 2021/08/05 10:42:25 Duration: 533.803µs Name: google.spanner.v1.Spanner.Commit Code: 10 Message: Transaction 2 aborted due to active transaction 1. The emulator
only supports one transaction at a time. Some best practices to avoid ABORT errors in Cloud Spanner service are
:
2021-08-05T10:42:25.2087265Z 1. Avoid use of nested transactions
.
2021-08-05T10:42:25.2088255Z 2. Explicitly Rollback failed transactions
.
2021-08-05T10:42:25.2090414Z 3. All transactions should be running inside of retry loops
.
2021-08-05T10:42:25.2091644Z Attributes: map[Client:true FailFast:true] Annotations: [
]
2021-08-05T10:42:25.2092587Z ParentSpanID: f4f88a04fc14f3a8 SpanID: cfc9196389b9ca23 ChildSpanCount:
0
Slide 15
Slide 15 text
ͬͯΈͨ
ͦͯ͠Timeout…
2021-08-05T10:42:25.1409165Z === RUN TestSpanne
r
2021-08-05T10:42:25.2019180Z 2021/08/05 10:42:25 Duration: 68.2807ms Name: google.spanner.admin.database.v1.DatabaseAdmin.CreateDatabase Code: 0 Message: Attributes: map[Client:true
FailFast:true] Annotations: [
]
2021-08-05T10:42:25.2030092Z ParentSpanID: 66d0f65276a786dc SpanID: 41a185b1034c5e57 ChildSpanCount:
0
2021-08-05T10:42:25.2034069Z INFO: create spanner client with projects/tbf-test/instances/tbf-test-instance/databases/for-ut-a76
3
2021-08-05T10:42:25.2035431Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_utils.go:7
0
2021-08-05T10:42:25.2037531Z 2021/08/05 10:42:25 Duration: 179.701µs Name: cloud.google.com/go/spanner.NewClient Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2038616Z ParentSpanID: 66d0f65276a786dc SpanID: 1c72141091f035d2 ChildSpanCount:
0
2021-08-05T10:42:25.2050051Z 2021/08/05 10:42:25 Duration: 2.842616ms Name: google.spanner.v1.Spanner.BatchCreateSessions Code: 0 Message: Attributes: map[Client:true FailFast:true]
Annotations: [
]
2021-08-05T10:42:25.2051837Z ParentSpanID: d21332cdab39e5c7 SpanID: ade4c02b39debc42 ChildSpanCount:
0
2021-08-05T10:42:25.2054728Z 2021/08/05 10:42:25 Duration: 2.925417ms Name: cloud.google.com/go/spanner.BatchCreateSessions Code: 0 Message: Attributes: map[] Annotations:
[***Time:2021-08-05 10:42:25.201796419 +0000 UTC m=+157.694579279 Message:Creating a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204703536 +0000 UTC m=+157.697486496
Message:Received a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204718536 +0000 UTC m=+157.697501396 Message:Finished creating 2 sessions Attributes:map[]***
]
2021-08-05T10:42:25.2056889Z ParentSpanID: 0000000000000000 SpanID: d21332cdab39e5c7 ChildSpanCount:
1
2021-08-05T10:42:25.2058866Z 2021/08/05 10:42:25 Duration: 410.303µs Name: google.spanner.v1.Spanner.BeginTransaction Code: 0 Message: Attributes: map[Client:true FailFast:true]
Annotations: [
]
2021-08-05T10:42:25.2060645Z ParentSpanID: f742a36e1e950d4d SpanID: 88b54f8ac68294bd ChildSpanCount:
0
2021-08-05T10:42:25.2061510Z INFO: before ReadRo
w
2021-08-05T10:42:25.2062577Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2
0
2021-08-05T10:42:25.2064464Z 2021/08/05 10:42:25 Duration: 9.8µs Name: cloud.google.com/go/spanner.Read Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2065690Z ParentSpanID: f742a36e1e950d4d SpanID: 6386dee853276c38 ChildSpanCount:
1
2021-08-05T10:42:25.2067399Z 2021/08/05 10:42:25 Duration: 555.803µs Name: google.spanner.v1.Spanner.StreamingRead Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations:
[
]
2021-08-05T10:42:25.2069446Z ParentSpanID: 3e576d47e1cb43b3 SpanID: 1928fca56e701b2e ChildSpanCount:
0
2021-08-05T10:42:25.2071376Z 2021/08/05 10:42:25 Duration: 748.605µs Name: cloud.google.com/go/spanner.RowIterator Code: 0 Message: Attributes: map[] Annotations: [
]
2021-08-05T10:42:25.2072630Z ParentSpanID: 6386dee853276c38 SpanID: 3e576d47e1cb43b3 ChildSpanCount:
1
2021-08-05T10:42:25.2074888Z WARNING: err: spanner: code = "NotFound", desc = "row not found(Table: Organization, PrimaryKey: (\"111\"))
"
2021-08-05T10:42:25.2076611Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2
9
2021-08-05T10:42:25.2078426Z INFO: before Apply with ApplyAtLeastOnc
e
2021-08-05T10:42:25.2081087Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:4
9
2021-08-05T10:42:25.2084536Z 2021/08/05 10:42:25 Duration: 533.803µs Name: google.spanner.v1.Spanner.Commit Code: 10 Message: Transaction 2 aborted due to active transaction 1. The emulator
only supports one transaction at a time. Some best practices to avoid ABORT errors in Cloud Spanner service are
:
2021-08-05T10:42:25.2087265Z 1. Avoid use of nested transactions
.
2021-08-05T10:42:25.2088255Z 2. Explicitly Rollback failed transactions
.
2021-08-05T10:42:25.2090414Z 3. All transactions should be running inside of retry loops
.
2021-08-05T10:42:25.2091644Z Attributes: map[Client:true FailFast:true] Annotations: [
]
2021-08-05T10:42:25.2092587Z ParentSpanID: f4f88a04fc14f3a8 SpanID: cfc9196389b9ca23 ChildSpanCount:
0
Slide 16
Slide 16 text
Ͳ͏Δʁ
•ςετͷͱ͖͚ͩTraceΛstdoutʹग़ྗ͢
ΔExporterΛ͏
ͱͬͯ؆୯
Slide 17
Slide 17 text
TextTraceExporter
͜Μ͚ͩ
type textTraceExporter struct{
}
var _ trace.Exporter = (*textTraceExporter)(nil
)
func NewTextTraceExporter() trace.Exporter
{
return &textTraceExporter{
}
}
func (exporter *textTraceExporter) ExportSpan(sd *trace.SpanData)
{
if os.Getenv("DUMP_TEST_TRACE") != "TRUE"
{
retur
n
}
if sd == nil
{
retur
n
}
rawlog.Printf
(
"Duration: %s Name: %s Code: %d Message: %s Attributes: %+v Annotations: %+v\n\tParentSpanID:
%s SpanID: %s ChildSpanCount: %d"
,
sd.EndTime.Sub(sd.StartTime), sd.Name, sd.Code, sd.Message, sd.Attributes, sd.Annotations
,
sd.ParentSpanID, sd.SpanID, sd.ChildSpanCount
,
)
}
Slide 18
Slide 18 text
TextTraceExporter
TestMainͱ͔Ͱৗʹग़͠·͢
exporter := NewTextTraceExporter(
)
trace.RegisterExporter(exporter
)
trace.ApplyConfig(trace.Config{DefaultSampler:
trace.AlwaysSample()})
Slide 19
Slide 19 text
͓·͚
Spannerͷઃఆྫ
spanner.ClientConfig
{
// ଓͷઃఆΛߦ͏
// DUMP_TEST_TRACE Λ TRUE ʹͨ͠ͱ͖ɺBatchCreateSessions,
BeginTransaction, DeleteSession ͱ͔ͷtrace͕ΊͪΌΊͪΌग़ΔͷͰ
Λগͳ͍ͨ͘͠
SessionPoolConfig: spanner.SessionPoolConfig
{
MinOpened: 2, // RWTx + ApplyAtLeastOnc
e
MaxOpened: 5
,
MaxIdle: 0
,
MaxBurst: 5
,
WriteSessions: 0.5
,
HealthCheckWorkers: 2
,
HealthCheckInterval: 0
,
TrackSessionHandles: true
,
}
,
},
Slide 20
Slide 20 text
༨ஊ
ຊͷࡶஊͰ͢
Slide 21
Slide 21 text
ݱঢ়ͷࠔΓ
•spanner.CommitTimestam
p
•Txޙʹࠩ͠ସ͍͑ͨ(frontʹฦ͢ͷͰ
Slide 22
Slide 22 text
ݱঢ়ͷࠔΓ
•Schema managementͲ͏͠Α
•wrench?
Slide 23
Slide 23 text
ݱঢ়ͷࠔΓ
•TextTraceExporterΑΓGUI͕΄͍͠
…
•ݟΔͷ͕ຯʹΊΜͲ͏
Slide 24
Slide 24 text
ࣗ࡞πʔϧ
• sq
b
• Datastore likeͳॻ͖ຯͷDMLϏϧμ
• si
g
• Goͷstruct͔ΒSQLΈཱͯ༻ͷίʔυΛੜ͢Δ
• sid
x
• DDL͔ΒIndexใΛ࣋ͭίʔυΛੜ͢Δ
• cursor(Ծ
)
• Datastore likeͳcursorੜϥΠϒϥϦ
ݱঢ়ະެ։
Slide 25
Slide 25 text
No content