Slide 1
Slide 1 text
DISTRIBUTED SCHEDULING | Spring Boot
the challenges and pitfalls of implementing a background job
Rafael Ponte | @rponte
Slide 2
Slide 2 text
https://twitter.com/rponte/status/1492168535421992960
Slide 3
Slide 3 text
But implementing a
background job…
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
https://deniseyu.io/art/
Slide 6
Slide 6 text
Concurrent and
Distributed Computing
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
Where is
my Credit Card?
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
db
server
Slide 14
Slide 14 text
db
server
Slide 15
Slide 15 text
db
server
Slide 16
Slide 16 text
db
server
Slide 17
Slide 17 text
db
server
Slide 18
Slide 18 text
db
server
Slide 19
Slide 19 text
db
server
Slide 20
Slide 20 text
db
server
Slide 21
Slide 21 text
db
server
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
How to schedule
tasks with
Spring Boot?
Slide 26
Slide 26 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// lógica do job vai aqui
}
}
Slide 27
Slide 27 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// lógica do job vai aqui
}
}
Slide 28
Slide 28 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// job logic goes here
}
}
Slide 29
Slide 29 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// job logic goes here
}
}
Slide 30
Slide 30 text
@Component
public class OneJob {
public void runQuiteOften() {
// job logic goes here
}
}
Slide 31
Slide 31 text
@Component
public class OneJob {
@Scheduled
public void runQuiteOften() {
// job logic goes here
}
}
Slide 32
Slide 32 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// job logic goes here
}
}
Slide 33
Slide 33 text
@Component
public class OneJob {
@Scheduled(fixedDelay = 60_000)
public void runQuiteOften() {
// job logic goes here
}
}
Slide 34
Slide 34 text
Coming back…
Slide 35
Slide 35 text
db
server
Slide 36
Slide 36 text
server
db
Slide 37
Slide 37 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 38
Slide 38 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 39
Slide 39 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 40
Slide 40 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 41
Slide 41 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 42
Slide 42 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 43
Slide 43 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 44
Slide 44 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 45
Slide 45 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 46
Slide 46 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 47
Slide 47 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 48
Slide 48 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 49
Slide 49 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 50
Slide 50 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 51
Slide 51 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Naive Solution
Slide 52
Slide 52 text
@rponte
Rafael Ponte
Brazilian Software Engineer at
aka Prince of the Ocean
aka Maharaja of the Legacies
19+ years of experience working
with Java and distributed systems
Loves working with "boring techs"
Slide 53
Slide 53 text
DISTRIBUTED SCHEDULING | Spring Boot
the challenges and pitfalls of implementing a background job
Rafael Ponte | @rponte
Slide 54
Slide 54 text
Naive
Solution
Slide 55
Slide 55 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 56
Slide 56 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 57
Slide 57 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
Slide 58
Slide 58 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 59
Slide 59 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 60
Slide 60 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 61
Slide 61 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 62
Slide 62 text
What if…
Slide 63
Slide 63 text
…the
application
crashes?
Slide 64
Slide 64 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 65
Slide 65 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 66
Slide 66 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 67
Slide 67 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 68
Slide 68 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Inserts a
new card
Updates a
proposal with
its card
Slide 69
Slide 69 text
-- Inserts a new credit card
INSERT INTO card
(id, cardNumber)
VALUES
(42, ’5260-9991-9040-0001’);
-- Updates the proposal with its credit card
UPDATE proposal
SET card_id = 42,
status = 'ELIGIBLE_WITH_ATTACHED_CARD'
WHERE
id = 7031;
Slide 70
Slide 70 text
-- Inserts a new credit card
INSERT INTO card
(id, cardNumber)
VALUES
(42, ’5260-9991-9040-0001’);
-- Updates the proposal with its credit card
UPDATE proposal
SET card_id = 42,
status = 'ELIGIBLE_WITH_ATTACHED_CARD'
WHERE
id = 7031;
Slide 71
Slide 71 text
-- Inserts a new credit card
INSERT INTO card
(id, cardNumber)
VALUES
(42, ’5260-9991-9040-0001’);
-- Updates the proposal with its credit card
UPDATE proposal
SET card_id = 42,
status = 'ELIGIBLE_WITH_ATTACHED_CARD'
WHERE
id = 7031;
Slide 72
Slide 72 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 73
Slide 73 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard); // begins and commits a transaction
proposal.attachTo(newCard);
proposalRepository.save(proposal); // begins and commits a transaction
});
}
Slide 74
Slide 74 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard); // begins and commits a transaction
proposal.attachTo(newCard);
proposalRepository.save(proposal); // begins and commits a transaction
});
}
it’s similar to
auto_commit = true😱
Slide 75
Slide 75 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 76
Slide 76 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 77
Slide 77 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 78
Slide 78 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 79
Slide 79 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 80
Slide 80 text
ID STATUS CARD_ID
7028 NOT_ELIGIBLE 39
7029 NOT_ELIGIBLE 40
7030 ELIGIBLE_WITH_ATTACHED_CARD 41
7031 ELIGIBLE
7032 ELIGIBLE_WITH_ATTACHED_CARD 96
... ... ...
ID CARD_NUMBER
40 5260-9991-9040-0040
41 5260-9991-9040-0041
42 5260-9991-9040-00042
Proposal Card
Database
Slide 81
Slide 81 text
ID STATUS CARD_ID
7028 NOT_ELIGIBLE 39
7029 NOT_ELIGIBLE 40
7030 ELIGIBLE_WITH_ATTACHED_CARD 41
7031 ELIGIBLE
7032 ELIGIBLE_WITH_ATTACHED_CARD 96
... ... ...
ID CARD_NUMBER
40 5260-9991-9040-0040
41 5260-9991-9040-0041
42 5260-9991-9040-00042
Proposal Card
Database
Slide 82
Slide 82 text
ID STATUS CARD_ID
7028 NOT_ELIGIBLE 39
7029 NOT_ELIGIBLE 40
7030 ELIGIBLE_WITH_ATTACHED_CARD 41
7031 ELIGIBLE
7032 ELIGIBLE_WITH_ATTACHED_CARD 96
... ... ...
ID CARD_NUMBER
40 5260-9991-9040-0040
41 5260-9991-9040-0041
42 5260-9991-9040-00042
Proposal Card
Database
Slide 83
Slide 83 text
ID STATUS CARD_ID
7028 NOT_ELIGIBLE 39
7029 NOT_ELIGIBLE 40
7030 ELIGIBLE_WITH_ATTACHED_CARD 41
7031 ELIGIBLE
7032 ELIGIBLE_WITH_ATTACHED_CARD 96
... ... ...
ID CARD_NUMBER
40 5260-9991-9040-0040
41 5260-9991-9040-0041
42 5260-9991-9040-00042
Proposal Card
Database
Slide 84
Slide 84 text
Maybe…
Slide 85
Slide 85 text
https://twitter.com/rponte/status/1492168535421992960
Slide 86
Slide 86 text
Runs every 1 minute
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
Data corruption error
Slide 87
Slide 87 text
Runs every 1 minute Runs every 5 minutes
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
Data corruption error Fix corrupted data
Slide 88
Slide 88 text
Runs every 1 minute Runs every 5 minute
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
ID COL1 COL2
1
2
3
4
Data corruption error Fix corrupted data
No!
Slide 89
Slide 89 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 90
Slide 90 text
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 91
Slide 91 text
No content
Slide 92
Slide 92 text
@Transactional
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 93
Slide 93 text
@Transactional
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 94
Slide 94 text
@Transactional
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
Slide 95
Slide 95 text
@Transactional
@Scheduled(…)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderBy…;
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(…);
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
You got Recoverability!
Slide 96
Slide 96 text
What if…
Slide 97
Slide 97 text
…data volume
grows?
Slide 98
Slide 98 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 99
Slide 99 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 100
Slide 100 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
Slide 101
Slide 101 text
No content
Slide 102
Slide 102 text
No content
Slide 103
Slide 103 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 104
Slide 104 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findAllByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 105
Slide 105 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findAllByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 106
Slide 106 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findAllByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 107
Slide 107 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findAllByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 108
Slide 108 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findAllByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 109
Slide 109 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
Slide 110
Slide 110 text
List findAllByStatusOrderByCreatedAtAsc(...);
Slide 111
Slide 111 text
List findAllByStatusOrderByCreatedAtAsc(...);
Slide 112
Slide 112 text
List findTop50ByStatusOrderByCreatedAtAsc(...);
Slide 113
Slide 113 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
Slide 114
Slide 114 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
limit 50
Slide 115
Slide 115 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 116
Slide 116 text
But…
Slide 117
Slide 117 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
Slide 118
Slide 118 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
this loop
will end very
EARLY!
Slide 119
Slide 119 text
@Component
public class AttachCardsToProposalsJob {
private CardsClient cardsClient;
private CardRepository cardRepository;
private ProposalRepository proposalRepository;
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
It will only
run on the NEXT
execution!
Slide 120
Slide 120 text
while(true)
Slide 121
Slide 121 text
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
proposals.forEach(proposal -> {
// process each proposal
});
}
Slide 122
Slide 122 text
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
// process each proposal
});
}
}
Slide 123
Slide 123 text
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
// process each proposal
});
}
}
Slide 124
Slide 124 text
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
// process each proposal
});
}
}
Slide 125
Slide 125 text
But after
running…
Slide 126
Slide 126 text
No content
Slide 127
Slide 127 text
What?
Slide 128
Slide 128 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
Slide 129
Slide 129 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
- fetching only 50 objects at a time
Slide 130
Slide 130 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
- creating a new reference
- fetching only 50 objects at a time
Slide 131
Slide 131 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
- exiting the loop at the right time
- creating a new reference
- fetching only 50 objects at a time
Slide 132
Slide 132 text
Why?
Slide 133
Slide 133 text
@Component
public class AttachCardsToProposalsJob {
// dependências e constantes
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
Slide 134
Slide 134 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findAllByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 135
Slide 135 text
Revisiting
JPA and Hibernate
Slide 136
Slide 136 text
Job
Slide 137
Slide 137 text
Repository
Job
Slide 138
Slide 138 text
Repository EM
Job
Slide 139
Slide 139 text
Repository EM
Job
Slide 140
Slide 140 text
Repository EM
Job
Slide 141
Slide 141 text
Repository EM
Job
Slide 142
Slide 142 text
Repository EM
Job
Slide 143
Slide 143 text
Repository EM
Job
Slide 144
Slide 144 text
Repository EM
Job
Slide 145
Slide 145 text
That’s what we expected…
Slide 146
Slide 146 text
Long-running transaction
Slide 147
Slide 147 text
Repository EM
Job
Long-running transaction
Slide 148
Slide 148 text
@Transactional
Repository EM
Job
Long-running transaction
Slide 149
Slide 149 text
@Transactional
Repository EM
Job
Long-running transaction
Slide 150
Slide 150 text
@Transactional
Repository EM
Job
Long-running transaction
Slide 151
Slide 151 text
@Transactional
Repository EM
Job
(First-Level Cache)
Long-running transaction
Slide 152
Slide 152 text
@Transactional
Repository EM
Job
(First-Level Cache)
Long-running transaction
Slide 153
Slide 153 text
The idea here is to understand…
Slide 154
Slide 154 text
@Transactional
Repository EM
Job
(First-Level Cache)
Long-running transaction
Slide 155
Slide 155 text
@Transactional
EM
(First-Level Cache)
Long-running transaction
Slide 156
Slide 156 text
@Transactional
EM
(First-Level Cache)
Long-running transaction
Slide 157
Slide 157 text
EM
(First-Level Cache)
@Transactional
Long-running transaction
Slide 158
Slide 158 text
EM
(First-Level Cache)
@Transactional
Long-running transaction
Slide 159
Slide 159 text
Repository EM
Job
(First-Level Cache)
@Transactional
Long-running transaction
Slide 160
Slide 160 text
Repository EM
Job
(First-Level Cache)
Long-running transaction
Slide 161
Slide 161 text
Ahhh!!
Slide 162
Slide 162 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
Slide 163
Slide 163 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
begins and ends
a transaction
Slide 164
Slide 164 text
But…
Slide 165
Slide 165 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
}
}
}
Slide 166
Slide 166 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard); // begins and commits a transaction
proposal.attachTo(newCard);
proposalRepository.save(proposal); // begins and commits a transaction
});
}
}
}
We gave up on
data consistency! 🤡
Slide 167
Slide 167 text
@Component
public class AttachCardsToProposalsJob {
// dependencies and constants
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(ELIGIBLE);
if (proposals.isEmpty()) {
break;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard); // begins and commits a transaction
proposal.attachTo(newCard);
proposalRepository.save(proposal); // begins and commits a transaction
});
}
}
}
We give up on
data consistency!
Nooo!
Slide 168
Slide 168 text
Short-running transaction
Slide 169
Slide 169 text
Job
Transaction
Job
Slide 170
Slide 170 text
Job
Transaction
Job
start of job end of job
Slide 171
Slide 171 text
Job
Transaction
Job
Tx1
Slide 172
Slide 172 text
start of
transaction
end of
transaction
Job
Job
Transaction
Tx1
Slide 173
Slide 173 text
Job
Transaction
Job
Long-running transaction
Tx1
Slide 174
Slide 174 text
We need to minimize it…
Slide 175
Slide 175 text
Job
Transaction
Job
Slide 176
Slide 176 text
Job
Transaction
Job
Tx1
Slide 177
Slide 177 text
Job
Transaction
Job
start of
transaction
end of
transaction
Tx1
Slide 178
Slide 178 text
Tx1
Job
Transaction
Job
Short-running transaction
Slide 179
Slide 179 text
Job
Transaction
Job
Tx1
Slide 180
Slide 180 text
Job
Transaction
Job
Tx1 Tx2
Slide 181
Slide 181 text
Job
Transaction
Job
Tx1 Tx2 Tx3
Slide 182
Slide 182 text
Job
Transaction
Job
Tx1 Tx2 Tx3 Tx4
Slide 183
Slide 183 text
Job
Transaction
Job
Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 TxN
Slide 184
Slide 184 text
Job
Transaction
Job
Short-running transaction
Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 TxN
Slide 185
Slide 185 text
Repository EM
Job
(First-Level Cache)
Short-running transaction
Slide 186
Slide 186 text
Repository EM
Job
@Transactional
Short-running transaction
(First-Level Cache)
Slide 187
Slide 187 text
Repository EM
Job
@Transactional
Short-running transaction
(First-Level Cache)
Slide 188
Slide 188 text
Repository EM
Job
@Transactional
Short-running transaction
(First-Level Cache)
Slide 189
Slide 189 text
Repository EM
Job
@Transactional
Short-running transaction
(First-Level Cache)
Slide 190
Slide 190 text
Transaction Management
PROGRAMMATIC
Slide 191
Slide 191 text
@Component
public class AttachCardsToProposalsJob {
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
// executes the logic within a transactional scope
}
}
}
Slide 192
Slide 192 text
@Component
public class AttachCardsToProposalsJob {
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
// executes the logic within a transactional scope
}
}
}
Slide 193
Slide 193 text
@Component
public class AttachCardsToProposalsJob {
@Transactional
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});;
}
}
}
Slide 194
Slide 194 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 195
Slide 195 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 196
Slide 196 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 197
Slide 197 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 198
Slide 198 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 199
Slide 199 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
It does
NOT
execute! 😵💫
Slide 200
Slide 200 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
@Transactional.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 201
Slide 201 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
transactionManager.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 202
Slide 202 text
@Component
public class AttachCardsToProposalsJob {
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
while (true) {
transactionManager.execute(transaction -> {
// executes the logic within a transactional scope
});
}
}
}
Slide 203
Slide 203 text
After refactoring…
Slide 204
Slide 204 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 205
Slide 205 text
No content
Slide 206
Slide 206 text
start of job
end of job
Slide 207
Slide 207 text
start of job
end of job
You got batch processing!
Slide 208
Slide 208 text
What if…
Slide 209
Slide 209 text
…we run the
application in a
cluster?
Slide 210
Slide 210 text
No content
Slide 211
Slide 211 text
No content
Slide 212
Slide 212 text
No content
Slide 213
Slide 213 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 214
Slide 214 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 215
Slide 215 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 216
Slide 216 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 217
Slide 217 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 218
Slide 218 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
1004 ELIGIBLE
Slide 219
Slide 219 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
1004 ELIGIBLE
Slide 220
Slide 220 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE
Slide 221
Slide 221 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE
Slide 222
Slide 222 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE_WITH_ATTACHED_CARD
Slide 223
Slide 223 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE_WITH_ATTACHED_CARD
Slide 224
Slide 224 text
No content
Slide 225
Slide 225 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 226
Slide 226 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 227
Slide 227 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 228
Slide 228 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
2x
Slide 229
Slide 229 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
1004 ELIGIBLE
2x
Slide 230
Slide 230 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
1004 ELIGIBLE
2x
2x
Slide 231
Slide 231 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE
2x
2x
Slide 232
Slide 232 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE
2x
2x
2x
Slide 233
Slide 233 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE_WITH_ATTACHED_CARD
2x
2x
2x
Slide 234
Slide 234 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
1004 ELIGIBLE_WITH_ATTACHED_CARD
2x
2x
2x
2x
Slide 235
Slide 235 text
Oops!
Slide 236
Slide 236 text
Synchronized?
Slide 237
Slide 237 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(...)
public synchronized void execute() {
// executes the logic here
}
}
Slide 238
Slide 238 text
No content
Slide 239
Slide 239 text
No content
Slide 240
Slide 240 text
No content
Slide 241
Slide 241 text
No content
Slide 242
Slide 242 text
@Component
public class AttachCardsToProposalsJob {
@Scheduled(...)
public synchronized void execute() {
// executes the logic here
}
}
No!
Slide 243
Slide 243 text
Distributed Lock?
Slide 244
Slide 244 text
@Scheduled(...)
public void execute() {
if (isLeader()) {
// executes the logic here
}
}
Slide 245
Slide 245 text
@Scheduled(...)
public void execute() {
if (isLeader()) {
// executes the logic here
}
}
Slide 246
Slide 246 text
No content
Slide 247
Slide 247 text
No content
Slide 248
Slide 248 text
No content
Slide 249
Slide 249 text
No content
Slide 250
Slide 250 text
No content
Slide 251
Slide 251 text
No content
Slide 252
Slide 252 text
No content
Slide 253
Slide 253 text
No content
Slide 254
Slide 254 text
No content
Slide 255
Slide 255 text
No content
Slide 256
Slide 256 text
No content
Slide 257
Slide 257 text
No!
Slide 258
Slide 258 text
How about…
Slide 259
Slide 259 text
No content
Slide 260
Slide 260 text
No content
Slide 261
Slide 261 text
@Repository
public interface ProposalRepository extends JpaRepository {
public List findTop50ByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 262
Slide 262 text
@Repository
public interface ProposalRepository extends JpaRepository {
@Lock(LockModeType.PESSIMISTIC_WRITE)
public List findTop50ByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 263
Slide 263 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
limit 50
Slide 264
Slide 264 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
limit 50
for update
Slide 265
Slide 265 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 266
Slide 266 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 267
Slide 267 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 268
Slide 268 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 269
Slide 269 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 270
Slide 270 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 271
Slide 271 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 272
Slide 272 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 273
Slide 273 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 274
Slide 274 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 275
Slide 275 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 276
Slide 276 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 277
Slide 277 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 278
Slide 278 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 279
Slide 279 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
... ...
Slide 280
Slide 280 text
Yess!
Slide 281
Slide 281 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE_WITH_ATTACHED_CARD
... ...
You got high-availability!
Slide 282
Slide 282 text
What if…
Slide 283
Slide 283 text
…we need
more
throughput?
Slide 284
Slide 284 text
No content
Slide 285
Slide 285 text
No content
Slide 286
Slide 286 text
No content
Slide 287
Slide 287 text
No content
Slide 288
Slide 288 text
No content
Slide 289
Slide 289 text
No content
Slide 290
Slide 290 text
No content
Slide 291
Slide 291 text
To distribute the load…
Slide 292
Slide 292 text
9630 9631
QUEUE
9629
Slide 293
Slide 293 text
9630 9631
producer
9629
produces:
1000 items/min
QUEUE
Slide 294
Slide 294 text
9630 9631
producer
consumer
9629
consumes:
600 items/min
produces:
1000 items/min
QUEUE
Slide 295
Slide 295 text
9630 9631
producer
consumer
9629
consumer
consumes:
400 items/min
consumes:
600 items/min
produces:
1000 items/min
QUEUE
Slide 296
Slide 296 text
consumes:
400 items/min
consumes:
600 items/min
produces:
1000 items/min
9630 9631
producer
consumer
9629
consumer
QUEUE
Slide 297
Slide 297 text
No content
Slide 298
Slide 298 text
No!
Slide 299
Slide 299 text
How about…
Slide 300
Slide 300 text
No content
Slide 301
Slide 301 text
@Repository
public interface ProposalRepository extends JpaRepository {
@QueryHints({
@QueryHint(
name = "javax.persistence.lock.timeout",
value = LockOptions.SKIP_LOCKED) // org.hibernate.LockOptions
})
@Lock(LockModeType.PESSIMISTIC_WRITE)
public List findTop50ByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 302
Slide 302 text
@Repository
public interface ProposalRepository extends JpaRepository {
@QueryHints({
@QueryHint(
name = "javax.persistence.lock.timeout",
value = LockOptions.SKIP_LOCKED) // org.hibernate.LockOptions
})
@Lock(LockModeType.PESSIMISTIC_WRITE)
public List findTop50ByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 303
Slide 303 text
@Repository
public interface ProposalRepository extends JpaRepository {
@QueryHints({
@QueryHint(
name = "javax.persistence.lock.timeout",
value = LockOptions.SKIP_LOCKED) // org.hibernate.LockOptions
})
@Lock(LockModeType.PESSIMISTIC_WRITE)
public List findTop50ByStatusOrderByCreatedAtAsc(ProposalStatus status);
}
Slide 304
Slide 304 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
limit 50
for update
Slide 305
Slide 305 text
select p.*
from proposal p
where p.status = 'ELIGIBLE'
order by p.created_at asc
limit 50
for update skip locked
Slide 306
Slide 306 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 307
Slide 307 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 308
Slide 308 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 309
Slide 309 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 310
Slide 310 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 311
Slide 311 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 312
Slide 312 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
... ...
Slide 313
Slide 313 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 314
Slide 314 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 315
Slide 315 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 316
Slide 316 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 317
Slide 317 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 318
Slide 318 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 319
Slide 319 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 320
Slide 320 text
ID STATUS
1001 ELIGIBLE_WITH_ATTACHED_CARD
1002 ELIGIBLE_WITH_ATTACHED_CARD
1003 ELIGIBLE
... ...
Slide 321
Slide 321 text
In other words…
Slide 322
Slide 322 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 323
Slide 323 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
Slide 324
Slide 324 text
Yess!
Slide 325
Slide 325 text
ID STATUS
1001 ELIGIBLE
1002 ELIGIBLE
1003 ELIGIBLE
1004 ELIGIBLE
You got scalability!
Slide 326
Slide 326 text
What if…
Slide 327
Slide 327 text
…we need
more
performance?
Slide 328
Slide 328 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 329
Slide 329 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 330
Slide 330 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 331
Slide 331 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 332
Slide 332 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 333
Slide 333 text
@Autowired
private TransactionTemplate transactionManager;
@Scheduled(fixedDelay = 60_000)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(transactionStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Inserts a
new card
Updates a
proposal with
its card
Slide 334
Slide 334 text
For each iteration in the loop…
Slide 335
Slide 335 text
2 roundtrips
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
Slide 336
Slide 336 text
2 roundtrips
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
Slide 337
Slide 337 text
2 roundtrips
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
Slide 338
Slide 338 text
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
INSERT INTO card(cardNumber) VALUES (' 5513-8141-9261-0002');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2022;
4 roundtrips
Slide 339
Slide 339 text
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
INSERT INTO card(cardNumber) VALUES (' 5513-8141-9261-0002');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2022;
INSERT INTO card(cardNumber) VALUES (' 5599-4154-6354-0003');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2023;
6 roundtrips
Slide 340
Slide 340 text
100 roundtrips
INSERT INTO card(cardNumber) VALUES ('5260-9991-9040-0001');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2021;
INSERT INTO card(cardNumber) VALUES (' 5513-8141-9261-0002');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2022;
INSERT INTO card(cardNumber) VALUES (' 5599-4154-6354-0003');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2023;
INSERT INTO card(cardNumber) VALUES (' 5419-8753-1126-0004');
UPDATE proposal SET status = 'ELIGIBLE_WITH_ATTACHED_CARD' WHERE id = 2024;
INSERT INTO ...
UPDATE proposal SET ...
Slide 341
Slide 341 text
Latency
total=120ms
server database
40ms
80ms
Slide 342
Slide 342 text
Hibernate batch size
Slide 343
Slide 343 text
spring:
jpa:
properties:
hibernate:
jdbc:
batch_size: 50
order_inserts: true
order_updates: true
application.yml
Slide 344
Slide 344 text
spring:
jpa:
properties:
hibernate:
jdbc:
batch_size: 50
order_inserts: true
order_updates: true
application.yml
Slide 345
Slide 345 text
For each batch…
Slide 346
Slide 346 text
2 roundtrips
INSERT INTO card(cardNumber) VALUES (?)
Params: [
('5513-8141-9261-0001'),
('5513-8141-9261-0002'),
(‘5513-8141-9261-0003'),
(‘5513-8141-9261-0004'),
...
]
UPDATE proposal SET status = ? WHERE id = ?;
Params: [
(‘ELIGIBLE_WITH_ATTACHED_CARD’, 2021),
('ELIGIBLE_WITH_ATTACHED_CARD', 2022),
('ELIGIBLE_WITH_ATTACHED_CARD', 2023),
('ELIGIBLE_WITH_ATTACHED_CARD', 2024),
...
]
Slide 347
Slide 347 text
2 roundtrips
INSERT INTO card(cardNumber) VALUES (?)
Params: [
('5513-8141-9261-0001'),
('5513-8141-9261-0002'),
(‘5513-8141-9261-0003'),
(‘5513-8141-9261-0004'),
...
]
UPDATE proposal SET status = ? WHERE id = ?;
Params: [
(‘ELIGIBLE_WITH_ATTACHED_CARD’, 2021),
('ELIGIBLE_WITH_ATTACHED_CARD', 2022),
('ELIGIBLE_WITH_ATTACHED_CARD', 2023),
('ELIGIBLE_WITH_ATTACHED_CARD', 2024),
...
]
Slide 348
Slide 348 text
INSERT INTO card(cardNumber) VALUES (?)
Params: [
('5513-8141-9261-0001'),
('5513-8141-9261-0002'),
(‘5513-8141-9261-0003'),
(‘5513-8141-9261-0004'),
...
]
2 roundtrips
UPDATE proposal SET status = ? WHERE id = ?;
Params: [
(‘ELIGIBLE_WITH_ATTACHED_CARD’, 2021),
('ELIGIBLE_WITH_ATTACHED_CARD', 2022),
('ELIGIBLE_WITH_ATTACHED_CARD', 2023),
('ELIGIBLE_WITH_ATTACHED_CARD', 2024),
...
]
Slide 349
Slide 349 text
INSERT INTO card(cardNumber) VALUES (?)
Params: [
('5513-8141-9261-0001'),
('5513-8141-9261-0002'),
(‘5513-8141-9261-0003'),
(‘5513-8141-9261-0004'),
...
]
2 roundtrips
UPDATE proposal SET status = ? WHERE id = ?;
Params: [
(‘ELIGIBLE_WITH_ATTACHED_CARD’, 2021),
('ELIGIBLE_WITH_ATTACHED_CARD', 2022),
('ELIGIBLE_WITH_ATTACHED_CARD', 2023),
('ELIGIBLE_WITH_ATTACHED_CARD', 2024),
...
]
You got optimal batch processing!
Slide 350
Slide 350 text
What if…
Slide 351
Slide 351 text
…something goes
wrong with the
network?
Slide 352
Slide 352 text
@Scheduled(...)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(txStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 353
Slide 353 text
@Scheduled(...)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(txStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 354
Slide 354 text
@Scheduled(...)
public void execute() {
boolean pending = true;
while (pending) {
pending = transactionManager.execute(txStatus -> {
List proposals = proposalRepository.findTop50ByStatusOrderByCreatedAtAsc(…);
if (proposals.isEmpty()) {
return false;
}
proposals.forEach(proposal -> {
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Card newCard = cardData.toModel();
cardRepository.save(newCard);
proposal.attachTo(newCard);
proposalRepository.save(proposal);
});
return true;
});
}
}
Slide 355
Slide 355 text
https://deniseyu.io/art/
Slide 356
Slide 356 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
Slide 357
Slide 357 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
waits in
fi
nitely?
Slide 358
Slide 358 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
waits in
fi
nitely? timeout
Slide 359
Slide 359 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
transient errors?
waits in
fi
nitely? timeout
Slide 360
Slide 360 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
transient errors? retry+backo
f
waits in
fi
nitely? timeout
Slide 361
Slide 361 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
can I send the
request again?
transient errors? retry+backo
f
waits in
fi
nitely? timeout
Slide 362
Slide 362 text
CardDataResponse cardData = cardsClient.findCardByProposalId(proposal.getId());
can I send the
request again? idempotency
transient errors? retry+backo
f
waits in
fi
nitely? timeout
Slide 363
Slide 363 text
WHAT
WHAT
WHAT
WHAT WHAT
WHAT
WHAT WHAT
WHAT IF…
WHAT IF…
WHAT IF…
WHAT
WHAT IF…
HAT IF… WHAT IF…
WHAT
WHAT IF…
WHAT IF…
WHAT IF…
WHAT IF…
AT IF…
WHAT IF…
F…WHAT
Slide 364
Slide 364 text
Concluding
Slide 365
Slide 365 text
A background job
can be hard… 😬
Slide 366
Slide 366 text
Recap…
Slide 367
Slide 367 text
Now, we have a
background job
that…
Slide 368
Slide 368 text
No content
Slide 369
Slide 369 text
…can recover from crashes
Slide 370
Slide 370 text
…can handle huge volumes of data
…can recover from crashes
Slide 371
Slide 371 text
…remains available in the face of failures
…can handle huge volumes of data
…can recover from crashes
Slide 372
Slide 372 text
…designed and built to be scalable
…remains available in the face of failures
…can handle huge volumes of data
…can recover from crashes
Slide 373
Slide 373 text
…designed and built to be scalable
…remains available in the face of failures
…can handle huge volumes of data
…optimized for high-throughput
…can recover from crashes
Slide 374
Slide 374 text
No content
Slide 375
Slide 375 text
But it also
works on all
other
platforms
Slide 376
Slide 376 text
@rponte
Rafael Ponte
Thank you!!