Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Tuning Linux for Databases
Search
Alexey Lesovsky
October 15, 2016
Education
0
49
Tuning Linux for Databases
Slides from my talk at HDConf 2016 Minsk, Belarus
Alexey Lesovsky
October 15, 2016
Tweet
Share
More Decks by Alexey Lesovsky
See All by Alexey Lesovsky
PostgreSQL Scaling Usecases
lesovsky
0
100
Patroni failure stories, or How to crash yout PostgreSQL cluster
lesovsky
0
420
PostgreSQL High Availability in 2019
lesovsky
0
360
Top developer's mistakes when working with PostgreSQL
lesovsky
1
140
Let's Pull the Plug On the Autovacuum (EN)
lesovsky
0
1.2k
Troubleshooting PostgreSQL for Developers (RU)
lesovsky
2
350
Troubleshooting PostgreSQL Streaming Replication
lesovsky
0
980
Call of Postgres: Advanced Operations. Part I.
lesovsky
0
64
Call of Postgres: Advanced Operations. Part II.
lesovsky
0
79
Other Decks in Education
See All in Education
中間活動報告会 人材育成WG・技術サブWG / 20250808-oidfj-eduWG-techSWG
oidfj
0
640
2026 g0v 零時政府年會啟動提案 / g0v Summit 2026 Kickstart
rschiang
0
270
~キャラ付け考えていますか?~ AI時代だからこそ技術者に求められるセルフブランディングのすゝめ
masakiokuda
7
460
技術勉強会 〜 OAuth & OIDC 入門編 / 20250528 OAuth and OIDC
oidfj
5
1.8k
アントレプレナーシップ教育 ~ 自分で自分の幸せを決めるために ~
yoshizaki
0
170
シリコンバレーでスタートアップを共同創業したファウンディングエンジニアとしての学び
tomoima525
1
1.2k
Data Management and Analytics Specialisation
signer
PRO
0
1.5k
学びは趣味の延長線
ohmori_yusuke
0
100
小学校女性教員向け プログラミング教育研修プログラム「SteP」の実践と課題
codeforeveryone
0
110
【Discordアカウント作成ガイド】
ainischool
0
110
ARアプリを活用した防災まち歩きデータ作成ハンズオン
nro2daisuke
0
180
核燃料政策を問う─英国の決断と日本
hide2kano
0
180
Featured
See All Featured
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
810
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
A better future with KSS
kneath
239
17k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Fireside Chat
paigeccino
39
3.6k
The Cost Of JavaScript in 2023
addyosmani
53
8.9k
GitHub's CSS Performance
jonrohan
1032
460k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
We Have a Design System, Now What?
morganepeng
53
7.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
112
20k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
Transcript
None
About PostgreSQL DBA. Linux system administrator. PostgreSQL-Consulting.com: • 24/7 support.
• Audit, performance optimizations. • Consulting and Training. • Monitoring and Emergency. • Capacity planning. Slides: https://goo.gl/awmZ2H
Agenda RDBMS on Linux, why? Databases and Resources. OS subsystems.
CPU, Process scheduling, Power saving policies. Memory, VM, NUMA, Huge pages. Storage, File Systems, Input/Output. Other misc.
Why Linux? Linux is a good choice: • Active development
& Community support. • A lot of features & Fast implementation. • Stable & Mature & Durable.
Databases & Resources Concurrency Query speed Sort, group, hash,... OS
page cache DB buffer pool Local process cache DB data files Transaction Log Cold start CPU Memory Storage
Databases & Resources CPU Scheduling NUMA Power Saving Virtual Memory
NUMA Huge Pages File Systems Storage I/O CPU Memory Storage
Resources CPU scheduler. Virtual memory and NUMA. Huge pages. File
systems. Storage IO. Power saving policy. Others.
CPU scheduling CPU scheduler responsible for proper processes planning: Sysctl:
• kernel.sched_migration_cost_ns = 5000000 (default: 500000). • kernel.sched_autogroup_enabled = 0 (default: 1). http://www.postgresql.org/message-id/
[email protected]
http://kernelnewbies.org/Linux_2_6_38#head-59575a6aeafa38490226a560ee02de89829a5b20
CPU scheduling CPU scheduler responsible for proper processes planning: Sysctl:
• kernel.sched_migration_cost_ns = 5000000 (default: 500000). • kernel.sched_autogroup_enabled = 0 (default: 1). http://www.postgresql.org/message-id/
[email protected]
http://kernelnewbies.org/Linux_2_6_38#head-59575a6aeafa38490226a560ee02de89829a5b20 Be aware on Ubuntu: 12.04 #1055222 and 14.04 #1422016. Use noautogroup kernel param instead of sysctl.conf.
Virtual Memory What is it? Allocator, Caching, Dirty pages and
Writeback.
Virtual Memory
Virtual Memory Sysctl: vm.dirty_background_ratio & vm.dirty_ratio = disable it. vm.dirty_background_bytes
& vm.dirty_bytes = depends on ... RAID cache size, 64MB/128MB otherwise
Virtual Memory Out-of-memory & OOM-Killer Sysctl: vm.swappiness = 1 (default:
60)
NUMA S — Socket C — CPU core M —
Memory bank
NUMA BIOS: enable memory node interleaving. Kernel boot: numa=off. numactl
utility. Sysctl: • vm.zone_reclaim_mode = 0 (default: 0). • kernel.numa_balancing = 0 (default: 0).
Huge Pages Huge pages vs. Transparent huge pages. Huge pages
are supported by many RDBMS. Always disable transparent huge pages.
Huge Pages Huge pages vs. Transparent huge pages. Huge pages
are supported by many RDBMS. Always disable transparent huge pages. /etc/rc.local: • echo never > /sys/kernel/mm/transparent_hugepage/enabled • echo never > /sys/kernel/mm/transparent_hugepage/defrag
Filesystems Ext3 vs Ext4 vs XFS: what is better? Filesystem
Barriers.
Filesystems Ext3 vs Ext4 vs XFS: what is better? Filesystem
Barriers. Disable Write Cache: • hdparm -W0 /dev/device • MegaCli64 -LDSetProp -DisDskCache -Lall -aALL
Filesystems Ext3 vs Ext4 vs XFS: what is better? Filesystem
Barriers. Disable Write Cache: • hdparm -W0 /dev/device • MegaCli64 -LDSetProp -DisDskCache -Lall -aALL Hardware RAID + BBU = barrier=0 (disable). Software RAID = barrier=1 (enable).
Filesystems Ext3 vs Ext4 vs XFS: what is better? Filesystem
Barriers. Disable Write Cache: • hdparm -W0 /dev/device • MegaCli64 -LDSetProp -DisDskCache -Lall -aALL Hardware RAID + BBU = barrier=0 (disable). Software RAID = barrier=1 (enable). Enterprise SSD with Power Loss Protection = barrier=0 (disable).
Storage IO SATA/SAS vs SSD. IO elevators.
Storage IO SATA/SAS vs SSD. IO elevators: • noop: SSD,
PCIe SSD, hi-end storages. • deadline: RAID, SATA/SAS. • cfq: good default. • none (multi-queue block IO): SSD, PCIe SSD.
Storage IO SATA/SAS vs SSD. IO elevators: • noop: SSD,
PCIe SSD, hi-end storages. • deadline: RAID, SATA/SAS. • cfq: good default. • none (multi-queue block IO): SSD, PCIe SSD. # echo 'elevator_name' > /sys/block/<device>/queue/scheduler kernel boot: elevator=<name> /sys/block/*/queue/: rotational, rq_affinity, read_ahead_kb
Power Saving Policy Drivers: acpi_cpufreq vs. intel_pstate. scaling_governor.
Power Saving Policy Drivers: acpi_cpufreq vs. intel_pstate. scaling_governor: • /sys/devices/system/cpu/cpuX/cpufreq/scaling_available_governors
• /sys/devices/system/cpu/cpuX/cpufreq/scaling_governor
Power Saving Policy Drivers: acpi_cpufreq vs. intel_pstate. scaling_governor: • /sys/devices/system/cpu/cpuX/cpufreq/scaling_available_governors
• /sys/devices/system/cpu/cpuX/cpufreq/scaling_governor acpi_cpufreq + performance. intel_pstate + powersave.
Misc: Clocksources What is clocksource? acpi_pm vs. hpet vs. tsc.
/sys/devices/system/clocksource/clocksource0/available_clocksource. /sys/devices/system/clocksource/clocksource0/current_clocksource.
Summary Linux is a good choice for RDBMS: Modern, Universal,
Flexible, Stable. Adapt Linux for your workloads. Test → Change → Test → Commit/Rollback.
Questions? Alexey Lesovsky
[email protected]
PostgreSQL-Consulting.com: Data maintenance at its best
https://postgresql-consulting.com