Aaron Morton - Meetup Sept 25th 2014

Cassandra London Meetup Group

Transcript

1. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License CASSANDRA LONDON SEPT 2014 LESSER KNOWN FEATURES OF CASSANDRA 2.1 Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder & Principal Consultant
2. About The Last Pickle. ! Work with clients to deliver and improve Apache Cassandra based solutions. Apache Cassandra Committer, DataStax MVP, Hector Maintainer, Apache Usergrid Committer. Based in New Zealand & USA.
3. Configuration CQL3
   Tools Internals
4. Change Logging to Logback (CASSANDRA-5883)
   Faster Auto Reloading Auto Archiving logback.xml
5. Change Logging to Logback (CASSANDRA-5883)
   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${cassandra.logdir}/system.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"> <fileNamePattern>${cassandra.logdir}/system.log.%i.zip</ fileNamePattern> <minIndex>1</minIndex> <maxIndex>20</maxIndex> </rollingPolicy> </appender> ! <logger name=“org.apache.cassandra.Foo” level="ERROR"/>
6. Change Logging to Logback (CASSANDRA-5883)
   TIP: Change compression so zless and zgrep work. ! ${cassandra.logdir}/system.log.%i.gz
7. Change Logging to Logback (CASSANDRA-5883)
   $ ls -lah logs/ 963K system.log 129K system.log.1.gz 158K system.log.2.gz 158K system.log.3.gz 158K system.log.4.gz 158K system.log.5.gz 158K system.log.6.gz 158K system.log.7.gz 158K system.log.8.gz 157K system.log.9.gz
8. Command Line Parameters
9. New Default Paths (CASSANDRA-7136)
   // bin/cassandra.in.sh CASSANDRA_HOME="`dirname "$0"`/.." cassandra_storagedir=“$CASSANDRA_HOME/data" ! // bin/cassandra -Dcassandra.logdir=$CASSANDRA_HOME/logs" -Dcassandra.storagedir=$cassandra_storagedir"
10. New Default Paths (CASSANDRA-7136)
    No change for package installs. ! Leave YAML settings blank for new defaults.
11. New Default Paths (CASSANDRA-7136)
    $ ls -lh cassandra/v2.1/ -rw-r--r--@ CHANGES.txt -rw-r--r--@ LICENSE.txt -rw-r--r--@ NEWS.txt -rw-r--r--@ NOTICE.txt drwxr-xr-x@ bin drwxr-xr-x@ conf drwxr-xr-x data drwxr-xr-x@ lib drwxr-xr-x logs drwxr-xr-x@ pylib drwxr-xr-x@ tools
12. New Default Paths (CASSANDRA-7136)
    $ ls -lah cassandra/ drwxr-xr-x@ apache-cassandra-1.0.12 drwxr-xr-x@ apache-cassandra-1.1.12 drwxr-xr-x@ apache-cassandra-1.2.18 drwxr-xr-x@ apache-cassandra-2.0.10 drwxr-xr-x@ apache-cassandra-2.1.0-rc7 lrwxr-xr-x current -> v2.0/ lrwxr-xr-x v1.0 -> apache-cassandra-1.0.12 lrwxr-xr-x v1.1 -> apache-cassandra-1.1.12 lrwxr-xr-x v1.2 -> apache-cassandra-1.2.18/ lrwxr-xr-x v2.0 -> apache-cassandra-2.0.10 lrwxr-xr-x v2.1 -> apache-cassandra-2.1.0-rc7
13. Custom Configuration Loader (CASSANDRA-5045) -Dcassandra.config.loader ! Provide a custom class that implements o.a.c.config.ConfigurationLoader.
14. Custom Configuration Loader (CASSANDRA-5045) Useful when embedding or storing configuration elsewhere.
15. Add QueryHandler Interface (CASSANDRA-6659) -Dcassandra.custom_query_handler_class ! Provide a custom class that implements o.a.c.cql3.QueryHandler.
16. Add QueryHandler Interface (CASSANDRA-6659) Custom implementation to handle all CQL3 queries. ! For example logging, slow query log, stubbing Cassandra.
17. Add QueryHandler Interface (CASSANDRA-6659) public interface QueryHandler { ResultMessage process(String query, QueryState state,…); ! ResultMessage.Prepared prepare(String query, QueryState state); ! ParsedStatement.Prepared getPrepared(MD5Digest id); ! CQLStatement getPreparedForThrift(Integer id); ! ResultMessage processPrepared(CQLStatement statement, …); ! ResultMessage processBatch(BatchStatement statement, …);
18. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430) ! -Dcassandra.metricsReporterConfigFile ! https://github.com/addthis/metrics- reporter-config
19. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430) Reporters push metrics to Graphite, Ganglia, Riemann etc.
20. Pluggable o.a.c.metrics Reporters (CASSANDRA-4430) riemann: - period: 1 timeunit: 'SECONDS' hosts: - host: 'localhost' port: 5555 predicate: color: "white" useQualifiedName: true patterns: - "^org.apache.cassandra.metrics.ClientRequest.+" - "^org.apache.cassandra.metrics.DroppedMessage.+"
21. Selective Commit Log Replay (CASSANDRA-4809) -Dcassandra.replayList ! -Dcassandra.replayList=Foo.Bar,…
22. Selective Commit Log Replay (CASSANDRA-4809) Use for selective Point in Time recovery using Commit Log Archiving.
23. Add Option To Disable STCS In L0 (CASSANDRA-6621) ! -Dcassandra.disable_stcs_in_l0 ! May be useful when Bootstrapping.
24. Use JMX_PORT For The RMI Port (CASSANDRA-7087)
    ! // conf/cassandra-env.sh -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT
25. YAML Configuration
26. Enabled Hints Per Data Center (CASSANDRA-6157) ! hinted_handoff_enabled: DC1,DC2 ! Disable Hints for a troubled DC. ! !
27. Add Paranoid Disk Failure Option (CASSANDRA-6646) ! disk_failure_policy: stop_paranoid ! Implement STOP behaviour for CorruptSSTableException.
28. Commit Log Failure Handling (CASSANDRA-6364) ! commit_failure_policy: stop !
29. Commit Log Failure Handling (CASSANDRA-6364) ! ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2014-… CommitLog.java:350 - Failed to persist commits to disk. Commit disk failure policy is stop; terminating thread ! java.io.IOException: Something broke. at org.apache.cassandra.db.commitlog.AbstractCommitLogService $1.run(AbstractCommitLogService.java:123) at java.lang.Thread.run(Thread.java:744)
30. Add file_cache_size_in_mb setting (CASSANDRA-5661) ! # Total memory to use for sstable-reading buffers. # Defaults to # the smaller of 1/4 of heap or 512MB # file_cache_size_in_mb: 512
31. Add file_cache_size_in_mb setting (CASSANDRA-5661) File Readers have a 64 KB buffer. Compressed File Readers have two 64 KB buffers.
32. Add file_cache_size_in_mb setting (CASSANDRA-5661) Discard cached file readers 512ms after last access for a file.
33. Add file_cache_size_in_mb setting (CASSANDRA-5661) Maintain a maximum combined buffer size for all file readers.
34. Improve 2.1 flush defaults (CASSANDRA-7551)
    ! # memtable_flush_writers defaults to the smaller of (number of # disks, number of cores), with a minimum of 2 and a maximum of 8. # # If your data directories are backed by SSD, you should increase # this to the number of cores. # memtable_flush_writers: 8
35. New concurrent_compactors Defaults (CASSANDRA-7139) ! # concurrent_compactors defaults to the smaller of (number # of disks, number of cores), with a minimum of 2 and a # maximum of 8. # # If your data directories are backed by SSD, you should # increase this to the number of cores. # concurrent_compactors: 1
36. Throw Error When Frame Too Large (CASSANDRA-5981) ! # native_transport_max_frame_size_in_mb: 256 ! Improves Existing Behaviour, InvalidRequestException now thrown.
37. Add broadcast_rpc_address Option (CASSANDRA-5899) ! broadcast_rpc_address: 1.2.3.4 ! Broadcast to clients via node discovery.
38. Add listen_interface and rpc_interface (CASSANDRA-7417) ! rpc_interface: eth1 ! listen_interface: eth0
39. Add listen_interface and rpc_interface (CASSANDRA-7417) Must have a only one IP Address.
40. Drop queries reading many Tombstones (CASSANDRA-6117) Replaces tombstone_debug_threshold
41. Drop queries reading many Tombstones (CASSANDRA-6117) ! tombstone_warn_threshold: 1000
42. Drop queries reading many Tombstones (CASSANDRA-6117) WARN [main] … ! SliceQueryFilter.java:236 - Read 2 live and 4 tombstoned cells in system.schema_columnfamilies (see tombstone_warn_threshold). 2147483631 columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
43. Drop query that using Tombstones (CASSANDRA-6117) ! tombstone_failure_threshold: 100000
44. Drop query that using Tombstones (CASSANDRA-6117) ! ERROR [SharedPool-Worker-7] … ! SliceQueryFilter.java:212 - Scanned over 100000 tombstones in system.schema_columns; query aborted (see tombstone_fail_threshold)
45. Log a Warning for Large Batches (CASSANDRA-6487) ! batch_size_warn_threshold_in_kb: 5
46. Log a Warning for Large Batches (CASSANDRA-6487) ! WARN [Native-Transport-Requests:553]… ! BatchStatement.java (line 228) Batch of prepared statements for [Keyspace1.Standard1] is of size 6165, exceeding specified threshold of 5120 by 1045.
47. Preemptive opening compaction result (CASSANDRA-6916) Replaces compaction_preheat_key_cache: true, preheat_kernel_page_cache: false and populate_io_cache_on_flush table property.
48. Preemptive opening compaction result (CASSANDRA-6916) With sstable_preemptive_open_interval_in_mb: 50
49. Preemptive opening compaction result (CASSANDRA-6916) Pre heats Key Cache and IO Cache, and reads from partial SSTables.
50. Throttle Inter-DC Streaming Traffic (CASSANDRA-6596) ! inter_dc_stream_throughput_outbound_megabits_per_sec: 0 Applied after stream_throughput_outbound_megabits_per_sec: 200
51. Configuration CQL3
    Tools Internals
52. Alias Support For SELECT Statement (CASSANDRA-5075)
    ! cqlsh:dev> select foo as total_foo from foo; ! total_foo ----------- bar baz ! (2 rows)
53. An Official Way To Disable Compaction (CASSANDRA-5074) ! create table no_compaction_for_you ( foo text, bar text, primary key (foo, bar) ) with compaction = { 'enabled' : false, 'class' : 'SizeTieredCompactionStrategy' };
54. Aggressive Tombstone Compactions (CASSANDRA-6563) ! CREATE TABLE foo ( foo text primary key, bar text ) WITH compaction = { 'class' : 'SizeTieredCompactionStrategy', 'unchecked_tombstone_compaction' : 'true' };
55. Aggressive Tombstone Compactions (CASSANDRA-6563) Disables check for other row fragments.
56. Configuration CQL3
    Tools Internals
57. cqlsh
58. Pass cqlsh Statements via Command Line CASSANDRA-7172 ! bin/cqlsh --execute="select distinct foo from dev.foo;" ! foo ------- three one five two four ! (5 rows)
59. Include CQL in Error Messages (CASSANDRA-7111) ! cqlsh:dev> SELECT foo bar from foo; ! <ErrorMessage code=2000 [Syntax error in CQL query] message="line 1:11 no viable alternative at input 'bar'"> cqlsh:dev> !
60. nodetool
61. Set & Get Logging Levels via nodetool (CASSANDRA-7090) ! nodetool getlogginglevels Logger Name Log Level ROOT INFO com.thinkaurelius.thrift ERROR ! nodetool setlogginglevel org.apache.cassandra.db.Row DEBUG ! nodetool getlogginglevels Logger Name Log Level ROOT INFO com.thinkaurelius.thrift ERROR org.apache.cassandra.db.Row DEBUG
62. Add listsnapshots To nodetool (CASSANDRA-5742) ! nodetool listsnapshots ! Snapshot Details: Snapshot name Keyspace name Column family True size Size on disk pre-migration Keyspace1 Standard1 508.03 MB 555.59 MB all-good Keyspace1 Standard1 549.01 MB 803.68 MB ! Total TrueDiskSpaceUsed: 1.03 GB
63. Add Snapshot Space Used To cfstats (CASSANDRA-6231) ! Table: Standard1 SSTable count: 4 Space used (live), bytes: 76875505 Space used (total), bytes: 76875505 Space used by snapshots (total), bytes: 1108391939
64. Save Compaction History (CASSANDRA-5078) ! nodetool compactionhistory ! id keyspace columnfamily compacted_at bytes_in bytes_out 31...9d Keyspace1 Standard1 1410287780455 488451903 174360500 51...9d Keyspace1 Standard1 1410287834272 527847260 256812363
65. Repairing Between Specific Replicas (CASSANDRA-6440) ! nodetool repair [(-dc <specific_dc> | --in-dc <specific_dc>)...] [(-hosts <specific_host> | --in-hosts <specific_host>)]
66. Reset SSTable LCS Compaction Level (CASSANDRA-5271) ! sstablelevelreset --really-reset Keyspace1 Standard1 ! ! Changing level from 1 to 0 on …/Keyspace1-Standard1-ka-35-Data.db ! Skipped .../Keyspace1-Standard1-ka-33-Data.db since it is already on level 0 ! Changing level from 1 to 0 on …/Keyspace1-Standard1-ka-32-Data.db ! Skipped .../Keyspace1-Standard1-ka-34-Data.db since it is already on level 0 !
67. Configuration Schema
    Tools Internals
68. Promote Partition Index (CASSANDRA-2319) Index of Cells created when partition is larger than column_index_size_in_kb (64).
69. Promote Partition Index (CASSANDRA-2319) Column Index promoted into the Key Cache. Improves reading from wide Partitions.
70. Remove Keyspace.switchLock (CASSANDRA-5549) Removed memtable_flush_queue_size Added memtable_cleanup_threshold
71. Remove Keyspace.switchLock (CASSANDRA-5549) When the Flush Queue was full Mutations would be blocked. (Blocked FlushWriter tasks.)
72. Unique Column Family Directories (CASSANDRA-5202) Prevents “re-using”data from dropped Column Families.
73. Unique Column Family Directories (CASSANDRA-5202) ls -lah data/data/Keyspace1/ ! drwxr-xr-x Counter1-89df6990386511e48bb41d4c6fbb569d drwxr-xr-x Counter3-89f20730386511e48bb41d4c6fbb569d drwxr-xr-x Standard1-89df4280386511e48bb41d4c6fbb569d
74. Specify Cells To Cache Per Row (CASSANDRA-5357) create table row_cache ( foo text, bar text, primary key (foo, bar) ) with rows_per_partition_to_cache = 10;
75. Use hibernate when join_ring is false (CASSANDRA-6961) Use when returning a node that as been down for longer than the Hint window.
76. Use hibernate when join_ring is false (CASSANDRA-6961) nodetool join
77. Add Manifest Describing Snapshot Files (CASSANDRA-6326) ! { "files": [ "Keyspace1-Standard1-ka-1-Data.db", "Keyspace1-Standard1-ka-3-Data.db", "Keyspace1-Standard1-ka-2-Data.db", "Keyspace1-Standard1-ka-5-Data.db", "Keyspace1-Standard1-ka-4-Data.db" ] }
78. Track min/max Clustered Values (CASSANDRA-5514) ! CREATE TABLE temperature ( weatherstation_id text, event_time timestamp, temperature text, PRIMARY KEY (weatherstation_id,event_time) );
79. Track min/max Clustered Values (CASSANDRA-5514) ! SELECT event_time, temperature FROM temperature WHERE weatherstation_id='1234ABCD' AND event_time >= '2013-04-03 07:01:00' AND event_time <= '2013-04-03 07:04:00';
80. Track min/max Clustered Values (CASSANDRA-5514) Skips SSTables that will not contain relevant data.
81. The Take Away Use Cassandra v2.0.10 ! Get testing on v2.1.0
82. Thanks. !
83. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton ! Co-Founder & Principal Consultant www.thelastpickle.com ! Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License