MongoDB Review Checklist and Tools

Tools

1. Analyze the mongostat results: https://docs.mongodb.com/manual/reference/program/mongostat/
2. And mongotop to see query highlight https://docs.mongodb.com/manual/reference/program/mongotop/
3. Enable slow queries: db.setProfilingLevel(1,50)
4. Dex: to analyze slow queries: http://dex.mongolab.com/
5. mtools to visualize the query usage https://github.com/rueckstiess/mtools
6. System metrics: top and iostat

Checklist

1. Usage and location of the Replica Set instances
2. Check system resources usage: CPU and disk IOPS
3. Check network latency, bandwidth  and Application level Read Preference and Write Concern
4. Check mongo instances usage (mongostat)
5. Check top slow queries (mongotop, slow queries)
6. Check queries and command usage using 
7. Check backup strategy
8. Check storage engine and MongoDB version
9. Check monitoring tools

Enable Slow Queries Analysis on Postgres RDS (pg_stat_statements)

Analyzing slow queries on Postgres can be done by enabling slow queries, but it is even more fun with the builtin extension: pg_stat_statements

1. Enable tracking by changing parameters group values (if you did not create a custom parameter group follow these instrcutions and don't forget to reboot the instance to take effect):
pg_stat_statements.track ALL

2. Connect to Postgres
psql -U ADMIN_USER -h RDS_PATH.rds.amazonaws.com -d postgres

3. Create the extension
CREATE EXTENSION pg_stat_statements;

4. Verify the extension was created:
SELECT * 
FROM pg_available_extensions 
WHERE 
    name = 'pg_stat_statements' and 

    installed_version is not null;

5. Analyze the queries usage:
SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent FROM pg_stat_statements ORDER BY total_time DESC LIMIT 20;

Keep Performing,
Moshe Kaplan

Kafka Cluster Hardware Best Practices

Reg. Kafka case studies

Some of the largest Kafla installations in Israel are Adtech (Kenshoo, Taboola...) and Cyber (Verint...) with >40TB or 75M messages per day 

Although the literature claims that Kafka async writes favours magnetic disks (or at least minimizes their benefit [1]) The practice shows that using attached storage 1TB SSD is the best (Kenshoo for example), and it confirmed by confluent that IO can bound performance if disks are slow (our case) [3]

Micron Benchmark shows x4 to x20 performance when using SSD/NVMe [4]

You may avoid using NAS/SAN/external storage machine [2]

You should avoid placing OS and logs on the Kafka disks [2]

One last thing, verify you are using 10Gb NICs [5]

Keep Performing,

Moshe Kaplan

[1] https://blog.cloudera.com/deploying-apache-kafka-a-practical-faq/ 

Using SSDs instead of spinning disks has not been shown to provide a significant performance improvement for Kafka, for two main reasons:

Kafka writes to disk are asynchronous. That is, other than at startup/shutdown, no Kafka operation waits for a disk sync to complete; disk syncs are always in the background. That’s why replicating to at least three replicas is critical—because a single replica will lose the data that has not been sync’d to disk, if it crashes.
Each Kafka Partition is stored as a sequential write ahead log. Thus, disk reads and writes in Kafka are sequential, with very few random seeks. Sequential reads and writes are heavily optimized by modern operating systems.

 

[2] https://docs.confluent.io/current/kafka/deployment.html   

[3] https://docs.confluent.io/current/streams/sizing.html   

[4]  https://www.micron.com/about/blog/2017/october/improve-apache-kafka-performance-with-flash-storage   

[5] https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.1/bk_planning-your-deployment/content/ch_hardware-sizing.html