Dec 19, 2021

MongoDB DataOps: Fork, Lookup , mongostat, Custom Replication and Taking Care of Failed System

Why fork is used?

fork is the right way to make sure your MongoDB runs as a daemon. In this case, fork should be defined as true in the configuration file. If you want to run it as an interactive process. just set it to false.

How to Perform Lookup?

db.classes.aggregate( [
{ $lookup: {
  from: "members",
  localField: "enrollmentlist",
  foreignField: "name",
  as: "enrollee_info"
}}
])

mongostat and mongotop

These two great tools are provided as part of the MongoDB installation and can be used to detect the queries types and collections that most heavily affect the MongoDB performance.

Simple Backup and Restore

mongodump --out /tmp/dump/
mongorestore mongodb://127.0.0.1 /tmp/dump/

How to Reset Your MongoDB Instance?

Stop the service, remove all content from the /var/lib/mongodb directory, and start the service again:
sudo service mongod stop
sudo rm -rf /var/lib/mongodb
sudo service mongod start

Setting Custom Replication Source

MongoDB replication automatically balances between replication from the Primary (most updated values) and secondaries w/ a low networking latency to minimize WAN network usage and optimize network resources.
However, your can control this behavior and configure specific source rs.syncFrom() command. Please notice that this command result is temporary and will reset to default once mongod is restarted, connection closed between nodes or 30 sec timeout is reached.

Initiate Your Replicaset:

1. Set the replicaset name in the mongod.conf file
2. Change the bindIP to 0.0.0.0 in the config file
3. Run on the first node the rs.initiate command
rs.initiate(
   {
      _id: "rs",
      version: 1,
      members: [
         { _id: 0, host : "172.30.0.158:27017" },
         { _id: 1, host : "172.30.0.151:27017" },
         { _id: 2, host : "172.30.0.219:27017" }
      ]
   }
)

Search for Delta in the Oplog

db.oplog.rs.find({"ts":{$gt : Timestamp(1640074543, 1)}})

Upgrade from MySQL 5.6 to 8.0

Verify your system configuration to select the right package and method for installation

uname -r
> 2.6.32-696.3.2.el6.x86_64  ==> 64bit

uname -r
> Description:    CentOS release 6.9 (Final) => CentOS 6.9

Upgrade to the latest 5.6

Look for the latest MySQL 5.6 minor version and get the package:

wget https://downloads.mysql.com/archives/get/p/23/file/MySQL-5.6.51-1.el6.x86_64.rpm-bundle.tar

Remove current package
sudo service mysqld stop
sudo yum -y remove mysql-community-common-5.6.36-2.el6.x86_64

tar -xvf MySQL-5.6.51-1.el6.x86_64.rpm-bundle.tar
sudo rpm -i MySQL-client-5.6.51-1.el6.x86_64.rpm
sudo rpm -i  MySQL-server-5.6.51-1.el6.x86_64.rpm
sudo service mysql start
sudo mysql_upgrade

Upgrade to the latest 5.7
wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.35-1.el6.x86_64.rpm-bundle.tar
sudo service mysql stop
sudo yum -y remove MySQL-server-5.6.51-1.el6.x86_64 MySQL-client-5.6.51-1.el6.x86_64
tar -xvf mysql-5.7.35-1.el6.x86_64.rpm-bundle.tar
sudo rpm -i mysql-community-libs-5.7.35-1.el6.x86_64.rpm mysql-community-common-5.7.35-1.el6.x86_64.rpm mysql-community-client-5.7.35-1.el6.x86_64.rpm mysql-community-server-5.7.35-1.el6.x86_64.rpm
sudo service mysqld start
sudo mysql_upgrade

Upgrade to the latest 8.0
Look for latest MySQL 8.0 minor version
wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.26-1.el6.x86_64.rpm-bundle.tar

sudo service mysqld stop
sudo yum -y remove mysql-community-libs-5.7.35-1.el6.x86_64 mysql-community-common-5.7.35-1.el6.x86_64 mysql-community-client-5.7.35-1.el6.x86_64 mysql-community-server-5.7.35-1.el6.x86_64
tar -xvf mysql-8.0.26-1.el6.x86_64.rpm-bundle.tar
sudo rpm -i mysql-community-client-plugins-8.0.26-1.el6.x86_64.rpm mysql-community-common-8.0.26-1.el6.x86_64.rpm mysql-community-libs-8.0.26-1.el6.x86_64.rpm mysql-community-client-8.0.26-1.el6.x86_64.rpm mysql-community-server-8.0.26-1.el6.x86_64.rpm
sudo service mysqld start


Comparing SQL to MongoDB Query Syntax

#MongoDB Aggregation Pipeline

db.documents.aggregate([
{"$match": {"name": "/^asdasd/"}},
{"$group": {"_id": "$name", {"cnt": {$sum:1}}},
{"$match": {"cnt": {$gt: 3}}},
{"$lookup": {from: "meta", localField:"name", foreignField:"name", as: "meta"}}
{"$match": {"meta": {$exists: True}}},
{"$out": "debug_collection"}
]);

#SQL Syntax

SELECT *
FROM documents

SELECT *
FROM documents
WHERE name like 'asdasd%'

SELECT name, COUNT(*) AS cnt
FROM documents
WHERE name like 'asdasd%'
GROUP BY name

SELECT name, COUNT(*)
FROM documents
WHERE name like 'asdasd%'
GROUP BY name
HAVING COUNT(*) > 3

SELECT doc.*, meta.*
FROM meta INNER JOIN (
SELECT name, COUNT(*)
FROM documents
WHERE name like 'asdasd%'
GROUP BY name
HAVING COUNT(*) > 3) AS doc ON doc.name = meta.name

Dec 12, 2021

Your First MongoDB, C# and .NET Core App/My C# and MongoDB Cheat Sheet


Download .NET core SDK

Setup you project and your console project

  1. Select a folder
  2. In the command line: 
    dotnet new console -o console
    dotnet add console package MongoDB.Driver
    cd console
    code console
  3. vsoode will be opened

Create a program.cs file:
namespace HelloWorld
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Run the code:
dotnet run


Now add a document and write it
using MongoDB.Driver;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;


namespace MongoData
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
            var mongoClient = new MongoClient("MONGODB_URL");
            var mongoDatabase = mongoClient.GetDatabase("trial_db");
            var collection = mongoDatabase.GetCollection<TrialDoc>("trial_cl");

            TrialDoc trialDoc = new TrialDoc();
            trialDoc.BookName = "123";
            collection.InsertOne(trialDoc);
            Console.WriteLine("Done!");
        }
    }

   
    public class TrialDoc
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public string? Id { get; set; }

        [BsonElement("Name")]
        public string BookName { get; set; } = null!;

        public int Price { get; set; }

        public string Category { get; set; } = null!;

        public string Author { get; set; } = null!;
    }
}


Add Read Operation: Find All
            var list = collection.Find(_ => true).ToList();
            foreach (var item in list) {
                Console.WriteLine(item.BookName);
            }

Add Read Operation: Specific Record
            var single = collection.Find(x => x.BookName == "123".ToString()).FirstOrDefault();
            Console.WriteLine(single.Id);

Update the record
            collection.ReplaceOne(x => x.Id == single.Id, new TrialDoc() {Id = single.Id, BookName = "456"}, new ReplaceOptions {IsUpsert = true});

var _client = new MongoClient();
var _database = _client.GetDatabase("users");
var counters = _database.GetCollection<BsonDocument>("counters");
var counterQuery = Builders<BsonDocument>.Filter.Eq("_id", "eventId");

var findAndModifyResult = counters.FindOneAndUpdate(counterQuery,
              Builders<BsonDocument>.Update.Set("web", "testweb"));
            FilterDefinition<TrialDoc> filter =
                Builders<TrialDoc>.Filter.Eq(x => x.BookName, "123");
            ProjectionDefinition<TrialDoc> project = Builders<TrialDoc>
                .Projection.Include(x => x.BookName);

            var results = collection.Find(filter).Project(project).ToList();
            foreach (var item in results) {
                Console.WriteLine(item);
            }
Don't forget to create a full text search index first
            var results_list = collection.Find(Builders<TrialDoc>.Filter.Text("123")).ToList();
            foreach (var item in results_list) {
                Console.WriteLine(item);
            }




using MongoDB.Driver.GeoJsonObjectModel;
#Add to document
public GeoJson2DCoordinates Location { get; set; } = null!;
trialDoc.Location = new GeoJson2DCoordinates(31.11, 19.12);
var keys = Builders<BsonDocument>.IndexKeys.Geo2DSphere("Location");
await collection.Indexes.CreateOneAsync(keys);

Nov 29, 2021

How to Setup Your Development Environment for a MongoDB and .Net based Project

1. Get your MongoDB Atlas Replicaset 

2. Make sure you can access the MongoDB Atlas Replicaset using port 27017/TCP

3. Setup your desktop with the following as described in the embedded Loom:

 - Install MongoDB Compass on the desktop 

 - Install a coding environment with recent VS Code ready for .NET Core

Now when everything is ready, you start coding your project and enjoy the combined power of .NET Core and MongoDB

Keep Performing,
Moshe Kaplan

How to Setup a Free MongoDB Atlas Replicaset

 Having a MongoDB Relicaset was always simple, but with MongoDB Atlas, it is simple than ever.

In this two sessions video we'll learn:

1. How to open a MongoDB Atlas account, create a project and a new cluster (Replicaset)

2. How to verify you can connect your Replicaset



Nov 21, 2021

How to Install MongoDB Communitry Edition 5.0.4 on Ubuntu 18.04

Well it is super simple, and you can actually try the MongoDB guide or my embedded video:

I used for this task a t2.medium AWS instance with 2 burstable cores, 4GB RAM and 8GB gp2 SSD disks.

Keep Performing,

Moshe Kaplan

Sep 22, 2021

An Open Source Data Masking Solution for MySQL

Today we'll disucss the need for data masking due to privacy regulations such as GDPR that becone more and more common in the industry.

In order to deploy such a solution we'll utlize two great products:

1. Percona Server 8.0.17 that has recently introduced the data masking plugin (that is compatible w/ MySQL Enterprise one. This plugin exposes multiple functions that translate sensitive strings such as SSN and emails to masked strings.

2. ProxySQL a proxy server that supports modifying SQL queries on the fly. For example replacing SELECT ssn FROM users; with SELECT mask_ssn(ssn) FROM users;

The Percona Server will serve as our MySQL solution (you can use it as a slave instance if you need it for analyst purposes only). while the ProxySQL will serve as a Proxy that modifies SQL queries to utilize the Percona server data masking functions. You may also need to limit users access from the network to the Percona server.

Bottom Line

New times bring new products that can serve us to create novel solutions

Keep Performing,

Moshe Kaplan

Jun 9, 2021

MongoDB Monitoring Using Prometheus

If you environment is bound to Prometheus it is best we focus on this platform as a baseline.


We will need multiple aspects:

System Counters
Main metrics needed:
1. Machines CPU
2. Read and Write IOPS to disks
3. Memory utilization
4. Disk utilization
5. Incoming and outgoing network traffic

MongoDB Counters
For this task it is best to use the MongoDB exporter
There are also 3 Grafana dashboards provided that we can use to monitor the performance
MongoDB Overview;
MongoDB ReplSet;
MongoDB WiredTiger;

These will be able to provide us key indicators such as locks, replicaset status, 

MongoDB Slow Queries
It will also be wise to enable the MongoDB slow query w/ an initial limit of 100ms to gain some overview of the cluster slow queries:
db.setProfilingLevel(1,100)

Feb 10, 2021

Why 4 Nodes MongoDB Replicaset is not a Good Idea

Customer Question:

We have a 4 nodes replicaset w/ leading nodes each with different priority. From time to time, when we suffer from network issues, all the replicaset goes down.

Why is this Happens?

Since two of the nodes are a single site and the two other on two other sites, when the first site goes offline, no site can create a majority. Therefore, you must remove one of the nodes in the first site to avoid these downtimes.

Moreover, the MongoDB selects the primary node by priority. If the primary is unstable, everytime it will go offline, a secondary will be chosen w/ a possible few secondds replicaset downtime. When the high priority node goes back online, it will be selected again to primary (and causing another possible downtime).

Therefore, if there is no good reason, avoid specifing variable priorities.

How to fix It?

1. Perform the task during off peak hours

2. Remove the arbiter node by rs.remove() 

3. Verify replicaset is okay by running rs.status()

4. Modify the remaining nodes configuration:

cfg = rs.conf()

cfg.members[0].priority = 1

cfg.members[1].priority = 1

cfg.members[2].priority = 1

rs.reconfig(cfg)

5. Verify again.

Bottom Line

Keep it Simple :-)


Keep Performing,

Moshe Kaplan

ShareThis

Intense Debate Comments

Ratings and Recommendations