From Chaos to Clarity: Mastering MongoDB Aggregation for Your Data Needs
I am sure you all must have used JOIN in SQL to combine and merge tables to get the data which you desire. Also I know that you all have…
From Chaos to Clarity: Mastering MongoDB Aggregation for Your Data Needs
I am sure you all must have used JOIN in SQL to combine and merge tables to get the data which you desire. Also I know that you all have wondered is that applicable in MongoDB as well? Just like SQL joins bring together data from multiple tables to create a delightful feast of information, MongoDB’s aggregation framework allows you to mix and match your documents, uncovering insights hidden beneath the surface. So, grab your plate, and let’s dig into the world of aggregation pipelines, where every document is a dish waiting to be served!
There are various pipeline stages which we come across while wanting to get various data of our desire but here in this blog, let us have a hands on with a few most commonly used pipeline stages. Here, we look upon match group project unwind and lookup stages. Also, we will write a proper query using these stages in order to enhance our understanding on the topic. And GoLang developers, cheer up, you will have a wonderful exposure as we will learn the mongodb pipelining using the language as well.
First, let us understand what these stages are. You can check the official MongoDB documentation for the aggregation pipeline(its simple to understand, have a read of documentation as well 👀)
Here are the five pipelines and what they do:
$matchSimplest stage among all. It is used to filter documents that match specified criteria.
{
$match: {
<query>
}
}
// this is how we use $match
$groupThe$groupstage in MongoDB's aggregation pipeline is used to group documents by a specified identifier and perform operations on the grouped data, such as counting, summing, or averaging. It is similar to grouping in the SQL, where you group the documents based on a criteria and hence use them to perform various query operations.
{
$group: {
_id: <expression>, // Field to group by
<field1>: { <accumulator1> }, // Accumulator operations
<field2>: { <accumulator2> },
...
}
}
// a simple syntax for group
$projectAs the name suggests, it is used to project/show the required fields and hide the unwanted fields in the pipeline. It allows you to include, exclude, or add new fields, as well as modify existing fields in the documents that pass through the pipeline.
{
$project: {
<field1>: <expression1>,
<field2>: <expression2>,
...
}
}
4.$lookup
Ever heard of left outer join? This is the mongoDB version of left outer join. $lookup allows you to combine documents from one collection with documents from another based on a specified field. Here, you need to pass in some parameter values as mentioned in the syntax:
{
$lookup: {
from: <foreignCollection>, // The name of the collection to join
localField: <field>, // The field from the input documents
foreignField: <field>, // The field from the documents in the foreign collection
as: <outputArrayField> // The name of the array field to add to the output documents
}
}
$unwindWhenever you get array result in your collection, you probably want to deconstruct the array so that each element of the array could be easily accessible, don’t you? This is what is achieved by using unwind, flatten those arrays into individual documents, allowing for easier processing and analysis of the data.
{
$unwind: {
path: <field>, // The array field to unwind
includeArrayIndex: <string>, // (optional) Name for the index of the array element
preserveNullAndEmptyArrays: <boolean> // (optional) Keep documents with null or empty arrays
}
}
Okay, since we have gotten a basic understanding of these five pipeline stages, shall we implement them all at once in a simple querying?
Yes! This will probably make you understand using pipelines and write your own pipeline functions if ever needed (you will probably need to write them one day or the other).
Wait wait wait, before I forget telling you all, a very important thing about pipeline. I am making this statement in bold so that you would not forget:
The result of one pipeline stage is passed to another pipeline stage in mongodb.
Here is a problem. Actually I was working in one of my personal project, and I had to write a pipeline for getting desirable data and then I remembered, this is worth sharing. So, I was making a project where I had a database and four collections: food order order_items and table
I have written the models for them in Golang
// schema for OrderItem
type OrderItem struct {
ID primitive.ObjectID `bson:"_id"`
Quantity *int `json:"quantity" validate:"required"`
UnitPrice *float64 `json:"unit_price" validate:"required"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
FoodId *string `json:"food_id" validate:"required"`
OrderItemId string `json:"order_item_id"`
OrderId string `json:"order_id" validate:"required"`
}
// schema for Order
type Order struct {
ID primitive.ObjectID `bson:"_id"`
OrderDate time.Time `json:"order_date" validate:"required"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
OrderId string `json:"order_id"`
TableId *string `json:"table_id" validate:"required"`
}
// schema for Food
type Food struct {
ID primitive.ObjectID `bson:"_id"`
Name *string `json:"name" validate:"required,min=2,max=100"`
Price *float64 `json:"price" validate:"required"`
ImageUrl *string `json:"image_url" validate:"required"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
FoodId string `json:"food_id"`
MenuId *string `json:"menu_id" validate:"required"`
}
// schema for table
type Table struct {
ID primitive.ObjectID `bson:"_id"`
NumbersAccomodated *int `json:"numbers_accomodated" validate:"required"`
TableNumber *int `json:"table_number" validate:"required"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
TableId string `json:"table_id"`
}
Seeshh! With these schemas, we can populate our mongodb collections in the database as well. I am using mongodb atlas for the project. So, I am adding these data into the schemas:
// order-items
[
{
"_id": "64e43b5e5f1f2c4a1b6bace1",
"quantity": 2,
"unit_price": 12.99,
"created_at": "2024-10-17T12:34:56Z",
"updated_at": "2024-10-17T12:34:56Z",
"food_id": "64e43c1f5f1f2c4a1b6bace4",
"order_item_id": "OI1001",
"order_id": "O1001"
},
{
"_id": "64e43b6f5f1f2c4a1b6bace2",
"quantity": 1,
"unit_price": 5.99,
"created_at": "2024-10-17T12:40:12Z",
"updated_at": "2024-10-17T12:40:12Z",
"food_id": "64e43c2f5f1f2c4a1b6bace5",
"order_item_id": "OI1002",
"order_id": "O1001"
},
{
"_id": "64e43c7f5f1f2c4a1b6bace3",
"quantity": 3,
"unit_price": 8.99,
"created_at": "2024-10-18T11:15:00Z",
"updated_at": "2024-10-18T11:15:00Z",
"food_id": "64e43c3f5f1f2c4a1b6bace6",
"order_item_id": "OI1003",
"order_id": "O1002"
},
{
"_id": "64e43c8e5f1f2c4a1b6bace4",
"quantity": 2,
"unit_price": 9.49,
"created_at": "2024-10-18T11:25:00Z",
"updated_at": "2024-10-18T11:25:00Z",
"food_id": "64e43c4f5f1f2c4a1b6bace7",
"order_item_id": "OI1004",
"order_id": "O1002"
},
{
"_id": "64e43d9f5f1f2c4a1b6bace5",
"quantity": 1,
"unit_price": 7.49,
"created_at": "2024-10-19T10:10:00Z",
"updated_at": "2024-10-19T10:10:00Z",
"food_id": "64e43c5f5f1f2c4a1b6bace8",
"order_item_id": "OI1005",
"order_id": "O1003"
}
]
// order
[
{
"_id": "64e43b7e5f1f2c4a1b6bace3",
"order_date": "2024-10-17T12:30:00Z",
"created_at": "2024-10-17T12:30:00Z",
"updated_at": "2024-10-17T12:45:00Z",
"order_id": "O1001",
"table_id": "T1001"
},
{
"_id": "64e43d5e5f1f2c4a1b6bace4",
"order_date": "2024-10-18T11:10:00Z",
"created_at": "2024-10-18T11:10:00Z",
"updated_at": "2024-10-18T11:45:00Z",
"order_id": "O1002",
"table_id": "T1002"
},
{
"_id": "64e43e6f5f1f2c4a1b6bace5",
"order_date": "2024-10-19T10:00:00Z",
"created_at": "2024-10-19T10:00:00Z",
"updated_at": "2024-10-19T10:30:00Z",
"order_id": "O1003",
"table_id": "T1003"
}
]
// food
[
{
"_id": "64e43c1f5f1f2c4a1b6bace4",
"name": "Burger",
"price": 12.99,
"image_url": "https://example.com/burger.jpg",
"created_at": "2024-10-17T12:00:00Z",
"updated_at": "2024-10-17T12:15:00Z",
"food_id": "64e43c1f5f1f2c4a1b6bace4"
},
{
"_id": "64e43c2f5f1f2c4a1b6bace5",
"name": "Fries",
"price": 5.99,
"image_url": "https://example.com/fries.jpg",
"created_at": "2024-10-17T12:05:00Z",
"updated_at": "2024-10-17T12:20:00Z",
"food_id": "64e43c2f5f1f2c4a1b6bace5"
},
{
"_id": "64e43c3f5f1f2c4a1b6bace6",
"name": "Pizza",
"price": 8.99,
"image_url": "https://example.com/pizza.jpg",
"created_at": "2024-10-18T11:00:00Z",
"updated_at": "2024-10-18T11:20:00Z",
"food_id": "64e43c3f5f1f2c4a1b6bace6"
},
{
"_id": "64e43c4f5f1f2c4a1b6bace7",
"name": "Pasta",
"price": 9.49,
"image_url": "https://example.com/pasta.jpg",
"created_at": "2024-10-18T11:05:00Z",
"updated_at": "2024-10-18T11:25:00Z",
"food_id": "64e43c4f5f1f2c4a1b6bace7"
},
{
"_id": "64e43c5f5f1f2c4a1b6bace8",
"name": "Salad",
"price": 7.49,
"image_url": "https://example.com/salad.jpg",
"created_at": "2024-10-19T10:00:00Z",
"updated_at": "2024-10-19T10:20:00Z",
"food_id": "64e43c5f5f1f2c4a1b6bace8"
}
]
// table
[
{
"_id": "64e43b9e5f1f2c4a1b6bace6",
"numbers_accomodated": 4,
"table_number": 10,
"created_at": "2024-10-17T11:30:00Z",
"updated_at": "2024-10-17T11:45:00Z",
"table_id": "T1001"
},
{
"_id": "64e43d6e5f1f2c4a1b6bace7",
"numbers_accomodated": 2,
"table_number": 15,
"created_at": "2024-10-18T11:20:00Z",
"updated_at": "2024-10-18T11:35:00Z",
"table_id": "T1002"
},
{
"_id": "64e43e7f5f1f2c4a1b6bace8",
"numbers_accomodated": 6,
"table_number": 20,
"created_at": "2024-10-19T09:30:00Z",
"updated_at": "2024-10-19T09:45:00Z",
"table_id": "T1003"
}
]
You can use your own data for this purpose and insert them into your MongoDB collection, but if you want to follow along, adding these data to test out would be fine as well. And by the way, I asked ChatGPT to create these data for me, so they are all dummy ones. Smart huh??😂
How can I retrieve the order items for a specific order ID, including the food details, table information, and total payment due, using a MongoDB aggregation pipeline?
Do you get the question? I am provided with an order ID, and I need to get all the order items based on the order.
Okay, now that you might have understood the question, let us start writing the pipeline.
But before that, remember what I said earlier: The result of one pipeline stage is passed to another pipeline stage. And also, pipeline stages are written in an array
First off, let us consider the order ID to be **O1002
**We are performing the aggregation pipeline in OrderItems collection. So, first off we filter out the data in the collection which have the order id as mentioned.
For filtering we use$match
[
{
$match: {
order_id: "O1002"
}
},
]
/*
This will filter out all the items with order_id of 01002 from the
collection
*/

Result with first match
Now, we use the $lookup stage in the above result. We do the left outer join with the food collection and get the details of the food item from it, using food_id as the joining field.
[
{
$match: {
order_id: "O1002"
}
},
{
$lookup: {
from: "food",
localField: "food_id",
foreignField: "food_id",
as: "food"
}
},
]
/*
from order_item collection lookup to food collection
the local field in orderItem collection is food_id which is matched with
field food_id in food collection
the result is stored as food for this pipeline
*/

Result after first lookup stage
In the above result, we can see clearly a new field with a name of food is added and it is in the form of an array. Now, we need to access the data inside of the array. So what can we do?
Just remember a thing, whenever we get into such situations and encounter an array, always perform $unwind
Like mentioned earlier, it helps in deconstruction.
[
{
$match: {
order_id: "O1002"
}
},
{
$lookup: {
from: "food",
localField: "food_id",
foreignField: "food_id",
as: "food"
}
},
{
$unwind: {
path: "$food",
preserveNullAndEmptyArrays: true
}
},
]
/*
performed unwind operation
here, we must provide the path -> the array item which you wish to deconstruct
preserveNullAndEmptyArray set to true means deconstruction is done even
for array which is empty and the field will not be removed.
*/

Result of unwind stage
See! After the unwind being performed, you can see the food which was an array earlier is now deduced into an object. And now we can easily access their fields using the dot(.) operation.
The next thing which is to be done is to get the details for the order field based on the order_id
So, why not do the same with order collection? We can $lookup into order collection and to the result of that pipeline, we can use $unwind
[
...
earlier code
...
{
$lookup: {
from: "order",
localField: "order_id",
foreignField: "order_id",
as: "order"
}
},
{
$unwind: {
path: "$order",
preserveNullAndEmptyArrays: true
}
},
]
This will provide us the result, where we can now access the field of order collection as well.

Now, what we need to do is get the table details. For that we need to perform $lookup into table collection. But what would be the local field? Yes, there in order object which we received with the previous pipeline result, we can see table_id. We can surely use that for lookup purpose. Thus we perform the same lookup and unwind operation to get the table details as well.
[
...
{
$lookup: {
from: "table",
localField: "order.table_id",
foreignField: "table_id",
as: "table"
}
},
{
$unwind: {
path: "$table",
preserveNullAndEmptyArrays: true
}
},
]

Our resulting documents contain many unnecessary fields. So why not just keep the field we require and want to pass to the next stage of pipeline? We can achieve that by using $project
So by using $project now, we will only keep the required fields and pass it furthur for querying so that our document is more readable.
[
...
{
$project: {
_id: 0,
amount: "$food.price",
total_count: 1,
food_name: "$food.name",
food_image: "$food.image_url",
table_number: "$table.table_number",
table_id: "$table.table_id",
order_id: "$order.order_id",
quantity: 1
}
},
]
Here, we only want these many fields to be present now. Here, _id: 0 means we do not want to have the field _id in the result any furthur. Similarly, quantity: 1 tells that quantity field is taken in the further processing. Here is the result it gives.

Result after project stage
The result has become clean and more readable, isn’t it? And I am sure the people at the frontend would not scratch their head any more because they would get cleaner data.
Now, we need to get all the order items which are there based on the order. We can just group all the orders with the same order_id so we would have only one result. Thus we now perform the $group pipeline operation.
[
...
{
$group: {
_id: {
order_id: "$order_id",
table_id: "$table_id",
table_number: "$table_number",
},
payment_due: {
$sum: "$amount"
},
total_count: {
$sum: 1
},
order_items: {
$push: "$$ROOT",
}
}
},
]
Okay, this can be a bit more to digest. Let me explain this part.
_id field here means the parameter for grouping. How are we grouping them? Here, we group the documents together whose order_id table_id and table_number are same to that of what we got in the output of previous pipeline.
We also add few accumulators here:
payment_due-> for all the grouped items, we perform$sumoperation on the amounts. So, amount of all the items in the group are added and is thus saved inpayment_duetotal_count-> here, how many items are ordered, that is counted. For each document being grouped based on the criteria, 1 is added to it allowing the total count to be found.order_items-> here we perform$push: “$$ROOT”operation."$push": "$$ROOT"in a MongoDB aggregation pipeline is used within the$groupstage to collect and store the entire document (i.e., the current document being processed) into an array field. So,"order_items": { "$push": "$$ROOT" }means that for each group, the entire document (the current item) will be pushed into an array namedorder_items.
{
"_id": {
"order_id": "O1002",
"table_id": "T1002",
"table_number": 15
},
"payment_due": 18.48,
"total_count": 2,
"order_items": [
{
"food_name": "Pizza",
"food_image": "https://example.com/pizza.jpg",
"table_number": 15,
"table_id": "T1002",
"order_id": "O1002",
"quantity": 3,
"amount": 8.99
},
{
"quantity": 2,
"amount": 9.49,
"food_name": "Pasta",
"food_image": "https://example.com/pasta.jpg",
"table_number": 15,
"table_id": "T1002",
"order_id": "O1002"
}
]
}
This is what I get as a result of the pipeline up until now.
On performing $group, we have the _id which has order_id table_id and table_number
payment_due has performed the sum of amounts of all the items and order_items has the array containing the total current documents.

Result after the group stage
Now, I got what I needed. I can just send this to my frontend or some more cleanup can be performed to just make sure our frontend engineer do not scratch their head. I again use $project now to just make the final result more appropriate and more readable.
[
...
{
$project: {
_id: 0,
payment_due: 1,
total_count: 1,
table_number: "$_id.table_number",
order_items: 1
}
}
]
From the above stage, we only provide payment_due total_count order_items and a new field we add with table_number

Final result
This is what we get as a final result. Here is the complete pipeline query which we performed and the result we get:
[
{
$match: {
order_id: "O1002"
}
},
{
$lookup: {
from: "food",
localField: "food_id",
foreignField: "food_id",
as: "food"
}
},
{
$unwind: {
path: "$food",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: "order",
localField: "order_id",
foreignField: "order_id",
as: "order"
}
},
{
$unwind: {
path: "$order",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: "table",
localField: "order.table_id",
foreignField: "table_id",
as: "table"
}
},
{
$unwind: {
path: "$table",
preserveNullAndEmptyArrays: true
}
},
{
$project: {
_id: 0,
amount: "$food.price",
total_count: 1,
food_name: "$food.name",
food_image: "$food.image_url",
table_number: "$table.table_number",
table_id: "$table.table_id",
order_id: "$order.order_id",
quantity: 1
}
},
{
$group: {
_id: {
order_id: "$order_id",
table_id: "$table_id",
table_number: "$table_number",
},
payment_due: {
$sum: "$amount"
},
total_count: {
$sum: 1
},
order_items: {
$push: "$$ROOT",
}
}
},
{
$project: {
_id: 0,
payment_due: 1,
total_count: 1,
table_number: "$_id.table_number",
order_items: 1
}
}
]
Result:
{
"total_count": 2,
"order_items": [
{
"order_id": "O1002",
"quantity": 3,
"amount": 8.99,
"food_name": "Pizza",
"food_image": "https://example.com/pizza.jpg",
"table_number": 15,
"table_id": "T1002"
},
{
"order_id": "O1002",
"quantity": 2,
"amount": 9.49,
"food_name": "Pasta",
"food_image": "https://example.com/pasta.jpg",
"table_number": 15,
"table_id": "T1002"
}
],
"table_number": 15,
"payment_due": 18.48
}
Here it is. The result from our pipeline is clean and much more understandable. Below is the total pipelines which we performed to get the desired results. Thus, MongoDB aggregation pipeline has a very strong advantage for cleaning and querying the larger data to get the desired data. You would need to have a good idea of writing pipelines if you want to excel in fetching the required data from the database.

Okay now hey GoLang devs, here is a function in Go to write the above pipelines. Yes, writing pipelines in Go is a headache but hey, if you open up the documentation of aggregation pipeline alongside while you write the pipeline functions, I bet you would write them correct in your very first attempt. Here is the pipeline function:
// function with pipeline to get the order_items by order_id
// order_id is taken as a parameter
// returns either OrderItem -> a slice of primitive.M(map[string]interface{})
// or an error
func ItemsByOrder(id string) (OrderItem []primitive.M, err error) {
// create a context
var ctx, cancel = context.WithTimeout(context.Background(), 100*time.Second)
defer cancel()
// code for match stage
matchStage := bson.D{
{
Key: "$match",
Value: bson.D{
{
Key: "order_id",
Value: id,
},
},
},
}
// lookup for the food
// all the parameters should be in the order
lookupFoodStage := bson.D{
{
Key: "$lookup", Value: bson.D{
{
Key: "from", Value: "food",
},
{
Key: "localField", Value: "food_id",
},
{
Key: "foreignField", Value: "food_id",
},
{
Key: "as", Value: "food",
},
},
},
}
// code for unwind stage
unwindFoodStage := bson.D{
{
Key: "$unwind", Value: bson.D{
{
Key: "path",
Value: "$food",
},
{
Key: "preserveNullAndEmptyArrays",
Value: true,
},
},
},
}
lookupOrderStage := bson.D{
{
Key: "$lookup", Value: bson.D{
{
Key: "from", Value: "order",
},
{
Key: "localField", Value: "order_id",
},
{
Key: "foreignField", Value: "order_id",
},
{
Key: "as", Value: "order",
},
},
},
}
unwindOrderStage := bson.D{
{
Key: "$unwind", Value: bson.D{
{
Key: "path", Value: "$order",
},
{
Key: "preserveNullAndEmptyArrays", Value: true,
},
},
},
}
lookupTableStage := bson.D{
{
Key: "$lookup", Value: bson.D{
{
Key: "from", Value: "table",
},
{
Key: "localField", Value: "order.table_id",
},
{
Key: "foreignField", Value: "table_id",
},
{
Key: "as", Value: "table",
},
},
},
}
unwindTableStage := bson.D{
{
Key: "$unwind", Value: bson.D{
{
Key: "path", Value: "table",
},
{
Key: "preserveNullAndEmptyArrays", Value: true,
},
},
},
}
// project and provide only required parameters to the next stage
projectStage := bson.D{
{
Key: "$project", Value: bson.D{
{Key: "id", Value: 0},
{Key: "amount", Value: "$food.price"},
{Key: "total_count", Value: 1},
{Key: "food_name", Value: "$food.name"},
{Key: "food_image", Value: "$food.image_url"},
{Key: "table_number", Value: "$table.table_number"},
{Key: "table_id", Value: "$table.table_id"},
{Key: "order_id", Value: "$order.order_id"},
{Key: "quantity", Value: 1},
},
},
}
// group stage
// add _id and accumulators as discussed earlier
groupStage := bson.D{
{
Key: "$group", Value: bson.D{
{
Key: "_id", Value: bson.D{
{
Key: "order_id", Value: "$order_id",
},
{
Key: "table_id", Value: "$table_id",
},
{
Key: "table_number", Value: "$table_number",
},
},
},
{
Key: "payment_due", Value: bson.D{
{
Key: "$sum", Value: "$amount",
},
},
},
{
Key: "total_count", Value: bson.D{
{
Key: "$sum", Value: 1,
},
},
},
{
Key: "order_items", Value: bson.D{
{
Key: "$push", Value: "$$ROOT",
},
},
},
},
},
}
// final project result which would be the result of the whole pipeline
projectResultStage := bson.D{
{
Key: "$project", Value: bson.D{
{
Key: "id", Value: 0,
},
{
Key: "payment_due", Value: 1,
},
{
Key: "total_count", Value: 1,
},
{
Key: "table_number", Value: "$_id.table_number",
},
{
Key: "order_items", Value: 1,
},
},
},
}
// assuming the collection is orderItemCollection
// perform Aggregate method using go mongo-driver package
// add all the stages into mongo.Pipeline in the order
// returns a result or an error
res, err := orderItemCollection.Aggregate(ctx, mongo.Pipeline{
matchStage,
lookupFoodStage,
unwindFoodStage,
lookupOrderStage,
unwindOrderStage,
lookupTableStage,
unwindTableStage,
projectStage,
groupStage,
projectResultStage,
})
// check for error, if found, log it
if err != nil {
panic(err)
}
// decode all the result into OrderItem
if err = res.All(ctx, &OrderItem); err != nil {
panic(err)
}
defer cancel()
// return the parameters
return OrderItem, err
}
Sooo yessss. These are some of the commonly used aggregation pipeline stages. A lot more of them are there. You can definitely check the official documentation of mongodb aggregation pipeline. But I am sure after reading through this and practicing, you would be able to write your own pipelines based on your requirements. Now I guess we shall wind up. So okay, Adios Amigos.!
메타데이터
- post_id
- 05264e0331b6
- slug
- from-chaos-to-clarity-mastering-mongodb-aggregation-for-your-data-needs-05264e0331b6
- url
- https://medium.com/@drupd17/from-chaos-to-clarity-mastering-mongodb-aggregation-for-your-data-needs-05264e0331b6
- canonical_url
- https://medium.com/@drupd17/from-chaos-to-clarity-mastering-mongodb-aggregation-for-your-data-needs-05264e0331b6
- author_url
- https://medium.com/@drupd17
- status
- ok
- fetched_at
- 2026-07-22 11:47:57