Skip to main content

Implement Multi Threading in Nodejs with a real world use-case

In this article, we will see a problem statement and how you can solve that using nodejs multi threading concept.Implement Multi Threading in Nodejs with a real world use-case.

As we all know, Nodejs is a single threaded run time environment. If you are completely new to Node single thread concept. read this article to get a better understanding of it.

Since, Nodejs is single thread. Main constraint that Nodejs developers faced is running CPU intensive tasks in the Node applications.

But Node.js v11 came up with one the important features. that is, worker threads in Nodejs.

What are Worker Threads

worker threads in the Nodejs solves the problem of multi threading. there are some scenarios where you might need to run the task in different thread rather than running it in main thread.

In that case, worker threads takes the task without blocking the main thread of the application.

If you want to know more about the basic implementation of worker threads, refer this article to get better understanding of basics.

worker threads

Implementation - Problem Statement

Let's see where we might need worker threads in Nodejs with an use-case. Let's say that you are building an web application which has lot of stock videos.

When user uploads the video, you have to add the watermark of your application logo for copyright issues.

let's see how you can implement this scenario using worker threads.

Adding a watermark to video can be implemented using ffmpeg. but the problem in using that in nodejs application is, it is cpu intensive.

If you directly implement that in your application. it will block the main thread and affects the performance.

Solution

So, let's implement that using worker threads.

As always, create a directory and initialize the project using npm init --yes

Install the dependencies for the project.

1npm i express body-parser ffmpeg hbs multer
  • express - Nodejs Framework to handle the request and response
  • body-parser - it handles the request post body.
  • ffmpeg - This library is used to handle adding watermark to the uploaded video.
  • hbs - handlebars is a template engine to render the views in express applications.
  • multer - it is used to handle the file upload in nodejs applications.

Here, we are going to use babel to use ES6 in node application. babelbasically compiles the ES6 to javascript.

1npm i --save-dev @babel/cli @babel/core @babel/node @babel/preset-env dotenv nodemon

create a file called .babelrc and add the following code

1{
2 "presets": [
3 [
4 "@babel/preset-env",
5 {
6 "targets": {
7 "node": true
8 }
9 }
10 ]
11 ]
12}

After that, create a directory viewsand inside that directory, create a file index.hbs and add the following code,

1<!DOCTYPE html>
2<html lang="en">
3 <head>
4 <meta charset="UTF-8" />
5 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6 <meta http-equiv="X-UA-Compatible" content="ie=edge" />
7 <title>Video Watermark</title>
8 <link
9 rel="stylesheet"
10 href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css"
11 />
12 </head>
13
14 <body>
15 <h1>Watermarkify</h1>
16
17 {{# if error }}
18 <div class="ui negative message">
19 <i class="close icon"></i>
20 <div class="header">
21 {{error}}
22 </div>
23 </div>
24 {{/if}}
25
26 <form
27 method="POST"
28 class="ui form"
29 enctype="multipart/form-data"
30 action="/upload-video"
31 >
32 <div class="field">
33 <label>Upload Video</label>
34 <input type="file" name="ssvideo" accept="video/*" />
35 </div>
36 <button class="ui button">upload</button>
37 </form>
38 </body>
39</html>

basically, we have an html form which takes video file as an input.

Once, form is submitted, we need to handle the route to get the data in express.

1import express from "express"
2import * as bodyParser from "body-parser"
3import * as path from "path"
4import multer from "multer"
5
6require("dotenv").config()
7
8const storage = multer.diskStorage({
9 destination: "./uploads/",
10 filename: function(req, file, cb) {
11 cb(
12 null,
13 file.fieldname + "-" + Date.now() + path.extname(file.originalname)
14 )
15 },
16})
17
18const upload = multer({ dest: "uploads", storage: storage })
19
20const app = express()
21
22app.use(bodyParser.json())
23app.use(bodyParser.urlencoded({ extended: false }))
24
25app.set("views", path.join(__dirname, "views"))
26app.set("view engine", "hbs")
27
28app.get("/", (req, res) => {
29 res.render("index")
30})
31
32app.post("/upload-video", upload.single("ssvideo"), (req, res) => {
33 console.log(req.file)
34})
35
36const PORT = process.env.PORT
37
38app.listen(PORT, () => {
39 console.log(`Server is running on PORT ${PORT}`)
40})

Here, we setup the handlebar template engine along with multer in express application.

After that, we have two routes which handles the default index page and upload video url.

we will get the uploaded file data in /upload-video url, you can process the video in the route.

Now, it is time to implement the worker threads.

Worker Threads - Concepts

There are three main concepts in worker threads. they are WorkerisMainThread and workerData.

Mainly, Worker is used to create a worker thread and isMainThread check if the application is running on main thread and worker data is used to pass the data from main thread to worker thread.

In /upload-video route, check if it is running in main thread, if it is, create a worker thread and pass the uploaded file data to worker thread.

1app.post("/upload-video", upload.single("ssvideo"), (req, res) => {
2 if (isMainThread) {
3 let thread = new Worker("./threads/threaderone.js", {
4 workerData: {
5 file: req.file.path,
6 filename: req.file.filename,
7 watermark_image_url: image_url,
8 },
9 })
10
11 thread.on("message", data => {
12 res.download(data.file, req.file.filename)
13 })
14
15 thread.on("error", err => {
16 console.error("thread", err)
17 })
18
19 thread.on("exit", code => {
20 if (code != 0) console.error(`Worker stopped with exit code ${code}`)
21 })
22 }
23})

Here, we have three callback functions, they are message,error and exit. to get values from different events.

Creating Worker Threads

create a file called threaderone.js inside directory thread and add the following code.

1let ffmpeg = require("ffmpeg")
2const fs = require("fs")
3const { workerData, parentPort } = require("worker_threads")
4
5let dest = "/dest/video.mp4"
6
7try {
8 let process = new ffmpeg(workerData.file)
9
10 process.then(video => {
11 video.fnAddWatermark(
12 __dirname + "/watermark.png",
13 __dirname + "/" + workerData.filename,
14 {
15 position: "C",
16 },
17 function(err, file) {
18 if (!err) {
19 console.log("New video file is" + file)
20
21 parentPort.postMessage({ status: "Done", file: file })
22 }
23 }
24 )
25 })
26} catch (e) {
27 console.log(e.code)
28 console.log(e.msg)
29}

Basically, it gets the workerData from the main thread and do all the process of embedding watermark to the video and post the message to main thread.

Main thread gets the modified video and send it as response to the user.

In this process, main thread doesn't get blocked by the CPU intensive task.

Comments

Popular posts from this blog

How to use Ngx-Charts in Angular ?

Charts helps us to visualize large amount of data in an easy to understand and interactive way. This helps businesses to grow more by taking important decisions from the data. For example, e-commerce can have charts or reports for product sales, with various categories like product type, year, etc. In angular, we have various charting libraries to create charts.  Ngx-charts  is one of them. Check out the list of  best angular chart libraries .  In this article, we will see data visualization with ngx-charts and how to use ngx-charts in angular application ? We will see, How to install ngx-charts in angular ? Create a vertical bar chart Create a pie chart, advanced pie chart and pie chart grid Introduction ngx-charts  is an open-source and declarative charting framework for angular2+. It is maintained by  Swimlane . It is using Angular to render and animate the SVG elements with all of its binding and speed goodness and uses d3 for the excellent math functio...

Understand Angular’s forRoot and forChild

  forRoot   /   forChild   is a pattern for singleton services that most of us know from routing. Routing is actually the main use case for it and as it is not commonly used outside of it, I wouldn’t be surprised if most Angular developers haven’t given it a second thought. However, as the official Angular documentation puts it: “Understanding how  forRoot()  works to make sure a service is a singleton will inform your development at a deeper level.” So let’s go. Providers & Injectors Angular comes with a dependency injection (DI) mechanism. When a component depends on a service, you don’t manually create an instance of the service. You  inject  the service and the dependency injection system takes care of providing an instance. import { Component, OnInit } from '@angular/core'; import { TestService } from 'src/app/services/test.service'; @Component({ selector: 'app-test', templateUrl: './test.component.html', styleUrls: ['./test.compon...

How to solve Puppeteer TimeoutError: Navigation timeout of 30000 ms exceeded

During the automation of multiple tasks on my job and personal projects, i decided to move on  Puppeteer  instead of the old school PhantomJS. One of the most usual problems with pages that contain a lot of content, because of the ads, images etc. is the load time, an exception is thrown (specifically the TimeoutError) after a page takes more than 30000ms (30 seconds) to load totally. To solve this problem, you will have 2 options, either to increase this timeout in the configuration or remove it at all. Personally, i prefer to remove the limit as i know that the pages that i work with will end up loading someday. In this article, i'll explain you briefly 2 ways to bypass this limitation. A. Globally on the tab The option that i prefer, as i browse multiple pages in the same tab, is to remove the timeout limit on the tab that i use to browse. For example, to remove the limit you should add: await page . setDefaultNavigationTimeout ( 0 ) ;  COPY SNIPPET The setDefaultNav...