Skip to main content

Scenario-based Learning - A New Learning Perspective. Node.js

Why it is Important?

Every time, we learn something. we go and find something to implement it to get a good grasp of it. From, a beginner perspective of the technology. it is great.

But, there is a gap between a beginner who learns the technology to a person who works in the production level application. we really don't know what kind of a problem the particular technology solves in real industrial applications until we work for a company/ a freelancing project

What I am going to do is, I will be sharing all the problem scenarios that I faced in the production. So, the beginner for the particular technology can replicate the scenario on his own and learn from it.

Basically, he/she is going to gain my experience through learning on his own. So, In the future, if he faces the same problem scenario, he/she can tackle it in an efficient way.

Node.js Experience

This Blog series is from my Node.js Experience. Basically, I am a React,Node.js and MongoDB Developer.

Soon, I will share problem scenarios for React as well. I will start with a simple scenario this week. In upcoming articles, I will share more complex scenarios where you can learn from it.

Scenario based Learning A New Learning Perspective

Problem Scenario

Recently, I faced a situation where I need to read a large sized file from internet and write it in my server.

To do this in Node.js, you can think of it like just read the file and write directly into the server.

But there is a problem with this approach, Let's say that we implement something like this

1const fs = require("fs")
2
3fs.readFileSync("sample.mkv", (err, data) => {
4  if (err) throw err
5
6  fs.writeFileSync("output", data, err => {
7    if (err) throw err
8  })
9})

the problem is,

bad code

There is a limit in Node.js Buffer, we can't store more than the Buffer size.

To address this problem, we need something called Stream in Node.js.

What is Stream?

Storing a complete file in Memory is so expensive. also, we need to store the file without having this problem. To solve this, we use Stream which is nothing but processing the data in chunks.

Stream process the huge data in a chunk by chunk which store the chunk in memory one at a time.

Solution

we need to create a readable stream which reads the data from the source and writable streamwhich writes the data to the destination.

If you are new to Node.js, I would suggest you watch some tutorials and try this problem scenarios.then, it will be easy to understand what is going on.

Solution Code

1const fs = require("fs")
2const stream = require("stream")
3
4//creating a readable stream
5const readable = fs.createReadStream("sample")
6
7//creating a writable stream
8const writable = fs.createWriteStream("output")
9
10fs.stat("sample", (err, stats) => {
11  this.filesize = stats.size
12
13  this.counter = 1
14
15  //this is an event which handles the data in chunks
16  readable.on("data", chunk => {
17    let percentageCopied = ((chunk.length * this.counter) / this.fileSize) * 100
18    process.stdout.clearLine()
19    process.stdout.cursorTo(0)
20    process.stdout.write(`${Math.round(percentageCopied)}%`)
21
22    //writing the chunk into writable stream output
23    writable.write(chunk)
24    this.counter += 1
25  })
26
27  readable.on("end", e => {
28    console.log("Read is completed")
29  })
30
31  readable.on("error", e => {
32    console.log("Some error occured: ", e)
33  })
34
35  writable.on("finish", () => {
36    console.log("Successfully created the file copy!")
37  })
38})

It reads the data from the local file and writes it again from another local file. For the conceptual purpose, I have used the local file itself rather than a file from the internet.

There is also a problem with this approach, if you analyze the Memory Manager in your machine while this code runs. it will take a lot of memory.

The reason being is, Disk write will not cope up with a speed of Disk Read . Reading a disk will be faster than Writing into a disk.

Beauty is, we can solve this problem too.

Efficient Solution

Node.js Stream has a solution for the above problem which is backpressure

1const stream = require("stream")
2const fs = require("fs")
3
4let fileName = "sample"
5
6const readabale = fs.createReadStream(fileName)
7const writeable = fs.createWriteStream("output")
8
9fs.stat(fileName, (err, stats) => {
10  this.fileSize = stats.size
11  this.counter = 1
12  this.fileArray = fileName.split(".")
13
14  try {
15    this.fileoutputName =
16      "output" + "/" + this.fileArray[0] + "_Copy." + this.fileArray[1]
17  } catch (e) {
18    console.exception("File name is invalid")
19  }
20
21  process.stdout.write(`File: ${this.fileoutputName} is being created:`)
22
23  readabale.on("data", chunk => {
24    let percentage = ((chunk.length * this.counter) / this.fileSize) * 100
25    process.stdout.clearLine() // clear current text
26    process.stdout.cursorTo(0)
27    process.stdout.write(`${Math.round(percentage)}%`)
28    this.counter += 1
29  })
30
31  //Note this line : Read Stream pipes the Write Streams
32  readabale.pipe(writeable)
33
34  // In case if we have an interruption while copying
35  writeable.on("unpipe", e => {
36    process.stdout.write("Write Failed!")
37  })
38})

The only change that we did with the previous solution is to pipe the readable stream to the writable stream

it will automatically control the disk read and write speed, thus it will not choke the RAM.

this is a simple solution to implement. This same concept can be used in some other Technical problem scenarios also

Scenario #2

consider a system where we have implemented a crawler which feeds the data to the Kafka. we need to get the data from Kafka pipeline and store it to Database.

Scenario #3

A User is uploading a huge size of files, we need to store it but, we can't able to store the size after a certain file size limit. what we can do is, implement the stream which reads the data and compresses it. store it in the server.

That's it for this article, Hope you like this series. I am planning to write more articles on this series if I can get a good response from this initiative.

Comments

Post a Comment

Popular posts from this blog

4 Ways to Communicate Across Browser Tabs in Realtime

1. Local Storage Events You might have already used LocalStorage, which is accessible across Tabs within the same application origin. But do you know that it also supports events? You can use this feature to communicate across Browser Tabs, where other Tabs will receive the event once the storage is updated. For example, let’s say in one Tab, we execute the following JavaScript code. window.localStorage.setItem("loggedIn", "true"); The other Tabs which listen to the event will receive it, as shown below. window.addEventListener('storage', (event) => { if (event.storageArea != localStorage) return; if (event.key === 'loggedIn') { // Do something with event.newValue } }); 2. Broadcast Channel API The Broadcast Channel API allows communication between Tabs, Windows, Frames, Iframes, and  Web Workers . One Tab can create and post to a channel as follows. const channel = new BroadcastChannel('app-data'); channel.postMessage(data); And oth...
Post a Comment
Read more

Certbot SSL configuration in ubuntu

  Introduction Let’s Encrypt is a Certificate Authority (CA) that provides an easy way to obtain and install free  TLS/SSL certificates , thereby enabling encrypted HTTPS on web servers. It simplifies the process by providing a software client, Certbot, that attempts to automate most (if not all) of the required steps. Currently, the entire process of obtaining and installing a certificate is fully automated on both Apache and Nginx. In this tutorial, you will use Certbot to obtain a free SSL certificate for Apache on Ubuntu 18.04 and set up your certificate to renew automatically. This tutorial will use a separate Apache virtual host file instead of the default configuration file.  We recommend  creating new Apache virtual host files for each domain because it helps to avoid common mistakes and maintains the default files as a fallback configuration. Prerequisites To follow this tutorial, you will need: One Ubuntu 18.04 server set up by following this  initial ...
Post a Comment
Read more

Working with Node.js streams

  Introduction Streams are one of the major features that most Node.js applications rely on, especially when handling HTTP requests, reading/writing files, and making socket communications. Streams are very predictable since we can always expect data, error, and end events when using streams. This article will teach Node developers how to use streams to efficiently handle large amounts of data. This is a typical real-world challenge faced by Node developers when they have to deal with a large data source, and it may not be feasible to process this data all at once. This article will cover the following topics: Types of streams When to adopt Node.js streams Batching Composing streams in Node.js Transforming data with transform streams Piping streams Error handling Node.js streams Types of streams The following are four main types of streams in Node.js: Readable streams: The readable stream is responsible for reading data from a source file Writable streams: The writable stream is re...
Post a Comment
Read more