NodeJS Buffer Explained

Before TypedArray didn't come into the picture, the JavaScript had no mechanism for manipulating or reading binary streams. Binary streams are the streams for processing binary data i.e 0 and 1.

In this article, I am going to explain about Buffers in NodeJS. NodeJS introduced buffers module which provides a way of handling streams of binary data.

Before going to discuss in deep about Buffer, Let's understand first What is Buffer?

What is Buffer?

Buffers are the mechanism for reading and manipulating streams of binary data i.e. 0 and 1. Buffers(The Buffers are the global object in node and you can use them anywhere in your nodejs application. there is no need to require it using require keyword) are the instances of the Buffer class in nodejs, which are designed to handle raw binary data. Each buffer represents some raw memory which is allocated outside V8.

Typical example and definition

We know that a stream is the movement of data from one end to another end. but how exactly they are moved in nodejs?

Suppose, the rate at which data arrives is faster than the rate which a process consumes the data, the excess data needs to be stored somewhere for this data to be processed.

On the other side, if the process is consuming data faster than the rate it arrives, the some data that arrives earlier needs to wait or stayed for a certain amount of data for processing later.

The waiting area, where this excess data waits, is the buffer!. It is a small physical location in our computer(usually in the RAM), where data are temporally stored, wait, and are sent out for processing eventually during streaming.

A typical example we generally see. You could see buffer in action when you’re streaming a video online. If your internet speed is fast enough, the speed of the streams will be fast to instantly fill the buffer and send it out for processing. On the other side, if the internet speed is slow then your streaming video paused for some time and display a loading icon until a buffer is full and sent out for processing. This process continuously goes on until the streams are finished.

Where You See Buffers

Buffers are generally seen when we talk about binary data coming from binary streams, such as fs.createReadStream. fs.createReadStream is used to read the content of a file.

Usage

There are a few ways to create new buffers:

var buffer = new Buffer(8);
This buffer is uninitialized and contains 8 bytes.

var buffer = new Buffer([ 8, 6, 7, 5, 3, 0, 9]);

The above piece of code initializes the buffer to the contents to this array.

var buffer = new Buffer("I'm a string!", "utf-8");

This initializes the buffer to a binary encoding of the first string as specified by the second argument (in this case, utf-8). 

Buffer also supports the following:
 
ascii: This encoding is way fast, but is limited to the ASCII character set. Moreover, it will convert null characters into spaces, unlike the UTF-8 encoding.

ucs2: A two-byte, little-endian encoding. Can encode a subset of Unicode.

base64: Base64 string encoding.

binary: This is the "binary string" format mentioned earlier, and is in the process of being deprecated. Avoid its use.

Writing to Buffers

Given that there is already a buffer created:

var buffer = new Buffer(16);

we can start writing strings to it:

buffer.write("Hello", "utf-8")
5

The first argument of buffer.write represent the string that will be  written to the buffer, and second argument is the encoding of first argument. utf-8 encoding is the default.

buffer.write returned 5. This means that we wrote to five bytes of the buffer.

buffer.write(" world!", 5, "utf-8")
7

When we provide three arguments to buffer.write, the second argument is an offset, or index of the buffer to start writing at given index.

Reading from Buffers

toString:

Probably the most common way to read buffers is to use the toString() method, since many buffers contain text:

buffer.toString('utf-8')
'Hello world!\u0000k\t'

Again, the first argument is the encoding. In this case, we can see that the entire buffer is not used! because we know that how many bytes we have written to this buffer. 

buffer.toString("utf-8", 0, 12)
'Hello world!'

Buffers Methods

Buffer.isBuffer(object)

Returns true if object is a buffer, It is similar to Array.isArray.

Buffer.byteLength(string, encoding)

Check the number of bytes required to encode a string with a given encoding(which defaults to utf-8). For example:

var snowman = "?";
snowman.length
1
Buffer.byteLength(snowman)
3

Buffer.length

This is the length of your buffer, and represents how much memory is allocated. For example:

var buffer = new Buffer(16)
buffer.write(snowman)
3
buffer.length
16

In this example, the contents written to the buffer only consist of three groups (since they represent the single-character snowman), but the buffer's length is still 16, as it was initialized.

Buffer.copy

buffer.copy(target, targetStart=0, sourceStart=0, sourceEnd=buffer.length)
 

buffer.copy function allows you to copy the contents of one buffer into other. The first argument is the buffer on which we'll copy the contents of buffer, and the rest of the arguments allows copying only a subset of the source. For example:

var frosty = new Buffer(24)
var snowman = new Buffer("?", "utf-8")
frosty.write("Happy birthday! ", "utf-8")
16
snowman.copy(frosty, 16)
3
frosty.toString("utf-8", 0, 19)
'Happy birthday! ?'

Buffer.slice

buffer.slice(start, end=buffer.length)
 

Buffer.slice provides the same functionality as Array.prototype.slice. The slice is not a new buffer and it references a subset of the memory space. For example:

var puddle = frosty.slice(16, 19)
puddle.toString()
'?'
puddle.write("___")
3
frosty.toString("utf-8", 0, 19)
'Happy birthday! ___'