How to Upload CSV file on S3 Bucket using NodeJS

Posted By : Ankit Uniyal | 25-Dec-2017

nodejs

We will discuss about how to upload CSV file on S3 bucket in NodeJS.

Below is the code snippet :

var s3 = new AWS.S3();
var params = {
    Bucket: bucketName,
    Key: filePath,
    Body: csvFileContent,
    ContentType: 'application/octet-stream',
    ContentDisposition: contentDisposition(filePath, {
        type: 'inline'
    }),
    CacheControl: 'public, max-age=86400'
}
s3.putObject(params, function(err, data) {
    if (err) {
        console.log("Error at uploadCSVFileOnS3Bucket function", err);
        next(err);
    } else {
        console.log("File uploaded Successfully");
        next(null, filePath);
    }
});

Each header field specification:

1.Bucket : My Bucket name on S3 server.

2.Key : My file path in which my CSV file will be save.

3.Body : Content of my CSV file.

4.ContentType : We will be using 'application/octet-stream'.

5.ContentDisposition : We need to add "ContentDisposition" as a header field in my params object to successfully upload my CSV file on S3 bucket.

6.CacheControl: This is optional, when CDN request content from S3 bucket then this value is provided by S3 to CDN upto that time time CDN cache will not expire and CSN will then request to S3 after that time elapsed.

Also, I am using "json-2-csv" npm module for generating csv file content from JSON.

Below is the code :

var generateCSVFile = function(next, callback, csvFileContent) {
    console.log("generateCSVFile function started", csvFileContent);
    if (csvFileContent && csvFileContent.length > 0) {
        var options = {
            'prependHeader': true,
            'delimiter': {
                'wrap': '"',
                'field': ',',
                'eol': '\n'
            },
            'trimHeaderFields': true,
            'trimFieldValues': true,
            'checkSchemaDifferences': false
        };
        converter.json2csv(csvFileContent, function(err, csv) {
            if (err) {
                console.log("Error in csv file generation", err);
                next(err);
            } else {
                console.log("csv file is", csv);
                next(null, csv);
            }
        }, options);
    } else {
        next(null, []);
    }
}

Each field specification:

1.Delimiter :

a.Field : Default is ','

b.Wrap : The character in which field values will get wrapped.

Default : ' '

c.eol : Line Delimiter end.

Default : '\n'

2.trimHeaderFields : Do header fields needs to get trimmerd ?

Default : false

3.trimFieldValues : Do Field values needs to get trimmered ?

Default : false

4.checkSchemaDifferences : Is all documents have the same schema ?