Hugo + Node.js Koa App Connected to MongoDB

This project demonstrates how to create a development environment utilizing a Docker stack of Nginx to serve the static website, Nodejs for the api applications, MongoDB for the api data and Traefik for reverse proxy. This stack is suitable for deployment to staging and production environments.

Prerequisites

These products will need to be installed in order to complete this tutorial.

Project Setup

Create a directory for the entire project, e.g., hugo-koa-mongo. All of the project files will go in this folder. This folder will be referred to as the project root.

Hugo Static Website Generator

To get started, open a terminal in the project root and create a new Hugo site in a folder named www as follows.

hugo new site www

Add a Theme

There are numerous themes available at themes.gohugo.io to choose from. You can install one of them if you prefer or use this example to install my hugo-starter theme. Download and extract the theme into the www/themes/starter folder, or use Git and clone the theme from it’s git repository. For example,

git init
cd www
git submodule add https://github.com/jimfrenette/hugo-starter.git themes/starter

After the theme has been installed, update the config.toml site configuration file to use the theme. For example,

config.toml
theme = "starter"

Preview the site on the hugo dev server

cd www

hugo server

If the site loads, we’re ready to move onto the next step.

MongoDB

We will spin up a MongoDB Docker container for the api database. To demonstrate, we need to populate it with some data. For this, I have exported tables from the Chinook database into csv files which can then be imported using mongoimport.

You can download the csv files within the source code for this project or complete the process on your own as follows.

  1. Download the Chinook_Sqlite.sqlite database.

  2. Open it with DB Browser for SQLite

  3. Export these tables to csv files:

    • Album.csv
    • Artist.csv
    • Genre.csv
    • MediaType.csv
    • Track.csv

We’re going to copy an entrypoint folder with a shell script and all the csv files we exported into the MongoDB Docker image in order to populate the database. In the project root, create a new folder named docker with an entrypoint-initdb.d folder as follows.

mkdir -p docker/entrypoint-initdb.d

Copy or move all of the exported csv files into the docker/entrypoint-initdb.d folder.

In the docker folder, create a mongo.dockerfile that will create an image from mongo and copy the files in entrypoint-initdb.d into the docker-entrypoint-initdb.d folder of the new image.

mongo.dockerfile
FROM mongo

COPY ./entrypoint-initdb.d/* /docker-entrypoint-initdb.d/

In the docker/entrypoint-initdb.d folder, create this importChinook.sh script. This script will run when the image is created to populate MongoDB using the csv files.

importChinook.sh
mongoimport --db chinook --collection Album --type csv -f AlbumId,Title,ArtistId --file /docker-entrypoint-initdb.d/Album.csv
mongoimport --db chinook --collection Artist --type csv -f ArtistId,Name --file /docker-entrypoint-initdb.d/Artist.csv
mongoimport --db chinook --collection Genre --type csv -f GenreId,Name --file /docker-entrypoint-initdb.d/Genre.csv
mongoimport --db chinook --collection MediaType --type csv -f MediaTypeId,Name --file /docker-entrypoint-initdb.d/MediaType.csv
mongoimport --db chinook --collection Track --type csv -f TrackId,Name,AlbumId,MediaTypeId,GenreId,Composer,Milliseconds,Bytes,UnitPrice --file /docker-entrypoint-initdb.d/Track.csvnpm i nodemon -D

Node.js Koa API

The API is built using Koa.js Next generation web framework for Node.js. This app will accept requests to /api and return json data from the MongoDB Docker container.

In the project root, create a folder named api with src/server/chinook and src/server/routes folders within. For example,

mkdir -p api/src/server/{chinook,routes}

In the api/src/server/routes folder, create a chinook folder for the respective routes.

Project structure

  • hugo-koa-mongo
    • api
      • src
        • server
          • chinook
          • routes
            • chinook
    • docker
      • entrypoint-initdb.d
        • Album.csv
        • Artist.csv
        • Genre.csv
        • importChinook.sh
        • MediaType.csv
        • Track.csv
      • mongo.dockerfile
    • www

Initialize the Node.js app with npm init to create the package.json manifest file that will include all of the application dependency definitions and npm script commands for starting and building the app. For example,

cd api

npm init -y

In the src/server/chinook folder, create the connect.js module for our MongoDB connectivity.

connect.js
const bluebird = require('bluebird');
const mongoose = require('mongoose');

const connStr =  'mongodb://mongo:27017';

const options = {
    dbName: 'chinook',
    useNewUrlParser: true/*,
    useCreateIndex: true,
    useFindAndModify: false,
    autoIndex: false, // Don't build indexes
    reconnectTries: Number.MAX_VALUE, // Never stop trying to reconnect
    reconnectInterval: 500, // Reconnect every 500ms
    poolSize: 10, // Maintain up to 10 socket connections
    // If not connected, return errors immediately rather than waiting for reconnect
    bufferMaxEntries: 0,
    connectTimeoutMS: 10000, // Give up initial connection after 10 seconds
    socketTimeoutMS: 45000, // Close sockets after 45 seconds of inactivity
    family: 4 // Use IPv4, skip trying IPv6
    */
};

// Set mongoose Promise to Bluebird
mongoose.Promise = bluebird

// Retry connection
const connectWithRetry = () => {
  console.log('MongoDB connection with retry')
  return mongoose.connect(connStr, options)
}

// Exit application on error
mongoose.connection.on('error', err => {
  console.log(`MongoDB connection error: ${err}`)
  setTimeout(connectWithRetry, 5000)
  // process.exit(-1)
})

mongoose.connection.on('connected', () => {
  console.log('MongoDB is connected')
})

// if (config.env === 'development') {
//   mongoose.set('debug', true)
// }

const connect = () => {
  connectWithRetry()
}

module.exports = connect;

The following npm i or npm install commands are run from the api directory. When the install commands are run, the package.json file is updated with the respective package version info.

Install the MongoDB Node.js driver, mongodb.

npm i mongodb

Install connect.js dependencies, bluebird and mongoose.

npm i bluebird mongoose

Bluebird is a fully-featured Promise library and Mongoose is a Object Document Mapper (ODM) for MongoDB.

Models

In the src/server/chinook folder, create the data models. For example,

album.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const AlbumSchema = new Schema({
    AlbumId: Number,
    Name: String,
    ArtistId: Number
},{ 
    collection: 'Album'
});

module.exports = mongoose.model('Album', AlbumSchema);
artist.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

/*
 notice there is no ID. That's because Mongoose will assign
 an ID by default to all schemas

 by default, Mongoose produces a collection name by passing the model name to
 the utils.toCollectionName method.
 This method pluralizes the name Artist to Artists.
 Set this option if you need a different name for your collection.
*/

const ArtistSchema = new Schema({
    ArtistId: Number,
    Name: String
},{ 
    collection: 'Artist'
});

module.exports = mongoose.model('Artist', ArtistSchema);
track.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const TrackSchema = new Schema({
    TrackId: Number,
    Name: String,
    AlbumId: Number,
    MediaTypeId: Number,
    GenreId: Number,
    Composer: String,
    Milliseconds: Number,
    Bytes: Number,
    UnitPrice: String
},{ 
    collection: 'Track'
});

module.exports = mongoose.model('Track', TrackSchema);

Koa

Install koa and koa-router.

npm i koa koa-router

Routes

In the src/server/routes folder, create the default api route. For example,

index.js
const Router = require('koa-router');
const router = new Router();

router.get('/api/', async (ctx) => {
  ctx.body = {
    status: 'success',
    message: 'hello, world!'
  };
})

module.exports = router;

In the src/server/routes/chinook folder, create the api/chinook routes. For example,

album.js
const Router = require('koa-router');

const connect = require('../../chinook/connect');

connect();

const router = new Router();
const BASE_URL = `/api/chinook`;

const Album = require('../../chinook/album');

function getAlbums(artist) {
    return new Promise((resolve, reject) => {
        var query = Album.find({ 'ArtistId': artist });
        query.exec((err, results) => {
            if (err) return handleError(err);
            resolve(results);
        });
    });
}

router.get(BASE_URL + '/albums/:artist', async (ctx) => {
    try {
        ctx.body = await getAlbums(ctx.params.artist);
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;
artist.js
const Router = require('koa-router');

const connect = require('../../chinook/connect');

connect();

const router = new Router();
const BASE_URL = `/api/chinook`;

const Artist = require('../../chinook/artist');

function getArtists() {
    return new Promise((resolve, reject) => {
        var query = Artist.find();
        query.exec((err, results) => {
            if (err) return handleError(err);
            resolve(results);
        });
    });
}

router.get(BASE_URL + '/artists', async (ctx) => {
    try {
        ctx.body = await getArtists();
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;
track.js
const Router = require('koa-router');

const connect = require('../../chinook/connect');

connect();

const router = new Router();
const BASE_URL = `/api/chinook`;

const Track = require('../../chinook/track');

function getTracks(album) {
    return new Promise((resolve, reject) => {
        var query = Track.find({ 'AlbumId': album });
        query.exec((err, results) => {
            if (err) return handleError(err);
            resolve(results);
        });
    });
}

router.get(BASE_URL + '/tracks/:album', async (ctx) => {
    try {
        ctx.body = await getTracks(ctx.params.album);
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;

App Entrypoint

Create a src/server/index.js application entrypoint file as follows to tie everything together.

index.js
const Koa = require('koa');
const indexRoutes = require('./routes/index');
const artistRoutes = require('./routes/chinook/artist');
const albumRoutes = require('./routes/chinook/album');
const trackRoutes = require('./routes/chinook/track');

const app = new Koa();
const PORT = process.env.PORT || 1337;

app.use(indexRoutes.routes());
app.use(artistRoutes.routes());
app.use(albumRoutes.routes());
app.use(trackRoutes.routes());


const server = app.listen(PORT, () => {
  console.log(`Server listening on port: ${PORT}`);
});

module.exports = server;

npm-run-script

To build the respective dev or prod versions of the api server, in the package.json file under scripts, define the dev and start commands. These commands are executed when the Docker container is started based on the settings in the docker-compose.yml.

package.json
...

"scripts": {
    "dev": "nodemon ./src/server/index.js",
    "start": "node ./src/server/index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
}

Since nodemon is needed to watch and rebuild our api app in dev mode, let’s install it and save it as a dev dependency.

npm i nodemon -D

Docker Compose

To install the docker images, create our containers and start up our environment, add this docker-compose.yml file to the project root. Note that the volume paths map the project files to their paths within the Docker containers. For example, the Hugo publish directory www/public maps to the nginx server path for html, /usr/share/nginx/html.

version: "3"

services:

  app:
    image: node:alpine
    container_name: "hkm_node"
    user: "node"
    working_dir: /home/node/app
    labels:
      - 'traefik.backend=hkm_node'
      - 'traefik.frontend.rule=Host: localhost; PathPrefix: /api'
    environment:
      - NODE_ENV=production
    volumes:
      - ./api:/home/node/app
      - ./api/node_modules:/home/node/node_modules
    expose:
      - "1337"
    # command: "node ./src/server/index.js"
    command: "npm run dev"
    depends_on:
      - mongo

  mongo:
    build:
      context: ./docker
      dockerfile: mongo.dockerfile
    container_name: "hkm_mongo"
    labels:
      - 'traefik.backend=hkm_mongo'
    ports:
      - "27017:27017"
    volumes:
      - mongodata:/data/db

  nginx:
    image: nginx
    container_name: "hkm_nginx"
    labels:
      - 'traefik.backend=hkm_nginx'
      - 'traefik.frontend.rule=Host: localhost'
    volumes:
      - ./www/public:/usr/share/nginx/html

  traefik:
    image: traefik
    container_name: "hkm_traefik"
    command: -c /dev/null --docker --logLevel=INFO
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  mongodata:

In the project root, run docker-compose up -d which starts the containers in the background and leaves them running. The -d is for detached mode.

nginx 403

If you get a 403 Forbidden nginx server message, it’s because we didn’t publish the Hugo site.

cd www

hugo

To see the published Hugo site, restart the services in the project root using docker-compose. The -d switch is for disconnected mode, for example,

docker-compose down

docker-compose up -d

API Test

Load localhost/api/chinook/artists in a browser to see the json response.

For troubleshooting, view the docker conatainer logs or spin up in connected mode by omitting the -d switch, e.g., docker-compose up.


All of the source code for this tutorial is available on GitHub.

Source Code

comments powered by Disqus