Hugo + Node.js Koa App Connected to MongoDB

This project demonstrates how to create a development environment utilizing a Docker stack of Nginx to serve the static website, Nodejs for the api applications, MongoDB for the api data and Traefik for reverse proxy. This stack is suitable for deployment to staging and production environments.

Prerequisites

These products will need to be installed in order to complete this tutorial.

Project Setup

Create a directory for the entire project, e.g., hugo-koa-mongo. All of the project files will go in this folder. This folder will be referred to as the project root.

Hugo Static Website Generator

To get started, open a terminal in the project root and create a new Hugo site in a folder named www as follows. 

hugo new site www

Add a Theme

There are numerous themes available at themes.gohugo.io to choose from. You can install one of them if you prefer or use this example to install my hugo-starter theme. Download and extract the theme into the www/themes/starter folder, or use Git and clone the theme from it’s git repository. For example,

git init
cd www
git submodule add https://github.com/jimfrenette/hugo-starter.git themes/starter

If you get an error, git submodule add https://github.com/jimfrenette/hugo-starter.git themes/starter 'www/themes/starter' already exists in the index. Unstage it using git rm --cached themes/starter.

After the theme has been installed, update the config.toml site configuration file to use the theme. For example,

config.toml
theme = "starter"

Preview the site on the hugo dev server

cd www

hugo server

If the site loads, we’re ready to move onto the next step.

MongoDB

We will spin up a MongoDB Docker container for the api database. To demonstrate, we need to populate it with some data. For this, I have exported tables from the Chinook database into csv files which can then be imported using mongoimport.

You can download the csv files within the source code for this project or complete the process on your own as follows.

  1. Download the Chinook_Sqlite.sqlite database.

  2. Open it with DB Browser for SQLite

  3. Export these tables to csv files:

    • Album.csv
    • Artist.csv
    • Genre.csv
    • MediaType.csv
    • Track.csv

We’re going to copy an entrypoint folder with a shell script and all the csv files we exported into the MongoDB Docker image in order to populate the database. In the project root, create a new folder named docker with an entrypoint-initdb.d folder as follows.

mkdir -p docker/entrypoint-initdb.d

Copy or move all of the exported csv files into the docker/entrypoint-initdb.d folder.

In the docker folder, create a mongo.dockerfile that will create an image from mongo and copy the files in entrypoint-initdb.d into the docker-entrypoint-initdb.d folder of the new image.

mongo.dockerfile
FROM mongo

COPY ./entrypoint-initdb.d/* /docker-entrypoint-initdb.d/

In the docker/entrypoint-initdb.d folder, create this importChinook.sh script. This script will run when the image is created to populate MongoDB using the csv files.

importChinook.sh
mongoimport --db chinook --collection Album --type csv -f AlbumId,Title,ArtistId --file /docker-entrypoint-initdb.d/Album.csv
mongoimport --db chinook --collection Artist --type csv -f ArtistId,Name --file /docker-entrypoint-initdb.d/Artist.csv
mongoimport --db chinook --collection Genre --type csv -f GenreId,Name --file /docker-entrypoint-initdb.d/Genre.csv
mongoimport --db chinook --collection MediaType --type csv -f MediaTypeId,Name --file /docker-entrypoint-initdb.d/MediaType.csv
mongoimport --db chinook --collection Track --type csv -f TrackId,Name,AlbumId,MediaTypeId,GenreId,Composer,Milliseconds,Bytes,UnitPrice --file /docker-entrypoint-initdb.d/Track.csvnpm i nodemon -D

Node.js Koa API

The API is built using Koa.js Next generation web framework for Node.js. This app will accept requests to /api and return json data from the MongoDB Docker container.

In the project root, create a folder named api with src/server/chinook and src/server/routes folders within. For example,

mkdir -p api/src/server/{chinook,routes}

In the api/src/server/routes folder, create a chinook folder for the respective routes.

Project structure

  • hugo-koa-mongo
    • api
      • src
        • server
          • chinook
          • routes
            • chinook
    • docker
      • entrypoint-initdb.d
        • Album.csv
        • Artist.csv
        • Genre.csv
        • importChinook.sh
        • MediaType.csv
        • Track.csv
      • mongo.dockerfile
    • www

Initialize the Node.js app with npm init to create the package.json manifest file that will include all of the application dependency definitions and npm script commands for starting and building the app. For example,

cd api

npm init -y

The following npm i or npm install commands are run from the api directory. When the install commands are run, the package.json file is updated with the respective package version info.

Install the MongoDB Node.js driver, mongodb.

npm i mongodb

Install mongoose for a schema-based solution to model the application data. It also includes built-in type casting, validation, query building, business logic hooks and more.

npm i mongoose

Models

In the src/server/chinook folder, create the data models. For example,

album.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const AlbumSchema = new Schema({
    AlbumId: Number,
    Name: String,
    ArtistId: Number
},{
    collection: 'Album'
});

const chinook = mongoose.connection.useDb('chinook');

module.exports = chinook.model('Album', AlbumSchema);
artist.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

/*
 notice there is no ID. That's because Mongoose will assign
 an ID by default to all schemas

 by default, Mongoose produces a collection name by passing the model name to
 the utils.toCollectionName method.
 This method pluralizes the name Artist to Artists.
 Set this option if you need a different name for your collection.
*/

const ArtistSchema = new Schema({
    ArtistId: Number,
    Name: String
},{
    collection: 'Artist'
});

const chinook = mongoose.connection.useDb('chinook');

module.exports = chinook.model('Artist', ArtistSchema);
track.js
const mongoose = require('mongoose');
const Schema = mongoose.Schema;

const TrackSchema = new Schema({
    TrackId: Number,
    Name: String,
    AlbumId: Number,
    MediaTypeId: Number,
    GenreId: Number,
    Composer: String,
    Milliseconds: Number,
    Bytes: Number,
    UnitPrice: String
},{
    collection: 'Track'
});

const chinook = mongoose.connection.useDb('chinook');

module.exports = chinook.model('Track', TrackSchema);

Koa

Install koa and koa-router.

npm i koa koa-router

Routes

In the src/server/routes folder, create the default api route. For example,

index.js
const Router = require('koa-router');
const router = new Router();

router.get('/api/', async (ctx) => {
  ctx.body = {
    status: 'success',
    message: 'hello, world!'
  };
})

module.exports = router;

In the src/server/routes/chinook folder, create the api/chinook routes. For example,

album.js
const Router = require('koa-router');

const router = new Router();
const BASE_URL = `/api/chinook`;

const Album = require('../../chinook/album');

function getAlbums(artist) {
    return new Promise((resolve, reject) => {
        var query = Album.find({ 'ArtistId': artist });
        query.exec((err, results) => {
            if (err) {
                resolve(err);
            } else {
                resolve(results);
            }
        });
    });
}

router.get(BASE_URL + '/albums/:artist', async (ctx) => {
    try {
        ctx.body = await getAlbums(ctx.params.artist);
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;
artist.js
const Router = require('koa-router');

const router = new Router();
const BASE_URL = `/api/chinook`;

const Artist = require('../../chinook/artist');

function getArtists() {
    return new Promise((resolve, reject) => {
        var query = Artist.find();
        query.exec((err, results) => {
            if (err) {
                resolve(err);
            } else {
                resolve(results);
            }
        });
    });
}

router.get(BASE_URL + '/artists', async (ctx) => {
    try {
        ctx.body = await getArtists();
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;
track.js
const Router = require('koa-router');

const router = new Router();
const BASE_URL = `/api/chinook`;

const Track = require('../../chinook/track');

function getTracks(album) {
    return new Promise((resolve, reject) => {
        var query = Track.find({ 'AlbumId': album });
        query.exec((err, results) => {
            if (err) {
                resolve(err);
            } else {
                resolve(results);
            }
        });
    });
}

router.get(BASE_URL + '/tracks/:album', async (ctx) => {
    try {
        ctx.body = await getTracks(ctx.params.album);
    } catch (err) {
        console.log(err)
    }
})

module.exports = router;

App Entrypoint

Create a src/server/index.js application entrypoint file as follows to initiate the app, routes and configure the MongoDB connection.

index.js
const Koa = require('koa');
const mongoose = require('mongoose');
const indexRoutes = require('./routes/index');
const artistRoutes = require('./routes/chinook/artist');
const albumRoutes = require('./routes/chinook/album');
const trackRoutes = require('./routes/chinook/track');

/**
 * Koa app */
const app = new Koa();
const PORT = process.env.PORT || 1337;
const server = app.listen(PORT, () => {
    console.log(`Server listening on port: ${PORT}`);
});

/**
 * MongoDB connection */
const connStr =  'mongodb://mongo:27017/default';
mongoose.connect(connStr, {useNewUrlParser: true});
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
    console.log('connected');
});

app.use(indexRoutes.routes());
app.use(artistRoutes.routes());
app.use(albumRoutes.routes());
app.use(trackRoutes.routes());

module.exports = server;

npm-run-script

To build the respective dev or prod versions of the api server, in the package.json file under scripts, define the dev and start commands. These commands are executed when the Docker container is started based on the settings in the docker-compose.yml.

package.json
...

"scripts": {
    "dev": "nodemon ./src/server/index.js",
    "start": "node ./src/server/index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
}

Since nodemon is needed to watch and rebuild our api app in dev mode, let’s install it and save it as a dev dependency.

npm i nodemon -D

Docker Compose

To install the docker images, create our containers and start up our environment, add this docker-compose.yml file to the project root. Note that the volume paths map the project files to their paths within the Docker containers. For example, the Hugo publish directory www/public maps to the nginx server path for html, /usr/share/nginx/html.

version: "3"

services:

  app:
    image: node:alpine
    container_name: "${DEV_PROJECT_NAME}_node"
    user: "node"
    working_dir: /home/node/app
    labels:
      - 'traefik.backend=${DEV_PROJECT_NAME}_node'
      - 'traefik.frontend.rule=Host: ${DEV_PROJECT_HOST}; PathPrefix: /api'
    environment:
      - NODE_ENV=production
    volumes:
      - ./api:/home/node/app
      - ./api/node_modules:/home/node/node_modules
    expose:
      - "1337"
    # command: "node ./src/server/index.js"
    command: "npm run dev"
    depends_on:
      - mongo

  mongo:
    build:
      context: ./docker
      dockerfile: mongo.dockerfile
    container_name: "${DEV_PROJECT_NAME}_mongo"
    labels:
      - 'traefik.backend=${DEV_PROJECT_NAME}_mongo'
    ports:
      - "27017:27017"
    volumes:
      - mongodata:/data/db

  nginx:
    image: nginx
    container_name: "${DEV_PROJECT_NAME}_nginx"
    labels:
      - 'traefik.backend=${DEV_PROJECT_NAME}_nginx'
      - 'traefik.frontend.rule=Host: ${DEV_PROJECT_HOST}'
    volumes:
      - ./www/public:/usr/share/nginx/html

  traefik:
    image: traefik:v1.7
    container_name: "${DEV_PROJECT_NAME}_traefik"
    command: -c /dev/null --docker --logLevel=INFO
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  mongodata:

update: note that we’re pulling a traefik version 1 image in the docker-compose.yml above. The traefik:latest image switched to version 2 a few months after this post was originally published. A configuration migration guide can be found here should you want to use version 2.

I like to use an .env file to configure docker-compose variables. In the project root, create this .env file.

### PROJECT SETTINGS

DEV_PROJECT_NAME=hkm
DEV_PROJECT_HOST=localhost

In the project root, run docker-compose up -d which starts the containers in the background and leaves them running. The -d is for detached mode.

nginx 403

If you get a 403 Forbidden nginx server message, it’s because we didn’t publish the Hugo site.

cd www

hugo

If you get permissions access error when publishing using hugo for www/public or any of its files or directories. Delete the public folder and try again.

To see the published Hugo site, restart the services in the project root using docker-compose. The -d switch is for disconnected mode, for example,

docker-compose down

docker-compose up -d

API Test

Load localhost/api/chinook/artists in a browser to see the json response.

For troubleshooting, view the docker conatainer logs or spin up in connected mode by omitting the -d switch, e.g., docker-compose up.

All of the source code for this tutorial is available on GitHub.

Source Code

Part 1 of 2 in the hugo-koa-mongo series.

Hugo + Node.js Koa App Connected to MongoDB Part II

comments powered by Disqus