Setting up a GTFS app

This tutorial and its repository show how to build the core setup used by most of MobilityDataDev's GTFS projects. They contains starter code/configuration for:

  • a frontend web application
  • a NodeJS backend server
  • a PostgreSQL database (containing GTFS data)
  • and Docker containers to run it all.

Let's get started!

0. Prerequisites

First, make sure you have the required programmes and a repository available on your machine.

  • Install Visual Studio Code.
  • Install Docker Desktop.
  • Install NodeJS.
  • Create a new directory on your machine and open it in VSCode.
  • Save your work on GitHub with:
    • git init
    • git branch -M main
    • git add .
    • git commit -m"v1.0.0"
    • git remote add origin https://github.com/[username]/[repository name].git
    • git push -u origin main

If you want, you can allocate a specific amount of resources to Docker. I'm using 4GB of memory and 2 processors, set with this command: " [wsl2]```n memory=4GB```n processors=2" | Out-File -FilePath "$env:USERPROFILE\.wslconfig" -Encoding utf8 followed by wsl --shutdown. You can also check your current settings, if they exist, with cat "$env:USERPROFILE\.wslconfig".

Ok, you're all set up. Let's build the foundations for a GTFS application!

1. NodeJS app

Start with a super basic app that only logs a message, just to have visual confirmation that it works.

  1. Create a ./src/ directory at the root of the repository. This will hold all the JavaScript (and later Typescript) files, which basically constitute the brain of the app.

  2. Inside that subdirectory, create the app's entry file, i.e. ./src/index.js, containing the message:

console.log(`I'm an app!`);
  1. Run npm init to generate the default ./package.json file, giving ./src/index.js as the entry point when prompted.

Test it by running node ./src/index.js. It will display the "I'm an app!" message in the terminal. Perfect.

Actually, nearly perfect. It would be better in a Docker container.

2. Docker containers

Create the following files at the root of the repository to build a NodeJS container.

Containerizing your Node application has numerous benefits. First, Docker's friendly, CLI-based workflow lets any developer build, share, and run containerized Node applications. Second, developers can install their app from a single package and get it up and running in minutes. Third, Node developers can code and test locally while ensuring consistency from development to production. – Source

./Dockerfile

A Dockerfile is a blueprint for a custom Docker image. Ours is the combination of NodeJS' default image and of the app's packages listed in ./packages.json (i.e. the app's dependencies).

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. – Source

An image is a read-only template with instructions for creating a Docker container. Often, an image is based on another image, with some additional customization. – Source

Containers have everything that your code needs in order to run, down to a base operating system. – Source

A package in Node.js contains all the files you need for a module. Modules are JavaScript libraries you can include in your project. – Source

### Custom Node image for gtfs-app ###

# Uses Node's default image as the template.
FROM node:alpine

# Creates the app's directory inside the container.
WORKDIR /gtfs-app

# Copies the app's package(-lock).json files into the directory above.
# These files lists the app's dependencies/packages.
COPY package*.json ./

# Installs the listed dependencies in the app's directory.
RUN npm install

# Generates a new image based on the contents of the current directory
# (= the first dot) and saves it in the current directory (= the second dot).
COPY . .

./docker-compose.yml

This file defines all the containers used in the project. So far you only need a NodeJS container (made from the custom image defined in the ./Dockerfile). Later, this will also include a PostgreSQL database and a pgAdmin user interface.

Docker Compose is a tool for defining and running multi-container applications. – Source

version: '3.9'

services:
  node-app-service:
    container_name: node-app-container
    build:
      context: .
      dockerfile: Dockerfile
    command: sh -c 'node ./src/index.js'
    volumes:
      - ./src/:/gtfs-app/src/

3. Launch!

Run docker-compose up --build in VSCode's terminal. You should still see "I'm an app!" appear in the logs.

Congratulations, you now have a NodeJS app running on your machine through Docker!

How?

docker-compose up triggers the creation of the container defined in ./docker-compose.yml. By appending the --build flag, it also triggers the creation of the Docker image defined in the ./Dockerfile. Then, it runs the container's command which starts the app from the ./src/ codebase.

Note that it takes time to download/install all the dependencies to generate the Docker image. So from now on, as long as the Docker image is still valid (i.e. no edits to ./Dockerfile and no new packages), the --build flag should be omitted.

Try it: run docker-compose up. The "I'm an app!" message appears instantly this time.

4. TypeScript

Next, let's replace JavaScript with TypeScript. This really improves the developer experience.

TypeScript is JavaScript with syntax for types. TypeScript adds additional syntax to JavaScript to support a tighter integration with your editor. Catch errors early in your editor. – Source

  1. Run npm install typescript @types/node ts-node --save-dev in VSCode's terminal to enable TypeScript.

  2. Create a .gitignore file at the root of the repository to exclude this directory created by npm and the build directory:

build/
node_modules/
  1. Run tsc --init to generate the default ./tsconfig.json file, then uncomment and edit the following lines:
{
  "target": "es6",
  "rootDir": "./src",
  "moduleResolution": "node",
  "baseUrl": "./src",
  "outDir": "./build"
}

  1. Change the extension of the app's entry file, ./src/index.js, to .ts.

  2. In ./package.json, update the file extension in the line "main": "./src/index.js" to .ts and add two commands to the scripts property:

{
  "scripts": {
    "build": "tsc",
    "start": "node ./build/index.js"
  }
}
  1. Update the command in ./docker-compose.yml so that it compiles the TypeScript codebase into JavaScript, then launches the compiled code:
services:
  gtfs-import-node-app-service:
    command: sh -c "npm run build && npm run start"
  1. Create a .dockerignore file at the root of the repository to tell Docker what to omit when building an image:
.git/
.gitignore
.vscode/
build/
LICENCE
node_modules/
nodemon.json
public/
README.md
sample-data/

Re-run docker-compose up --build. You should still get the "I'm an app!" message, but this time it's from a TypeScript app. Cool!

5. Nodemon

Final bit of environment setup: nodemon. This isn't necessary, but it makes the local development experience a bit nicer.

"nodemon is a tool that helps develop Node.js based applications by automatically restarting the node application when file changes in the directory are detected." -- Source

  1. Run npm install nodemon --save-dev.

  2. Create ./nodemon.json to watch for changes in the codebase and automatically restart the NodeJS app when they occur:

{
  "verbose": true,
  "ignore": ["src/**/*.spec.ts"],
  "watch": ["src/**/*.ts"],
  "exec": "node --inspect=0.0.0.0:9229 --nolazy -r ts-node/register"
}
  1. In ./package.json, add a "dev" command to the scripts property:
{
  "scripts": {
    "dev": "nodemon -L ./src/index.ts",
    "build": "tsc",
    "start": "node ./build/index.js"
  }
}
  1. Update the command in ./docker-compose.yml to launch the app using the dev script, and add a volume for nodemon's configuration file:
services:
  node-app-service:
    command: sh -c "npm run dev"
    volumes:
      - ./nodemon.json:/gtfs-app/nodemon.json

Update the containers once more with docker-compose up --build. You see the "I'm a NodeJS app!" message again, but also a bunch of new lines prefixed with [nodemon] including this one: [nodemon] watching 1 file. This "1 file" is ./src/index.ts. Edit it to change the message to the standard "Hello World", save, and watch the app automatically reload and display the new message!

Phew. We're done with the foundations. Now for the fun part.

6. Express server

So far, your NodeJS app only outputs a message, which is pretty useless. The next step is to replace this with an Express server able to handle requests from the frontend and respond with GTFS data.

Express.js is the most popular web framework for Node.js. It is designed for building web applications and APIs and has been called the de facto standard server framework for Node.js. – Source

  1. Run npm install express and npm install @types/express --save-dev.

  2. Overwrite ./src/index.ts with Express' starter code, enriched with some TypeScript typing:

//
import express, { Request, Response, Application } from 'express';

//
const app: Application = express();

//
app.get('/', (request: Request, response: Response) => {
  response.send(`I'm an app!`);
});

//
app.listen(4000, () => {
  console.log(`Server listening on http://localhost:4000/`);
});
  1. Map your browser's 4000 port to the container's in ./docker-compose.yml:
services:
  node-app-service:
    ports:
      - '4000:4000'

  1. Run docker-compose up --build again to include the new dependencies. The app loads and... there's no more message! It makes sense, it's gone from the codebase. But there's something else to see...

  2. Open http://localhost:4000/.

And there it is: "I'm an app!". The app now delivers content to the browser.

7. Frontend

Ok the browser is showing data from the backend, that's great but it's just ~an ugly~ a simple line of text. Time to turn this into a proper web page. We'll use the Pug template engine for that.

A template engine enables you to use static template files in your application. At runtime, the template engine replaces variables in a template file with actual values, and transforms the template into an HTML file sent to the client. – Source

  1. Run npm install pug and npm install @types/pg --save-dev.

  2. Create two new directories at the root of the repository: ./views/ and ./public/.

  3. In ./docker-compose.yml, add the new directories as volumes:

services:
  node-app-service:
    volumes:
      - ./public/:/gtfs-app/public/
      - ./views/:/gtfs-app/views/
  1. Create a ./views/index.pug file containing basic HTML starter code and one test variable:
doctype html
html(lang="en")
  head
    meta(charset="UTF-8")
    meta(name="viewport", content="width=device-width, initial-scale=1.0")
    title GTFS App
    link(rel="stylesheet", href="/style.css")
  body
    p= test
  1. Create a ./public/style.css file to make the text a bit nicer, and red for obvious visual confirmation that it works:
p {
  font-family: Helvetica, Arial, sans-serif;
  font-size: 15px;
  color: red;
}
  1. Edit ./src/index.ts to define pug as the template engine, set the views and public directories, and render ./views/index.pug instead of simply logging a message:
import express, { Request, Response, Application } from 'express';
//
import path from 'path';

const app: Application = express();

//
app.set('view engine', 'pug');
//
app.set('views', path.join(__dirname, '..', 'views'));
//
app.use(express.static(path.join(__dirname, '..', 'public')));

app.get('/', (request: Request, response: Response) => {
  //
  response.render('index', { test: `I'm an app!` });
});

app.listen(4000, () => {
  console.log(`Server listening on http://localhost:4000/`);
});

Run docker-compose up --build again, refresh the browser, the message is still there, in red this time. It works but it's still pretty bare.

  1. Use Bootstrap's starter template for a nice-looking base layout:
doctype html
html(lang="en")
  head
    meta(charset="UTF-8")
    meta(name="viewport", content="width=device-width, initial-scale=1.0")
    title GTFS App
    link(
      rel="stylesheet"
      type="text/css"
      href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css"
      integrity="sha384-QWTKZyjpPEjISv5WaRU9OFeRpok6YctnYmDr5pNlyT2bRjXh0JMhjY6hW+ALEwIH"
      crossorigin="anonymous")
    link(rel="stylesheet", href="/style.css")
  body
    nav.navbar.navbar-dark.bg-dark
      a.navbar-brand.px-2(href="/") GTFS App
    div.main.container-fluid
      div.row
        div.col
          h1.p-5.text-center= test
    script(
      src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js"
      integrity="sha384-YvpcrYf0tY3lHB60NNkmXc5s9fDVZLESaAA55NDzOxhy9GkcIdslK1eN7N6jIeHz"
      crossorigin="anonymous"
    )

It looks much better already. Now let's finally get some GTFS data in here instead of this random "I'm an app!" message.

8. ProgreSQL database

To run a database locally, you're going to need a new Docker container plus an extra package, because NodeJS doesn't have ready-made features to connect and interact with a database.

There are several databases available, including NoSQL (e.g. MongoDB), MySQL, and PostgreSQL. We recommend the latter: it's the richest, most powerful of the three options, and it comes with a PostGIS extension which is very handy to work on GTFS data. Next, there are yet more options to use that database in a NodeJS app, such as ORMs, pg-promise and node-postgres. The latter is the lowest level of all, which usually means it's more efficient, so that sounds good.

PostgreSQL: The World's Most Advanced Open Source Relational Database – Source

PostGIS extends the capabilities of the PostgreSQL relational database by adding support for storing, indexing, and querying geospatial data. – Source

Object Relational Mapping (ORM) is a technique used in creating a "bridge" between object-oriented programs and, in most cases, relational databases. – Source

[pg-promise is a] PostgreSQL interface for Node.js (...) built on top of node-postgres. – Source

node-postgres is a collection of node.js modules for interfacing with your PostgreSQL database. – Source

Read: Difference between High Level and Low level languages

  1. In ./docker-compose.yml, add a new container definition based on PostgreSQL's latest official image:
services:
  postgresql-database-service:
    container_name: postgresql-database-container
    image: postgres
    restart: always
    ports:
      - '5432:5432'
    environment:
      - POSTGRES_USER=root
      - POSTGRES_PASSWORD=root
      - POSTGRES_DB=root
  1. Load sample data on build by creating a new ./sample-data/ directory, filling it with SQL files from an existing database dump (like ours), and adding it as an "entry point" volume in ./docker-compose.yml:
services:
  postgresql-database-service:
    volumes:
      - ./sample-data/:/docker-entrypoint-initdb.d/
  1. Edit ./docker-compose.yml once more to share the new database's credentials with the NodeJS app, and make it a dependency:
services:
  node-app-service:
    depends_on:
      - postgresql-database-service
    environment:
      - POSTGRES_HOST=postgresql-database-container
      - POSTGRES_PORT=5432
      - POSTGRES_USER=root
      - POSTGRES_PASSWORD=root
      - POSTGRES_DB=root

  1. Install node-postgres with npm install pg and npm install @types/pg --save-dev.

  2. Create a new ./src/getDatabasePool.ts file to create a database connection as described in pg's documentation:

import { Pool } from 'pg';

export const getDatabasePool = (): Pool => {
  // "process" is a core module of NodeJS. Its "env" property hosts the
  // environment variables set when the process was started (i.e. those given
  // in ./docker-compose.yml).
  const { POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_DB, POSTGRES_PASSWORD } =
    process.env;

  return new Pool({
    host: POSTGRES_HOST,
    port: parseInt(POSTGRES_PORT, 10),
    database: POSTGRES_DB,
    user: POSTGRES_USER,
    password: POSTGRES_PASSWORD
  });
};
  1. Import and use the connection in ./src/index.ts to fetch GTFS stops from the database, wrapping the request in a try-catch for good measure:
import { getDatabasePool } from './getDatabasePool';

const pool = getDatabasePool();

app.get('/', async (request: Request, response: Response) => {
  try {
    const stops = await pool.query('SELECT stop_name FROM stops LIMIT 1');
    const message = `I'm a GTFS stop. My name is ${stops.rows[0].stop_name}`;
    response.render('index', { test: message });
  } catch (error) {
    console.error(error);
    response.status(500).send('Internal Server Error');
  }
});

You're nearly there! Run docker-compose up --build, wait for the new components to get installed, then refresh http://localhost:4000/. And voilà. It's done!

You now have a web page displaying a GTFS stop name, that was fetched from a database through a backend server. Good job!

9. pgAdmin user interface

Last chapter, promise! It's an easy one, too.

It can be hard working on a database without a graphical user interface. So let's add one. The most popular option for a PostgreSQL database is pgAdmin, which you can run as a local web-based application using yet another container.

A graphical user interface (GUI) is a digital interface in which a user interacts with graphical components such as icons, buttons, and menus. In a GUI, the visuals displayed in the user interface convey information relevant to the user, as well as actions that they can take. – Source

  1. In docker-compose.yml, add a new container definition for pgAdmin and a couple of volumes to connect it to the database:
services:
  postgresql-database-service:
    volumes:
      - local_pgdata:/var/lib/postgresql/data

  pgadmin-gui-service:
    container_name: pgadmin-gui-container
    image: dpage/pgadmin4
    restart: always
    ports:
      - '8001:80'
    environment:
      PGADMIN_DEFAULT_EMAIL: user@mail.com
      PGADMIN_DEFAULT_PASSWORD: root
      PGADMIN_CONFIG_WTF_CSRF_CHECK_DEFAULT: 'False'
      PGADMIN_CONFIG_WTF_CSRF_ENABLED: 'False'
    volumes:
      - pgadmin-data:/var/lib/pgadmin

volumes:
  local_pgdata:
  pgadmin-data:

Start the containers one last time with docker-compose up --build. It takes a bit of time for the new image to download, but when it's done the logs will display "pgadmin-gui-container (...) Listening at: http://[::]:80".

  1. Open http://localhost:8001/ in your browser.

  2. Log in using the credentials set in ./docker-compose.yml, i.e. email "user@mail.com" and password "root".

  3. In the top navigation bar, click Object > Register > Server... then fill in the credentials from our PostgreSQL database:

    • General:
      • Name: GTFS
    • Connection:
      • Host: postgresql-database-container
      • Port: 5432
      • Username: root
      • Password: root

  4. Click Save.

  5. In the sidebar, expand Databases > root > Schemas > public > Tables.

And there they are: GTFS tables with all of their columns and rows.

What's next?

Now, you can use this setup to build many different GTFS-based applications. Either build upon what you already have here, or fork our repository to start from a fresh copy.

A fork is a new repository that shares code and visibility settings with the original "upstream" repository. Forks are often used to iterate on ideas or changes before they are proposed back to the upstream repository, such as in open source projects or when a user does not have write access to the upstream repository. – Source