Service configuration

A service's configuration is the YAML files in the config/ folder. They are read by the AppConfig class from rock-me-ts: it gathers all the files, runs them through the EJS templating engine, merges the documents, and hands the service a single object with typed access to its sections.

How it is loaded#

1. dotenv reads .env, then .env.local (values from it override .env). Both files are only needed for local runs; in Docker the variables are set by the container environment.
2. All files matching config/**/*.yml are collected, including nested folders like config/custom/. The order is not sorted, so files are conventionally numbered: 02_config.yml, 11_clickhouse_struct.yml, custom/14_rst.yml.
3. The contents of all files are concatenated into a single text and rendered with EJS. Available in templates: env (environment variables), pkg (the contents of package.json), envName (dev, stage or production) and consts.
4. The result is read as a multi-document YAML and deep-merged left to right: whatever appears later overrides what came earlier. Arrays are replaced wholesale.
5. The dev and prod sections are extracted from the final object, and one of them is applied on top of the common part depending on NODE_ENV (production or prod — the prod section, otherwise dev).

Thus in chwriter the base table schema lives in config/11_clickhouse_struct.yml, while the additional columns for a specific installation live in config/custom/*.yml, which simply add keys to the same clickhouse.tables section.

Standard sections#

The framework expects them, and they exist in every service:

---
title: Rockstat kafka streamer
version: "<%= pkg.version %>"

rpc:
  name: kafkastream          # the service name = the name of its direct channel in Redis
  listen_all: true
  listen_direct: true

log:
  use: pino
  pino:
    safe: true
    level: <%= env.LOG_LEVEL || 'info' %>
    prettyPrint: false

redis:
  dsn: <%= env.REDIS_DSN %>

metrics:
  statsd:
    host: <%= env.STATSD_HOST || '127.0.0.1' %>
    port: 8125
    debug: false
    prefix: krn

Section	Access in code	Purpose
rpc	config.rpc	name — the service name; service_group — the group for enrichers; channels — the list of channels; listen_all, listen_direct — flags for AppRunner
log	config.log	pino settings: level, name, safe
redis	config.redis	dsn of the form redis://host:6379/0
metrics	config.meter	statsd — host, port, metrics prefix; without this section metrics are simply not sent

Your own section#

Put the service's settings in a separate section named after it, and describe its type in src/types.ts. AppConfig is parameterized with this type, and config.get('kafkastream') returns a typed object:

kafkastream:
  clientId: "rst-kafkastream"
  brokers: [<%= env.KAFKA_BROKERS || '' %>]
  ssl: false
  mappings:
    - ['in.gen.track.ping', null]
    - ['in.gen.track.', [rstat.web, uid]]

// src/types.ts
export type KafkaConfig = {
  clientId: string;
  brokers: Array<string>;
  ssl?: boolean;
  mappings: Array<[string, [string, string] | null]>;
};

export type ModuleConfig = {
  name: string;
  kafkastream: KafkaConfig;
};

const config = new AppConfig<ModuleConfig>();
const options = config.get('kafkastream');   // KafkaConfig

The dev: and prod: sections let you keep different values for environments in a single file:

clickhouse:
  uploadInterval: 2

dev:
  clickhouse:
    uploadInterval: 1

Environment variables#

The framework and services read variables directly or through configuration templates:

Variable	Who reads it	Value
NODE_ENV	AppConfig	production enables the prod section, everything else — dev
REDIS_DSN	configuration	the Redis address, in Docker defaults to redis://redis:6379
LOG_LEVEL	configuration	the pino level: debug, info, warn
STATSD_HOST	configuration	the StatsD host, defaults to 127.0.0.1
TZ	Node.js	always UTC: event dates are computed in UTC
DATACENTER_ID, WORKER_ID	TheIds	parts of the snowflake identifier, default to 1 and 1
SERVICE_GROUP	mmgeo	the enricher group, the key of its result in the event

A local .env usually looks like this:

TZ=UTC
LOG_LEVEL=debug
REDIS_DSN=redis://127.0.0.1:6379/0
STATSD_HOST=127.0.0.1

Multiple instances of the same service#

If a service runs in several copies, each one needs its own name, otherwise they will answer the same direct requests. mmgeo builds the name from the group and the worker number:

rpc:
  service_group: <%= env.SERVICE_GROUP || 'unknown_enrich' %>
  name: <%= env.SERVICE_GROUP || 'unknown_enrich' %>_<%= env.WORKER_ID || '1' %>
  channels:
    - enrich
    - <%= env.SERVICE_GROUP || 'unknown_enrich' %>_<%= env.WORKER_ID || '1' %>

Front is built the same way: its instances are named front_1, front_2 and so on.