Service layout

The easiest way to start a new service is to copy kafkastream: it has a minimum of code and nothing but the skeleton and a single module. Below is what this skeleton contains and why.

Files#

my-service/
├── config/
│   └── 02_config.yml      # configuration; every *.yml in the folder is read recursively
├── src/
│   ├── index.ts           # entry point: imports and exports appServer
│   ├── AppServer.ts       # assembling dependencies, subscribing to the bus, setup()
│   ├── constants.ts       # channel and method names
│   ├── types.ts           # ModuleConfig — the types of your own configuration sections
│   └── kafka/             # module with the service logic
├── @types/                # declarations for packages without types (an empty folder with a .keep works)
├── .env                   # environment variables for local runs
├── .env.local             # local overrides, not committed
├── Dockerfile
├── makefile               # building and publishing the image
├── package.json
└── tsconfig.json

package.json#

Among the dependencies, the required ones are the framework itself, reflect-metadata (imported first in AppServer.ts) and source-map-support for readable stack traces in production:

{
  "name": "@rockstat/my-service",
  "version": "1.0.0",
  "scripts": {
    "start:dev": "LOG_LEVEL=debug ts-node-dev -r 'tsconfig-paths/register' -r 'source-map-support/register' ./src/index.ts | pino-pretty -L debug",
    "start:prod": "TS_NODE_BASEURL=./dist node -r 'tsconfig-paths/register' -r 'source-map-support/register' dist/index",
    "build": "rimraf dist && tsc -p tsconfig.json",
    "clean": "rimraf dist",
    "watch": "tsc -w"
  },
  "dependencies": {
    "@rockstat/rock-me-ts": "file:../rockmets",
    "js-yaml": "^4.1.0",
    "reflect-metadata": "^0.2.1",
    "source-map-support": "^0.5.21"
  },
  "devDependencies": {
    "@types/node": "^20.11.26",
    "pino-pretty": "^11.0.0",
    "rimraf": "^5.0.5",
    "ts-node": "^10.9.2",
    "ts-node-dev": "^2.0.0",
    "tsconfig-paths": "^4.2.0",
    "typescript": "^5.4.2"
  }
}

The framework is wired in one of two ways:

• Locally — "@rockstat/rock-me-ts": "file:../rockmets", if the rockme-framework repository is cloned alongside (that is how kafkastream does it).
• In Docker — npm link @rockstat/rock-me-ts. The base image rockstat/band-base-ts already contains the built and linked framework, see Build and run.

tsconfig.json#

{
  "compilerOptions": {
    "target": "esnext",
    "module": "commonjs",
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "strictNullChecks": true,
    "noImplicitAny": true,
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "sourceMap": true,
    "outDir": "./dist/",
    "typeRoots": [
      "node_modules/@types",
      "node_modules/@rockstat/rock-me-ts/@types",
      "@types"
    ],
    "baseUrl": "./src/",
    "paths": {
      "@app/*": ["*"]
    }
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

What matters here:

• baseUrl and paths set up the alias @app/* for src/*. During development it is resolved by tsconfig-paths/register, and in production — by the same package with TS_NODE_BASEURL=./dist.
• typeRoots includes the framework's @types: that is where the declarations for redis-fast-driver, flake-idgen, xxhash and merge-options live, since they have no types of their own.
• emitDecoratorMetadata and experimentalDecorators are required by reflect-metadata.

Entry point#

src/index.ts does nothing on its own — the server instance is created when the AppServer module is imported:

import { appServer } from '@app/AppServer';
export default appServer;

If there is no module-alias in package.json, the line import 'module-alias/register' is not needed: the aliases are resolved by tsconfig-paths.

AppServer#

The skeleton is the same in all services. The Deps container is passed into modules so that they take the logger, metrics, configuration and RPC from it, without knowing anything about the order in which they are created:

import 'reflect-metadata';
import {
  RPCAdapterRedis, RPCAgnostic, AgnosticRPCOptions, Meter, RedisFactory,
  TheIds, Logger, AppConfig, RPCAdapter, BROADCAST
} from '@rockstat/rock-me-ts';
import { ModuleConfig } from '@app/types';
import { KafkaManager } from '@app/kafka';

export class Deps {
  log: Logger;
  id: TheIds;
  config: AppConfig<ModuleConfig>;
  meter: Meter;
  rpc: RPCAgnostic;
  rpcAdaptor: RPCAdapter;
  redisFactory: RedisFactory;
  constructor(obj: { [k: string]: any } = {}) {
    Object.assign(this, obj);
  }
}

export class AppServer {
  log: Logger;
  deps: Deps;
  name: string;
  rpc: RPCAgnostic;
  kafkamanager: KafkaManager;

  constructor() {
    const config = new AppConfig<ModuleConfig>();
    const log = new Logger(config.log);
    const meter = new Meter(config.meter);
    this.name = config.rpc.name;
    this.deps = new Deps({ id: new TheIds(), log, config, meter });
    this.log = log.for(this);
    this.log.info('Starting service');
    try {
      const redisFactory = this.deps.redisFactory = new RedisFactory({ log, meter, ...config.redis });
      const channels = [config.rpc.name, BROADCAST];
      const rpcOptions: AgnosticRPCOptions = { channels, redisFactory, log, meter, ...config.rpc };
      this.deps.rpcAdaptor = new RPCAdapterRedis(rpcOptions);
      this.rpc = this.deps.rpc = new RPCAgnostic(rpcOptions);
      this.rpc.setup(this.deps.rpcAdaptor);
      this.kafkamanager = new KafkaManager(this.deps);
    } catch (exc) {
      this.log.error(exc);
      this.stop();
    }
    this.setup()
      .then(() => this.log.info('Setup completed'))
      .catch(exc => { this.log.error(exc); this.stop(); });
  }

  /** Connections to external systems and registration of bus methods */
  async setup() {
    await this.kafkamanager.init();
    this.rpc.register(BROADCAST, this.kafkamanager.write);
  }

  private stop() {
    this.log.info('Stopping...');
    process.exit(0);
  }
}

export const appServer = new AppServer();

The order in the constructor matters: first the configuration, then the logger and metrics (they need the configuration), then Redis and RPC. Everything asynchronous — connecting to Kafka or ClickHouse, checking the schema — lives in setup(), and the registration of bus methods happens after the service is ready to accept data.

The logic module#

Keep the service's logic in a separate class that receives Deps in the constructor and takes what it needs from it. This way the module can be tested by substituting its dependencies:

import { Deps } from '@app/AppServer';
import { Logger, Meter } from '@rockstat/rock-me-ts';
import { KafkaConfig } from '@app/types';

export class KafkaManager {
  log: Logger;
  meter: Meter;
  options: KafkaConfig;

  constructor(deps: Deps) {
    const { log, meter, config } = deps;
    this.log = log.for(this);
    this.meter = meter;
    this.options = config.get('kafkastream');
  }

  async init() { /* connection */ }

  write = async (msg: { [k: string]: any }) => { /* event handling */ return {}; }
}