Documentation
ruen

Documentation

Lifecycle and addressing

Incoming requests#

Rockstat is built on microservices. Let's walk through an example:

  1. The SDK, or an external tool or service, sends a request.
  2. The Front service receives all incoming requests, validates them, adds metadata, after which they become events (event).
  3. Information about how the event was received and about its context is added to it. Each event also has a routing key, which services use to decide whether they need to process it or not. Example: in.gen.track.page — an incoming request of the generic type generic, which ultimately must be processed by the track service, and the name of the event for that service is page (a page view). There is no actual service named track, and there is no need for one. But other services can use this label to tell what kind of event this is.
  4. The event enricher (the event was previously a request) enricher. Front knows the list of services (received from the Director service) that want to take part in enriching events with a given key. For example, for the key in.gen.track.page there will be the services MaxMind ip2geo, Sypex ip2geo, UserAgent parser. All of them receive the event and generate their own additions based on its contents. They resolve the geographic location from the IP address, and from the User-Agent they determine which device, operating system and browser were used. The results are sent back to Front, where they are appended to the event's data. /// Continued below ///
  1. The next stage is passing the event to the main handler (handler). This role is responsible for taking action based on the received event. Based on the received data, the handler can send a signal or data back to the browser, can message someone through a chat bot or send an email, or simply pass a signal to another service. You can even use machine learning models in real time, for example, to compute the probability of advancing to the next funnel step or of falling into CHURN.
  2. After the handler, the event is passed to all listeners (listener) at once. They perform database writes, data streaming to other systems, passive statistics collection, and so on. These include the services ClickHouse Writer and Send Mixpanel, which ship in the base package and whose names make it easy to tell what they are responsible for. By the way, they receive each and every piece of incoming data and save/send it, so there is no need to worry about that.
  3. The response phase begins. Front takes into account the parameters returned by the handler (if it returned any) and the parameters of the incoming data, and based on them chooses exactly how to respond: return data in JSON format, return a 1x1 gif pixel, or perform an HTTP redirect to another address. This is needed, for example, to handle Cookie-Sync logic, where the pixel must redirect to a third-party pixel, which appends its own User-ID and returns it back, and at that point you just need to show the pixel and store the matching data.
  4. The original request initiator has received the response, and at this point the processing cycle ends.

Addressing#

All incoming requests arrive at the Front microservice, which passes them on, and it figures out exactly where to based on the key that the data came in with. The key has the format aa.bb.cc.dd, where:

  • aa: the direction in/out, meaning incoming/outgoing
  • bb: the request type; almost everywhere the generic type is used right now
  • cc: the service name; usually this holds the name of the service that generated the data or the name of the service that receives it
  • dd: the name of the action or operation of that service.

Example: in.gen.track.page

Converting HTTP paths into keys#

For incoming data the key is not specified; when the data is received, the Front service builds it on its own. It does so based on the segments of the address the request came to. The first segment is the service, the second is the action name. The format is: https://yourdomain/{service}/{name}, for example:

https://subdomain.yourdomain/appmetrika/install -> `in.gen.appmetrika.install`

This is done because in most cases there is no way to influence the format of the data being sent, whereas specifying the address where it is sent is almost always possible.

Converting structures into keys#

If the data is sent on its own, it is more convenient to send the service and action name together with everything else. In this case the service and name parameters are used. For example, this request:

{
  "service": "track",
  "name": "page",
  "data": {},
  "projectId": 1627632140,
  "uid": "6410748886712844288",
  "user": {
    "gaId": "2039255293.1537236640",
    "ymId": "1537242940246737376",
    "ts": 1537243840171,
  //....

will be assigned the key in.gen.track.page