Topics

Andy X topics are the categories used to organize messages. Each topic has a name that is unique inside a Component in Andy X Cluster.

Overview #

This section gives a high-level overview of how the topics works and an introduction to the configuration topic settings for tuning.

To see examples how to create and configure a topic, refers to these guides. For additional examples, including using of Buildersoft Cloud, refers to Buildersoft Cloud docs.

Concepts #

When an event stream enters Andy X, it is persisted as a topic. In Andy X’s universe, a topic is a materialized event stream. In other words, a topic is a stream at rest. As in other pub-sub systems, topics in Andy X are named channels for transmitting messages from producers to consumers. Topic names well defined and are stored inside the Tenant/Product/Component/Topic

    e.g,
    {tenant}/{Product}/{Component}/{Topic}
    default/transactions/application/txn-received

Andy X topics are the categories used to organize messages. Each topic has a name that is unique inside a Component in Andy X Cluster. Messages are sent to and read from specific topics. In other words, producers write data to topics, and consumers read data from topics. Andy X topics are multi-subscriber. This means that a topic can have zero, one, or multiple subscriptions to that topic and the data written to it.

In Andy, topics are sharded and replicated across nodes (depends on Cluster Configuration) throughout the implementation. Nodes refers to each of the Main Nodes in Andy X Cluster. Shards are important because they enable parallelization of topics, enabling high message throughput. Below, is illustrating how a topic is created in Andy X Cluster.

Topic #

Andy X was built with highly scalable persistent storage of message data as a primary objective. Topics enable you to persistently store as many messages as you need while preserving message ordering. By default, Andy X stores all messages produced to a topic.

Storage Configurations #

Topics can have different configurations on the storage level. These configurations should appled for each use-case you may have, e.g., There should be different configurations for a topic that stores billion of messages for day, from the topic has has thousand of messages for day. In the table bellow are shown the settings you can apply to a topic.

MemTopic (aka. Memory Topic) is an in-memory data structure holding data before they are flushed to SST files. It serves both read and write - new writes always insert data to memtopic, and reads has to query memtopic before reading from SST files, because data in memtopic is newer. Once a memtopic is full, it becomes immutable and replaced by a new memtopic. A background thread will flush the content of the memtopic into a SST file, after which the memtopic can be destroyed.

Default values for each config key is

When a new topic is created, default configuration values will be passed to that topic from the table above. But, this does not mean that will be always the same, as you can configure default values in storage_config.json at

~/data/active/storage_config.json

Default topic settings for storage.

{
  "DefaultWriteBufferSizeInBytes": 100000000,
  "DefaultMaxWriteBufferNumber": 4,
  "DefaultMaxWriteBufferSizeToMaintain": 0,
  "DefaultMinWriteBufferNumberToMerge": 2,
  "DefaultMaxBackgroundCompactionsThreads": 1,
  "DefaultMaxBackgroundFlushesThreads": 1,
}

Transport Configuration #

Same as Storage Configuration for topic. Also, Transport Layer is important for different use-cases you may have. The only drawback for the transport layer is that if you change configuration it will be applied for all topics connected in the node. In the table below are shown the transport configuration.

Default values for each config key is

Transport Configuration can be configured in transport_config.json config file at

~/data/active/transport_config.json

Changing Transport Configurations, it will require to restart the cluster.