The Magistral Data Stream Network

The elastic infrastructure to build and connect your realtime IoT apps

Documentation

Overview

Magistral is an elastic high-throughput cluster-centric Cloud messaging system. It enables quick and fast development, integration and scale of customer applications with our real-time and low-latency network without downtime.

Magistral provides scalable, distributed and replicated infrastructure, that guarantees strong durability and fault-tolerance. More than 10 SDKs for different platforms and programming languages enable seamless integration of IoT, mobile, desktop, server and other platforms, applications and systems.

Design

Magistral Network is a cluster of Magistral Nodes. Each Magistral Node is a cluster itself, which replicates and distributes data passing through across several machines. It has throughput in hundreds of thousands messages per-second, thus cluster of Magistral Nodes increases performance of the Network proportionally.

Magistral adheres publish/subscribe model. Magistral Node instantly delivers sent by publishers messages to connected subscribers balancing data loads across its machines on the fly.

Let’s go through the Magistral terminology to know definitions we use and to understand how it works.

Terminology

App

App is one of the first things you meet when starting to work with Magistral. It is logical unit that separates available resources, keeping them in some namespace. Every application has it’s own name, that normally denotes meaning of the task you use Magistral resources for (e.g. Vehicle Tracking or Facility Monitoring).

Every application has its own publish and subscribe keys, assigned to that app which you need to specify when creating Magistral application instance to connect to the Network and exchange data afterwards.

Topic

It is a composite data feed where publishers send messages and subscribers consume from. Topic has its own unique name, which can consist only of alphanumeric symbols and supports nested structure with wildcards.

For example, for transport company (Vehicle Tracking App) it could be topics bus, tram.

Channel

Channel is a discrete part of the topic. Each channel represents strongly ordered sequence of messages. All published messages sent by publisher go through one or more channels and remain available for “historical data” readings within retention period.

Currently, channels can have only sequent numeric identifiers (support of string channel identifiers planned in the near future). For example, via Vehicle Tracking App - bus topic - channel 1426 we can receive all tracking coordinates of the bus by its (1426) unique identifier.

Group

Magistral make use of consumer groups. Each group of subscribers receives a message, that will be consumed only by one random participant of this group. If you want to achieve that all consumers listening to some specific topic receive published message, then you need to use one unique group identifier for each consumer.

Keys

There are 4 different keys you will deal with when using Magistral — publish, subscribe, secret and optional AES encryption key (cipher):

Publish key — used to identify application and enable publish functionality for Magistral application instance.

Subscribe key — similar to publish key gives you access to subscribe operations in context of app it belongs to.

Secret key — unique key, that identifies user and his permissions to access Magistral resource access.

Cipher — optional key that can be used align with API to cipher message data on client side and transfer it encrypted through the Magistral network. The data can be decrypted only by key owner, thus neither Magistral service nor any other potential recipient of such message can read it without personal key.

SDK

SDK is a toolkit to connect, publish, subscribe and do other unified operations with your messaging resources via our Magistral API. More than 10 SDK available in different programming languages.

Retention period

Data sent through the Magistral stored in our servers and remain available for import and repeated readings via API within some period of time, which is different for available plans and customizable for enterprise Private Cloud / On-Premise deployments.

Getting started - Web Interface

App management

The next step you need to make after registration in Magistral service is to create an app. It can be made from application management view. Just click on “Applications” item in left menu and you will be forwarded to the page, where you can add new one, change name or delete existing one.

Caveat! Deletion of application releases all resources linked to the app and deletes all data sent through the Magistral and still stored in its servers. This operation cannot be undone.

Clicking on the app in the list you will be forwarded to special page for this app, where you can find publish and subscribe keys needed to make connection to the Network.

Topic management

In topic management view (“Topic” item in left tree menu) you can perform standard operations with topics.

During creation you can specify number of channels for each topic and modify this “composite” factor later on. Change topic name operation is not supported in the system, thus for this purpose you might need to delete existing (with deletion of all sent messages) and create a new one.

Removal of existing resources cannot be undone and lead to release of these resources from Magistral.

User management

In this component customer can create users and manage their permissions to work with available Magistral messaging resources. Permissions can be granted on permanent or temporary basis up to per-channel granularity, with read and/or write rights.

Each user has one or set of permissions, each of them linked to a special secret key. This key, together with app’s publish and subscribe keys, needed in order to connect to the Magistral network.

Getting started - API

Create Magistral instance

The creation of Magistral application instance is a first point in order to get started with Magistral service.

Magistral magistral = new Magistral(String pubKey, String subKey, String secretKey);
Magistral magistral = new Magistral(String pubKey, String subKey, String secretKey, boolean ssl);
Magistral magistral = new Magistral(String pubKey, String subKey, String secretKey, String cipher);
Magistral magistral = new Magistral(String pubKey, String subKey, String secretKey, boolean ssl, String cipher);

There are several constructors available. For all of them you need to know at least publish, subscribe and secret keys (can be found in app management and user management views). Constructors also contain optional fields, which indicate if SSL-encrypted connection should be established and if AES client-side encryption with user-specified encryption key should be activated.

Topic/channel description

Magistral API provides methods to receive detailed description of available resources. The description information contains list of topics and channels user can use to read or write messages to.

magistral.topic(“bus”, new Callback() {
     public void success(List<TopicMeta> meta) {
          TopicMeta topicInfo = meta.get(0);
          System.out.println("For topic '" + topicInfo.getName() + "' following channels are available [" + topicInfo.getСhannels() + "]");
     }
     public void error(MagistralException ex) {
          System.out.println(ex.getMessage());
     }
});

Permissions

Permissions functionality describe user rights for all resources he has been granted. It contain per-channel permission descriptions with read/write rights.

magistral.permissions("bus", new Callback() {
     public void success(List<PermMeta> meta) {
          for (PermMeta pm : meta) {
               for (int ch : pm.channels()) {
                    System.out.println("Permissions for [" + pm.topic() + ":" + ch + "] are "
                    + "[read : " + pm.readable(ch) + ", write : " + pm.writable(ch) + "]");
               }
          }
     }
     public void error(MagistralException ex) {
          System.out.println(ex.getMessage());
     }
});

Publishing/Subscribing

This is a main functionalities Magistral API provides. By analogy with other known messaging system you need to indicate topic to send or receive messages from. Plus, in Magistral you can also indicate up-to channel subscription, ignoring data exchange activities in other existing channels in presented for choosen topic.

Subscription example:

magistral.subscribe(“bus”, “operator”, 52, new NetworkListener() {
     public void messageReceived(MessageEvent e) {
          String topic = e.topic();
          int channel = e.channel();
          String msg = new String(e.msgBody(), StandardCharsets.UTF_8);
          System.out.println(“Location of ” + topic + “ No ” + channel + “ is ” + msg);
     }
     public void disconnected(String topic) {
          System.out.println("Disconnected from '" + topic + "'");
     }
     public void reconnect(String topic) {
          System.out.println("Reconnect on '" + topic + "'");
     }
     public void connected(String topic) {
          System.out.println("Connected to '" + topic + "'");
     }
     public void error(MagistralException ex) {
          System.out.println(ex.getMessage());
     }
});

Publish example:

magistral.publish(“bus”, 52, “55.598707, 37.035291”.getBytes(StandardCharsets.UTF_8), new Callback() {
     public void success(PubMeta meta) {
          String topic = meta.getTopic();
          int channel = meta.getChannel();
          System.out.println("Send updated coordinates for " + topic + " No " + channel);
     }
     public void error(MagistralException ex) {
          System.out.println(ex.getMessage());
     }
});

History

Magistral API also provides possibility to rewind the message flow and get all historical messages sent through the network and still stored in servers within retention period.

Example:

final int msgCount = 100;
magistral.history("bus", 52, msgCount, new io.magistral.client.data.Callback() {
     public void success(History history) {
          System.out.println("Last " + msgCount + " messages are: ");
          for (Message msg : history.getMessages()) {
               System.out.println(new String(msg.getBody(), StandardCharsets.UTF_8) + " at " + msg.getTime());
          }
     }
     public void error(MagistralException ex) {
          System.out.println(ex.getMessage());
     }
});

Check out our API page for more details.