To help integration team, we installed Spider on PRM. And it works out of the box :)

The main difficulty was to run the Whisperer on an old Ubuntu 14.04. But, thanks to Docker, it runs without knowing it :)

• As the only Docker version available on Ubuntu 14.04 is a very old one, I had to copy the Docker image manually on the server.
• But thanks to very stable interfaces, this old Docker version can still run new Dockers!! So great!

Whisperers have got a technical upgrade:

• Upgraded to Node 10 and async await programming pattern
• Upgraded to latest Node pcap library

To follow Whisperers migrations, I added a Whisperer version and it is sent with the status to the server. The monitoring now shows this version in the Current Status grid:

Moreover, this grid has been improved to show:

• In orange the whisperers that have not communicated since 2 minutes
• In dark red, those that have not communicated for more than 1 day

The Timeline is one of the component that took me most brain hours, with the map and the grid.

To enable Flowbird's Streetsmart project to use it, I decided to opensource it. This is my first opensource project... We'll see how it goes ;)

Repository​

The component is available since a couple of months, and has been integrated successfully in Streetsmart monitoring UI. It embeds a small test application packaged as a Docker that allows testing, and even development with hot reloading.

• Npm package: https://www.npmjs.com/package/@spider-analyzer/timeline
• Gitlab repo: https://gitlab.com/TincaTibo/timeline
• Trello for project plan and plan changes: https://trello.com/b/DxjkiuFB/timeline-react-component

Changes​

Since its packaging as an opensource and standalone component, the Timeline has evolved (Changelog) thanks to Streetsmart UI team ideas, and all changes are accessible both on Spider and Streetsmart.

• React 16 refactor
• Lots of options to control the behavior
• Left margin adapt to legend length
• Small icon displayed when cursor is out of view

• Autosliding with regular fetching when dragging cursor left of right of the area
• Visual improvements and bug fixes

I added Elastic Metricbeat inside Docker Swarm cluster to gather metrics and performance information of all Dockers in Cluster.

Nothing apart Dockers runs on Spider cluster now.

This allowed adding graphic in monitoring with containers CPU and RAM usage.

This allowed also to assert the load of Traefik and Beats.

Since a long time, user settings were saved in local storage, but if you opened a new session on another computer, you had to set all as you wanted again.

Now, settings are saved on the server. Settings are split in 2: users own settings and global settings:

User settings

• Grid settings
• Timeline settings
• Refresh delays
• Workspace composition (panels opened and width)
• Selected view, mode and options
• Usage stats and errors sending to server
• ...

Global settings

• Url templates
• Merge patterns for replicas and clients
• Correlation tokens
• Stats configurations
• Saved queries
• ...

Features

• Settings are stored after a change on one of them, but with a delay of x seconds. As settings are often not changed alone, it reduces server calls
  • Then user has to wait a bit before settings is saved
  • The settings icon is shown with white fill when a save is waiting
  • The settings icon is rotated on saving

• Date of last saving is displayed in settings panel

• List of saved settings is available in source tab of settings panel
• Defaults
  • Saving settings is enabled by default on start (cannot be changed)
  • Saving settings can be disabled in settings panel
  • Saving settings are disabled by default when loading an external link (not to override user's settings)
• Global settings will be able to be overridden by team settings on coming months

Very neat to avoid mistake or save time when setting up a new Whisperer:

• Now, when editing a Whisperer configuration, you may clone the configuration values of an existing Whisperer to fasten your edition.
  • The edition is still in progress, you may accept or cancel the changes.
• This is possible for all 3 current configurations tabs:
  • Capture config
  • Parsing config
  • Share
• The list of Whisperers to clone is taken from the filtered list in the whisperers selection input of the menu. Thus, you can use Elasticsearch power to filter the ones you want to clone.
  • Easy and neat. Just have to know it ;)

 

I don't know why it took my so long to start implementing this, I overuse it already :) So neat !!

It is now possible to save existing queries  and give them a name, to reload them afterwards.

Note: The query only includes the generated filter in the search input, no information is saved on the selected time, the current view or the filters input.

Saving a query​

• To save a query, click on the save icon

• In input is show to give it a name
  • If the query has already been saved (and is not modified), the name of the current query is shown
  • Changing the name will update / replace the saved query name

• You may give the same name to different query. It is up to you to remove old ones.
• The saved queries are linked to the selected View (HTTP, TCP...)

Loading a saved query​

• Once saved, a query can be reloaded by clicking the load icon

• The list of saved query shows up with alphabetical sorting

• The search input is filled with the selected query and search is executed

Filters selection dropdown has been improved.

• Previously, the select applied the current query to show you the available options.
  • If you had previously selected a value, no other value could be displayed, as the query was filtering them out;
• Now, the select gathers the available options by running the current query, but without applying the current filter selection. Thus allowing you to add more values in the selected options of this filter.

This is much more useful than showing a blank select dropdown ;-)

With Streetsmart team help and Ace one, we improved Lucene syntax highlighting in Ace, and then in Spider.

The custom parser I put in place better parses:

• Field identifiers
• Numbers
• Dates
• Wilcards
• Regular expressions
• IP adresses
• Ranges

And has better parenthesis matching.

All in all, editing queries by hand has never been so easy :-)

Before going public, I needed extended features in user account management:

• User needed to be able to reinitialize his password when he forgot it
  • Without contacting the support
  • With security and identity confirmation: getting a link by mail to enter a new password
• User needed to be able to change email in a secured way
  • Since email is used for password recovery, changing mail must be secured by something stronger than current token
  • User then needed to enter its password when changing email
• User needed to be notified by mail when changing email or password
• Since email was getting very important, emails needed to be confirmed
  • When setting an new email adress, user need to confirm the email by cliking on a link sent to him by email
  • At user account creation, or email change
• Last but not least, a new user needs to be able to create its account with or without admin confirmation

First, I needed to create an email service.

Mail sender service​

This is a simple mail over REST service, integrated with a defined account.

• It can use SMTP, secured SMTP or Gmail account.
• It uses nodemailer library
• It integrates with gmail by OAuth tokens following these tips:
  • https://nodemailer.com/smtp/oauth2/
  • https://medium.com/@nickroach_50526/sending-emails-with-node-js-using-smtp-gmail-and-oauth2-316fe9c790a1
• API

Then I needed to define and develop the flows

Password recovery​

Process is the following:

• On login screen, click on the password forgotten link, and enter your email adress

• A mail is sent to your adress (if the account exists) with a link to change your password

• The link opens a new window were you may set a new password The link has a limit in validity and won't work after a short period of time The link can only be used once

Changing an email or a password​

When changing email or password in profile edition tab, Spider is now requesting that you provide current password:

Then,

• If you change email
  • An informative email is sent to previous mail to tell about mail change

• • Another email is sent to new email address with a link to confirm new mail
• If you change password
  • An informative email is sent to confirm change of password

Checking an email​

At account creation, or at email change, an activation email is sent to user's email address to confirm his email:

This email contains a link to confirm the new address. It is valid only once and for a limited time.

• Clicking on the link validates the email change.

• Not doing so on time... locks down the account, and only an administrator can unlock it.

Creating an account​

A new user can now create an account from the login screen:

A creation form opens to input user details:

At the bottom of it, a comment input allows to give information for the administrator validating the account (if need be).

The new user will receive an email to confirm his email address.

Depending on platform settings, the account may need validation/activation by an administrator.

• The latter will then receive an email automatically, once the email is confirmed.
• He can also associates whisperers to the new account, if requested.