PerfAvore 1.0.1

Perf Avore: A Rule Based Performance Analysis and Monitoring Tool in FSharp

Introduction

"Perf Avore" is a Rule Based Performance Analysis and Monitoring Tool that uses Rules specified by the user to match conditions and invoke actions on either an .ETL Trace files or real time processing. More in depth details about the implementation, process amongst other topics are mentioned in the accompanying notebook.

The use case of Perf Avore would be to detect and diagnose performance issues effectively by specifying details that are pertinent to performance issues in the rule itself. For example, spikes in allocations can put pressure on the GC and inevitably slow down the process; specifying a rule that tracks AllocationAmount on the GC/AllocationTick event if it goes above a specified amount and then print the call stack for it can shed light on the impetus behind the increased pressure.

High Level Overview

1. Users provide rules.
   1. Rules consist of conditions and actions.
   2. Conditions Include:
      1. The Name of the Trace Event and the property they'd like to trace.
      2. The condition or case for which they'd like to act on.
      3. Examples:
         1. GC/AllocationTick.AllocationAmount > 200000 : Print Alert
         2. ThreadPoolWorkerThreadAdjustment/Stats.Throughput < 4: Print CallStack
         3. GC/HeapStats.GenerationSize0 isAnomaly DetectIIDSpike : Print Chart
2. Based on either a given trace or by real time monitoring, conditions are checked for and actions are invoked.

Rules

A rule consists of a Condition and an Action. An explanation of a rule is best explained with an example:

GC/AllocationTick.AllocationAmount > 200000 : Print Alert

Here, the user requests that for the said process, an alert will be printed if the AllocationAmount of the GC/AllocationTick event is greater than 200,000 bytes. The action if the condition is met is that of alerting the user by outputting a message.

A rule, more generally, is of the following format: EventName.PropertyName ConditionalOperator ConditionalOperand : ActionOperator ActionOperand

where:

Condition Types

The following are the currently implemented Conditions worth mentioning:

The currently implemented algorithm is that of IID Spike Detector based on the ML.NET algorithm that can be found here. More details about this algorithm can be found in the Notebook.

Action Types

The following are the currently implemented action types:

Running Perf-Avore

Perf-Avore can be run by cd'ing into the src/PerfAvore/PerfAvore.Console directory and then running:

1. dotnet restore
2. dotnet run -- --processname <ProcessName> [--tracepath <TracePath>] [--rulespath <RulesPath>].

Command Line Arguments

Developing Perf-Avore

This project was heavily inspired by maoni0's realmon, a project I had such a great time contributing to, I wanted to generalize the use case. The name is a play on Italian's Per Favore meaning please where the purpose of the project is to please the most disgruntled of perf engineers by aiding them with investigations.

This project was developed in VSCode using Ionide and the dotnet cli. The prototypes were all done using DotNet Interactive Notebooks.

TODO For Later On

1. Linux / MacOs Compatibility.
2. Audit report.
3. Unit Testing!

References

1. Taking Stock of Anomalies with F# And ML.NET
2. A CPU Sampling Profiler in Less Than 200 Lines
3. Tutorial: Detect anomalies in time series with ML.NET
4. Plug-in martingales for testing exchangeability on-line - arXiv:1204.3251