API Endpoint - Discovery & Control

Objective

This document provides instructions on how to enable markup and analysis of API endpoints for your application. Volterra discovers API endpoints for your application services and performs behavioural analysis on the various logs collected from the endpoints using advanced machine learning. Thus, the Probability Distribution Function (PDF) for metrics related to each endpoint is generated. The analysis is performed periodically and the PDFs are updated accordingly. To know more about behaviour analysis, see Behavioural Firewall.

The API EP is a tuple of URL and method for the API. There could be absolute paths or there could be variables in the path such as G namespace/$namespace where $namespace indicates the namespace name. These type of URLs are called as collapisble URLs and Volterra learns such kind of URLs and presents them in the collapsed format.

Enabling the API Endpoint (EP) markup and analysis results in the following benefits:

  • Dynamic discovery of all the API endpoints of your application.
  • Monitoring the performance and trends for your APIs.
  • Determining which APIs are supposed to be between a set of services and enhancing security to allow only those APIs.
  • Obtaining insights such as which API is most hit and the associated request size.

The PDFs are obtained for the following metrics:

  • Request size and response size
  • Latency with data and without data
  • Request rate
  • Error rate
  • Response throughput

Using the instructions provided in this document, you can enable the API endpoint markup for your application, inspect the discovered API endpoints, and monitor the related PDFs in Volterra service mesh.


Prerequisites


Configuration

The following image describes the configuration work-flow for enabling API EP markup for your application:

seq apiep
Figure: Work-flow for Enabling API EP Markup

Configuration Sequence

The following table presents the sequence of activities in enabling the TSA detection:

Activity Description
Create App Type Create app type and configure the API EP markup functionality.
Assign App Type Label to Load Balancers Set the app type label to the load balancers and services for which API EP markup needs to be performed.
Monitor API EPs and PDFs Monitor the service mesh or load balancer for API endpoints and the associated PDFs.

Create App Type

To enable API EP markup for your application services, it is required to first enable the associated machine learning model for those services using the app type object.

The app type object is created in the shared namespace. The load balancers of that app type in different namespaces need to be assigned with the label of the app type object.

Perform the following to create app type and enable generating the anomaly model.

Step 1: Log into the VoltConsole and navigate to app type configuration.

Select the Shared namespace on the namespace selector. Select Security from the configuration menu and AI & ML -> App Types from the options. Click Add app type to start app type creation.

nav atype new
Figure: Navigate to App Type Configuration

Step 2: Configure app type object settings.

Enter the configuration in the app type object creation form as per the following guidelines:

  • Enter a name for the app type. This is the value for the app type label to be assigned to the load balancers for which the TSA needs to be enabled.
  • Click Add item in the Features field of the Application Type Features section. Select API Discovery for the AI/ML Feature Type field.
  • Click Add item again and select Per API Request Analysis for the AI/ML Feature Type field.
  • Optionally, select Enable learning from redirect traffic in the Business Logic Markup Setting section. This enables AI engine to learn the endpoints from redirected traffic.
  • Click Save and Exit to complete creating the app type object.

apptype cnf new
Figure: App Type Feature Configuration


Assign App Type Label to Load Balancers

After creating the app type, it is required to assign the app type label to the load balancers for which you want to enable API EP markup.

Note: Enabling API EP markup for all load balancers in a namespace requires you to apply the app type label to all load balancers in that namespace.

Perform the following to assign the app type label to your load balancers.

Step 1: Log into the VoltConsole and navigate to load balancer management.

Change to your application namespace and select Manage -> Load Balancers from the configuration menu and HTTP Load Balancers from the options. Click ...->Edit for the oad balancer for which the app type label needs to be assigned.

lb edit
Figure: Navigate to load balancer Edit Configuration

Step 2: Assign the app type label.
  • Select ves.io/app_type for the Labels field and type.

at label new
Figure: App Type Label Selection

  • Type the name of the app type object created in the previous step and click Assign Custom Value to add the app type label.

label value new
Figure: App Type Label Addition

  • Click Save and Exit to apply the label to the load balancer. This enables the learning and API EP markup for the load balancer.

Adding app type label to more than one load balancer groups the data of all such load balancers into a single learning model and presents the API EP markup analysis in the service mesh.


Monitor API EPs and PDFs

After enabling the load balancers for API EP markup, you can monitor and inspect the API EP markup and PDFs in the following 2 ways:

  • From the service mesh option in your namespace - This displays all endpoints of all load balancers with the app type label.
  • From the loadbalancers option in your namespace - This displays all endpoints for that load balancer.

Note: Learning of the API EPs and associated PDFs get updated every 4 hours and is incremental in nature. The APIs that do not get traffic for a time interval required for the model, those APIs are aged out and will not be displayed in the markup.

Perform the following to inspect the API EPs and PDFs:

Step 1: Log into the VoltConsole and navigate to service mesh.

Change to your application namespace and select Mesh from the configuration menu and Service Mesh from the options. Click on your application tile from the displayed list to load its service mesh monitoring.

nav sm new
Figure: Navigate to Service Mesh

Step 2: Load the the endpoints view.
  • The service mesh loads service graph by default. Click Endpoints tab to load the API EP markup view. The endpoint paths are shown in a hierarchical structure with root and leaf relation ships presented in segments.

sm apiep new
Figure: Service Mesh Endpoints View

  • Select a service edge from the All Endpoints drop-down to display the API endpoints specific for that service interaction.

srv edge apiep new
Figure: API EP Markup for Specific Service Interaction

Note: You can also load API EP markup for a specific service interaction from the Service Graph view. Click on an edge to load the quick view for that edge and click on Endpoints in the quick view to load the endpoints view for that specific service interaction. By inspecting the edge level API EP markup, you can determine which APIs are supposed to be functional between those nodes(services) and apply further security using service policies to restrict the traffic to those APIs only.

  • Click on the Search drop-down and select a specific API to display the hierarchy for that path.

specific apiep new
Figure: API EP Markup for Specific API

The following sample shows collapsed URL presented as a dynamic component:

collapsed url
Figure: Dynamic Component in API EP Markup

Note: The API EP markup also displays static resources. These are regular static resources such as javascript files that a web application uses.

  • Click on any path to expand or collapse it. If a path displays the method or PDFs|<METHOD>, click on it to display a quick PDF view.

hover pdf new
Figure: Quick PDF View from the API EP View

  • Hover over any PDF in the quick view to display the PDF percentile and mean values. Click on any PDF to display full PDF view for that specific metric.

specific pdf
Figure: PDF Information for a Metric in PDF Quick View

Step 3: Load the PDFs view.
  • Click on the Table option to display the PDFs view for the API endpoints. This view shows tabular list for all API endpoints and displays the collpased URLs, PDFs for metrics, and last updated time.

pdfs full new
Figure: API EP PDFs Full View

  • Hover over any PDF to display the PDF percentile and mean value for that metric. Click on any PDF to display that specific PDF's full view in graph format.

reqsize pdf
Figure: Detailed PDF View for a Specific Metric

Note: The X-axis represents the metric value and Y-axis represents the probability density.

  • Hover any where on the PDF graph to display the probablity density for a given metric value. You can also change metric from here using the X Axis drop-down menu.

Note: In case the learning model does not get enough data for an API, it displays blank entries for the PDFs for that API and displays a message on the tool-tip mentioning that not enough data is available.

Step 4: Monitor API EP markup for a specific load balancer.

Similarly you can navigate to Virtual Hosts -> HTTP Load Balancers in the application namespace. Click on your load balancer from the displayed list to load its dashboard and click API Endpoints to load the API EP markup for that load balancer. Monitoring features are same as that of the service mesh.


Concepts


API References