Integration with Splunk ITSI Module for Application Performance Monitoring
Defining AppInternals to Aternity Mapping
Defining Business Processes for Aternity Data
Troubleshooting FAQ
Application Description
The objective of the Splunk App for Aternity and Riverbed is to provide analytics and visualizations of the ingested AppInternals data:
Objects under monitoring and their topology – such as an instance running on (which servers), and which server tags are used to groups the servers
Metrics and delays for objects under monitoring; for example counts and aggregations of normal, slow, very slow transactions
Alerts – generated by AppInternals and sent via SNMP
Metrics and location data for Business Activities ingested from Aternity
Data normalization and enrichment for out of the box integration with Splunk ITSI Module for Application Performance Monitoring
Technical Details:
The Application works with data supplied by the IT Squared / Cloud-Native Integration for Riverbed APM
EUEM dashboards also are compatible with data supplied by the Riverbed technical add-on for Aternity
The Application is installed on the Search Heads
Correct configuration / editing of the Lookup File is required (see Lookup Instructions)
Note:Before updating the application please save a copy of the Lookup File. It is required to add the File back after completing the update.
Installation
Splunk Requirements
Splunk Enterprise 6.6 or later
Installation steps:
Install latest version of the IT Squared App for Aternity and Riverbed from Splunkbase. It has to be installed on the Search Heads only.
Configure Lookup File (see Lookup Instructions)
The Application works with data supplied by the IT Squared / Cloud-Native Integration for Riverbed APM. EUEM dashboards also are compatible with data supplied by the Riverbed technical add-on for Aternity.
Once everything is configured, open the
Application in Splunk Web UI and work with data ingested from AppInternals using
Application / Transaction / Instance / Server / Server tag dashboards, and with
Aternity data using EUEM dashboards.
The Application also provides 6 (six) pre-built panels globally available for embedding into your own dashboards (see Using Pre-built Panels).
Lookup Instructions
This section describes how to edit lookup .csv files defining mapping between various AppInternals and Aternity entities (names, ID, URL, SNMP traps, etc.).
There are several fields in various events coming from the AppInternals server that are referring to the server itself:
“Pretty” host name for both metric and delay events.
Host name parsed from SNMP traps.
URL
used for drilldowns.
Reliable mapping between these fields is required. This can be achieved by entering correct values into aix_host_mapping.csv file lookup.
Lookup File Editing
The IT Squared App for Aternity and Riverbed looks for mapping configured in the aix_host_mapping.csv and aternity_mapping.csv files, located within the App at $SPLUNK_HOME/etc/apps/appinternals/ lookups. It is possible to either edit the files directly from the operating system, or use the following instructions:
Install Lookup File Editor
Download the “Lookup File Editor” App from Splunkbase and install it alongside the IT Squared App for Aternity and Riverbed.
2. Edit the mapping
Once the Look File Editor is installed, launch it and enter “aix” or “aternity” into the “Filter by name” field. This action will return the aix_host_mapping.csv or aternity_mapping.csv link. Click on the link to edit corresponding .csv file.
Editing using the Lookup Editor is
straightforward; it is possible to right-click the table for editing options. Add/Edit
rows in the table and enter the relevant mappings:
“pretty” host name, the corresponding host name for SNMP traps, and the URL for drilldowns for AppInternals
Account Name, Account ID and the corresponding server URL for drilldowns for Aternity
Note:
Make sure URLs notation is defined without
last “/”.
Using Pre-built Panels
After installation of the Application 6 (six) pre-built panels will be globally available for embedding into dashboards.
Adds the Table Top N Transactions with the worst Server Time
Offers a range of Appinternals drilldowns for the selected transaction from the table
Input Tokens:
appint_host_token – Chosen AppInternals Server
aixurl_token – URL of chosen AppInternals Server
appint_app_token – Chosen Application
appint_topN – Number of results to show
tokDurationAix – Dashboard earliest time converted to AppInternals format
tokEarliestAix – Dashboard latest time converted to AppInternals format
Output Tokens (available for other panels):
trname_token – Chosen Transaction Name
trid_token – Chosen Transaction Id
For a sample Dashboard demonstrating the panel, please see “Transactions Panels Sample Dashboard”.
3. AppInternals_Instance_Filters
This panel performs the following functions:
Adds Inputs to selected Appinternals Server, Infrastructure Server and Instance
Generates the necessary parameters and restrictions for the correct operation of the AppInternals_Instance_Details panel
Offers a range of Appinternals drilldowns for the selected Instance
Prerequisites: Correctly configured Lookup File (See Lookup Instructions)
Input Tokens: None
Output Tokens (available for other panels):
appint_host_token – Chosen AppInternals Server
aixurl_token – URL of chosen AppInternals Server
appint_srv_token – Chosen infrastructure Server
appint_inst_token – Chosen Instance
tokDurationAix – Dashboard earliest time converted to AppInternals format
tokEarliestAix – Dashboard latest time converted to AppInternals format
For a sample Dashboard demonstrating the panel, please see “Instance Panels Sample Dashboard”.
4. AppInternals_Instance_Details
Adds Requests # timechart for choosen Instance & Server
Adds Server Performance timechart for chosen Instance & Server
Adds Delay times by Category timechart for chosen Instance & Server
Adds Delay times by Package timechart for chosen Instance & Server
Adds Threads timechart for chosen Instance & Server
Adds Garbage Collecting Time timechart for chosen Instance & Server
Input Tokens:
appint_host_token – Choosen AppInternals Server
aixurl_token – URL of choosen AppInternals Server
appint_srv_token – Choosen infrastructure Server
appint_inst_token – Choosen Instance
tokDurationAix – Dashboard earliest time converted to AppInternals format
tokEarliestAix – Dashboard latest time converted to AppInternals format
Output Tokens (available for other panels): None
For a sample Dashboard demonstrating the panel, please see “Instance Panels Sample Dashboard”.
5. AppInternals_Server_Filters
Adds Inputs to selected Appinternals Server and Infrastructure Server
Generates the necessary parameters and restrictions for the correct operation of the AppInternals_Server_Details panel
Offers a range of Appinternals drilldowns for the selected Infrastructure Server
Prerequisites: Correctly configured Lookup File (See Lookup Instructions)
Input tokens: None
Output tokens (available for other panels):
appint_host_token – Chosen AppInternals Server
aixurl_token – URL of chosen AppInternals Server
appint_srv_token – Chosen infrastructure Server
tokDurationAix – Dashboard earliest time converted to AppInternals format
tokEarliestAix – Dashboard latest time converted to AppInternals format
For a sample Dashboard demonstrating the panel, please see “Server Panels Sample Dashboard”.
6. AppInternals_Server_Details
Adds CPU (Busy) Avg value for chosen Infrastructure Server
Adds CPU Statistics timechart for chosen Infrastructure Server
Adds Avg RAM Usage value for chosen Infrastructure Server
Adds Memory Statistics timechart for chosen Infrastructure Server
Adds Disk I/O Utilization timechart for chosen Infrastructure Server
Adds Network Utilization timechart for chosen Infrastructure Server
Input tokens:
appint_host_token – Chosen AppInternals Server
aixurl_token – URL of chosen AppInternals Server
appint_srv_token – Chosen infrastructure Server
tokDurationAix – Dashboard earliest time converted to AppInternals format
tokEarliestAix – Dashboard latest time converted to AppInternals format
Output tokens (available for other panels): None
For a sample Dashboard demonstrating the panel, please see “Server Panels Sample Dashboard”.
Integration with Splunk ITSI Module for Application Performance Monitoring
The IT Squared App for Aternity and Riverbed adds configuration out of the box for data enrichment and normalization for integration with Splunk ITSI deployment, in particular Splunk ITSI Module for Application Performance Monitoring.
If you are using Splunk ITSI, the itoa_admin role needs access to all indexes with data that you want to monitor in ITSI. You have to add the following configuration change on each search head running Splunk IT Service Intelligence. If you have a search head cluster, perform these steps on one node and the Splunk platform replicates the configuration change to all cluster nodes.
Go to Settings > Access controls, then click Roles.
Click itoa_admin to edit it.
Scroll down to Indexes searched by default.
Click on the index that contain data ingested from AppInternals to move it into the list of indexes searched by default.
Click Save.
Defining AppInternals to Aternity Mapping
Transactions
ingested from AppInternals can be mapped to Business Activities coming from
Aternity. One AppInternals transaction can be mapped to one or many Business
Activities.
On AppInternals
side we are operating with application_id
+ transaction_id pairs, and on Aternity side with application_name + activity_name pairs. In current Application
version mapping should be defined with the help of Aternity_Mapping calculated
field.
To add/change
mapping follow these steps:
Go to Web UI -> Settings -> Fields -> Calculated fields and create Aternity_Mapping field (or edit if it already exists).
If you are creating the field, enter aix_metric as sourcetype and Aternity_Mapping as field name.
Add/Edit eval expression according to the following notation. We are using operator case to define multiple individual mappings. In the example below blue mapping defines one-to-one relationship, and green mapping defines one-to-many relationship. There is also 1=1 final expression catching all unmapped pairs and setting Aternity_Mapping as null for them.
An example:
case(application_id=2 AND transaction_id=44, “(YourApp)-(Home)”, application_id=2 AND transaction_id=55, mvappend(“(YourApp)-(Securities Landing)”, “(SAP)-(Search Account)”), 1=1, null)
Note: In the future
Aternity_Mapping field would be defined during data ingestion,
Defining Business Processes for Aternity Data
Contextually the IT
Squared App for Aternity and Riverbed Business Process is a set of Business
Activities that, once completed, accomplish an organizational goal. In other
words you define Business Process as a collection of Application + Business
Activities pairs ingested from Aternity.
The Application is looking on tags for Business Process definition. Tags in Splunk allow assigning names to specific single field and value combinations, so to accommodate Application + Business Activities pairs Event Types are used.
To add/change Business Process definitions follow these steps:
Go to Web UI -> Settings -> Event types
Create/Edit Business Process eventtype definition
Enter a unique Eventtype name. We suggest the following notation for future management convenience: BP-(Business Process Name)
Add/Edit search string defining which Application + Business Activity pairs belong to this Business Process. Make sure sourcetype is aternity*. Below is search string example:
(sourcetype=aternity* (APPLICATION_NAME=”Microsoft Outlook” AND (ACTIVITY_NAME=”Send Mail To Outbox” OR ACTIVITY_NAME=”Open Mail” OR ACTIVITY_NAME=”Preview Mail”)) OR (APPLICATION_NAME=”SAP” AND (ACTIVITY_NAME=”Search Account” OR ACTIVITY_NAME=”Create Account” OR ACTIVITY_NAME=”Save Record”)))
5. Add/Edit Tag(s) for this event type. You cannot define tags with spaces in Splunk, so substitute them with underscores. Business Process tag notation is Business_Process_Name_biz . Note that _biz suffix is mandatory, it is being used to filter Business Process definition tags from any other tags set up by other Splunk applications.
Troubleshooting FAQ
If Application dashboards show no search results:
Make sure that you user role has access to indexes with data ingested from AppInternals and/or Aternity and they are included into the list of indexes searched by default – check this in role settings
Make sure that index with data ingested from AppInternals holds results for the time frame selected on dashboards – check this with direct search on index.
If Business Processes and Activities dashboard ends up with empty panels, make sure that Business Processes are correctly defined for the Aternity Data (see Defining Business Processes for Aternity Data).
If drilldowns to EUEM to APM dashboards end up with empty panels, make sure that Aternity_Mapping calculated field containing AppInternals App/TX mapping to one or many Aternity App/Business Activities is defined correctly (see Defining Appnternals to Aternity Mapping).
If drilldowns from the Application dashboards do not pass selected time frame into AppInternals, it means that your time format is different from default. Contact us and we will change it according to your Splunk settings.
Comments