

Notable events are created first, which then lead to one or more traditional alerts. We’ll dig in to these configurations later, but for now, we just want to acknowledge the difference between the two concepts.Īt the risk of being pedantic, what is an alert anyway? Is it an email? A text message? A ticket to a ticketing system? A flashing red light in the NOC? Something else? Within ITSI, we take a two-tiered approach to generating alerts. KPI severities are viewable in the service analyzer dashboard, deep dives, and other UI locations, but in and of themselves don’t generate alerts.Īlerts are generated from additional configurations, driven from KPI severity and service health score changes.

Thresholds apply only to KPIs they dictate when a KPI severity (or status, as they are sometimes referred) changes from normal to critical, high, low, etc. Let’s first clarify the difference between thresholds and alerts-in ITSI, these are related but separate concepts. Dependent services are also optional and are simply references to other already configured ITSI services on which this service depends. KPIs are optional and when defined, will require threshold configurations. Dependent services (Optional sometimes referred to as subservices)Įach service will always have a health score, which is computed based on the status of the KPIs and subservices defined for that service.We'll refer back to these concepts during alert configuration, so having a basic working understanding of this hierarchy is important. To understand how service issues will ultimately result in meaningful alerts, we should briefly revisit the hierarchy of KPIs and services. In this multi-part blog, we'll outline some practical guidance to get you going. The changes should take effect immediately.Excited about your shiny new Splunk IT Service Intelligence (ITSI) license? Well, you should be! But navigating from your first service creation to meaningful and trusted alerts takes some care and planning. Set the maxSockets attribute to specify the number of sockets that should be available for REST HTTP operations.In the stanza, set the maxThreads attribute to specify the number of threads for REST HTTP operations that Splunk Enterprise should use.

In the $SPLUNK_HOME\etc\system\local, create or edit nf.

Increasing the number of threads can increase the amount of memory that the Splunk Enterprise instance uses. You can override this automatic configuration by making changes to nf. Override automatic socket and thread configuration Both must be present before Splunk Enterprise can make REST calls. The number of available file descriptors is different than the number of threads. For example, if the number of open file descriptors is 36000, then Splunk Enterprise reserves 12000 for sockets for REST HTTP operations. The result is the number of file descriptors available for sockets for REST HTTP operations. It then checks the number of available file descriptors for the system, as configured by the ulimit command. Splunk Enterprise uses the following formulas to compute the thread limit for HTTP REST: Splunk Enterprise thus reserves threads and file descriptors to use for these services. If Splunk Enterprise runs out of either HTTP sockets or threads, it can't complete REST calls to its backend and any such calls fail. Threads let the processes perform tasks, and sockets let the processes communicate with the network. Splunk Enterprise needs threads and file descriptors to perform REST HTTP operations. How Splunk Enterprise calculates threads and sockets for REST HTTP operations
ERROR CODE 32 SPLUNK ITSI SOFTWARE
This error occurs because, as of Splunk Enterprise 6.0 and later, the software limits the number of REST HTTP connections an instance uses to prevent service failure caused by resource exhaustion.
