Troubleshooting Incident Trigger

An Incident may not trigger for one or more of the following reasons:

  1. Rule filter conditions may not be satisfied, or events may be present. In this case, Fortinet recommends the following:
    • Test the rule with real events. If the rule does not trigger, then there may be some event attributes that are not parsed, or the rule is not written correctly.
    • View the rule and from the Filter condition page, and run the Rule as a Query for a previous time-period to see if there are matches.
  2. The reporting device may be in maintenance mode, in which case, its events are ignored.
  3. The Rule exception condition may be satisfied – this is expected behavior.
  4. Events from Collector are delayed more than dropping_time_threshold when received by Worker. In this case, the log PH_DROP_EVENT_FROM_SHARED_BUFFER will be generated by the Rule Worker.
  5. Rule Worker failed to upload packed aggregated result to Rule Master. This could be a networking issue. In this case, the log PH_RULEMOD_SUMMARY_UPLOAD_FAILED will be generated by the Rule Worker.
  6. Rule Worker failed to pack summary events, before sending to Rule Master because of send buffer limit exceeded. This can happen if a rule is loosely written and the group by table size is very large at Rule Worker level. In this case, the log PH_REPORT_PACK_FAILED will be generated by the Rule Worker.
  7. Rule Master failed to upload Incident to App Server. In this case, the log PH_UTIL_NOTIFICATION_UPLOAD_FAILURE will be generated by the Rule Master.
  8. Incident dropped because of too many incidents from the same rule, or too many incidents in general. In this case, the log PH_DROP_INCIDENT will be generated by the Rule Master. This is done to protect the system from getting flooded with Incidents. There are 6 parameters (in phoenix_config.txt) to control the incident rate:



    Short term Incident generation thresholds:
    • incident_rate_short_term_time_gap = 1 #unit: minute
    • incident_rate_short_term_per_rule_limit = 20
    • incident_rate_short_term_all_rules_limit = 200;

    Long term Incident generation thresholds:

    • incident_rate_long_term_time_gap = 60 #uint: minute
    • incident_rate_long_term_per_rule_limit = 300;
    • incident_rate_long_term_all_rules_limit = 3000;

    For the above parameters, if the following "too many incidents" condition is met, then incident generation pauses until the next hour boundary. For example, if this happens at 12:35PM, then Incident generation pauses from 12:35PM to 1:00PM.

    Excessive Incident Generation Condition:

    In 1 minute

    • For one rule, more than 20 incidents fired OR
    • For all rules, more than 200 incidents fired

    OR in 60 minutes

    • For one rule, more than 300 incidents fired OR
    • For all rules, more than 3000 incidents fired