As [Wang2017] and [Lee1998] state, IDS detection can be classified either as signature-based or anomaly-based detection.
If you need a little primer on intrusion detection systems, read my post on defining IDS in the wild.

Signature-based detection

Signature-based detection, also known as misuse detection, uses “patterns of well-known attacks or weak spots of the system to identify intrusions” [Lee1990]. This bears similarity with Anti-Virus solutions, as their detection model is also (partly) based on attack signatures.
The main advantage of this approach is the high performance and low False Positive rates.
The main disadvantage is its high maintenance need, as attack signatures have to be generated by hand / by another tool.
Signature-based detection is also not able to detect novel (“zero day”) attacks on the system under observation, as no established rules match novel attacks.

Anomaly-based detection

Anomaly-based detection is sometimes also called behaviour-based detection.
In recent times it is often associated with machine learning, as this is one of the major use cases for ML in the context of IDS systems.
The idea is to train your detector with a “normal state” of the system under observation. It “tries to determine wheter deviation from the established normals usage patterns can be flagged as intrusions” [Lee1990].
The main advantage is the detection of any unusual activity, be it novel or not.
The main disadvantage however is the high computing cost while training various machine learning algorithms which can pose a problem with online/realtime-processing of data. Besides the computing cost, anomaly-based detection is prone to a higher False Positive rate than signature-based methods, as it needs a representative normal state dataset during training phase.