Ansible Role logstash¶
This role installs and configures a Logstash server with support for multiple pipelines.
Note that this role does NOT let you specify a particular Logstash version. It simply installs the latest available Logstash version from the repos configured in the system. If you want or need to install a specific version, have a look at the linuxfabrik.lfops.repo_elasticsearch role (Logstash uses the Elasticsearch repository).
Available since LFOps 6.0.0.
Dependent Roles¶
Any LFOps playbook that installs this role runs these for you. Optional ones can be disabled via the playbook’s skip variables.
The official elasticsearch repository must be enabled (role: linuxfabrik.lfops.repo_elasticsearch).
Optional Role Variables¶
logstash__elasticsearch_ca_cert
ASCII-armored PEM CA certificate for TLS connections to Elasticsearch. Should match the CA used by Elasticsearch.
Type: String.
Default: unset
logstash__grok_patterns__host_var / logstash__grok_patterns__group_var
List of custom grok pattern file definitions.
Subkeys:
name:Mandatory. Filename in
/etc/logstash/patterns/.Type: String.
content:Mandatory. Pattern definitions (format:
PATTERN_NAME regex).Type: String.
state:Optional.
presentorabsent.Type: String.
Default:
'present'
Type: List of dictionaries.
Default:
[]
logstash__java_opts
Additional Java options passed to Logstash via
LS_JAVA_OPTS. By default, sets the temp directory to{{ logstash__path_data }}/tmpbecause/tmpon CIS-hardened systems is mounted with noexec.Type: String.
Default:
'-Djava.io.tmpdir={{ logstash__path_data }}/tmp'
logstash__log_level
The log level. Valid values are:
fatal,error,warn,info,debug,trace.Type: String.
Default:
'info'
logstash__monitoring_cluster_uuid
Elasticsearch Cluster UUID. Binds the metrics of Logstash to this specific cluster.
Type: String.
Default: unset
logstash__monitoring_enabled
Enables or disables default collection of Logstash monitoring data.
Type: Bool.
Default:
true
logstash__node_name
A descriptive name for the node.
Type: String.
Default:
'{{ ansible_facts["nodename"] }}'
logstash__path_data
Path to the directory where Logstash stores its data.
Type: String.
Default:
'/var/lib/logstash'
logstash__path_logs
Path to the directory where Logstash stores its logs.
Type: String.
Default:
'/var/log/logstash'
logstash__pipelines__host_var / logstash__pipelines__group_var
List of pipeline definitions.
Subkeys:
pipeline_id:Mandatory. Unique identifier for the pipeline. Used as filename (
<pipeline_id>.conf).Type: String.
content:Mandatory. The pipeline configuration content (input/filter/output sections).
Type: String.
state:Optional.
presentorabsent.Type: String.
Default:
'present'
pipeline_workers:Optional. Number of worker threads for this pipeline.
Type: Number.
pipeline_batch_size:Optional. Maximum number of events per batch.
Type: Number.
pipeline_batch_delay:Optional. Maximum delay in milliseconds before dispatching an undersized batch.
Type: Number.
pipeline_ordered:Optional. Event ordering mode:
auto,true, orfalse.Type: String.
queue_type:Optional. Queue type:
memoryorpersisted.Type: String.
queue_max_bytes:Optional. Maximum queue capacity (e.g.,
'1024mb').Type: String.
queue_page_capacity:Optional. Page data file size for persisted queues.
Type: String.
queue_max_events:Optional. Maximum number of unread events in the queue.
Type: Number.
queue_checkpoint_acks:Optional. Maximum number of acked events before forcing a checkpoint.
Type: Number.
queue_checkpoint_writes:Optional. Maximum number of written events before forcing a checkpoint.
Type: Number.
dead_letter_queue_enable:Optional. Enable dead letter queue for this pipeline.
Type: Bool.
dead_letter_queue_max_bytes:Optional. Maximum size of the dead letter queue.
Type: String.
Type: List of dictionaries.
Default:
[]
logstash__raw
Raw content which will be appended to the
logstash.ymlconfig file.Type: String.
Default:
''
logstash__service_enabled
Enables or disables the logstash service, analogous to
systemctl enable/disable --now.Type: Bool.
Default:
true
logstash__service_state
Controls the state of the logstash service, analogous to
systemctl start/stop/restart/reload. Possible options:started,stopped,restarted,reloaded.Type: String.
Default:
'started'
Example:
# optional
logstash__elasticsearch_ca_cert: '{{ lookup("ansible.builtin.file", "{{ inventory_dir }}/group_files/elasticsearch/ca.crt") }}'
logstash__grok_patterns__host_var:
- name: 'custom_app'
content: |
CUSTOM_TIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}
CUSTOM_LOGLEVEL (DEBUG|INFO|WARN|ERROR|FATAL)
CUSTOM_APPLOG %{CUSTOM_TIMESTAMP:timestamp} %{CUSTOM_LOGLEVEL:level} %{GREEDYDATA:message}
- name: 'old_patterns'
state: 'absent'
logstash__java_opts: '-Djava.io.tmpdir={{ logstash__path_data }}/tmp'
logstash__log_level: 'info'
logstash__monitoring_cluster_uuid: 'zk-HjQtYQGyL3NFSSu7InA'
logstash__monitoring_enabled: false
logstash__node_name: '{{ ansible_facts["nodename"] }}'
logstash__path_data: '/var/lib/logstash'
logstash__path_logs: '/var/log/logstash'
logstash__pipelines__host_var:
- pipeline_id: 'beats'
pipeline_workers: 4
content: |
input {
beats {
port => 5044
}
}
filter {
if [fields][type] == "syslog" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{SYSLOGLINE}" }
}
}
}
output {
elasticsearch {
hosts => ["https://elasticsearch.example.com:9200"]
ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
user => "logstash_writer"
password => "${LOGSTASH_ES_PASSWORD}"
index => "beats-%{+YYYY.MM.dd}"
}
}
- pipeline_id: 'syslog'
queue_type: 'persisted'
queue_max_bytes: '2048mb'
content: |
input {
syslog {
port => 514
}
}
output {
elasticsearch {
hosts => ["https://elasticsearch.example.com:9200"]
ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
index => "syslog-%{+YYYY.MM.dd}"
}
}
- pipeline_id: 'old-pipeline'
state: 'absent'
logstash__raw: |-
config.reload.automatic: true
config.reload.interval: 3s
logstash__service_enabled: true
logstash__service_state: 'started'