# What is Log Aggregation?

* It's a way of collecting and tagging application logs from many different services into a single dashboard that can easily be searched

### Popular Open Source Option: ELK

<figure><img src="/files/F5BgsfOrdftO0t7iEnUM" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/b78dP8dahquH1cBl0QEp" alt=""><figcaption></figcaption></figure>

#### Logstash:

* Gets all the messages
* Uses each field as a tag (let's say, `time, user, what the user did`
* Does not log the messages but sends them to Elasticsearch

<figure><img src="/files/RCIJShzomXKNqZCtetPP" alt=""><figcaption></figcaption></figure>

#### Elasticsearch

* Efficient database that logs all the info coming in from Logstash

#### Kibana

* You as the admin would connect to Kibana
* Kibana would query Elasticsearch (the DB)
* You would get the results needed to identify the fault

### How to use ELK to diagnose a PROD problem?

* Report: user saw error code 12345 when he tried to do "x" actions
* With ELK set up, all we have to do is go to Kibana and query recent logs for 12345
* That might give us the name of the service: "backend"
* Then we could narrow down the time to the five second interval around the error and filter for log messages emitted by either the backend or the database
* Finally, we'd read those log messages to find the context for the bug

### Bonus: Add log aggregation as an extra test

* Could be useful to see if there is any error messages coming back from running the stack

```docker
FROM vm/ubuntu:18.04

RUN docker-compose up -d
RUN docker-compose -f elk.yml up -d
RUN docker-compose exec web npm run test
RUN docker-compose exec elasticsearch curl -x GET 'http://localhost:9200/logs/_search
```

### Options of Log Aggregation platforms:

* ELK
* Fluentd
* DataDog
* LogDNA
* AWS CloudWatch Logs


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.arkannis.net/general-concepts/devops/what-is-log-aggregation.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.