# What is Log Aggregation?

* It's a way of collecting and tagging application logs from many different services into a single dashboard that can easily be searched

### Popular Open Source Option: ELK

<figure><img src="https://3885248957-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoE4wMO1dMVDOGDjh0En7%2Fuploads%2FA4RQPeakNGQlj7ab6waP%2Fimage.png?alt=media&#x26;token=b4420606-9902-4ebd-a422-e91ffda80d32" alt=""><figcaption></figcaption></figure>

<figure><img src="https://3885248957-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoE4wMO1dMVDOGDjh0En7%2Fuploads%2FCGPCVuTcE4KXY4UkZNar%2Fimage.png?alt=media&#x26;token=091a0bb7-a7c3-45af-829d-c0f8e39e5575" alt=""><figcaption></figcaption></figure>

#### Logstash:

* Gets all the messages
* Uses each field as a tag (let's say, `time, user, what the user did`
* Does not log the messages but sends them to Elasticsearch

<figure><img src="https://3885248957-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoE4wMO1dMVDOGDjh0En7%2Fuploads%2FH0KHI7zvGRPUQYVkpetO%2Fimage.png?alt=media&#x26;token=00b5f9f2-bfa8-465e-a398-00ce012fab77" alt=""><figcaption></figcaption></figure>

#### Elasticsearch

* Efficient database that logs all the info coming in from Logstash

#### Kibana

* You as the admin would connect to Kibana
* Kibana would query Elasticsearch (the DB)
* You would get the results needed to identify the fault

### How to use ELK to diagnose a PROD problem?

* Report: user saw error code 12345 when he tried to do "x" actions
* With ELK set up, all we have to do is go to Kibana and query recent logs for 12345
* That might give us the name of the service: "backend"
* Then we could narrow down the time to the five second interval around the error and filter for log messages emitted by either the backend or the database
* Finally, we'd read those log messages to find the context for the bug

### Bonus: Add log aggregation as an extra test

* Could be useful to see if there is any error messages coming back from running the stack

```docker
FROM vm/ubuntu:18.04

RUN docker-compose up -d
RUN docker-compose -f elk.yml up -d
RUN docker-compose exec web npm run test
RUN docker-compose exec elasticsearch curl -x GET 'http://localhost:9200/logs/_search
```

### Options of Log Aggregation platforms:

* ELK
* Fluentd
* DataDog
* LogDNA
* AWS CloudWatch Logs