logback-gelf

A Logback appender that encodes logs to
GELF and transports them
to Graylog servers.

Support for dropwizard logging.

Dependency information

Latest version:

Gradle

compile "com.bornconfused:logback-gelf:0.6"

Maven

<dependency>
  <groupId>com.bornconfused</groupId>
  <artifactId>logback-gelf</artifactId>
  <version>0.6</version>
</dependency>

Features

• Append via TCP or UDP (with chunking) to a remote graylog server
• MDC k/v converted to fields
• Fields may have types
• Auto include logger_name
• Auto include Markers
• Auto include Thread name
• Static fields (E.g facility)
• Very Few dependencies (Logback and GSON)

Configuring Logback

The minimal possible logback.xml you can write is something like.

<configuration>
    <appender name="GELF UDP APPENDER" class="com.bornconfused.logbackgelf.GelfUDPAppender">
        <encoder class="com.bornconfused.logbackgelf.GZIPEncoder">
            <layout class="com.bornconfused.logbackgelf.GelfLayout"></layout>
        </encoder>
    </appender>
   <root level="debug">
    <appender-ref ref="GELF UDP APPENDER" ></appender-ref>
  </root>
</configuration>

A more complete example that shows how you would overwrite many
default values:

<configuration>
    <!--Use TCP instead of UDP-->
    <appender name="GELF TCP APPENDER" class="com.bornconfused.logback.net.SocketEncoderAppender">
        <remoteHost>somehost.com</remoteHost>
        <port>12201</port>
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="com.bornconfused.logbackgelf.GelfLayout">
                <!--An example of overwriting the short message pattern-->
                <shortMessageLayout class="ch.qos.logback.classic.PatternLayout">
                    <pattern>%ex{short}%.100m</pattern>
                </shortMessageLayout>
                <!-- Use HTML output of the full message. Yes, any layout can be used (please don't actually do this)-->
                <fullMessageLayout class="ch.qos.logback.classic.html.HTMLLayout">
                    <pattern>%relative%thread%mdc%level%logger%msg</pattern>
                </fullMessageLayout>
                <useLoggerName>true</useLoggerName>
                <useThreadName>true</useThreadName>
                <useMarker>true</useMarker>
                <host>Test</host>
                <additionalField>ipAddress:_ip_address</additionalField>
                <additionalField>requestId:_request_id</additionalField>
                <includeFullMDC>true</includeFullMDC>
                <fieldType>requestId:long</fieldType>
                <!--Facility is not officially supported in GELF anymore, but you can use staticFields to do the same thing-->
                <staticField class="com.bornconfused.logbackgelf.Field">
                  <key>_facility</key>
                  <value>GELF</value>
                </staticField>
            </layout>
        </encoder>
    </appender>
    <root level="debug">
        <appender-ref ref="GELF TCP APPENDER" ></appender-ref>
    </root>
</configuration>

GelfLayout

com.bornconfused.logbackgelf.GelfLayout

This is where most configuration resides, since it’s the part that
actually converts a log event into a GELF compatible JSON string.

• useLoggerName: If true, an additional field call “_loggerName”
  will be added to each gelf message. Its contents will be the fully
  qualified name of the logger. e.g: com.company.Thingo. Default:
  false
• useThreadName: If true, an additional field call “_threadName”
  will be added to each gelf message. Its contents will be the name of
  the thread. Default: false
• host The hostname of the host from which the log is being sent.
  Displayed under source on web interface. Default:
  getLocalHostName()
• useMarker: If true, and the user has set a
  slf4j Marker on their
  log, then the marker.toString() will be added to the gelf message
  as the field “_marker”. Default: false
• shortMessageLayout: The
  Layout used to create
  the gelf short_message field. Shows up in the message column of
  the log summary in the web interface. Default: "%ex{short}%.100m"
  (PatternLayout)
• fullMessageLayout: The
  Layout used to create
  the gelf full_message field. Shows up in the message field of the
  log details in the web interface. Default: "%rEx%m"
  (PatternLayout)
• additionalFields: See additional fields below. Default: empty
• fieldType: See field type conversion below. Default: empty
  (fields sent as string)
• staticFields: See static fields below. Note, now that facility
  is deprecated, use this to set a facility Default: empty
• staticAdditionalFields: deprecated. Use staticFields. Default:
  empty
• includeFullMDC: See additional fields below. Default: false

Transports

Both UDP and TCP transports are supported. UDP is the recommended
graylog transport.

UDP

UDP can be configured using the
com.bornconfused.logbackgelf.GelfUDPAppender appender. Once messages reach
a certain size, they will be chunked according to the
gelf spec. A maximum of
128 chunks can be sent per log. If the encoded log is bigger than
that, the log will be dropped. Assuming the default 512 max packet
size, this allows for 65536 bytes (64kb) total per log message
(unzipped).

• remoteHost: The remote graylog server host to send log messages
  to (DNS or IP). Default: "localhost"
• port: The remote graylog server port. Default: 12201
• maxPacketSize: The maximum number of bytes per datagram packet.
  Once the limit is reached, packets will be chunked. Default: 512

GZIP

For UDP, you have the option of Gzipping the Gelf JSON before sending
over UDP. To do this, replace the
ch.qos.logback.core.encoder.LayoutWrappingEncoder encoder with the
com.bornconfused.logbackgelf.GZIPEncoder encoder. E.g

<appender name="GELF UDP APPENDER" class="com.bornconfused.logbackgelf.GelfUDPAppender">
    <encoder class="com.bornconfused.logbackgelf.GZIPEncoder">
        <layout class="com.bornconfused.logbackgelf.GelfLayout"></layout>
    </encoder>
</appender>

Remember, The GZIP encoder should NOT be used with TCP

TCP

TCP transport can be configured using the
com.bornconfused.logback.net.SocketEncoderAppender appender. Unfortunately,
the built in Logback Socket
Appender
doesn’t give you control of how logs are encoded before being sent
over TCP, which is why you have to use this appender. Note
that due to an unresolved Graylog
issue, GZIP
is not supported when using TCP.

<appender name="GELF TCP APPENDER" class="com.bornconfused.logback.net.SocketEncoderAppender">
    <port>12201</port>
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="com.bornconfused.logbackgelf.GelfLayout">
            ....
        </layout>
    </encoder>
</appender>

• remoteHost: The remote graylog server host to send log messages
  to (DNS or IP). Default: "localhost"
• port: The remote graylog server port. Required.
• maxPacketSize: The number of logs to keep in memory while the
  graylog server can’t be reached. Default: 128
• acceptConnectionTimeout: Milliseconds to wait for a connection
  to be established to the server before failing. Default: 1000

Extra features

Additional Fields

Additional Fields are extra k/v pairs that can be added to the GELF
json, and thus searched as structured data using graylog. In the slf4j
world, MDC (Mapped Diagnostic
Context) is an excellent way of programmatically adding fields to your
GELF messages.

Let’s take an example of adding the ip address of the client to every
logged message. To do this we add the ip address as a key/value to the
MDC so that the information persists for the length of the request,
and then we inform logback-gelf to look out for this mapping every
time a message is logged.

1. Store IP address in MDC

// Somewhere in server code that wraps every request
...
org.slf4j.MDC.put("ipAddress", getClientIpAddress());
...

1. Inform logback-gelf of MDC mapping

<layout class="com.bornconfused.logbackgelf.GelfLayout">
    <additionalField>ipAddress:_ip_address</additionalField>
</layout>

If the property includeFullMDC is set to true, all fields from the
MDC will be added to the gelf message. Any key, which is not listed as
additionalField will be prefixed with an underscore. Otherwise the
field name will be obtained from the corresponding additionalField
mapping.

If the property includeFullMDC is set to false (default value) then
only the keys listed as additionalField will be added to a gelf
message.

Static Fields

Use static additional fields when you want to add a static key value
pair to every GELF message. Key is the additional field key (and
should thus begin with an underscore). The value is a static string.

Now that the GELF facility is deprecated, this is how you add a
static facility. StaticFields replace staticAdditionalFields

E.g in the appender configuration:

<layout class="com.bornconfused.logbackgelf.GelfLayout">
  <staticField class="com.bornconfused.logbackgelf.Field">
    <key>_facility</key>
    <value>GELF</value>
  </staticField>
  <staticField class="com.bornconfused.logbackgelf.Field">
    <key>_node_name</key>
    <value>www013</value>
  </staticField>
</layout>

Static Additional Fields (deprecated)

Static Additional fields have been deprecated and superceded by
staticFields. While they offered a more concise way of expressing the
key/value pair, it was impossible to include a colon in the value.
staticFields are fully structured and don’t have this problem.

Field type conversion

You can configure a specific field to be converted to a numeric type.
Key is the additional field key as inserted into the MDC, value is the
type to convert to. Currently supported types are int, long, float and double.

<layout class="com.bornconfused.logbackgelf.GelfLayout">
    <additionalField>requestId:_request_id</additionalField>
    <fieldType>requestId:long</fieldType>
</layout>

If the conversion fails, logback-gelf will leave the field value alone
(i.e.: send it as String) and print the stacktrace