refcodes-logger: Fancy runtime-logs and highly scalable cloud logging

README

The REFCODES.ORG codes represent a group of artifacts consolidating parts of my work in the past years. Several topics are covered which I consider useful for you, programmers, developers and software engineers.

What is this repository for?

This artifact provides flexible logging of any data to any data sink (including files, databases or the console). It supports straight forward, composite (clustering) or partitioning functionality provided by different implementations of the Logger type. The RuntimeLogger type harnesses the Logger type for logging your runtime log messages and integrates with SLF4J seamlessly (and may also act as an alternative data sink to log to when using SLF4J.

How do I get set up?

To get up and running, include the following dependency (without the three dots “…”) in your pom.xml:

1
2
3
4
5
6
7
8
9
<dependencies>
	...
	<dependency>
		<artifactId>refcodes-logger</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.0.4</version>
	</dependency>
	...
</dependencies>

The artifact is hosted directly at Maven Central. Jump straight to the source codes at Bitbucket. Read the artifact’s javadoc at javadoc.io.

Introduction

The RuntimeLogger implementations are being configured with a Logger implementation. You may use the RuntimeLogger type, which, depending on you configuring it, logs to a SimpleDB cluster, the console (with ANSI escape sequence support) or any I/O device or to an SLF4J logger. Being an alternative data sink for SLF4J, the RuntimeLogger type’s architecture settles upon a generic Logger type, which actually can be used to log high volume logs of any data type and not being restricted to runtime logs. Furthermore, the RuntimeLogger type adds functionality not found in other logging frameworks such as logging out the class- and method-names of the logging source without any configuration or additional lines of code.

See also Logging like the nerds log for a quick start on how to do fancy console logging of your application’s runtime logs!

How do I get started?

You configure your RuntimeLoggerFactoryImpl (and the RuntimeLoggerFactorySingleton sub-class) by providing a runtimelogger.ini file (also supported are *.toml, *.yaml, *.xml, *.properties or *.json notations) in one of those locations relative to your main class’s location:

  • .
  • ./config
  • ./etc
  • ./settings
  • ./.config
  • ./.settings
  • ../config
  • ../etc
  • ../settings
  • ../.config
  • ../.settings

Given your main class’s JAR file resides in the folder /opt/app/lib, then the valid locations for the runtimelogger.ini file are:

  • /opt/app/lib
  • /opt/app/lib/config
  • /opt/app/lib/etc
  • /opt/app/lib/settings
  • /opt/app/lib/.config
  • /opt/app/lib/.settings
  • /opt/app/config
  • /opt/app/etc
  • /opt/app/settings
  • /opt/app/.config
  • /opt/app/.settings

(see also the class ConfigLocator.ALL which is used to identify the configuration’s location)

In case you pass a JVM argument via -Dconfig.dir=path_to_your_config_dir (where path_to_your_config_dir stands for the path to the directory where you placed configuration files such as the runtimelogger.ini file), then your path_to_your_config_dir is placed first in the list of configuration directories to look at (in case the directory exists). See SystemProperty.CONFIG_DIR as well as ConfigLocator.ALL.

Use the JVM argument -Dconfig.dir=path_to_your_config_dir for your custom runtimelogger.ini configuration folder.

The runtimelogger.ini configuration is deadly simple:

1
2
3
4
5
6
7
8
9
10
11
12
[root]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/logger=org.refcodes.logger.SystemLogger

[com.acme]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/name=com.acme
runtimelogger/logger=org.refcodes.logger.SystemLogger

The sections in squared braces (e.g. [com.acme]) represent the Java package for which the afterwards configured RuntimeLogger is responsible; e.g. a log issued from inside a class located in the package com.acme (or in one of its sub-packages) will be handled by the com.acme logger. In case no logger is found for a given package, then the root logger found below the [root] section is used. Useful to know that a logger for a package namespace is only created once.

If you like logs colored nicely with ANSI escape codes, then you will love the ConsoleLoggerSingleton:

1
2
3
4
5
[root]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

Make sure to include the refcodces-logger-alt-console dependency in your build setup to include the ConsoleLoggerSingleton logger.

Supported notations

Below find the various supported notations by example. We configure a root logger as well as a logger for the package com.acme, both using the fancy ConsoleLoggerSingleton logger.

Please note that some notations use the this keyword to denote the value of the enclosing element as some notations do not support (or make it hard) to assign a value to an element which has child elements.

TOML or INI

Using TOML or INI based runtimelogger.toml or runtimelogger.ini properties:

1
2
3
4
5
6
7
8
9
10
11
[root]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

[com.acme]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=ERROR
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

An advanced example setting the row width, disabling ANSI escape codes and setting an ASCII table style looks as follows :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton
runtimelogger/logger/rowWidth=120
runtimelogger/logger/escapeCodes=false
runtimelogger/logger/style=ASCII
runtimelogger/logger/layout=HACKER

[com.acme]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=ERROR
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

Java properties

Using Java based runtimelogger.properties properties:

1
2
3
4
5
6
7
root/runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
root/runtimelogger/logPriority=INFO
root/runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

com/acme/runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
com/acme/runtimelogger/logPriority=ERROR
com/acme/runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

An advanced example setting the row width, disabling ANSI escape codes and setting an ASCII table style looks as follows :

1
2
3
4
5
6
7
8
9
10
11
root/runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
root/runtimelogger/logPriority=INFO
root/runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton
root/runtimelogger/logger/rowWidth=120
root/runtimelogger/logger/escapeCodes=false
root/runtimelogger/logger/style=ASCII
root/runtimelogger/logger/layout=HACKER

com/acme/runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
com/acme/runtimelogger/logPriority=ERROR
com/acme/runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

XML

Using XML based runtimelogger.xml properties:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<config>
	<root>
		<runtimelogger this="org.refcodes.logger.RuntimeLoggerImpl">
			<logPriority>INFO</logPriority>
			<logger this="org.refcodes.logger.alt.console.ConsoleLoggerSingleton" />
		</runtimelogger>
	</root>
	<com>
		<acme>
			<runtimelogger this="org.refcodes.logger.RuntimeLoggerImpl">
				<logPriority>ERROR</logPriority>
				<logger this="org.refcodes.logger.alt.console.ConsoleLoggerSingleton" />
			</runtimelogger>
		</acme>
	</com>
</config>

Please note that this notation uses the this attribute to denote the value of the enclosing element as this notations makes it hard to assign a value via mixed content to an element which has child elements.

An advanced example setting the row width, disabling ANSI escape codes and setting an ASCII table style looks as follows :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<config>
	<root>
		<runtimelogger this="org.refcodes.logger.RuntimeLoggerImpl">
			<logPriority>INFO</logPriority>
			<logger this="org.refcodes.logger.alt.console.ConsoleLoggerSingleton" />
			<logger>
				<rowWidth>120</rowWidth>
				<escapeCodes>false</escapeCodes>
				<style>ASCII</tableStyle>
				<layout>HACKER</layout>
			</logger>
		</runtimelogger>
	</root>
	<com>
		<acme>
			<runtimelogger this="org.refcodes.logger.RuntimeLoggerImpl">
				<logPriority>ERROR</logPriority>
				<logger this="org.refcodes.logger.alt.console.ConsoleLoggerSingleton" />
			</runtimelogger>
		</acme>
	</com>
</config>

YAML

Using YAML based runtimelogger.yaml properties:

1
2
3
4
5
6
7
8
9
10
11
12
root:
    runtimelogger:
        this: org.refcodes.logger.RuntimeLoggerImpl
        logPriority: INFO
        logger: org.refcodes.logger.alt.console.ConsoleLoggerSingleton

com:
    acme:
        runtimelogger:
            this: org.refcodes.logger.RuntimeLoggerImpl
            logPriority: ERROR
            logger: org.refcodes.logger.alt.console.ConsoleLoggerSingleton       

Please note that this notations uses the this keyword to denote the value of the enclosing element as this notations do not support to assign a value to an element which has child elements.

An advanced example setting the row width, disabling ANSI escape codes and setting an ASCII table style looks as follows :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
root:
    runtimelogger:
        this: org.refcodes.logger.RuntimeLoggerImpl
        logPriority: INFO
        logger: 
            this: org.refcodes.logger.alt.console.ConsoleLoggerSingleton
            rowWidth: 120
            escapeCodes: false
            style: ASCII
            layout: HACKER
com:
    acme:
        runtimelogger:
            this: org.refcodes.logger.RuntimeLoggerImpl
            logPriority: ERROR
            logger: org.refcodes.logger.alt.console.ConsoleLoggerSingleton     

JSON

Using JSON based properties runtimelogger.json properties:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
	"root": {
		"runtimelogger": {
			"this": "org.refcodes.logger.RuntimeLoggerImpl",
			"logPriority": "INFO",
			"logger": "org.refcodes.logger.alt.console.ConsoleLoggerSingleton"
		}
	},
	"com": {
		"acme": {
			"runtimelogger": {
				"this": "org.refcodes.logger.RuntimeLoggerImpl",
				"logPriority": "ERROR",
				"logger": "org.refcodes.logger.alt.console.ConsoleLoggerSingleton"
			}
		}
	}
}    

Please note that this notations uses the this keyword to denote the value of the enclosing element as this notations do not support to assign a value to an element which has child elements.

An advanced example setting the row width, disabling ANSI escape codes and setting an ASCII table style looks as follows :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
	"root": {
		"runtimelogger": {
			"this": "org.refcodes.logger.RuntimeLoggerImpl",
			"logPriority": "INFO",
			"logger": {
				"this": "org.refcodes.logger.alt.console.ConsoleLoggerSingleton",
				"rowWidth": "120",
				"escapeCodes": "false",
				"style": "ASCII"
				"layout": "HACKER"
			}
		}
	},
	"com": {
		"acme": {
			"runtimelogger": {
				"this": "org.refcodes.logger.RuntimeLoggerImpl",
				"logPriority": "ERROR",
				"logger": "org.refcodes.logger.alt.console.ConsoleLoggerSingleton"
			}
		}
	}
}

Runtime logging

Using the refcodes-logging framework in your code to do runtime logging (such as LOGGER.debug("Starting ...") or LOGGER.error("Earth control, we have a problem ...") is also deadly simple. Just declare a static member variable to your class where you want to do your logging:

private static RuntimeLogger LOGGER = RuntimeLoggerFactorySingleton.getInstance().createInstance();

The usage is very similar to a logger such as Log4j or SLF4J:

1
2
3
LOGGER.info( "Starting the application ..." );
LOGGER.error( "Failed to read file as of an IO exception!", e );
LOGGER.trace( "Start time is {0}, duration is {1} ms, end time is {2}", theStartTime, theDurationMs, theEndTime );

By default, the refcodes-logger logs out the additional following information:

  • The overall line number of the current logged line
  • The class and the method from which you produced a log line

Console logging

The artifact refcodes-logger-alt-console provides a good starting point playing around with the refcodes-logger tool-box. With the refcodes-logger-alt-console you can print out colorful console logs into your terminal. It auto-detects the features of your terminal such as the number of chars per row, whether you use a shell on a Unix alike operating system or CMD.EXE on Windows and determines whether your terminal supports coloring of the output (ANSI escape codes).

Add the following dependency to your pom.xml:

1
2
3
4
5
6
7
8
9
<dependencies>
	...
	<dependency>
		<artifactId>refcodes-logger-alt-console</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.0.4</version>
	</dependency>
	...
</dependencies>

It transitively pulls the refcodes-logger artifact. Then create a runtimelogger.ini similar to the one above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=INFO
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton
runtimelogger/logger/style=ASCII
runtimelogger/logger/layout=HACKER

[com.acme]

runtimelogger=org.refcodes.logger.RuntimeLoggerImpl
runtimelogger/logPriority=DEBUG
runtimelogger/name=com.acme
runtimelogger/logger=org.refcodes.logger.alt.console.ConsoleLoggerSingleton

There are some attributes which can be (optionally) configured when declaring the ConsoleLoggerSingleton in your runtimelogger.ini:

  • style: The table’s style, values as of the TableStyle enumeration:
    • ASCII
    • ASCII_BLANK_HEADER_ASCII_BLANK_BODY
    • ASCII_HEADER_ASCII_BODY
    • BLANK
    • BLANK_HEADER_BLANK_BODY
    • BOLD
    • BOLD_HEADER_BOLD_BODY
    • BOLD_HEADER_SINGLE_BODY
    • DOUBLE
    • DOUBLE_HEADER_DOUBLE_BODY
    • DOUBLE_HEADER_DOUBLE_SINGLE_BODY
    • DOUBLE_SINGLE_HEADER_DOUBLE_SINGLE_BODY
    • DOUBLE_SINGLE_HEADER_SINGLE_BODY
    • DOUBLE_SINGLE_HEADER_SINGLE_DASHED_BODY
    • HYBRID_BOLD_HEADER_SINGLE_BODY
    • SINGLE
    • SINGLE_BLANK_HEADER_SINGLE_BLANK_BODY
    • SINGLE_DOUBLE_HEADER_SINGLE_BODY
    • SINGLE_DOUBLE_HEADER_SINGLE_DOUBLE_BODY
    • SINGLE_HEADER_SINGLE_BODY
  • rowWidth: In case the row width cannot be determined automatically, you can either set the system property -Dconsole.width=n (see below) or you can provide the number of chars you desire per row with the rowWidth attribute, e.g. rowWidth=160.

  • layout: The setup of the rows to be displayed, values as of the ColumnLayout enumeration:
    • BASIC
    • ENDUSER
    • DEVELOPER
    • GRANDPA
    • HACKER
    • DEVOPS
    • SUPERUSER
    • ANALYST

The ConsoleLogger (and in turn as being its sub-class the ConsoleLoggerSingleton) by default uses the Terminal.getConsoleWidth() method, which determines the width in characters of the system’s console in use. In case you pass a -Dconsole.width=n JVM argument (where n stands for the number of chars you desire for your console row width), then your width is taken, else the actual console’s width is being tried to be determined (see also SystemProperty.CONSOLE_WIDTH).

A fancy output could look as follows:

such

The output as a console copy’n’paste dump (note that the tilde character ~ is used to truncate long lines; all above WARNING is not truncated):

steiner@proteus:~/Workspaces/org.refcodes/refcodes-runtime-ext/refcodes-runtime-ext-console$ java -jar target/refcodes-runtime-ext-console-0.1.1-SNAPSHOT.jar
───────┬───────────────────┬───────┬───────────────┬──────────────────────────────┬──────────────────────┬────────────────────────────────────────────────────
0000001│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │~12 12:19:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
0000002│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │Console width    := 158                             
0000003│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │Console height   := 24                              
0000004│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │ANSI support     := true                            
0000005│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │Operating system := UNIX                            
0000006│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │CLI              := SHELL                           
0000007│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │Encoding         := UTF8                            
0000008│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │Process (PID)    := 13082                           
0000009│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │~rg.refodes.runtime.ext.console.SystemInfoConsoleApp
0000010│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │stty size        :=                                 
0000011│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │tput cols        := 80                              
0000012│2015-04-04T11:21:19│INFO   │main           │~.console.SystemInfoConsoleApp│printInfo()           │tput lines       := 24                              
steiner@proteus:~/Workspaces/org.refcodes/refcodes-runtime-ext/refcodes-runtime-ext-console$ 

SLF4J support

Instead of logging to the console via refcodes-logger-alt-console, you can use the existing SLF4J binding refcodes-logger-alt-slf4j and make your refcodes-logger log via SLF4J just by including the below dependency (instead of the above one) in your pom.xml:

1
2
3
4
5
6
7
8
9
<dependencies>
	...
	<dependency>
		<artifactId>refcodes-logger-alt-slf4j</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.0.4</version>
	</dependency>
	...
</dependencies>

This will cause your …

private static RuntimeLogger LOGGER = RuntimeLoggerFactorySingleton.getInstance().createInstance();

Now logs are printed with your SLF4J binding, which could be a Log4J binding.

SLF4J binding

Via the refcodes-logger-ext-slf4j artifact, you can use the refcodes-logger framework itself as an SLF4J binding - in order to make, for example, your Tomcat or spring-boot log to the fancy refcodes-logger-alt-console or a NoSQL DB cluster (as of refcodes-logger-alt-simpledb):

1
2
3
4
5
6
7
8
9
<dependencies>
	...
	<dependency>
		<artifactId>refcodes-logger-ext-slf4j</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.0.4</version>
	</dependency>
	...
</dependencies>

In addition you have to add a dependency of one of the refcodes-logger-alt-* artifacts to your pom.xml and an according runtimelogger.ini pulling that logger implementation you want your SLF4J logs to log to …

Attention: Never use the SLF4J binding refcodes-logger-ext-slf4j together with the refcodes-logger-alt-slf4j logger implementation, else the one will look for the other, the other for the one … until you get an StackOverflowException :D

With the SLF4J binding, you can log all your SLF4J logs (also the ones of the libs you included in your app) to any of the refcodes-logger-alt-* implementations:

Make your microservices log into a NoSQL database cluster with the refcodes-logger-alt-simpledb artifact providing support for Amazon’s SimpleDB.

Scalability

Just some keywords on scalability: The refcodes-logger framework implements the composite pattern and various partitioning and composition strategies for high scalability and therewith high throughput.

  • The CompositeLoggerImpl is an asynchronous logger with a log lines queue. Depending on the availability of the encapsulated loggers, the one or the other encapsulated logger takes the next waiting log line in the queue and processes it. The encapsulated refcodes-loggers can be any instance implementing the Logger interface (e.g. one of the loggers found in the refcodes-logger artifact and the refcodes-logger-alt-* artifacts).

  • The PartedLoggerImpl implements partitioning functionality. You define a partitioning criteria for your log lines, e.g. a tenant’s ID (or the date or in case of the refcodes-logger a package, class name or method) and the parted-logger takes care to address the encapsulated responsible logger (which in turn can be another parted-logger or a composite-logger or any other refcodes-logger). The encapsulated refcodes-loggers can be any instance implementing the Logger interface (e.g. one of the loggers found in the refcodes-logger artifact and the refcodes-logger-alt-* artifacts).

Querying

As the data sink to which a refcodes-logger writes its output can be write only (e.g. a terminal window) or write, read and delete (e.g. a NoSQL database such as the SimpleDB logger), there are three flavors of the Logger interface. For example the composite-logger and the parted-logger provides them three flavors:

Under the hood

Logger

The Logger is the most plain logger definition allowing you to log any kinds of data provided in a Record and which can contain any number of Column elements, so that you are enabled to log big data of any structured information (e.g. high volume HTTP-Requests entering your RESTful microservice).

RuntimeLogger

The RuntimeLogger defines a plain simple interface for logging out runtime information generated by software systems. The RuntimeLoggerImpl implementation takes care of logging out the class and the method generating a log line.

The RuntimeLoggerImpl actually takes a Logger instance, whichs implementation to take is up to you:

Use the RuntimeLoggerFactorySingleton factory (the factory is implemented as a singleton) to obtain RuntimeLogger instances configured by a runtimelogger.ini file.

Composite logger

The CompositeLogger uses the composite patter to forward Logger functionality to a number encapsulated logger instances. Depending on the performance (and availability) of an encapsulated logger, the calls to the composite’s CompositeLogger.log(org.refcodes.tabular.Record) method are executed by the next encapsulated logger ready for execution. An invocation of the CompositeLogger.log(org.refcodes.tabular.Record) method is forwarded to exactly one of the encapsulated Logger instances. The actual instance being called depends on its availability (in case, partitioning is needed, take a look at the PartedLogger and its sub-classes).

Using the CompositeLogger, a huge number of Record instances can be logged in parallel by logging them to different physical data sinks (represented by the encapsulated logger instances), thereby avoiding a bottleneck which a single physical data sink would cause for logging.

Internally a log line queue (holding Record instances to be logged) as well as a daemon thread per encapsulated logger (taking elements from the log line queue) are used to decouple the encapsulated Logger instances from the CompositeLogger. A given number of retries are approached in case there is an overflow of the log line queue; this happens when the queue is full and there are none encapsulated Logger instances to take the next Record. To avoid a building up of the log line queue, eventually causing an out of memory, log lines not being taken into the log line queue (as it is full) within the given number of retries, them log lines are dismissed. In such a case a warning with a log-level WARN is printed out.

Query logger

The QueryLogger type extends the CompositeLogger type with query functionality as of QueryLogger.findLogs(org.refcodes.criteria.Criteria). In contrast to the CompositeLogger.log(org.refcodes.tabular.Record) method, which is forwarded to exactly one of the encapsulated Logger instances, the QueryLogger.findLogs() (and the loke) method calls are forwarded to all of the encapsulated Logger instances (in case, partitioning is needed, take a look at the PartedQueryLogger type).

Trim logger

The CompositeTrimLogger extends the CompositeLogger with trim functionality. In contrast to the CompositeLogger.log(org.refcodes.tabular.Record) method, which is forwarded to exactly one of the encapsulated Logger instances, the CompositeTrimLogger.deleteLogs(org.refcodes.criteria.Criteria) and CompositeTrimLogger.clear() method calls are forwarded to all of the encapsulated Logger instances (in case, partitioning is needed, take a look at the PartedTrimLogger type).

Parted logger

The PartedLogger is a partitioning Logger which encapsulates Logger instances or CompositeLogger instances (or sub-classes of it) representing partitions.

This means: A partition is regarded to be a dedicated physical data sink or a CompositeLogger containing Logger instances attached to physical data sinks. A physical data sink may be a database (SQL or NoSQL), a files-system or volatile memory (in memory). To be more concrete: A physical data sink may be a domain when using Amazon’s SimpleDB, it may be a database table when using MySQL, it may be a HashMap or a List when using in memory storage.

The Record instances as managed by the Logger instances are mapped to the fields of the physical data sink (e.g. table columns regarding databases).

The Record instances are stored to, retrieved from or deleted from dedicated partitions depending on partitioning Criteria contained in the Record instances (or the query Criteria instances). The Criteria (e.g. the column partition Criteria in a Record) as provided to the #log(Record) method is used by the PartedLogger to select the partition to be addressed. In case of query operations the query Criteria is used to determine the targeted partition (in case no partition can be determined and a fallback Logger has been configured, then data may get logged to the fallback Logger)

In practice there can be several (composite) Logger instances being the partitions of the PartedLogger, each individually addressed by the partitioning Criteria. This approach a) helps us scale horizontally per partition when using CompositeLogger instances per partition and b) helps limiting the traffic on those horizontally scaling (composite) Logger instances by partitioning the data per Criteria using the parted Logger (or its sub-classes): Partitioning simply means switching to the partition defined by the Criteria to perform the according Logger operation.

Not having the PartedLogger (or a sub-class of it) would cause all the traffic for all Criteria to hit just a single (composite) Logger, limiting the possibility to scale endlessly (this one Logger would be the bottleneck, even when being massively scaled horizontally): In particular this applies when looking at the extended versions of the PartedLogger such as the PartedQueryLogger and the PartedTrimLogger where query requests are passed only to the partition which contains the required data: Increasing query traffic is parted and does not hit increasingly a single (composite) Logger.

A Record instance to be assigned to a partition must provide a so called partition column, whose value is used to determine which partition is to be addressed. The partition identifying column is passed upon construction of the PartedLogger type. Specializations may hide this parameter from their constructors and pass their partitioning column from inside their constructor to the super constructor.

The PartedQueryLogger extends the PartedLogger with the functionality of a query Logger. Any query operations, such as PartedQueryLogger.findLogs(org.refcodes.criteria.Criteria), are targeted at that partition containing the queried data. For this to work, the query must obey some rules:

The query is to contain an EqualWithCriteria instance addressing the partition in an unambiguous way, AndCriteria instances as being part of the root Criteria hierarchy or an unambiguously nested AndCriteria element hierarchy. More than one partition gets detected when unambiguous OrCriteria instances are applied to the partition criteria. In such cases, the query is addressed to all potential partitions. If it was not possible to identify any partitions, then, as a fallback, all partitions are queried.

Query results are taken from from the invoked partitions (in normal cases this would be a single partition) round robin. the first result is taken from the first queried partition’s result set (Record instances), the next result from the next queried partition and so on to start over again with the first queried partition. Round robin has proven useful to prevent invalidation of physical data sinks’s result sets as of timeouts.

The PartedTrimLogger type extends the PartedQueryLogger type with the functionality of a TrimLogger type. Delete operations with a query such as PartedTrimLogger.deleteLogs(org.refcodes.criteria.Criteria) are applied to the partitions in the same manner as done for PartedTrimLogger.findLogs(org.refcodes.criteria.Criteria).

Contribution guidelines

  • Writing tests
  • Code review
  • Adding functionality
  • Fixing bugs

Who do I talk to?

  • Siegfried Steiner (steiner@refcodes.org)

Terms and conditions

The REFCODES.ORG group of artifacts is published under some open source licenses; covered by the refcodes-licensing (org.refcodes group) artifact - evident in each artifact in question as of the pom.xml dependency included in such artifact.