refcodes-serial: A serial communication toolkit for Java

refcodes-serial: A serial communication toolkit for Java

Siegfried Steiner
Siegfried Steiner
10 min read

Categories

Tags

README

“The REFCODES.ORG codes represent a group of artifacts consolidating parts of my work in the past years. Several topics are covered which I consider useful for you, programmers, developers and software engineers.”

What is this repository for?

This toolkit targets at programming serial communication at a high abstraction level, addressing issues such endianess, length allocations, CRC checksums, acknowledgement management or handshake processing whilst keeping you in full control over your bits and bytes.

How do I get set up?

To get up and running, include the following dependency (without the three dots “…”) in your pom.xml:

1
2
3
4
5
6
7
8
9

<dependencies>
	...
	<dependency>
		<artifactId>refcodes-serial</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.3.5</version>
	</dependency>
	...
</dependencies>

The artifact is hosted directly at Maven Central. Jump straight to the source codes at Bitbucket. Read the artifact’s javadoc at javadoc.io.

Add serial TTY (COM) port support

The refcodes-serial-alt-tty artifact bridges with jSerialComm to directly interact with your serial ports (aka COM ports):

1
2
3
4
5
6
7
8
9

<dependencies>
	...
	<dependency>
		<artifactId>refcodes-serial-alt-tty</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.3.5</version>
	</dependency>
	...
</dependencies>

The artifact is hosted directly at Maven Central. Jump straight to the source codes at Bitbucket. Read the artifact’s javadoc at javadoc.io.

Manage request/response handshake with error correction

Last but not least, the refcodes-serial-ext-handshake artifact provides handshake means in terms of request/response ping-pong alongside with retry and acknowledge management:

1
2
3
4
5
6
7
8
9

<dependencies>
	...
	<dependency>
		<artifactId>refcodes-serial-ext-handshake</artifactId>
		<groupId>org.refcodes</groupId>
		<version>3.3.5</version>
	</dependency>
	...
</dependencies>

The artifact is hosted directly at Maven Central. Jump straight to the source codes at Bitbucket. Read the artifact’s javadoc at javadoc.io.

How do I get started?

Use the static import syntactic sugar to easily harness the refcoces-serial features.

1
2

import static org.refcodes.serial.SerialSugar.*;
// ...

Key players for building data structures fit for serial communication are the types Segment as well as Sequence. A Segment is a high level representation for primitive types or complex types, taking care of endianess, length allocations for strings or arrays or CRC checksum validations and is used to construct arbitrary complex data structures. A Sequence is the low level representation of a Segment, a Segment can be converted to a Sequence forth and back. A Sequence represents the bytes representation of a Segment, with endianess, length allocations or CRC checksums already taken care of as specified by the Segment and it provides some useful methods to manipulate its bytes representation.

Snippets of interest

Below find some code snippets which demonstrate the various aspects of using the refcodes-serial artifact (and , if applicable, its offsprings). See also the example source codes of this artifact for further information on the usage of this artifact.

Our first data structure

A Segment represents a high level data structure capable to be send through a wire (after being converted to a Sequence), containing everything required to pump its bytes representation through a serial port. This means a Segment takes care of the endianess, length allocations for strings or arrays or CRC checksum validations of your data structure. Constructing such a data structure is as easy as this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// ...
BooleanSegment theFlag;
IntSegment theValue;
StringSection theString;

Segment theSegment = crcPrefixSegment(
	segmentComposite(
		theFlag = booleanSegment( true ),
		theValue = intSegment( 5161, Endianess.LITTLE_ENDIAN),
		allocSegment( 
			theString = stringSection("Hello world!"), 2, Endianess.LITTLE_ENDIAN
		)
	), CrcAlgorithmConfig.CRC_16_CCITT_FALSE
);
// ...

The above data structure consists of a boolean value (line 8), an integer value (line 9) as well as a string value (line 11). All values consisting of more than a single byte will be encoded using the little endianess representation (lines 9 and 11). For the length allocation of the string we use two bytes (line 11). The data structure is secured against data decay during transmission using a CRC-16/CCITT-FALSE checksum (line 13). Now we can play a little with this data structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

// ...
// Create the bytes representation:
byte[] theBytes = theSegment.toSequence().toBytes();
System.out.println( NumericalUtility.toHexString( " ", theBytes ) );
System.out.println( theFlag.getPayload() );
System.out.println( theValue.getPayload() );
System.out.println( theString.getPayload() );

// Change the payload and create the bytes representation again:
theFlag.setPayload( false );
byte[] theOtherBytes = theSegment.toSequence().toBytes();
System.out.println( NumericalUtility.toHexString( " ", theOtherBytes ) );
System.out.println( theFlag.getPayload() );
System.out.println( theValue.getPayload() );
System.out.println( theString.getPayload() );

// Restore the data structure from original bytes representation:
theSegment.fromTransmission( theBytes );
System.out.println( theFlag.getPayload() );
System.out.println( theValue.getPayload() );
System.out.println( theString.getPayload() );

// Print our data structure's schema:
System.out.println( theSegment.toSchema() );
// ...

In Line 3 we retrieve the Sequence representation of the Segment from which we get it’s bytes representation. The Sequence we retrieved has already been encoded with the endianess, the length allocations for strings or arrays as well as the calculated CRC checksum as specified by the Segment.

Line 4 prints the final bytes representation of our original data structure:

1
2

0x02 0x58 0x01 0x29 0x14 0x00 0x00 0x0c 0x00 0x48 0x65 0x6c 0x6c 0x6f 0x20 0x77
0x6f 0x72 0x6c 0x64 0x21

In line 10 we change a single boolean value of our payload and line 12 prints out this other bytes representation of our data structure:

1
2

0x1d 0x86 0x00 0x29 0x14 0x00 0x00 0x0c 0x00 0x48 0x65 0x6c 0x6c 0x6f 0x20 0x77
0x6f 0x72 0x6c 0x64 0x21

Comparing the two above hexadecimal representations of our data structure, we see that the first two bytes as well the third byte differ: The third byte is our boolean value, represented as a byte (true = 0x01, false = 0x00) and the first two bytes are the CRC checksum: As we have changed our boolean value from true to false we got a completely different checksum, this way we will detect erroneous transmissions (in most cases).

In line 18 we restore our data structure from the original byte array, having our boolean value set to true again.

Finally in line 24 we print out the SerialSchema of our data structure, useful for debugging and documentation purposes. Here we see the exact construction of our data structure:

The SerialSchema is a JSON alike representation of your expression and helps to further process, analyze, debug or document your expressions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

CrcSegmentDecorator: {
  CRC_ALGORITHM: "CRC-16/CCITT-FALSE",
  CRC_BYTE_WIDTH: 2,
  CRC_CHECKSUM: 22530,
  CRC_CHECKSUM_CONCATENATION_MODE: "PREPEND",
  CRC_CHECKSUM_HEX: { 0x02, 0x58 },
  CRC_CHECKSUM_LITTLE_ENDIAN_BYTES: { 0x02, 0x58 },
  CRC_ENDIANESS: "LITTLE",
  DESCRIPTION: "A segment decorator enriching the encapsulated segment with a CRC checksum.",
  LENGTH: 21,
  TYPE: "org.refcodes.serial.CrcSegmentDecorator",
  VALUE: { 0x02, 0x58, 0x01, 0x29, 0x14, 0x00, 0x00, 0x0c, 0x00, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21 },
  SegmentComposite: {
    DESCRIPTION: "A body containing a composite segment as payload.",
    LENGTH: 19,
    TYPE: "org.refcodes.serial.SegmentComposite",
    VALUE: { 0x01, 0x29, 0x14, 0x00, 0x00, 0x0c, 0x00, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21 },
    BooleanSegment: {
      ALIAS: "booleanSegment",
      DESCRIPTION: "A segment containing an boolean payload.",
      LENGTH: 1,
      TYPE: "org.refcodes.serial.BooleanSegment",
      VALUE: { 0x01 },
      VERBOSE: "true"
    },
    IntSegment: {
      ALIAS: "intSegment",
      DESCRIPTION: "A body containing an integer payload.",
      ENDIANESS: "LITTLE",
      LENGTH: 4,
      TYPE: "org.refcodes.serial.IntSegment",
      VALUE: { 0x29, 0x14, 0x00, 0x00 },
      VERBOSE: "5161"
    },
    AllocSectionDecoratorSegment: {
      ALLOC_LENGTH: 12,
      ALLOC_LENGTH_WIDTH: 2,
      DESCRIPTION: "An allocation decorator referencing a decoratee and prefixing the length of the decoratee in bytes.",
      ENDIANESS: "LITTLE",
      LENGTH: 14,
      TYPE: "org.refcodes.serial.AllocSectionDecoratorSegment",
      VALUE: { 0x0c, 0x00, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21 },
      StringSection: {
        ALIAS: "stringSection",
        DESCRIPTION: "A section containing a string payload.",
        LENGTH: 12,
        TYPE: "org.refcodes.serial.StringSection",
        VALUE: { 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21 },
        VERBOSE: "Hello world!"
      }
    }
  }
}

The SerialSchema gives you full information on the construction of your Segment.

The above SerialSchema tells us that our data structure uses a CRC checksum (line 1) for detecting transmission errors, calculated with the CRC-16/CCITT-FALSE algorithm (line 2), the length of the checksum is 2 bytes (line 3) and the checksum calculated for the data structure is 22530 (line 4). Moving further down in the JSON we see the nested SerialSchema structures such as the the allocated length for our string (line 36) or the number of bytes used to store the allocation width (line 37) of the string.

Using transmission metrics

To ease our life, we can define a TransmissionMetrics object declaring the bytes representation of our data structure without the need to repeat ourself: In the previous example, we have set the CRC checksum algorithm, the endianess and the number of bytes for representing the length of a string or an array (length allocation) individually for each data type in the data structure respectively. Using a TransmissionMetrics object, the previous example would now look as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// ...
TransmissionMetrics theMetrics = TransmissionMetrics.builder().withEndianess( Endianess.LITTLE_ENDIAN ).withLengthWidth( 2 ).withCrcAlgorithm( CrcAlgorithmConfig.CRC_16_CCITT_FALSE ).build();
BooleanSegment theFlag;
IntSegment theValue;
StringSection theString;

Segment theSegment = crcPrefixSegment(
	segmentComposite(
		theFlag = booleanSegment( true ),
		theValue = intSegment( 5161, theMetrics ),
		allocSegment( 
			theString = stringSection("Hello world!"), theMetrics
		)
	), theMetrics
);
// ...

In line 2 we globally defined our CRC checksum algorithm, the endianess and the number of bytes for representing the length of a string (length allocation). This makes live much easier e.g. changing the overall endianess from little to big is just an issue of changing the TransmissionMetrics in line 2.

Using a POJO as data structure

Given a Sensor class which provides data for a single sensor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

public class Sensor {

	public int _value;
	public String _name;

	public Sensor() {}

	public Sensor( String aName, int aValue ) {
		_name = aName;
		_value = aValue;
	}

	public String getName() {
		return _name;
	}

	public int getPayload() {
		return _value;
	}
}

We directly can use this type or even an array of this type to create a bytes representation which we can transmit forth and back (below we actually go for the array):

1
2
3
4
5

// ...
TransmissionMetrics theMetrics = TransmissionMetrics.builder().withEndianess( Endianess.LITTLE_ENDIAN ).withLengthWidth( 2 ).withCrcAlgorithm( CrcAlgorithmConfig.CRC_16_CCITT_FALSE ).build();
Sensor[] theSensors = new Sensor[] { new Sensor( "SensorA", 103343 ), new Sensor( "SensorB", 22109 ), new Sensor( "SensorC", 313773 ) };
ComplexTypeSegment<Sensor[]> theSegment = complexTypeSegment( theSensors, theMetrics );
// ...

Once again in line 2 we globally defined our CRC checksum algorithm, the endianess and the number of bytes for representing the length of a string (length allocation). In line 3 we define an array of Sensor objects which we want to use for serial transmission. Finally a Segment is constructed from the Sensor objects array in line 4.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// ...
byte[] theBytes = theSegment.toSequence().toBytes();
System.out.println( NumericalUtility.toHexString( " ", theBytes ) );

// Restore the byte representation back to the data structure:
ComplexTypeSegment<Sensor[]> theOtherSegment = complexTypeSegment( Sensor[].class, theMetrics );
theOtherSegment.fromTransmission( theBytes );
Sensor[] theOtherSensors = theOtherSegment.getPayload();

System.out.println( theOtherSensors[0].getName() + "=" + theOtherSensors[0].getPayload() );
System.out.println( theOtherSensors[1].getName() + "=" + theOtherSensors[1].getPayload() );
System.out.println( theOtherSensors[2].getName() + "=" + theOtherSensors[2].getPayload() );

// Print our data structure's schema:
System.out.println( theSegment.toSchema() );
// ...

Having constructed the Segment in the previous code snippet, we now retrieve the bytes representation of this Sensor objects array from that very Segment in line 2. An empty Segment for restoring the given bytes back to an identical Sensor objects array is instantiated in line 6. Eventually the original bytes are fed into this other Segment and an identical Sensor objects array is retrieved in line 8.

This example also works perfectly fine with the Record type first seen in Java 16.

Under the hood

As you might already have noticed, a unified representation of the serialized bytes of our data structures, being the Sequence type, is used to efficiently process and manipulate the byte representations of our data structures.

Examples

See the funcodes-playload P2P (Peer-to-Peer) command line application which makes use of all of the refcodes-serial, refcodes-serial-alt-tty as well as refcodes-serial-ext-handshake artifacts - please also refer to the PLAYLOAD manpage or the downloads section of this site. For usage information, please take a look at the according examples here. For usage information on the alternate serial port (aka COM port) bridging implementation (using jSerialComm), please take a look at the according examples here. For usage information on the full duplex (request/response) handshake extensions, please take a look at the according examples here.

Contribution guidelines

  • Report issues
  • Finding bugs
  • Helping fixing bugs
  • Making code and documentation better
  • Enhance the code

Who do I talk to?

  • Siegfried Steiner (steiner@refcodes.org)

Terms and conditions

The REFCODES.ORG group of artifacts is published under some open source licenses; covered by the refcodes-licensing (org.refcodes group) artifact - evident in each artifact in question as of the pom.xml dependency included in such artifact.

LinkedIn Reddit