Chronicle

Overview

Chronicle Software is about simplifying fast data. It is a suite of libraries to make it easier to write, monitor and tune data processing systems where performance and scalability are concerned.

Writing to a Queue

In Chronicle Queue we refer to the act of writing your data to the Chronicle queue, as storing an excerpt. This data could be made up from any data type, including text, numbers, or serialised blobs. Ultimately, all your data, regardless of what it is, is stored as a series of bytes.

Just before storing your excerpt, Chronicle Queue reserves an 8-byte header. Chronicle Queue writes the length of your data into this header. This way, when Chronicle Queue comes to read your excerpt, it knows how long each blob of data is. We refer to this 8-byte header, along with your excerpt, as a document. So strictly speaking Chronicle Queue can be used to read and write documents.

Within this 8-byte header we also reserve a few bits for a number of internal operations, such as locking, to make Chronicle Queue thread-safe across both processors and threads. The important thing to note is that because of this, you can’t strictly convert the 8 bytes to an integer to find the length of your data blob.

To write data to a Chronicle-Queue, you must first create an Appender

try (ChronicleQueue queue = SingleChronicleQueueBuilder.binary(path + "/trades").build()) {
   final ExcerptAppender appender = queue.acquireAppender();
}

Chronicle Queue uses the following low-level interface to write the data:

try (final DocumentContext dc = appender.writingDocument()) {
      dc.wire().write().text(“your text data“);
}

So, Chronicle Queue uses an Appender to write to the queue and a Tailer to read from the queue. Unlike other java queuing solutions, messages are not lost when they are read with a Tailer.

Each Chronicle Queue excerpt has a unique index.

try (final DocumentContext dc = appender.writingDocument()) {
    dc.wire().write().text(“your text data“);
    System.out.println("your data was store to index="+ dc.index());
}

The high-level methods below such as writeText() are convenience methods on calling appender.writingDocument(), but both approaches essentially do the same thing. The actual code of writeText(CharSequence text) looks like this:

/**
 * @param text to write a message
 */
void writeText(CharSequence text) {
    try (DocumentContext dc = writingDocument()) {
        dc.wire().bytes().append8bit(text);
    }
}

This is the highest-level API which hides the fact you are writing to messaging at all. The benefit is that you can swap calls to the interface with a real component, or an interface to a different protocol.

// using the method writer interface.
RiskMonitor riskMonitor = appender.methodWriter(RiskMonitor.class);
final LocalDateTime now = LocalDateTime.now(Clock.systemUTC());
riskMonitor.trade(new TradeDetails(now, "GBPUSD", 1.3095, 10e6, Side.Buy, "peter"));

You can write a “self-describing message”. Such messages can support schema changes. They are also easier to understand when debugging or diagnosing problems.

// writing a self describing message
appender.writeDocument(w -> w.write("trade").marshallable(
        m -> m.write("timestamp").dateTime(now)
                .write("symbol").text("EURUSD")
                .write("price").float64(1.1101)
                .write("quantity").float64(15e6)
                .write("side").object(Side.class, Side.Sell)
                .write("trader").text("peter")));

You can write “raw data” which is self-describing. The types will always be correct; position is the only indication as to the meaning of those values.

// writing just data
appender.writeDocument(w -> w
        .getValueOut().int32(0x123456)
        .getValueOut().int64(0x999000999000L)
        .getValueOut().text("Hello World"));

You can write “raw data” which is not self-describing. Your reader must know what this data means, and the types that were used.

// writing raw data
appender.writeBytes(b -> b
        .writeByte((byte) 0x12)
        .writeInt(0x345678)
        .writeLong(0x999000999000L)
        .writeUtf8("Hello World"));

This is the lowest level way to write data. You get an address to raw memory and you can write what you want.

// Unsafe low level
appender.writeBytes(b -> {
    long address = b.address(b.writePosition());
    Unsafe unsafe = UnsafeMemory.UNSAFE;
    unsafe.putByte(address, (byte) 0x12);
    address += 1;
    unsafe.putInt(address, 0x345678);
    address += 4;
    unsafe.putLong(address, 0x999000999000L);
    address += 8;
    byte[] bytes = "Hello World".getBytes(StandardCharsets.ISO_8859_1);
    unsafe.copyMemory(bytes, Unsafe.ARRAY_BYTE_BASE_OFFSET, null, address, bytes.length);
    b.writeSkip(1 + 4 + 8 + bytes.length);
});

You can print the contents of the queue. You can see the first two, and last two messages store the same data.

// dump the content of the queue System.out.println(queue.dump());

position: 262568, header: 0

— !!data #binary trade: { timestamp: 2016-07-17T15:18:41.141, symbol: GBPUSD, price: 1.3095, quantity: 10000000.0, side: Buy, trader: peter }

position: 262684, header: 1

— !!data #binary trade: { timestamp: 2016-07-17T15:18:41.141, symbol: EURUSD, price: 1.1101, quantity: 15000000.0, side: Sell, trader: peter }

position: 262800, header: 2

— !!data #binary !int 1193046 168843764404224 Hello World

position: 262830, header: 3

— !!data #binary 000402b0 12 78 56 34 00 00 90 99 00 90 99 00 00 0B ·xV4·· ········ 000402c0 48 65 6C 6C 6F 20 57 6F 72 6C 64 Hello Wo rld

position: 262859, header: 4

— !!data #binary 000402c0 12 · 000402d0 78 56 34 00 00 90 99 00 90 99 00 00 0B 48 65 6C xV4····· ·····Hel 000402e0 6C 6F 20 57 6F 72 6C 64

Finding the index at the end of a Chronicle Queue

Chronicle Queue appenders are thread-local. In fact when you ask for:

final ExcerptAppender appender = queue.acquireAppender();

the acquireAppender() uses a thread-local pool to give you an appender which will be reused to reduce object creation.

As such, the method call to:

long index = appender.lastIndexAppended();

will only give you the last index appended by this appender; not the last index appended by any appender.

If you wish to find the index of the last record written, then you have to call:

long index = queue.createTailer().toEnd().index();

Dumping a Chronicle Queue, cq4 file as text to the Command Line

Chronicle Queue stores its data in binary format, with a file extension of cq4:

\��@π�header∂�SCQStoreÇE���»wireType∂�WireTypeÊBINARYÕwritePositionèèèèß��������ƒroll∂�SCQSRollÇ*���∆length¶ÄÓ6�∆format ÎyyyyMMdd-HH≈epoch¶ÄÓ6�»indexing∂ SCQSIndexingÇN��� indexCount•��ÃindexSpacing�Àindex2Indexé����ß��������…lastIndexé� ���ß��������fllastAcknowledgedIndexReplicatedé������ߡˇˇˇˇˇˇˇ»recovery∂�TimedStoreRecoveryÇ����…timeStampèèèß���������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������� This can often be a bit difficult to read, so it is better to dump the cq4 files as text. This can also help you fix your production issues, as it gives you the visibility as to what has been stored in the queue, and in what order.

You have to use the chronicle-queue.jar, from any version 4.5.3 or later, and set up the dependent files in the class path.

$ java -cp chronicle-queue-4.5.5.jar net.openhft.chronicle.queue.DumpQueueMain 19700101-02.cq4

this will dump the 19700101-02.cq4 file out as text, as shown below:

— !!meta-data #binary header: !SCQStore { wireType: !WireType BINARY, writePosition: 0, roll: !SCQSRoll { length: !int 3600000, format: yyyyMMdd-HH, epoch: !int 3600000 }, indexing: !SCQSIndexing { indexCount: !short 4096, indexSpacing: 4, index2Index: 0, lastIndex: 0 }, lastAcknowledgedIndexReplicated: -1, recovery: !TimedStoreRecovery { timeStamp: 0 } }

… 4198044 bytes remaining

Reading from a Queue using a Tailer

Reading the queue follows the same pattern as writing, except there is a possibility there is not a message when you attempt to read it.

Start Reading

try (ChronicleQueue queue = SingleChronicleQueueBuilder.binary(path + "/trades").build()) {
   final ExcerptTailer tailer = queue.createTailer();
}

You can turn each message into a method call based on the content of the message.

// reading using method calls
RiskMonitor monitor = System.out::println;
MethodReader reader = tailer.methodReader(monitor);
// read one message
assertTrue(reader.readOne());

You can decode the message yourself.

assertTrue(tailer.readDocument(w -> w.read("trade").marshallable(
        m -> {
            LocalDateTime timestamp = m.read("timestamp").dateTime();
            String symbol = m.read("symbol").text();
            double price = m.read("price").float64();
            double quantity = m.read("quantity").float64();
            Side side = m.read("side").object(Side.class);
            String trader = m.read("trader").text();
            // do something with values.
        })));

You can read self-describing data values. This will check the types are correct, and convert as required.


assertTrue(tailer.readDocument(w -> {
    ValueIn in = w.getValueIn();
    int num = in.int32();
    long num2 = in.int64();
    String text = in.text();
    // do something with values
}));

You can read raw data as primitives and strings.

assertTrue(tailer.readBytes(in -> {
    int code = in.readByte();
    int num = in.readInt();
    long num2 = in.readLong();
    String text = in.readUtf8();
    assertEquals("Hello World", text);
    // do something with values
}));

or, you can get the underlying memory address and access the native memory.

assertTrue(tailer.readBytes(b -> {
    long address = b.address(b.readPosition());
    Unsafe unsafe = UnsafeMemory.UNSAFE;
    int code = unsafe.getByte(address);
    address++;
    int num = unsafe.getInt(address);
    address += 4;
    long num2 = unsafe.getLong(address);
    address += 8;
    int length = unsafe.getByte(address);
    address++;
    byte[] bytes = new byte[length];
    unsafe.copyMemory(null, address, bytes, Unsafe.ARRAY_BYTE_BASE_OFFSET, bytes.length);
    String text = new String(bytes, StandardCharsets.UTF_8);
    assertEquals("Hello World", text);
    // do something with values
}));

Tailers and File Handlers Clean up

Chronicle queue tailers may create file handlers, the file handlers are cleaned up whenever the associated chronicle queue is close() or whenever the Jvm runs a Garbage Collection.

ExcerptTailer.toEnd()

In some applications, it may be necessary to start reading from the end of the queue (e.g. in a restart scenario). For this use-case, ExcerptTailer provides the toEnd() method.

If it is necessary to read backwards through the queue from the end, then the tailer can be set to read backwards:

ExcerptTailer tailer = queue.createTailer();
tailer.direction(TailerDirection.BACKWARD).toEnd();

When reading backwards, then the toEnd() method will move the tailer to the last record in the queue. If the queue is not empty, then there will be a DocumentContext available for reading:

// this will be true if there is at least one message in the queue
boolean messageAvailable = tailer.toEnd().direction(TailerDirection.BACKWARD).
        readingDocument().isPresent();

Low GC

Ultra low GC means less than one minor collection per day.

the principles of Zero-copy eliminating unnecessary garbage collection and increased speed. Runtime code generation that reduces code size for efficient CPU cache usage and increased speed. Smart ordering for optimal parsing and you guessed it increased speed. All these combine to allow Chronicle FIX to achieve excellent performance results.

Low garbage rate

Minimising garbage is key to avoiding GC pauses. To use your L1 and L2 cache efficiently, you need to keep your garbage rates very low. If you are not using these cache efficiently your application can be 2-5x slower.

The garbage from Chronicle is low enough that you can process one million events without jstat detecting you have created any garbage. jstat only displays multiples of 4 KB, and only when a new TLAB is allocated. Chronicle does create garbage, but it is extremely low. i.e. a few objects per million events processes.

Once you make the GC pauses manageable, or non-existent, you start to see other sources of delay in your system. Take away the boulders and you start to see the rocks. Take away the rocks and you start to see the pebbles.

Chronicle has minimal interaction with the Operating System.

System calls are slow, and if you can avoid call the OS, you can save significant amounts of latency.

For example, if you send a message over TCP on loopback, this can add a 10 micro-seconds latency between writing and reading the data. You can write to a chronicle, which is a plain write to memory, and read from chronicle, which is also a read from memory with a latency of 0.2 micro-seconds. (And as I mentioned before, you get persistence as well)

No need to worry about running out of heap.

A common problem with unbounded queues and this uses an open ended amount of heap.

Chronicle solves this by not using the heap to store data, but instead using memory mapped files. This improve memory utilisation by making the data more compact but also means a 1 GB JVM can stream 1 TB of data over a day without worrying about the heap or how much main memory you have. In this case, an unbounded queue becomes easier to manage.

how it works

Chronicle uses a memory mapped file to continuously journal messages, chronicles file-based storage will slowly grow in size as more data is written to the queue, the size of the queue can exceed your available memory, you are only constrained by the amount of disk space you have on your server. Chronicle writes data directly into off-heap memory which is shared between java processes on the same server.

Chronicle is very fast, it is able to write and read a message in just two microseconds with no garbage. Typically at the end of each day, you archive the queue and start the next day with a fresh empty queue.

Chronicle Queue is a distributed unbounded persisted queue.

Chronicle Queue:

supports asynchronous RMI and Publish/Subscribe interfaces with microsecond latencies.

passes messages between JVMs in under a microsecond (in optimised examples)

passes messages between JVMs on different machines via replication in under 10 microseconds (in optimised examples)

provides stable, soft, real time latencies into the millions of messages per second for a single thread to one queue; with total ordering of every event.

Queue introduction

Chronicle Queue is a Java project focused on building a persisted low-latency messaging framework for high performance and critical applications.

Chronicle diagram 005 At first glance Chronicle Queue can be seen as simply another queue implementation. However, it has major design choices that should be emphasised.

Using non-heap storage options (RandomAccessFile), Chronicle Queue provides a processing environment where applications do not suffer from Garbage Collection (GC). When implementing high-performance and memory-intensive applications (you heard the fancy term “bigdata”?) in Java, one of the biggest problems is garbage collection.

Garbage collection may slow down your critical operations non-deterministically at any time. In order to avoid non-determinism, and escape from garbage collection delays, off-heap memory solutions are ideal. The main idea is to manage your memory manually so it does not suffer from garbage collection. Chronicle Queue behaves like a management interface over off-heap memory so you can build your own solutions over it.

Chronicle Queue uses RandomAccessFiles while managing memory and this choice brings lots of possibilities. RandomAccessFiles permit non-sequential, or random, access to a file’s contents. To access a file randomly, you open the file, seek a particular location, and read from or write to that file. RandomAccessFiles can be seen as “large” C-type byte arrays that you can access at any random index “directly” using pointers. File portions can be used as ByteBuffers if the portion is mapped into memory.

This memory mapped file is also used for exceptionally fast interprocess communication (IPC) without affecting your system performance. There is no garbage collection as everything is done off-heap.

Message type

  • TCP: Stream-oriented
  • UDP, SCTP: message-oriented .

On heap vs off heap memory usage

Overview

I was recently asked about the benefits and wisdom of using off heap memory in Java. The answers may be of interest to others facing the same choices.

Off heap memory is nothing special. The thread stacks, application code, NIO buffers are all off heap. In fact in C and C++, you only have unmanaged memory as it does not have a managed heap by default. The use of managed memory or “heap” in Java is a special feature of the language. Note: Java is not the only language to do this. new Object() vs Object pool vs Off Heap memory.

new Object()

Before Java 5.0, using object pools was very popular. Creating objects was still very expensive. However, from Java 5.0, object allocation and garbage cleanup was made much cheaper, and developers found they got a performance speed up and a simplification of their code by removing object pools and just creating new objects whenever needed. Before Java 5.0, almost any object pool, even an object pool which used objects provided an improvement, from Java 5.0 pooling only expensive objects obviously made sense e.g. threads, socket and database connections.

Object pools

In the low latency space it was still apparent that recycling mutable objects improved performance by reduced pressure on your CPU caches. These objects have to have simple life cycles and have a simple structure, but you could see significant improvements in performance and jitter by using them. Another area where it made sense to use object pools is when loading large amounts of data with many duplicate objects. With a significant reduction in memory usage and a reduction in the number of objects the GC had to manage, you saw a reduction in GC times and an increase in throughput. These object pools were designed to be more light weight than say using a synchronized HashMap, and so they still helped.

Take this StringInterner class as an example. You pass it a recycled mutable StringBuilder of the text you want as a String and it will provide a String which matches. Passing a String would be inefficient as you would have already created the object. The StringBuilder can be recycled. Note: this structure has an interesting property that requires no additional thread safety features, like volatile or synchronized, other than is provided by the minimum Java guarantees. i.e. you can see the final fields in a String correctly and only read consistent references.

public class StringInterner { private final String[] interner; private final int mask; public StringInterner(int capacity) { int n = Maths.nextPower2(capacity, 128); interner = new String[n]; mask = n - 1; }

private static boolean isEqual(@Nullable CharSequence s, @NotNull CharSequence cs) {
    if (s == null) return false;
    if (s.length() != cs.length()) return false;
    for (int i = 0; i < cs.length(); i++)
        if (s.charAt(i) != cs.charAt(i))
            return false;
    return true;
}

@NotNull
public String intern(@NotNull CharSequence cs) {
    long hash = 0;
    for (int i = 0; i < cs.length(); i++)
        hash = 57 * hash + cs.charAt(i);
    int h = (int) Maths.hash(hash) & mask;
    String s = interner[h];
    if (isEqual(s, cs))
        return s;
    String s2 = cs.toString();
    return interner[h] = s2;
} } Off heap memory usage

Using off heap memory and using object pools both help reduce GC pauses, this is their only similarity. Object pools are good for short lived mutable objects, expensive to create objects and long live immutable objects where there is a lot of duplication. Medium lived mutable objects, or complex objects are more likely to be better left to the GC to handle. However, medium to long lived mutable objects suffer in a number of ways which off heap memory solves.

Off heap memory provides;

Scalability to large memory sizes e.g. over 1 TB and larger than main memory. Notional impact on GC pause times. Sharing between processes, reducing duplication between JVMs, and making it easier to split JVMs. Persistence for faster restarts or replying of production data in test. The use of off heap memory gives you more options in terms of how you design your system. The most important improvement is not performance, but determinism.

Off heap and testing

One of the biggest challenges in high performance computing is reproducing obscure bugs and being able to prove you have fixed them. By storing all your input events and data off heap in a persisted way you can turn your critical systems into a series of complex state machines. (Or in simple cases, just one state machine) In this way you get reproducible behaviour and performance between test and production.

A number of investment banks use this technique to replay a system reliably to any event in the day and work out exactly why that event was processed the way it was. More importantly, once you have a fix you can show that you have fixed the issue which occurred in production, instead of finding an issue and hoping this was the issue.

Along with deterministic behaviour comes deterministic performance. In test environments, you can replay the events with realistic timings and show the latency distribution you expect to get in production. Some system jitter can’t be reproduce esp if the hardware is not the same, but you can get pretty close when you take a statistical view. To avoid taking a day to replay a day of data you can add a threshold. e.g. if the time between events is more than 10 ms you might only wait 10 ms. This can allow you to replay a day of events with realistic timing in under an hour and see whether your changes have improved your latency distribution or not.

By going more low level don’t you lose some of “compile once, run anywhere”?

To some degree this is true, but it is far less than you might think. When you are working closer the processor and so you are more dependant on how the processor, or OS behaves. Fortunately, most systems use AMD/Intel processors and even ARM processors are becoming more compatible in terms of the low level guarantees they provide. There is also differences in the OSes, and these techniques tend to work better on Linux than Windows. However, if you develop on MacOSX or Windows and use Linux for production, you shouldn’t have any issues. This is what we do at Higher Frequency Trading.

What new problems are we creating by using off heap?

Nothing comes for free, and this is the case with off heap. The biggest issue with off heap is your data structures become less natural. You either need a simple data structure which can be mapped directly to off heap, or you have a complex data structure which serializes and deserializes to put it off heap. Obvious using serialization has its own headaches and performance hit. Using serialization thus much slower than on heap objects.

In the financial world, most high ticking data structure are flat and simple, full of primitives which maps nicely off heap with little overhead.

How does Chronicle Queue work

Terminology

  • Messages are grouped by topics. A topic can contain any number of sub-topics which are logically stored together under the queue/topic.
  • An appender is the source of messages.
  • A tailer is a receiver of messages.
  • Chronicle Queue is broker-less by default. You can use Chronicle Engine to act as a broker for remote access.

Note We deliberately avoid the term consumer as messages are not consumed/destroyed by reading.

At a high level:

  • appenders write to the end of a queue. There is no way to insert, or delete excerpts.

  • tailers read the next available message each time they are called.

By using Chronicle Engine, a Java or C# client can publish to a queue to act as a remote appender, and you subscribe to a queue to act as a remote tailer.

Topics and Queue files

Each topic is a directory of queues. There is a file for each roll cycle. If you have a topic called mytopic, the layout could look like this:

mytopic/ 20160710.cq4 20160711.cq4 20160712.cq4 20160713.cq4 To copy all the data for a single day (or cycle), you can copy the file for that day on to your development machine for replay testing.

Appenders and tailers are cheap as they don’t even require a TCP connection; they are just a few Java objects.

File Retention

You can add a StoreFileListener to notify you when a file is added, or no longer used. This can be used to delete files after a period of time. However, by default, files are retained forever. Our largest users have over 100 TB of data stored in queues.

Every Tailer sees every message.

An abstraction can be added to filter messages, or assign messages to just one message processor. However, in general you only need one main tailer for a topic, with possibly, some supporting tailers for monitoring etc.

As Chronicle Queue doesn’t partition its topics, you get total ordering of all messages within that topic. Across topics, there is no guarantee of ordering; if you want to replay deterministically from a system which consumes from multiple topics, we suggest replaying from that system’s output.

Guarantees

Chronicle Queue provides the following guarantees;

for each appender, messages are written in the order the appender wrote them. Messages by different appenders are interleaved,

for each tailer, it will see every message for a topic in the same order as every other tailer,

when replicated, every replica has a copy of every message.

Use Cases

Chronicle Queue is most often used for producer-centric systems where you need to retain a lot of data for days or years.

What is a producer-centric system?

Most messaging systems are consumer-centric. Flow control is implemented to avoid the consumer ever getting overloaded; even momentarily. A common example is a server supporting multiple GUI users. Those users might be on different machines (OS and hardware), different qualities of network (latency and bandwidth), doing a variety of other things at different times. For this reason it makes sense for the client consumer to tell the producer when to back off, delaying any data until the consumer is ready to take more data.

Chronicle Queue is a producer-centric solution and does everything possible to never push back on the producer, or tell it to slow down. This makes it a powerful tool, providing a big buffer between your system, and an upstream producer over which you have little, or no, control.

For market data in particular, real time means in a few microseconds; it doesn’t mean intra-day (during the day).

Chronicle Queue is fast and efficient, and has been used to increase the speed that data is passed between threads. In addition, it also keeps a record of every message passed allowing you to significantly reduce the amount of logging that you need to do.

Latency Sensitive Micro-services

Chronicle Queue supports low latency IPC (Inter Process Communication) between JVMs on the same machine in the order of magnitude of 1 microsecond; as well as between machines with a typical latency of 10 microseconds for modest throughputs of a few hundred thousands. Chronicle Queue supports throughputs of millions of events per second, with stable microsecond latencies.

Log Replacement

As Chronicle Queue can be used to build state machines. All the information about the state of those components can be reproduced externally, without direct access to the components, or to their state. This significantly reduces the need for additional logging.

However, any logging you do need can be recorded in great detail. This makes enabling DEBUG logging in production practical. This is because the cost of logging is very low; less than 10 microseconds. Logs can be replicated centrally for log consolidation.

Chronicle Queue is being used to store 100+ TB of data, which can be replayed from any point in time.

Source code

MappedFile

package net.openhft.chronicle.bytes;

import net.openhft.chronicle.core.Jvm;
import net.openhft.chronicle.core.OS;
import net.openhft.chronicle.core.ReferenceCounted;
import net.openhft.chronicle.core.ReferenceCounter;
import net.openhft.chronicle.core.io.IORuntimeException;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.lang.ref.WeakReference;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;
import java.nio.channels.FileLock;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;

import static net.openhft.chronicle.core.io.Closeable.closeQuietly;

/**
 * A memory mapped files which can be randomly accessed in chunks. It has overlapping regions to
 * avoid wasting bytes at the end of chunks.
 */
public class MappedFile implements ReferenceCounted {
    private static final long DEFAULT_CAPACITY = 128L << 40;
    // A single JVM cannot lock a file more than once.
    private static final Object GLOBAL_FILE_LOCK = new Object();
    @NotNull
    private final RandomAccessFile raf;
    private final FileChannel fileChannel;

public interface BytesStore extends RandomDataInput, RandomDataOutput, ReferencedCount, CharSequence

public interface Memory { default long heapUsed() { Runtime runtime = Runtime.getRuntime(); return runtime.totalMemory() - runtime.freeMemory(); }

@Override
@ForceInline
public void writeByte(long address, byte b) {
    UNSAFE.putByte(address, b);
}

/**

  • Marker annotation for some methods and constructors in the JSR 292 implementation.

  • To utilise this annotation se Chronicle Enterprise Warmup module. */ @Target({ElementType.METHOD, ElementType.CONSTRUCTOR}) @Retention(RetentionPolicy.RUNTIME) public @interface ForceInline { }

Reference

  • https://github.com/OpenHFT/Chronicle-Queue
  • http://vanillajava.blogspot.com/2015/08/what-does-chronicle-software-do.html

2024

3 minute read

awesome mr W

2 minute read

You are not a drop in the ocean, you are the entire ocean in a drop.

Flyway Self Healing

less than 1 minute read

how to let your flyway database scheme migrate more robustly and self healing

Flyway Self Healing

less than 1 minute read

how to let your flyway database scheme migrate more robustly and self healing

Back to top ↑

2023

Az Cli

less than 1 minute read

To run commands in VMs in Azure

Cheap and flexible computing

1 minute read

whether it seems possible or not - go for it Cheaper X 2 to EC2, to use Fargate Spot With Fargate Spot you can run interruption tolerant Amazon ECS t...

Understanding React export a Component

1 minute read

A young idler, an old beggar. - William Shakespeare Understanding React export a Component In this blog post, we will dive into the code of the RepoU...

UUID deep dive

4 minute read

A young idler, an old beggar. - William Shakespeare

which-port-my-service-is-running

2 minute read

Summary As a Java developer, it’s important to know how to find out which port number your Spring service is running on. This information is useful when you ...

Elk Search Tips

less than 1 minute read

message:/'Invoking SP with quoteContext*werqewr-1234asdf-sdf23-9d83-asdf23*'/

what is StrictHostKeyChecking in ssh

less than 1 minute read

What’s and how to avoid error of the authenticity of host ‘xxx’ can’t be established You can suppress the “The authenticity of host ‘’ can’t be established” ...

Chinese Verb

less than 1 minute read

知其雄，守其雌 什么意思

What’s Difference Of Cny And Cnh

3 minute read

what’s difference of CNY and CNH CNY and CNH are both currencies used in China, but they are different in a few important ways:

Spring Cloud Master Piece 9

7 minute read

What’s spring cloud config Spring Cloud Config is a distributed configuration server that provides a centralized location to manage external properties for a...

Spring Cloud Master Piece 6

1 minute read

Sample me build a micro service payment system with spring cloud Here’s an example of building a microservice payment system using Spring Cloud:

Spring Cloud Master Piece 2

5 minute read

what’s usage of bootstrap yml In a Spring Boot application, the bootstrap.yml (or bootstrap.properties) file is used for configuring the application’s enviro...

Spring Cloud Master Piece 1

7 minute read

what’s API gateway An API Gateway is a key component in microservices architecture that acts as a single entry point for client requests to a microservices-b...

Master Microfrontends

4 minute read

“The only way to do great work is to love what you do.” - Steve Jobs

How To Config JFR Java Flight Control

2 minute read

“Climb the mountains and get their good tidings. Nature’s peace will flow into you as sunshine flows into trees. The winds will blow their own freshness i...

Google マップ内の写真のコメントが表示されない

less than 1 minute read

紹介 私は、私のOppo Androidスマートフォンのアプリ「Googleマップ」で奇妙な問題が発生していることに気づきました。Googleマップで特定の場所（例えば「中央公園」）を検索すると、通常、このアプリは公園の写真やコメントリストを表示するはずです。例えば、誰かが公園の芝生や川の写真を投稿し、便利な場所...

master-cglib-in-java

2 minute read

You are not a drop in the ocean, you are the entire ocean in a drop.

Back to top ↑

2022

Minium Workable Mvp Vimrc

6 minute read

”—————————————————————- “ 4. User interface “—————————————————————- “ Set X lines to the cursor when moving vertically set scrolloff=0

Linux Tips

5 minute read

Remember, some things have to end for better things to begin.

Back to top ↑

2021

How to user fire extinguisher

less than 1 minute read

Summary As you know, staff and your safety is paramount. So what if emergency take place, such as fire in office, how to help yourself and your colleagues by...

Deep dive into Kubernetes Client API

1 minute read

Summary To talk to K8s for getting data, there are few approaches. While K8s’ official Java library is the most widely used one. This blog will look into thi...

Whitelabel Error Page

less than 1 minute read

Summary Whitelabel Error Page is the default error page in Spring Boot web app. It provide a more user-friently error page whenever there are any issues when...

Debts in a nutshell

less than 1 minute read

A debt security represents a debt owed by the issuer to an investor. Here, the investor acts as a lender to the issuer which may be a government, organisatio...

Back to top ↑

2020

How to process data from S3 download URL

less than 1 minute read

S3 download URL As you know, AWS S3 object can be downloaded/processed by S3 download URL. I’m showing you two examples on how to process S3 Object by NIO f...

Debug Stuck IntelliJ

1 minute read

What happened to a debug job hanging in IntelliJ (IDEAS) IDE? You may find when you try to debug a class in Intellij but it stuck there and never proceed, e....

Awesome Kotlin

less than 1 minute read

Difference with Scala Kotlin takes the best of Java and Scala, the response times are similar as working with Java natively, which is a considerable advantag...

JVM热身

1 minute read

此文是作者英文原文的翻译文章，英文原文在：http://todzhang.com/posts/2018-06-10-jvm-warm-up/

Mock in kotlin

2 minute read

Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...

Mock in kotlin

2 minute read

Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...

Docker

1 minute read

Dockers Concepts

Curl

less than 1 minute read

Linux Curl command

AOP

less than 1 minute read

The concept of join points as matched by pointcut expressions is central to AOP, and Spring uses the AspectJ pointcut expression language by default.

Micrometer notes

less than 1 minute read

As a general rule it should be possible to use the name as a pivot. Dimensions allow a particular named metric to be sliced to drill down and reason about th...

Awesome SSL certificates and HTTPS

16 minute read

What’s TLS TLS (Transport Layer Security) and its predecessor, SSL (Secure Sockets Layer), are security protocols designed to secure the communication betwee...

JVM warm up by Escape Analysis

13 minute read

Why JVM need warm up I don’t know how and why you get to this blog. But I know the key words in your mind are “warm” for JVM. As the name “warm up” suggested...

Java Concurrent

54 minute read

This blog is about noteworthy pivot points about Java Concurrent Framework Back to Java old days there were wait()/notify() which is error prone, while fr...

Back to top ↑

2019

Conversations with God

less than 1 minute read

Feelings is the language of the soul. If you want to know what’s true for you about something, look to how your’re feeling about.

Kafka In Spring

4 minute read

Enable Kafka listener annotated endpoints that are created under the covers by a AbstractListenerContainerFactory. To be used on Configuration classes as fol...

Mifid

1 minute read

FX Spot is not covered by the regulation, as it is not considered to be a financial instrument by ESMA, the European Union (EU) regulator. As FX is considere...

Foreign Exchange

less than 1 minute read

currency pairs Direct ccy: means USD is part of currency pair Cross ccy: means ccy wihtout USD, so except NDF, the deal will be split to legs, both with...

Back to top ↑

2018

Seconds

less than 1 minute read

nano seconds

Citrix receiver

less than 1 minute read

“Cannot connect to remote desktop” with Citrix Receiver

Guice

7 minute read

A new type of Juice Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice’s @Inject as the new new. You wi...

YAML

less than 1 minute read

Key points All YAML files (regardless of their association with Ansible or not) can optionally begin with — and end with …. This is part of the YAML format a...

Mockito

less than 1 minute read

Feature

Protobuf

5 minute read

What are protocol buffers?

Sudo in a Nutshell

6 minute read

Sudo in a Nutshell Sudo (su “do”) allows a system administrator to give certain users (or groups of users) the ability to run some (or all) commands as root...

Zoo-keeper

15 minute read

ZK Motto the motto “ZooKeeper: Because Coordinating Distributed Systems is a Zoo.”

Cucumber

28 minute read

Acceptance testing vs unit test It’s sometimes said that unit tests ensure you build the thing right, whereas acceptance tests ensure you build the right thi...

Scala

less than 1 minute read

Scala String

akka framework of scala

27 minute read

philosophy The actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-o...

Apache Camel

8 minute read

Camel’s message model In Camel, there are two abstractions for modeling messages, both of which we’ll cover in this section. org.apache.camel.Message—The ...

JXM

less than 1 minute read

Exporting your beans to JMX The core class in Spring’s JMX framework is the MBeanExporter. This class is responsible for taking your Spring beans and registe...

Solace MQ

1 minute read

Solace PubSub+ It is a message broker that lets you establish event-driven interactions between applications and microservices across hybrid cloud environmen...

Apigee

4 minute read

App deployment, configuration management and orchestration - all from one system. Ansible is powerful IT automation that you can learn quickly.

Ansible

10 minute read

Ansible: What Is It Good For? Ansible is often described as a configuration management tool, and is typically mentioned in the same breath as Chef, Puppet, a...

flexbox

4 minute read

How Flexbox works — explained with big, colorful, animated gifs

KDB

18 minute read

KDB However kdb+ evaluates expressions right-to-left. There are no precedence rules. The reason commonly given for this behaviour is that it is a much simple...

Portactor

1 minute read

Better to use smart wait

Agile and SCRUM

less than 1 minute read

Key concept In Scrum, a team is cross functional, meaning everyone is needed to take a feature from idea to implementation.

rxjs pipe in depth

6 minute read

https://stormforger.com/blog/2016/07/08/types-of-performance-testing/

Strategy-Of-Openshift-Releases

less than 1 minute read

Release & Testing Strategy There are various methods for safely releasing changes to Production. Each team must select what is appropriate for their own ...

NodeJs Notes

less than 1 minute read

commands to read files var lineReader = require(‘readline’).createInterface({ input: require(‘fs’).createReadStream(‘C:\dev\node\input\git_reset_files.tx...

Minium Viable Product

less than 1 minute read

https://blog.leanstack.com/minimum-viable-product-mvp-7e280b0b9418

CORS :Cross-Origin Resource Sharing

1 minute read

Cross-Origin Request Sharing - CORS (A.K.A. Cross-Domain AJAX request) is an issue that most web developers might encounter, according to Same-Origin-Policy,...

ngrx

2 minute read

Why @Effects? In a simple ngrx/store project without ngrx/effects there is really no good place to put your async calls. Suppose a user clicks on a button or...

iOS programming

3 minute read

View A view is also a responder (UIView is a subclass of UIResponder). This means that a view is subject to user interactions, such as taps and swipes. Thus,...

Back to top ↑

2017

cloud computering

less than 1 minute read

openshift vs openstack The shoft and direct answer is `OpenShift Origin can run on top of OpenStack. They are complementary projects that work well together....

cloud computering

3 minute read

Concepts Cloud computing is the on-demand demand delivery of compute database storage applications and other IT resources through a cloud services platform v...

Redux

less than 1 minute read

whats @Effects You can almost think of your Effects as special kinds of reducer functions that are meant to be a place for you to put your async calls in suc...

reactive programing

less than 1 minute read

The second advantage to a lazy subscription is that the observable doesn’t hold onto data by default. In the previous example, each event generated by the in...

Container

2 minute read

The Docker project was responsible for popularizing container development in Linux systems. The original project defined a command and service (both named do...

promise vs observiable

less than 1 minute read

The drawback of using Promises is that they’re unable to handle data sources that produce more than one value, like mouse movements or sequences of bytes in ...

JDK source

less than 1 minute read

interface RandomAccess Marker interface used by List implementations to indicate that they support fast (generally constant time) random access. The primary ...

SSH SFTP

4 minute read

Secure FTP SFTP over FTP is the equivalant of HTTPS over HTTP, the security version

AWS Tips

less than 1 minute read

After establishing a SSH session, you can install a default web server by executing sudo yum install httpd -y. To start the web server, type sudo service htt...

Oracle

less than 1 minute read

ORA-12899: Value Too Large for Column

Spring notes

27 minute read

Spring Bean Life Cycle Callback Methods

Kindle notes

113 minute read

#《亿级流量网站架构核心技术》目录一览 TCP四层负载均衡 使用Hystrix实现隔离 基于Servlet3实现请求隔离 限流算法 令牌桶算法 漏桶算法 分布式限流 redis+lua实现 Nginx+Lua实现 使用sharding-jdbc分库分表 Disruptor+Redis...

Java Security Notes

7 minute read

Java Security well-behaved: programs should be prevent from consuming too much system resources

R Language

less than 1 minute read

s<-read.csv("C:/Users/xxx/dev/R/IRS/SHH_SCHISHG.csv") # aggregate s2<-table(s$Original.CP) s3<-as.data.frame(s2) # extract by Frequency ordered s3...

SSH and Cryptography

35 minute read

SFTP versus FTPS SS: Secure Shell An increasing number of our customers are looking to move away from standard FTP for transferring data, so we are ofte...

Eclipse notes

less than 1 minute read

How do I remove a plug-in? Run Help > About Eclipse > Installation Details, select the software you no longer want and click Uninstall. (On Macintosh i...

Java JVM

less than 1 minute read

Class loading subsystem

Maven-Notes

27 minute read

Maven philosophy “It is important to note that in the pom.xml file you specify the what and not the how. The pom.xml file can also serve as a documentatio...

Java New IO

less than 1 minute read

Notes JDK 1.0 introduced rudimentary I/O facilities for accessing the file system (to create a directory, remove a file, or perform another task), accessi...

IT-Architect

1 minute read

SOA SOA is a set of design principles for building a suite of interoperable, flexible and reusable services based architecture. top-down and bottom-up a...

Algorithm

15 minute read

This page is about key points about Algorithm

Java-Tricky-Tech-Questions.md

19 minute read

What is the difference between Serializable and Externalizable in Java? In earlier version of Java, reflection was very slow, and so serializaing large ob...

Compare-In-Java

6 minute read

Concepts If you implement Comparable interface and override compareTo() method it must be consistent with equals() method i.e. for equal object by equals(...

Java Collections Misc

3 minute read

Difference between equals and deepEquals of Arrays in Java Arrays.equals() method does not compare recursively if an array contains another array on oth...

HashMap in JDK

4 minute read

Hashmap in JDK Some note worth points about hashmap Lookup process Step# 1: Quickly determine the bucket number in which this element may resid...

Java 8 Tips

66 minute read

This blog is listing key new features introduced in Java 8

Back to top ↑

2016

Java GC notes

28 minute read

verbose:gc verbose:gc prints right after each gc collection and prints details about each generation memory details. Here is blog on how to read verbose gc

Hash Code Misc

12 minute read

contract of hashCode : Whenever it is invoked on the same object more than once during an execution of a Java application, the hashCode method must consis...

Angulary Misc

2 minute read

Dependency Injection Angular doesn’t automatically know how you want to create instances of your services or the injector to create your service. You must co...

Random number in java

less than 1 minute read

ThreadLocalRandom, SecureRandm, java.util.Random, java.math.Random

Java new features

less than 1 minute read

JDK Versions JDK 1.5 in 2005 JDK 1.6 in 2006 JDK 1.7 in 2011 JDK 1.8 in 2014 Sun之前风光无限，但是在2010年1月27号被Oracle收购。 在被Oracle收购后对外承诺要回到每2年一个realse的节奏。但是20...

Simpler chronicle of CI(Continuous Integration) “乱弹系列”之持续集成工具

less than 1 minute read

引言 有句话说有人的地方就有江湖，同样，有江湖的地方就有恩怨。在软件行业历史长河（虽然相对于其他行业来说，软件行业的历史实在太短了，但是确是充满了智慧的碰撞也是十分的精彩）中有一些恩怨情愁，分分合合的小故事，比如类似的有，从一套代码发展出来后面由于合同到期就分道扬镳，然后各自发展成独门产品的Sybase DB和微...

浅谈软件单元测试中的“断言” (assert)，从石器时代进步到黄金时代。

less than 1 minute read

大家都知道，在软件测试特别是在单元测试时,必用的一个功能就是“断言”（Assert)，可能有些人觉得不就一个Assert语句，没啥花头，也有很多人用起来也是懵懵懂懂，认为只要是Assert开头的方法，拿过来就用。一个偶然的机会跟人聊到此功能，觉得还是有必要在此整理一下如何使用以及对“断言”的理解。希望可以帮助大家...

Head First Blockchina 1

less than 1 minute read

深入浅出区块链系统：第一章. what you should know about blockchain

Kubernetes 与 Docker Swarm的对比

less than 1 minute read

Kubernetes 和Docker Swarm 可能是使用最广泛的工具，用于在集群环境中部署容器。但是这两个工具还是有很大的差别。

http methods

11 minute read

RFC origion http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.1.2)

Spark-vs-Storm

less than 1 minute read

The stark difference among Spark and Storm. Although both are claimed to process the streaming data in real time. But Spark processes it as micro-batches; wh...

微服务

less than 1 minute read

可以想像一下，之前的传统应用系统，像是一个大办公室里面，有各个部门，销售部，采购部，财务部。办一件事情效率比较高。但是也有一些弊端，首先，各部门都在一个房间里。

kibana, view layer of elasticsearch

less than 1 minute read

What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...

kibana, view layer of elasticsearch

less than 1 minute read

What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...

iConnect

less than 1 minute read

UI HTML5, AngularJS, BootStrap, REST API, JSON Backend Hadoop core (HDFS), Hive, HBase, MapReduce, Oozie, Pig, Solr

Business Analysis

less than 1 minute read

Purpose of BA 带来一些商业价值（收益） 解决业务痛点

Data Structure

2 minute read

Binary Tree A binary tree is a tree in which no node can have more than two children. A property of a binary tree that is sometimes important is that th...

equity trading

less than 1 minute read

Difference between mutal funds and hedge funds

SQL

less than 1 minute read

Differences between not in, not exists , and left join with null

HTTPS/2

less than 1 minute read

concepts

Github page commands notes

less than 1 minute read

404 error for customized domain (such as godday) 404 There is not a GitHub Pages site here. Go to github master branch for gitpages site, manually add CN...

RenMinBi International

1 minute read

RQFII RQFII stands for Renminbi Qualified Foreign Institutional Investor. RQFII was introduced in 2011 to allow qualified foreign institutional investors to ...

Load Balancing

less than 1 minute read

Concepts LVS means Linux Virtual Server, which is one Linux built-in component.

Python

less than 1 minute read

(‘—–Unexpected error:’, <type ‘exceptions.TypeError’>) datetime.datetime.now()

Setup Git in Mint Linux

less than 1 minute read

How to setup Git in Mint Linux =================================================

Microservices vs. SOA

1 minute read

Microservice Services are organized around capabilities, e.g., user interface front-end, recommendation, logistics, billing, etc. Services are small in ...

Java Class Loader

33 minute read

Codecache The maximum size of the code cache is set via the -XX:ReservedCodeCacheSize=N flag (where N is the default just mentioned for the particular com...

Back to top ↑