KDB
KDB
However kdb+ evaluates expressions right-to-left. There are no precedence rules. The reason commonly given for this behaviour is that it is a much simpler to understand system, many q-gods would agree, beginners however may not.
The kdb+ built-in commands are mostly a single letter. If a system command that is not built-in is entered, that command will be passed to the underlying operating system.
Created with financial institutions in mind, the database was developed as a central repository to store time series data that supports real-time analysis of billions of records.
Columnar databases return answers to some queries in a more efficient way than row-based database management systems.
kdb+ dictionaries, tables and nanosecond time stamps are native data types and are used to store time series data.
In 1998, Kx Systems released kdb, a database built on the language K written by Arthur Whitney. In 2003, kdb+ was released as a 64-bit version of kdb.
kdb/q/kdb+ is both a database (kdb) and a vector language (q). It’s used by almost every major financial institution
Kdb+ is an in-memory column-oriented database based on the concept of ordered lists. In-memory means it primarily stores its data in RAM. This makes it extremely fast with a much simplified database engine but it requires a lot of RAM (Which no longer poses a problem as servers with massive amounts of RAM are now inexpensive). Column oriented database means that each column of data is stored sequentially in memory
Column DB
The main reason why indexes dramatically improve performance on large datasets is that database indexes on one or more columns are typically sorted by value, which makes range queries operations (like the above “find all records with salaries between 40,000 and 50,000” example) very fast (lower time-complexity).
Kdb+ the Database - Column Oriented DB allowing fast timeseries analysis
Column-oriented systems
A column-oriented database serializes all of the values of a column together, then the values of the next column, and so on. For our example table, the data would be stored in this fashion: 10:001,12:002,11:003,22:004; Smith:001,Jones:002,Johnson:003,Jones:004; Joe:001,Mary:002,Cathy:003,Bob:004; 40000:001,50000:002,44000:003,55000:004; In this layout, any one of the columns more closely matches the structure of an index in a row-based system. This may cause confusion that can lead to the mistaken belief a column-oriented store “is really just” a row-store with an index on every column. However, it is the mapping of the data that differs dramatically. In a row-oriented indexed system, the primary key is the rowid that is mapped from indexed data. In the column-oriented system, the primary key is the data, which is mapped from rowids.[2] This may seem subtle, but the difference can be seen in this common modification to the same store:
…;Smith:001;Jones:002,004;Johnson:003;…
Whether or not a column-oriented system will be more efficient in operation depends heavily on the workload being automated. Operations that retrieve all the data for a given object (the entire row) are slower. A row-based system can retrieve the row in a single disk read, whereas numerous disk operations to collect data from multiple columns are required from a columnar database. However, these whole-row operations are generally rare. In the majority of cases, only a limited subset of data is retrieved. In a rolodex application, for instance, collecting the first and last names from many rows to build a list of contacts is far more common than reading all data for any single address. This is even more true for writing data into the database, especially if the data tends to be “sparse” with many optional columns. For this reason, column stores have demonstrated excellent real-world performance in spite of many theoretical disadvantages.
Why are most databases row-oriented?
Imagine we want to add one row somewhere in the middle of our data for 2011-02-26, on the row oriented database no problem, column oriented we will have to move almost all the data! Lucky since we mostly deal with time series new data only appends to the end
of our table.
Difference vs row based DB
a subtle point is that unlike most standard SQL which is based on set theory
, kdb+ is based on vectors of ordered lists
. Where standard SQL has struggled with queries like find the top 3 stocks by price, find the bottom 3 by market cap because it has no concept of order, kdb’s ordering significantly simplifies many queries. This ordered concept allows kdb+ to provide unique timeseries joins that would be be extremely difficult in other variations of SQL and require the use of slow cursors.
Language Q
the primary design objectives of q are expressiveness, speed and efficiency. In these, it is beyond compare. The design trade-off is a terseness that can be disconcerting to programmers coming from verbose traditional database programming environments – e.g., C++, Java, C# or Python – and a relational DBMS
Q evolved from APL (A Programming Language), which was first invented as a mathematical notation by Kenneth Iverson at Harvard University in the 1950s. APL was introduced in the 1960s by IBM as a vector programming language, meaning that it processes a list of numbers in a single operation. It was successful in finance and other industries that required heavy number crunching.
q is Kx’s proprietary language. It’s a powerful, concise and elegant array language, which means that a production system could just be a single page of code, not pages and pages of code and a nightmare to maintain. Clearly there is an initial investment in learning it, but the power it gives you to manipulate streaming, real-time and historical data makes that initial investment really worthwhile.
q language - fast, interpreted vector based language
Q is a interpreted vector based dynamically typed language built for speed and expressiveness.
Since q is interpreted you can enter commands straight into the console there is no waiting for compilation, feedback is instantaneous.
Key features of Q
Interpreted Q is interpreted, not compiled. During execution, data and functions live in an in-memory workspace. Iterations of the development cycle tend to be quick because all run-time information needed to test and debug is immediately available in the workspace. Q programs are stored and executed as simple text files called scripts. The interpreter’s eval and parse routines are exposed so that you can dynamically generate code in a controlled manner.
Types Q is a dynamically typed language, in which type checking is mostly unobtrusive. Each variable has the type of its currently assigned value and type promotion is automatic for most numeric operations. Types are checked on operations to homogenous lists.
Evaluation Order While q is entered left-to-right, expressions are evaluated right-to-left or, as the q gods prefer, left of right – meaning that a function is applied to the argument on its right. There is no operator precedence and function application can be written without brackets. Punctuation noise is significantly reduced.
Null and Infinity Values In classical SQL, the value NULL represents missing data for a field of any type and takes no storage space. In q, null values are typed and take the same space as non-nulls. Numeric types also have infinity values. Infinite and null values can participate in arithmetic and other operations with (mostly) predictable results.
Integrated I/O I/O is done through function handles that act as windows to the outside world. Once such a handle is initialized, passing a value to the handle is a write.
Table Oriented Give up objects, ye who enter here. In contrast to traditional languages, you’ll find no classes, objects, inheritance and virtual methods in q. Instead, q has tables as first class entities. The lack of objects is not as severe as might first appear. Objects are essentially glorified records (i.e., entities with named fields), which are modeled by q dictionaries. A table can be viewed as a list of record dictionaries.
Ordered Lists Because classical SQL is the algebra of sets – which are unordered with no duplicates – row order and column order are not defined, making time series processing cumbersome and slow. In q, data structures are based on ordered lists, so time series maintain the order in which they are created. Moreover, simple lists occupy contiguous storage, so processing big data is fast. Very fast.
Column Oriented SQL tables are organized as rows distributed across storage and operations apply to fields within a row. Q tables are column lists in contiguous storage and operations apply on entire columns.
In-Memory Database One can think of kdb+ as an in-memory database with persistent backing. Since data manipulation is performed with q, there is no separate stored procedure language. In fact, kdb+ comprises serialized q column lists written to the file system and then mapped into memory.
In q, data structures are based on ordered lists, so time series maintain the order in which they are created. Moreover, simple lists occupy contiguous storage, so processing big data is fast. Very fast.
Q tables are column lists in contiguous storage and operations apply on entire columns.
Sample of Q code
q)l:10 12 14 16 18 22 32 45
q)sum l
169
q)avg l
21.125
q)l*10
100 120 140 160 180 220 320 450
q)k:til 8
q)k
0 1 2 3 4 5 6 7
q)l+k
10 13 16 19 22 27 38 52
Notice in the example code above the absence of loops, no for/while/do yet we could easily express adding one array to another. This is because the vector/list is the primary unit of data in kdb+. Operations are intended to be performed and expressed as being on an entire set of data. Dictionaries can be defined using lists, they provide a hashmap datastructure for quick lookups. Tables are constructed from dictionaries and lists. This brevity of data structures is actually one of the attributes that gives q its ability to express concisely what would take many lines in other languages.
Some of the practical applications of being able to combine both streaming and real-time data together with historical, all in the same database?
The most important thing is the simplicity, which translates into speed and ease of doing analysis. For example, it allows you to do complicated, time-critical analysis, such as pre-trade risk. This means that you are likely to see interesting trading opportunities before those who are using the same off-the-shelf solution as everybody else. So you’re there first and you’re there so early, you can afford to do comprehensive pre-trade risk analysis, and you’re able to look for patterns you’ve seen in the past.
In addition to capturing market data, firms are using kdb+ for order-book management, algorithmic trading, and risk assessment. “They are using kdb+/q for queries being performed on both streaming or historical data — the latter easily accommodating research and back-testing,”
DeltaFlow, a platform from First Derivatives that’s based on kdb+, is used by traders for high volume, low-latency algorithmic trading and by regulators for real-time detection of market abuse and unauthorized trading activity across multiple asset classes.
Combined power of kdb+/q
What’s beautiful about kdb+ is that since tables are columns of vectors, all the power of the q language can be used as easily on table data as it was on lists. Where we had sum[l],avg[l],weightedAvg[l1;l2] of lists we can write similar qSQL: select avg price, sum volume, weightedAvg[time;price] from trade Want to apply a function to a timeseries, simply place it inline: select {a:avg x; sqrt avg (xx)-aa} price from trade
Q commadns
To obtain official console display of any q value, apply the built-in function show to it.
q)show a:42
comments
At least one whitespace character must separate / intended to begin a comment from any text to the left of it on a line.
boolean
Boolean values in q are stored in a single byte and are denoted as the binary values they really are with an explicit type suffix b. One way to generate boolean values is to test for equality.
q)42=40+2 1b
date
One interesting and useful feature of q temporal values is that, as integral values under the covers, they naturally participate in arithmetic. For example, to advance a date five days, add 5.
q)2000.01.01+5 _ Or to advance a time by one microsecond (i.e., 1000 nanoseconds) add 1000.
q)12:00:00.000000000+1000 _ Or to verify that temporal values are indeed their underlying values, test for equality.
q)2000.01.01=0
Symbols
Symbols are denoted by a leading back-quote (called “back tick” in q-speak) followed by characters. Symbols without embedded blanks or other special characters can be entered literally into the console. q)`aapl _
Since symbols are atoms, any two can be tested for equality.
q)aapl=
apl
_
list
The fundamental q data structure is a list, which is an ordered collection of items sequenced from left to right. The notation for a general list encloses items with ( and ) and uses ; as separator. Spaces after the semi-colons are optional but can improve readability.
q)(1; 1.2; `one) _
In the case of a homogenous list of atoms, called a simple list, q adopts a simplified format for both storage and display. The parentheses and semicolons are dropped. For example, a list of underlying numeric type separates its items with a space.
q)(1; 2; 3) 1 2 3
A simple list of booleans is juxtaposed with no spaces and has a trailing b type indicator.
q)(1b; 0b; 1b) 101b A simple list of symbols is displayed with no separating spaces.
q)(one;
two; three)
onetwo
three
basic operations
to construct and manipulate lists. The most fundamental is til, which takes a non-negative integer n and returns the first n integers starting at 0 (n itself is not included in the result).
q)til 10 0 1 2 3 4 5 6 7 8 9
til list tips
Be mindful that q always evaluates expressions from right to left and that operations work on vectors whenever possible.
q)1+til 10 1 2 3 4 5 6 7 8 9 10
Similarly, we obtain the first 10 even numbers and the first ten odd numbers.
q)2til 10 _ q)1+2til 10 _ Finally, we obtain the first 10 even numbers starting at 42.
q)42+2*til 10 _
Another frequently used list primitive is join , that returns the list obtained by concatenating its right operand to its left operand.
q)1 2 3,4 5 1 2 3 4 5
extract
To extract items from the front or back of a list, use the take operator #. Positive argument means take from the front, negative from the back.
q)2#til 10 0 1 q)-2#til 10
Applying # always results in a list.
In particular, the idiom 0# returns an empty list of the same type as the first item in its argument. Using an atom argument is a succinct way to create a typed empty list of the type of the atom.
q)0#1 2 3 `long$()
Should you extract more items than there are in the list, # restarts at the beginning and continues extracting. It does this until the specified number of items is reached.
q)5#1 2 3 1 2 3 1 2
As with atoms, a list can be assigned to a variable.
q)L:10 20 30 The items of a list can be accessed via indexing, which uses square brackets and is relative to 0.
q)L[0] 10
Function
Conceptually, a q function is a sequence of steps that produces an output result from an input value. Since q is not purely functional, these rules can interact with the world by reaching outside the context of the function. Such actions are called side effects and should be carefully controlled.
Function definition is delimited by matching curly braces { and }. Immediately after the opening brace, the formal parameters are names enclosed in square brackets [ and ] and separated by semi-colons. These parameters presumably appear in the body of the function, which follows the formal parameters and is a succession of expressions sequenced by semi-colons.
Following is a simple function that returns the square of its input. On the next line we assign the same function to the variable sq. The whitespace is optional.
q){[x] xx} _ q)sq:{[x] xx} _
Here is a function that takes two input values and returns the sum of their squares.
q){[x;y] a:xx; b:yy; a+b} _ q)pyth:{[x;y] a:xx; b:yy; a+b} _
Here are the previous functions applied to arguments.
q){[x] xx}[5] 25 q)sq[5] _ q){[x;y] a:xx; b:y*y; a+b}[3;4] 25 q)pyth[3;4] _
monadic function
In q, as in most functional languages, we don’t need no stinkin’ brackets for application of a monadic function – i.e., with one parameter. Simply separate the function from its argument by whitespace. This is called function juxtaposition.
q){xx} 5 _ q)f:{xx} q)f 5 _
x,y,z
It is common in mathematics to use function parameters x, y, or z. If you are content with these names (in the belief that descriptive names provide no useful information to the poor soul reading your code), you can omit their declaration and q will understand that you mean the implicit parameters x, y, and z in that order.
q){xx}[5] 25 q){a:xx; b:y*y; a+b}[3;4] 25
verbs
higher order functions, or as they are called in q, adverbs.
In words, we tell q to start with the initial value of 0 in the accumulator and then modify + with the adverb / so that it adds across the list.
q)0 +/ 1 2 3 4 5 15
In this situation we don’t really need the flexibility to specify the initial value of the accumulator. It suffices to start with the first item of the list and proceed across the rest of the list. There is an even simpler form for this case.
q)(+/) 1 2 3 4 5
for loop
If you are new to functional programming, you may think, “Big deal, I write for loops in my sleep.”
More importantly, you can focus on what you want done without the irrelevant scaffolding of how to set up control structures. This is called declarative programming.
What else can we do with our newfound adverb? Change addition to multiplication for factorial.
q)(*/) 1+til 10 3628800
larger vs smaller
The fun isn’t limited to arithmetic primitives. We introduce |, which returns the larger of its operands and &, which returns the smaller of its operands.
q)42|98 98 q)42&98
Use | or & with over and you have maximum or minimum. |
q)(|/) 20 10 40 30 40 q)(&/) 20 10 40 30
command ‘over’ adverb
Some applications of / are so common that they have their own names.
q)sum 1+til 10 55 q)prd 1+til 10 “o” _ q)max 20 10 40 30 _ q)min 20 10 40 30
At this point the / pattern should be clear: it takes a given function and produces a new function that accumulates across the original list, producing a single result. In particular, / converts a dyadic function to a monadic aggregate function – i.e., one that collapses a list to an atom.
data type
long¶
In q versions 3.0 and later, the basic integer type is a signed eight-byte integer, called long. A literal is identified as a long by the fact that it contains only numeric digits, with an optional leading minus sign, and no decimal point. It may also have an optional trailing type indicator j indicating it is a long and not another integer type. Here is a typical long integer value.
q)42 42 Observe that the type indicator j is accepted but redundant.
q)42j 42
The short type represents a two-byte signed integer and requires the trailing type indicator h. For example,
q)-123h _ Similarly, the int type represents a four-byte signed integer and requires the trailing type indicator i.
The float type represents an IEEE standard eight-byte floating-point number, often called “double” in traditional languages. A float can hold (at least) 15 decimal digits of precision. It is denoted by optionally signed numeric digits with either a decimal point or an optional trailing type indicator f. Observe that the console shortens the display of floats with no significant digits to the right of the decimal.
You can change this by using the \P command (note upper case) to specify a display width up to 16 digits. If you issue \P 0 the console will display all 17 decimal digits of the underlying binary representation, although the last digit is unreliable.
boolean¶
The boolean type uses one byte to store a bit and is denoted by the bit value with the trailing type indicator b. There are no keywords for ‘true’ or ‘false’, nor are there separate logical operators for booleans.
q)0b _ q)1b
Text Data¶
There are two atomic text types in q. They are more akin to the SQL types CHAR and VARCHAR than the character types of traditional languages.
2.4.1 char¶
A char holds an individual ASCII or 8-bit Unicode character that is stored in one byte. It corresponds to a SQL CHAR. It is denoted by a single character enclosed in double quotes.
q)”q”
Some keyboard characters – e.g., the double-quote – cannot be entered directly into a char since they have special meaning in the q console. As in C, special characters are escaped with a preceding back-slash . The console display somewhat confusingly displays the escape, but the following are all actually single characters.
q)”"” “"” q)”\” _ q)”\n” _ q)”\r” _ q)”\t” _ Also as in C, you can escape any ASCII character by specifying its underlying numeric value as three octal digits.
q)”\142” “b”
symbol¶
A symbol is an atom holding text. It is denoted by a leading back-quote, read “back tick” in q-speak.
q)q
_
q)
zaphod
_
a symbol is not a collection of char. The symbol `a and the char “a” are not the same, as we can see by asking q if they are identical.
q)`a~”a” 0b
list
Index Notation¶
To access the item at index i in a list, follow the list immediately with [i]. This is called item indexing. For example,
q)(100; 200; 300)[0]
Indexed Assignment¶
Items in a list can also be assigned via item indexing. Thus,
q)L:1 2 3 q)L[1]:42 q)L 1 42 3
An omitted index returns the entire list.
q)L:10 20 30 40 q)L[] 10 20 30 40
2022
Gemfire Geode Error on Peer or client version with ordinal xx not supported. Highest known version is 1.12.1 Client
Worries less, smile more!
Password must not null in gemfire and geode, but I’ve assigned password in yaml properties file
Worries less, smile more!
One page to cover most commonly found errors for fat jar in SpringBoot
Kill time, or kiss time!
Awesome Shortcuts to boost productivities
Shortcuts & tips zoom Cmd+Shift+A: mute CMd+W: leave meeting. Fn+F: toggle full screen
Core Java tips requried an interview
Tips of Java How to convert list of object to list of String
Tell me difference of truststore and keystore in short answer
Do what you say, say what you do.
GIT useful scripts or error solutions
Don’t wish for it, work for it.
Tell me difference of yarn install and npm install in short answer
Don’t find fault. Find a remedy.
Bible blog for most commonly found Gradle errors
People are smarter than you think. Give them a chance to prove themselves.
Pearls in Front end development
Be happy in front of people who don’t like you, it kills them.
Ruby from zero to hero
This is your life. Do what you love, and do it often.
Everything you’d know for Groovy interviews
Life is short. Don’t waste it with negative people who don’t appreciate you. Keep them in your heart but keep them out of your life.
To outstanding as professional MacBook pro user
The most effective way to do it, is to do it Homebrew The best practice is to run brew info before install new software. It will generally list what’s c...
Failed to install gem in Mac, incompatible architecture and missing psych
Burn your ego before it burns you.
IntelliJ Tips to boost your productivities 10 times
Shortcuts Win+F10: Put IntelliJ Menu in focust, so you can navigate menu items by keyboard Win+F9: Open run/debug configuration window Alt+F10: To sho...
Everything you’d know about state machine for interviews
Less expecting, more accepting.
Tips about algorithm resolving from Leetcode
Stay focused, believe that you can achieve at the highest level, surround yourself with others who believe in you and do not stray from your goals.
Solution center for Node errors
Fina a way. If there’s none, make one!
Triple your productivities by Visual studio code keyboard shortcuts
The sentence The quick brown fox jumps over the lazy dog uses every letter of the alphabet.
TypeScript noteworthy notes
The moment you start focusing on yourself, things start falling into place.
RXJS – reactive Programming like a hero
When love is real, it doesn’t lie, cheat, pretend or keep secrets.
Concurrency in Java
Little things make big things happens.
Linux Tips
Remember, some things have to end for better things to begin.
A taste of GraphQL
A good day starts with a good mindset!
What’s inside magic in Spring Data JPA
A good day starts with a good mindset!
What’s inside magic in Spring Data JPA
A good day starts with a good mindset!
Why Spring turn a column name from camelNaming to snake_Naming
Don’t spend another year doing the same shit.
Some mistakes you’d avoid in java
With great power comes great responsibilities.
Untold stories for Jupiter, any differences JUnit 5 vs Junit 4
Don’t tell people your plans. Just show them your results!
FileNotFound Exception when loading data file in IntelliJ
Take time to do what makes your soul happy!
How to ace AWS certification just like play a game
Life isn’t about finding yourself. Life is about creating yourself.
Java Deep Notes
Java Deep Notes
Code to draw a Big H with all stars
Coding is everything! Code Now!
Code to draw a Big H with all stars
Coding is everything! Code Now!
Single vs Double precisions, float vs double data type
Leave nothing for tomorrow which can be done today. -Abraham Lincoln.
SkipTest-Not-Work-In-Multiple-models-project
Leave nothing for tomorrow which can be done today. -Abraham Lincoln.
SkipTest-Not-Work-In-Multiple-models-project
Leave nothing for tomorrow which can be done today. -Abraham Lincoln.
Git commands you can show off for 100 years
How to delete all local files blocking you pull, unstash and merge git clean -n git clean -f
Maven error and solution on No such host is known
Don’t promis when you are hapy. Don’t reply when you’re angry and don’t decide when you’re sad Service keep on restarting If you spot service is restartin...
Maven error and solution on No such host is known
Don’t promis when you are hapy. Don’t reply when you’re angry and don’t decide when you’re sad
Gradle build stuck
Gradle build stuck, keep on running but never ending
2021
Save my eyes, let your cell phone to read screen content to you
Too much screen time
How logging system Bootstrapped in Spring Boot Application
Summary Following diagram demonstrated the process to bootstrap and use Logback for loggings in Spring Boot applciation.
SQLServer Error about This driver is not configured for integrated authentication
Symptoms When you are using integrated authentication (Kerberos connection) for MS SqlServer connection, there is one possible error :
How to copy files from resources folder in jar and save to a file
Why to extract resources from jar to local disk
Debug of SpringBoot run not working in IntelliJ
Normal approach to debug maven
How to watch specific kubenetes deployment by labels
How to watch specific kubenetes deployment by labels
Failed to talk to github.com from corporation network
Background It’s typical to get various network connection issues when you run commands within corporation network. For example, you’ll find diversed issues w...
Day-Day-Up-Java
More developer friendly Threa Sleep
How to user fire extinguisher
Summary As you know, staff and your safety is paramount. So what if emergency take place, such as fire in office, how to help yourself and your colleagues by...
Deep dive into ApplicationEvent in SpringBoot
Summary As you know, there are various event will be sent (multicast) when a specific story taken place.
2021-09-22-IT-Solutions-For-Remote-Learning
IT-Solutions-For-Remote-Learning.md
Deep dive into Kubernetes Client API
Summary To talk to K8s for getting data, there are few approaches. While K8s’ official Java library is the most widely used one. This blog will look into thi...
How to get CPU name, core, 64bit and speed in command line
Summary In windows operation system, if you want to get your CPU name, core, 64bit and speed in command line. Just follow below actions:
Whitelabel Error Page
Summary Whitelabel Error Page is the default error page in Spring Boot web app. It provide a more user-friently error page whenever there are any issues when...
Google maps no photos reviews
Summary I found a weird problem of the app Google Maps of my Oppo Android phone. That’s when you search a place in Google map, say “Central Park”, ideally th...
Shall I still need booster even after I got dose 3?
The greatest wealth is health!
Debts in a nutshell
A debt security represents a debt owed by the issuer to an investor. Here, the investor acts as a lender to the issuer which may be a government, organisatio...
2020
How to process data from S3 download URL
S3 download URL As you know, AWS S3 object can be downloaded/processed by S3 download URL. I’m showing you two examples on how to process S3 Object by NIO f...
Debug Stuck IntelliJ
What happened to a debug job hanging in IntelliJ (IDEAS) IDE? You may find when you try to debug a class in Intellij but it stuck there and never proceed, e....
Awesome Kotlin
Difference with Scala Kotlin takes the best of Java and Scala, the response times are similar as working with Java natively, which is a considerable advantag...
Awesome tips for Chrome
Shortcuts & tips
JVM热身
此文是作者英文原文的翻译文章,英文原文在:http://todzhang.com/posts/2018-06-10-jvm-warm-up/
Awesome tips and shortcuts for Slack
Shortcuts for Slack
Awesome Reactive programming
Key points of Reactive Programming
Awesome Swift for iOS
Frame in Swift
Mock in kotlin
Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...
Mock in kotlin
Argument Matching & Answers For example, you have mocked DOC with call(arg: Int): Intfunction. You want to return 1 if argument is greater than 5 and -1 ...
Docker
Dockers Concepts
How to decode path parameters in All REST WebServices calls
How to decode path parameters in All REST WebServices calls
Curl
Linux Curl command
AOP
The concept of join points as matched by pointcut expressions is central to AOP, and Spring uses the AspectJ pointcut expression language by default.
Micrometer notes
As a general rule it should be possible to use the name as a pivot. Dimensions allow a particular named metric to be sliced to drill down and reason about th...
Pigeons in holes principle
# Pigeonhole principle
Awesome solutions for algorithm questions
你就会发现只要涉及递归的问题,都是 树的问题。
A Facial Recognition utility in a dozen of LOC
A Facial Recognition utility in a dozen of python LOC (Lines Of Code)
Awesome SSL certificates and HTTPS
What’s TLS TLS (Transport Layer Security) and its predecessor, SSL (Secure Sockets Layer), are security protocols designed to secure the communication betwee...
JVM warm up by Escape Analysis
Why JVM need warm up I don’t know how and why you get to this blog. But I know the key words in your mind are “warm” for JVM. As the name “warm up” suggested...
Java Concurrent Column 2
This is the second half about Java Concurrent of my blog
Java Concurrent
This blog is about noteworthy pivot points about Java Concurrent Framework Back to Java old days there were wait()/notify() which is error prone, while fr...
Algorithm notes from Leecode – 1
Algorithm Leetcode
2019
Conversations with God
Feelings is the language of the soul. If you want to know what’s true for you about something, look to how your’re feeling about.
Kafka In Spring
Enable Kafka listener annotated endpoints that are created under the covers by a AbstractListenerContainerFactory. To be used on Configuration classes as fol...
Terraform
Why Terraform
Kafka
Kafka
Mifid
FX Spot is not covered by the regulation, as it is not considered to be a financial instrument by ESMA, the European Union (EU) regulator. As FX is considere...
Foreign Exchange
currency pairs Direct ccy: means USD is part of currency pair Cross ccy: means ccy wihtout USD, so except NDF, the deal will be split to legs, both with...
2018
Seconds
nano seconds
Citrix receiver
Simple Binary Encoding (SBE)
Citrix receiver
“Cannot connect to remote desktop” with Citrix Receiver
Guice
A new type of Juice Put simply, Guice alleviates the need for factories and the use of new in your Java code. Think of Guice’s @Inject as the new new. You wi...
YAML
Key points All YAML files (regardless of their association with Ansible or not) can optionally begin with — and end with …. This is part of the YAML format a...
Distruptor
multithreading
Mockito
Feature
Protobuf
What are protocol buffers?
Sudo in a Nutshell
Sudo in a Nutshell Sudo (su “do”) allows a system administrator to give certain users (or groups of users) the ability to run some (or all) commands as root...
Zoo-keeper
ZK Motto the motto “ZooKeeper: Because Coordinating Distributed Systems is a Zoo.”
Presto DB
WHAT IS PRESTO?
Chronicle
Overview
Cucumber
Acceptance testing vs unit test It’s sometimes said that unit tests ensure you build the thing right, whereas acceptance tests ensure you build the right thi...
Scala
Scala String
akka framework of scala
philosophy The actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-o...
File Util in Apache Camel
FileUtil.class
Apache Camel
Camel’s message model In Camel, there are two abstractions for modeling messages, both of which we’ll cover in this section. org.apache.camel.Message—The ...
QuickFixJ
Settings
JXM
Exporting your beans to JMX The core class in Spring’s JMX framework is the MBeanExporter. This class is responsible for taking your Spring beans and registe...
Solace MQ
Solace PubSub+ It is a message broker that lets you establish event-driven interactions between applications and microservices across hybrid cloud environmen...
Apigee
App deployment, configuration management and orchestration - all from one system. Ansible is powerful IT automation that you can learn quickly.
Ansible
Ansible: What Is It Good For? Ansible is often described as a configuration management tool, and is typically mentioned in the same breath as Chef, Puppet, a...
flexbox
How Flexbox works — explained with big, colorful, animated gifs
Jboss tips
commands:
Locking and multithreading
Single Writer principle
KDB
KDB However kdb+ evaluates expressions right-to-left. There are no precedence rules. The reason commonly given for this behaviour is that it is a much simple...
Foreign Exchange
Foreign Exchange markets
Portactor
Better to use smart wait
Agile and SCRUM
Key concept In Scrum, a team is cross functional, meaning everyone is needed to take a feature from idea to implementation.
DevOps-Philosophy
:100:DevOps Model Defined
rxjs pipe in depth
https://stormforger.com/blog/2016/07/08/types-of-performance-testing/
How to setup nodejs to install package from intranet
Error of ‘ECONNRESET’ You may face error ECONNRESET from intranet, even appropriate proxy tools (e.g. cntlm) is running. The errors may looks like ```bash $ ...
Strategy-Of-Openshift-Releases
Release & Testing Strategy There are various methods for safely releasing changes to Production. Each team must select what is appropriate for their own ...
NodeJs Notes
commands to read files var lineReader = require(‘readline’).createInterface({ input: require(‘fs’).createReadStream(‘C:\dev\node\input\git_reset_files.tx...
Minium Viable Product
https://blog.leanstack.com/minimum-viable-product-mvp-7e280b0b9418
What is difference between declarations, providers and import in NgModule
What is difference between declarations, providers and import in NgModule
CORS :Cross-Origin Resource Sharing
Cross-Origin Request Sharing - CORS (A.K.A. Cross-Domain AJAX request) is an issue that most web developers might encounter, according to Same-Origin-Policy,...
ngrx
Why @Effects? In a simple ngrx/store project without ngrx/effects there is really no good place to put your async calls. Suppose a user clicks on a button or...
iOS programming
View A view is also a responder (UIView is a subclass of UIResponder). This means that a view is subject to user interactions, such as taps and swipes. Thus,...
2017
cloud computering
openshift vs openstack The shoft and direct answer is `OpenShift Origin can run on top of OpenStack. They are complementary projects that work well together....
cloud computering
Concepts Cloud computing is the on-demand demand delivery of compute database storage applications and other IT resources through a cloud services platform v...
Redux
whats @Effects You can almost think of your Effects as special kinds of reducer functions that are meant to be a place for you to put your async calls in suc...
reactive programing
The second advantage to a lazy subscription is that the observable doesn’t hold onto data by default. In the previous example, each event generated by the in...
common errors in NPM or node
code E503 code E503 when run npm install packages, e.g.
Container
The Docker project was responsible for popularizing container development in Linux systems. The original project defined a command and service (both named do...
promise vs observiable
The drawback of using Promises is that they’re unable to handle data sources that produce more than one value, like mouse movements or sequences of bytes in ...
Openshift tips
Commands bible
google analysis
How Page Value is calculated
JDK source
interface RandomAccess Marker interface used by List implementations to indicate that they support fast (generally constant time) random access. The primary ...
SSH SFTP
Secure FTP SFTP over FTP is the equivalant of HTTPS over HTTP, the security version
Setup WebSphere profiles and application in command line
Setup WebSphere profiles and application in command line
AWS Tips
After establishing a SSH session, you can install a default web server by executing sudo yum install httpd -y. To start the web server, type sudo service htt...
Oracle
ORA-12899: Value Too Large for Column
Spring notes
Spring Bean Life Cycle Callback Methods
Kindle notes
#《亿级流量网站架构核心技术》目录一览 TCP四层负载均衡 使用Hystrix实现隔离 基于Servlet3实现请求隔离 限流算法 令牌桶算法 漏桶算法 分布式限流 redis+lua实现 Nginx+Lua实现 使用sharding-jdbc分库分表 Disruptor+Redis...
Java JIT compiler
This is talking about Java JIT (Just-In-Time) compiler
Java Security Notes
Java Security well-behaved: programs should be prevent from consuming too much system resources
SeriableVersionUID
Noteworthy points about SeriableVersionUID in Java
R Language
s<-read.csv("C:/Users/xxx/dev/R/IRS/SHH_SCHISHG.csv") # aggregate s2<-table(s$Original.CP) s3<-as.data.frame(s2) # extract by Frequency ordered s3...
SSH and Cryptography
SFTP versus FTPS SS: Secure Shell An increasing number of our customers are looking to move away from standard FTP for transferring data, so we are ofte...
Eclipse notes
How do I remove a plug-in? Run Help > About Eclipse > Installation Details, select the software you no longer want and click Uninstall. (On Macintosh i...
Java JVM
Class loading subsystem
Maven-Notes
Maven philosophy “It is important to note that in the pom.xml file you specify the what and not the how. The pom.xml file can also serve as a documentatio...
Java New IO
Notes JDK 1.0 introduced rudimentary I/O facilities for accessing the file system (to create a directory, remove a file, or perform another task), accessi...
Network Protocols
Net Protocols
IT-Architect
SOA SOA is a set of design principles for building a suite of interoperable, flexible and reusable services based architecture. top-down and bottom-up a...
Algorithm
This page is about key points about Algorithm
Dead Lock
Concept
Java-Tricky-Tech-Questions.md
What is the difference between Serializable and Externalizable in Java? In earlier version of Java, reflection was very slow, and so serializaing large ob...
NavigableMap Misc
What is NavigableMap
Compare-In-Java
Concepts If you implement Comparable interface and override compareTo() method it must be consistent with equals() method i.e. for equal object by equals(...
Java Collections Misc
Difference between equals and deepEquals of Arrays in Java Arrays.equals() method does not compare recursively if an array contains another array on oth...
HashMap in JDK
Hashmap in JDK Some note worth points about hashmap Lookup process Step# 1: Quickly determine the bucket number in which this element may resid...
Java 8 Tips
This blog is listing key new features introduced in Java 8
Arbitrage vs Heading
What is the difference between arbitrage and hedging?
Java Enum Misc
Enum Misc
2016
Java GC notes
verbose:gc verbose:gc prints right after each gc collection and prints details about each generation memory details. Here is blog on how to read verbose gc
Hash Code Misc
contract of hashCode : Whenever it is invoked on the same object more than once during an execution of a Java application, the hashCode method must consis...
Apache Tips
Apache
Angulary Misc
Dependency Injection Angular doesn’t automatically know how you want to create instances of your services or the injector to create your service. You must co...
Random number in java
ThreadLocalRandom, SecureRandm, java.util.Random, java.math.Random
Java new features
JDK Versions JDK 1.5 in 2005 JDK 1.6 in 2006 JDK 1.7 in 2011 JDK 1.8 in 2014 Sun之前风光无限,但是在2010年1月27号被Oracle收购。 在被Oracle收购后对外承诺要回到每2年一个realse的节奏。但是20...
用10几行代码自己写个人脸识别程序
用10几行代码自己写个人脸识别程序
Eslastic Search
Eslastic Search
JSON lines
JSON lines
Python Scraphy
Python Scraphy
Simpler chronicle of CI(Continuous Integration) “乱弹系列”之持续集成工具
引言 有句话说有人的地方就有江湖,同样,有江湖的地方就有恩怨。在软件行业历史长河(虽然相对于其他行业来说,软件行业的历史实在太短了,但是确是充满了智慧的碰撞也是十分的精彩)中有一些恩怨情愁,分分合合的小故事,比如类似的有,从一套代码发展出来后面由于合同到期就分道扬镳,然后各自发展成独门产品的Sybase DB和微...
Head First Blockchina 3
Hyperledger Fabric for Mortals
【原创】深入浅出区块链系统:第二章
使用Solidity创建以太坊(Ethereum)智能合约(Smart Contract)
How to customize Sublime syntax highlights
Reference Sublime Scope Naming Syntax Guide
浅谈软件单元测试中的“断言” (assert),从石器时代进步到黄金时代。
大家都知道,在软件测试特别是在单元测试时,必用的一个功能就是“断言”(Assert),可能有些人觉得不就一个Assert语句,没啥花头,也有很多人用起来也是懵懵懂懂,认为只要是Assert开头的方法,拿过来就用。一个偶然的机会跟人聊到此功能,觉得还是有必要在此整理一下如何使用以及对“断言”的理解。希望可以帮助大家...
JetBrains/IntelliJ tips
Shortcuts
Head First Blockchina 1
深入浅出区块链系统:第一章. what you should know about blockchain
Kubernetes 与 Docker Swarm的对比
Kubernetes 和Docker Swarm 可能是使用最广泛的工具,用于在集群环境中部署容器。但是这两个工具还是有很大的差别。
漫谈开发设计中的一些‘原则’及’设计哲学’
在开发设计中有一些常用原则或者潜规则,根据笔者的经验,这里稍微总结一下最最常用的,以飨读者。
http methods
RFC origion http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.1.2)
Spark-vs-Storm
The stark difference among Spark and Storm. Although both are claimed to process the streaming data in real time. But Spark processes it as micro-batches; wh...
微服务
可以想像一下,之前的传统应用系统,像是一个大办公室里面,有各个部门,销售部,采购部,财务部。办一件事情效率比较高。但是也有一些弊端,首先,各部门都在一个房间里。
unmodifiableList, unmodifiableSet,unmodifiableMap
What’s it Returns an unmodifiable view of the specified set. This method allows modules to provide users with “read-only” access to internal sets. Query ope...
kibana, view layer of elasticsearch
What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...
kibana, view layer of elasticsearch
What’s Kibana kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on...
Anatomy of ThreadLocal
Design philosophies
iConnect
UI HTML5, AngularJS, BootStrap, REST API, JSON Backend Hadoop core (HDFS), Hive, HBase, MapReduce, Oozie, Pig, Solr
Business Analysis
Purpose of BA 带来一些商业价值(收益) 解决业务痛点
Something about RESTful architect
REST API must be hypertext driver Roy’s interview
Data Structure
Binary Tree A binary tree is a tree in which no node can have more than two children. A property of a binary tree that is sometimes important is that th...
Useful bookmarks
eBooks list of various books Node.js
heavy load web application
Common solutions
tips in as400 IBM Emulator
Toggle crosshair
Something about authentication
It’s annoying to keep on repeating typing same login and password when you access multiple systems within office or for systems in external Internet. There a...
equity trading
Difference between mutal funds and hedge funds
SQL
Differences between not in, not exists , and left join with null
HTTPS/2
concepts
Github page commands notes
404 error for customized domain (such as godday) 404 There is not a GitHub Pages site here. Go to github master branch for gitpages site, manually add CN...
RenMinBi International
RQFII RQFII stands for Renminbi Qualified Foreign Institutional Investor. RQFII was introduced in 2011 to allow qualified foreign institutional investors to ...
JavaScript tips
includes() vs some()
Docker Errors and Fixes
Docker Errors
Load Balancing
Concepts LVS means Linux Virtual Server, which is one Linux built-in component.
Python
(‘—–Unexpected error:’, <type ‘exceptions.TypeError’>) datetime.datetime.now()
Storage Management
RAID RAID is Reductant Array Independent Disk,
CI and CD
Concepts
XA Transactions in 2PC
Description
Setup Git in Mint Linux
How to setup Git in Mint Linux =================================================
Database sharding
DB sharding in YHD
Microservices vs. SOA
Microservice Services are organized around capabilities, e.g., user interface front-end, recommendation, logistics, billing, etc. Services are small in ...
Java Class Loader
Codecache The maximum size of the code cache is set via the -XX:ReservedCodeCacheSize=N flag (where N is the default just mentioned for the particular com...