Daily Notes & Snippets

Postgres Isolation Level By Example

2024-01-13T16:06:36+00:00

As engineer, we don’t put much attention on the Postgres isolation level. Most of the time relying on the default isolation level, which is read committed + locking is enough for most cases.

However, in some specific cases, we need to change the isolation level either to repeatable read or serialiazed. Let’s take a look at the example to understand the different between them.

Given a bills table for customer.

CREATE TABLE bills (
    id SERIAL PRIMARY KEY,
    date DATE,
    customer VARCHAR(20),
    total NUMERIC(10,2)
);

Read Committed

Only show committed row.

/* session 1 */                                 /* session 2 */
BEGIN;                                          BEGIN; 

INSERT INTO bills (date, customer, total) 
VALUES ('2024-01-01', 'Agnes', 10);

                                                SELECT * FROM bills;
                                                id | date | customer | total
                                                ----+------+----------+-------
                                                (0 rows)

COMMIT;

                                                SELECT * FROM bills;
                                                id |    date    | customer | total
                                                ----+------------+----------+-------
                                                1 | 2024-01-01 | Agnes    | 10.00
                                                (1 row)

Repeatable Read

Repeatable read uses snapshot from Postgres MVCC. This isolation prevent us from phantom read & non repeatable read.

Add a seed data.

INSERT INTO bills (date, customer, total) VALUES ('2024-01-01', 'John', 100);
INSERT INTO bills (date, customer, total) VALUES ('2024-01-01', 'Agnes', 10);

Read Committed Isolation

/* session 1 */                                 /* session 2 */
BEGIN;                                          BEGIN; 

INSERT INTO bills (date, customer, total) 
VALUES ('2024-01-01', 'Tono', 50);
COMMIT;

                                                SELECT * FROM bills;
                                                id |    date    | customer | total
                                                ----+------------+----------+--------
                                                1 | 2024-01-01 | John     | 100.00
                                                2 | 2024-01-01 | Agnes    |  10.00
                                                3 | 2024-01-01 | Tono     |  50.00
                                                (3 rows)                

Customer Tono that is created from session 1, is visible to session 2 because of repeatable read.

Repeatable Read Isolation

/* session 1 */                                 /* session 2 */
BEGIN TRANSACTION ISOLATION                     BEGIN TRANSACTION ISOLATION
LEVEL REPEATABLE READ;                          LEVEL REPEATABLE READ;

INSERT INTO bills (date, customer, total) 
VALUES ('2024-01-01', 'Tono', 50);
COMMIT;

                                                SELECT * FROM bills;
                                                id |    date    | customer | total
                                                ----+------------+----------+--------
                                                1 | 2024-01-01 | John     | 100.00
                                                2 | 2024-01-01 | Agnes    |  10.00
                                                (2 rows)

Customer Tono that is created from session 1 is not visible to session 2. Because session 2 runs on his own snapshot and non repeatable read.

Serializable

The Serializable isolation level provides the strictest transaction isolation. Postgres will execute the session one by one in serialize. To guarantee true serializability PostgreSQL uses predicate locking.

Example case is when customer’s next month bill must be created from last month bill. This case require serializable isolation to prevent multiple bill for the same customer being created in the next month.

Add a seed data.

INSERT INTO bills (date, customer, total) VALUES ('2024-01-01', 'John', 100);
INSERT INTO bills (date, customer, total) VALUES ('2024-01-01', 'Agnes', 10);

Repeatable Read Isolation

/* session 1 */                                 /* session 2 */
BEGIN TRANSACTION ISOLATION                     BEGIN TRANSACTION ISOLATION
LEVEL REPEATABLE READ;                          LEVEL REPEATABLE READ;

INSERT INTO bills (date, customer, total) 
(
    SELECT 
        date + INTERVAL '1 MONTH', 
        customer, 
        total+100 
    FROM bills 
    WHERE customer='John' 
    ORDER BY date DESC LIMIT 1
);

                                                INSERT INTO bills (date, customer, total) 
                                                (
                                                    SELECT 
                                                        date + INTERVAL '1 MONTH', 
                                                        customer, 
                                                        total+100 
                                                    FROM bills 
                                                    WHERE customer='John' 
                                                    ORDER BY date DESC LIMIT 1
                                                );

COMMIT;                                         COMMIT;

SELECT * FROM bills;
 id |    date    | customer | total
----+------------+----------+--------
  1 | 2024-01-01 | John     | 100.00
  2 | 2024-01-01 | Agnes    |  10.00
  3 | 2024-02-01 | John     | 200.00
  4 | 2024-02-01 | John     | 200.00
(4 rows)

                                                SELECT * FROM bills;
                                                id |    date    | customer | total
                                                ----+------------+----------+--------
                                                1 | 2024-01-01 | John     | 100.00
                                                2 | 2024-01-01 | Agnes    |  10.00
                                                3 | 2024-02-01 | John     | 200.00
                                                4 | 2024-02-01 | John     | 200.00
                                                (4 rows)

Session 1 & Session 2 commits are accepted. This cause an issue for John, because on Feb 2024, he has 2 bills to pay.

Serializable

/* session 1 */                                 /* session 2 */
BEGIN TRANSACTION ISOLATION                     BEGIN TRANSACTION ISOLATION
LEVEL SERIALIZABLE;                             LEVEL SERIALIZABLE;

INSERT INTO bills (date, customer, total) 
(
    SELECT 
        date + INTERVAL '1 MONTH', 
        customer, 
        total+100 
    FROM bills 
    WHERE customer='John' 
    ORDER BY date DESC LIMIT 1
);

                                                INSERT INTO bills (date, customer, total) 
                                                (
                                                    SELECT 
                                                        date + INTERVAL '1 MONTH', 
                                                        customer, 
                                                        total+100 
                                                    FROM bills 
                                                    WHERE customer='John' 
                                                    ORDER BY date DESC LIMIT 1
                                                );

COMMIT;                                         COMMIT;

                                                ERROR:  could not serialize access due to read/write dependencies among transactions
                                                DETAIL:  Reason code: Canceled on identification as a pivot, during commit attempt.
                                                HINT:  The transaction might succeed if retried.

SELECT * FROM bills;
 id |    date    | customer | total
----+------------+----------+--------
  1 | 2024-01-01 | John     | 100.00
  2 | 2024-01-01 | Agnes    |  10.00
  3 | 2024-02-01 | John     | 200.00
(3 rows)

                                                SELECT * FROM bills;
                                                id |    date    | customer | total
                                                ----+------------+----------+--------
                                                1 | 2024-01-01 | John     | 100.00
                                                2 | 2024-01-01 | Agnes    |  10.00
                                                3 | 2024-02-01 | John     | 200.00
                                                (3 rows)

When transaction isolation level is serialized, then the commit execution will be done in order. Session 1 commit first & success. Session 2 get an error because the data already changes in Session 1.

Lesson Learned

Lesson learned during my experience working with Postgres transaction in the last 7 years.

Only changes the isolation level if you know what you are doing. Otherwise, keep it as default & apply row/table lock when you need strong consistency.
The default isolation level in Postgres is read committed. For people come from MySQL, the mental model needs to be changed, since MySQL default isolation level is repeatable read.
When setting the isolation level to repeatable read or serialized, handle the retrial in your code.
Keep monitor your query performance, long running transaction & deadlock.

Reference

Big Endian vs Little Endian

2024-01-04T16:06:36+00:00

I get a bug today when encoding some message from my IoT device. After reading their doc, turns out they send message in little endian order instead of big endian order.

I write a simple go code to simulate the different between big endian and little endian. Big endian sends most significant bytes from right to left. Little endian sends most least bytes from right to left.

package main

import (
	"encoding/binary"
	"encoding/hex"
	"fmt"
)

func main() {
	n := 325

	little := make([]byte, 2)
	binary.LittleEndian.PutUint16(little, uint16(n))
	fmt.Println("little hex", hex.EncodeToString(little))
	fmt.Printf("little binary %08b\n", little)

	fmt.Println("===========")

	big := make([]byte, 2)
	binary.BigEndian.PutUint16(big, uint16(n))
	fmt.Println("big hex", hex.EncodeToString(big))
	fmt.Printf("big binary %08b\n", big)
}

Output

little hex 4501
little binary [01000101 00000001]
===========
big hex 0145
big binary [00000001 01000101]

From what I understand there is no hard rule on using either one. But big endian is more common to be used in the network protocol. Meanwhile little endian is more dominant in modern computer architectures processors (x86, ARM), IoT device & embedded system.

Int64 Protobuf is converted to string in JSON, Why?

2022-09-18T16:06:36+00:00

Today I learn that int64 protobuf is converted to string when we encode it to JSON. This cause some issue on the javascript client side when we assume that the value we receive already in int64 instead of string.

Here is a snippet of the code.

syntax = "proto3";

option go_package="wilianto.com/proto-int64/order";

message Money {
    string currency = 1;
    int64 amount = 2;
    int32 precision = 3;
}

package main

import (
	"fmt"

	"google.golang.org/protobuf/encoding/protojson"
	"wilianto.com/proto-int64/order"
)

func main() {
	money := order.Money{
		Currency:  "SGD",
		Precision: 2,
		Amount:    10050,
	}
	moneyJSON, _ := protojson.Marshal(&money)
	fmt.Println(string(moneyJSON))
	// output
	// {"currency":"SGD","amount":"10050","precision":2}
}

After reading again the protobuf to JSON mapping doc here, it’s expected that int64, fixed64 & uint64 are gonna be converted to decimal string. However, it does not work that way for int32, fixed32 & uint32.

But why 64-bit number is serialized to string?

Based from what I understand, the issue is not from the protobuf. Instead the main reason is on the JSON protocol itself.

From JSON RFC-8259, we can learn that most of common software implement IEEE 754 double precision floating point to handle such a precision number. It works well to handle integers with range [-(2^53)+1, (2^53)-1] but not with the long PI number for example (3.141592653589793238462643383279). It may indicate potential interoperability problems.

So, to keep the precision between different system/software implementation, JSON with 64-bit precision should be written as string.

Reference

[In Bahasa] Mengurus Visa Kerja Spanyol

2019-12-05T16:06:36+00:00

Hola!

Di artikel ini saya mau berbagi tentang pengalaman saya mengurus visa kerja Spanyol. Tipe visa yang saya urus adalah High Qualified Profesional visa atau yang disingkat HQP. Syarat dapat visa HQP ini ga sulit kok, persyaratan utamanya adalah memiliki bidang pendidikan yang sesuai dengan bidang kerja. Untuk kasus saya sebagai Software Engineer, saya memerlukan ijazah S1 teknik informatika, sistem informasi, atau yang setara.

Pengurusan visa ini memakan waktu cukup lama untuk kasus saya, sekitar 4 bulan. Pengurusan dimulai pada awal April 2019, saat itu saya menerima offer letter dari sebuah perusahaan Spanyol di Barcelona. Secara keseluruhan prosesnya dibagi menjadi 3 tahap. Mari kita kupas satu per satu prosesnya.

Pertama, persiapan dokumen-dokumen yang diperlukan. Untuk kasus saya dokumen yang diperlukan adalah:

Ijazah terakhir
Surat Keterangan Catatan Kepolisian (wajib dari mabes polri! bukan dari polda, polres, polsek)
Curriculum vitae dalam bahasa Spanyol
Salinan seluruh halaman passport
Bukti pembayaran pajak di Indonesia (buat yang suka nunggak, bakal repot ya)

Untuk ijazah dan SKCK agar bisa digunakan di Spanyol, kita perlu legalisir di kedutaan besar Spanyol. Berikut prosesnya:

Legalisasi di kemenkumham
Legalisasi di kemenlu.
Setelah itu terjemahkan halaman utama & bukti legalisasi ke dalam bahasa Spanyol.
Cetak halaman terjemahan tapi jangan cetak bolak-balik ya!

Setelah semua dokumen siap, bawa seluruh dokumen bahasa Indonesia & terjemahan ke kedutaan besar Spanyol di Indonesia untuk dilegalisir. Proses legalisasi memakan waktu sekitar 2 minggu dari tanggal penerimaan.

Hasil legalisir dari kemenkumham, kemenlu & kedutaan besar Spanyol.

Kedua, setelah seluruh dokumen ready, kirim ke bagian legal perusahaan. Mereka akan proses pengajuan visa kerja di kementerian tenaga kerja di Spanyol. Di proses ini, tugas saya hanya perlu bersabar, karena memakan waktu yang cukup lama (3 minggu atau bahkan ada yang sampai berbulan bulan).

Ketiga, jika kabar bahagia dikirimkan oleh legal perusahaan. Proses berikutnya adalah penerbitan visa di keduataan Spanyol di Indonesia. Berikut adalah dokumen-dokumen yang diperlukan:

2 formulir permohonan visa nasional yang diisi dengan lengkap dan ditandatangani oleh pemohon (di halaman 3 dan 4)
2 pas foto berukuran 3x4 cm, berwarna dan berlatar belakang putih, ditempel di formulir
Paspor yang berlaku minimal 4 bulan serta fotokopi semua halaman
SKCK
Surat keterangan dokter yang baru (masa berlaku 3 bulan), dengan pernyataan bahwa pemohon tidak mengidap penyakit yang berdampak bagi kesehatan masyarakat, sesuai dengan peraturan kesehatan internasional tahun 2005 dalam bahasa Spanyol
Salinan kontrak kerja

Bagian yang cukup sulit di tahap ini adalah mendapatkan tulisan “pemohon tidak mengidap penyakit yang berdampak bagi kesehatan masyarakat, sesuai dengan peraturan kesehatan internasional tahun 2005” di surat keterangan dokter. Saya mencoba beberapa rumah sakit besar di Jakarta untuk mendapatkan tulisan ajaib ini. Namun hanya di RS Fatmawati saya bisa mendapatkannya (saya akan bahas di lain artikel).

Setelah seluruh dokumen ready, bawa ke kedutaan besar Spanyol. Waktu yang diperlukan untuk penerbitan visa sekitar 2 minggu dengan biaya 60 euro.

Seluruh isi artikel ini ditulis berdasarkan pengalaman pribadi. Waktu & dokumen yang diperlukan mungkin akan berbeda-beda. Untuk lebih jelasnya sebaiknya menghubungi kedubes Spanyol di emb.yakarta.sc@maec.es atau +62 21 3142355, mereka sangat responsive dan membantu selama proses pembuatan visa.

Get Unique Values of a CSV Column Using Bash Command

2018-08-09T16:06:36+00:00

Example input CSV file. The file is named transaction.csv

invoice_number,item_id,qty
#001,1,2
#001,2,2
#002,2,2
#003,1,2
#003,5,3
#004,2,2
#003,5,3
#005,2,2

The goal is to get the list of unique invoice_number

#001
#002
#003
#004
#005

Step

Run awk command split by comma (-F), then print the first column. awk -F ',' '{print $1}' transaction.csv
Add sorting with unique params sort -u. Use -r to get reverse order
Final command is awk -F ',' '{print $1}' transaction.csv | sort -u

Reference

More detail about awk command http://tldp.org/LDP/abs/html/awk.html
More detail about sort -u vs sort | uniq performance discussion at stackoverflow thread

Real-time Malicious Login Detector with Apache Flink CEP

2017-11-03T16:06:36+00:00

I was playing with Apache Flink several weeks ago. For exercise, I made a simple rule to detect malicious login pattern with Complex Event Processing (CEP). The rule is very simple, 3x login activities for the same username and IP in a minute. Then for the data source, I use Kafka.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.wilianto.blog.login</groupId>
    <artifactId>cep-malicious-login</artifactId>
    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.10</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka-0.10_2.10</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-cep_2.10</artifactId>
            <version>1.3.2</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>com.wilianto.blog.login.App</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

MaliciousLoginEvent.java

public class MaliciousLoginEvent {
    private String username;
    private String ip;
    private String time;

    //setter
    //getter
} 

The data will be saved in JSON format in the Kafka topic. So I created a deserializer for JSON to MaliciousLoginEvent POJO.

MaliciousLoginDeserializationSchema.java

public class MaliciousLoginDeserializationSchema 
    implements DeserializationSchema<MaliciousLoginEvent> {
    public MaliciousLoginEvent deserialize(byte[] bytes) 
        throws IOException {
        MaliciousLoginEvent maliciousLoginEvent = null;
        try {
            JSONObject jsonData = new JSONObject(new String(bytes));
            maliciousLoginEvent = new MaliciousLoginEvent();
            maliciousLoginEvent.setUsername(jsonData.getString("username"));
            maliciousLoginEvent.setIp(jsonData.getString("ip"));
            maliciousLoginEvent.setTime(jsonData.getString("time"));
        } catch (JSONException e) {
            System.out.println("Unable to deserialize Malicious Login Kafka Message");
        } finally {
            return maliciousLoginEvent;
        }
    }

    public boolean isEndOfStream(MaliciousLoginEvent maliciousLoginEvent) {
        return false;
    }

    public TypeInformation<MaliciousLoginEvent> getProducedType() {
        return TypeExtractor.getForClass(MaliciousLoginEvent.class);
    }
}

The main code does:

Add streaming data source from Kafka topic (FailLogin)
Create a malicious login pattern (3x failed with the same username & IP in a minute)
Match the pattern with coming data
Add sink to print out the malicious pattern

App.java

public class App {
    public static void main(String[] args) throws Exception {
        //set kafka properties
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "localhost:9092");

        //set stream execution environtment
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //stream kafka
        FlinkKafkaConsumer010<MaliciousLoginEvent> flinkKafkaConsumer = new FlinkKafkaConsumer010<MaliciousLoginEvent>("FailLogin", new MaliciousLoginDeserializationSchema(), properties);
        flinkKafkaConsumer.setStartFromLatest();
        DataStream<MaliciousLoginEvent> kafkaInputStream = env.addSource(flinkKafkaConsumer).keyBy("username", "ip");

        //create pattern
        Pattern<MaliciousLoginEvent, ?> pattern = Pattern.<MaliciousLoginEvent>begin("firstAttempt")
                .next("secondAttempt")
                .next("thirdAttempt")
                .within(Time.minutes(1));
        PatternStream<MaliciousLoginEvent> patternStream = CEP.pattern(kafkaInputStream, pattern);

        //add simple sink alarm
        DataStream<String> result = patternStream.select(new PatternSelectFunction<MaliciousLoginEvent, String>() {
            public String select(Map<String, List<MaliciousLoginEvent>> suspects) throws Exception {
                MaliciousLoginEvent firstAttempt = suspects.get("firstAttempt").get(0);
                MaliciousLoginEvent secondAttempt = suspects.get("secondAttempt").get(0);
                MaliciousLoginEvent thirdAttempt = suspects.get("thirdAttempt").get(0);
                String message = String.format("Suspected Login Activity! \n" +
                        "First event: %s \n" +
                        "Second event: %s \n" +
                        "Third event: %s \n", firstAttempt, secondAttempt, thirdAttempt);

                System.out.println(message);

                return message;
            }
        });

        //execute
        env.execute("Mallicious Login Detector");
    }
}

Then for testing purpose, I made a simple ruby code in Sinatra to simulate login request. When login failed, an event will be sent to a Kafka topic in JSON format.

login.rb

require "sinatra"
require "kafka"
require "json"

user = { username: "wilianto", password: "12345678" }

post "/login" do
  username = params["username"]
  password = params["password"]
  ip       = params["ip"] # should be request.ip, just for testing purpose

  if user[:username] == username && user[:password] == password
    "Login success"
  else
    event = {
      username: username,
      ip: ip,
      time: Time.now
    }

    # push to kafka topic FailLogin
    kafka = Kafka.new(
      seed_brokers: ["localhost:9092"]
    )
    kafka.deliver_message(JSON.dump(event), topic: "FailLogin")

    "Login failed"
  end
end

Compile the CEP Flink apps and run the JAR file on Flink. Run the ruby login code and simulate some requests.

Example Request:

curl -X POST \
  http://localhost:4567/login \
  -H 'cache-control: no-cache' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -H 'postman-token: cb6cf5f1-2361-ad2d-4d9b-addd7b71b474' \
  -F username=wilianto \
  -F password=wrongpassword \
  -F ip=127.0.0.1

Example Output:

Suspected Login Activity!
First event: MaliciousLogin[username: wilianto, ip: 127.0.0.1, time: 2017-11-04 18:55:06 +0700]
Second event: MaliciousLogin[username: wilianto, ip: 127.0.0.1, time: 2017-11-04 18:55:12 +0700]
Third event: MaliciousLogin[username: wilianto, ip: 127.0.0.1, time: 2017-11-04 18:55:13 +0700]

Thanks for reading. Feel free to share what’s on your mind in the comment form below.

HTTP Method Name is Case Sensitive

2017-10-31T16:06:36+00:00

Today I learn that HTTP method name in HTTP request must be in upper case.

I’ve been working with web-based application for around 7 years, but the “stupid” thing is I don’t aware that HTTP method does not support lowercase letter. I just realize it today, after stuck in around an hour with my pairing partner. We get a 400 Bad Request.

After searching, we get information about this in RFC 7231. Here is the point we get for HTTP method standard.

HTTP was originally designed to be usable as an interface to
distributed object systems.  The request method was envisioned as
applying semantics to a target resource in much the same way as
invoking a defined method on an identified object would apply
semantics.  The method token is "case-sensitive" because it might be
used as a gateway to object-based systems with "case-sensitive" method
names.

.
.
.

This specification defines a number of standardized methods that are
commonly used in HTTP, as outlined by the following table.  By
convention, standardized methods are defined in "all-uppercase"
US-ASCII letters.

+---------+-------------------------------------------------+-------+
| Method  | Description                                     | Sec.  |
+---------+-------------------------------------------------+-------+
| GET     | Transfer a current representation of the target | 4.3.1 |
|         | resource.                                       |       |
| HEAD    | Same as GET, but only transfer the status line  | 4.3.2 |
|         | and header section.                             |       |
| POST    | Perform resource-specific processing on the     | 4.3.3 |
|         | request payload.                                |       |
| PUT     | Replace all current representations of the      | 4.3.4 |
|         | target resource with the request payload.       |       |
| DELETE  | Remove all current representations of the       | 4.3.5 |
|         | target resource.                                |       |
| CONNECT | Establish a tunnel to the server identified by  | 4.3.6 |
|         | the target resource.                            |       |
| OPTIONS | Describe the communication options for the      | 4.3.7 |
|         | target resource.                                |       |
| TRACE   | Perform a message loop-back test along the path | 4.3.8 |
|         | to the target resource.                         |       |
+---------+-------------------------------------------------+-------+

All general-purpose servers MUST support the methods GET and HEAD.
All other methods are OPTIONAL.