Published in Distributed Systems Engineering·PinnedSpatial Partitioned RDD using KD Tree in SparkToday, we plan to cover a use case of the KD tree in the Big Data Framework. Reference Publications: The content I am presenting here is taken from “Parallel Algorithms for Constructing Range and Nearest-Neighbor Searching Data Structures”. “Spatial data management in apache spark: the GeoSpark perspective and beyond” Terminology: Big Data refers…Apache Spark8 min read
Published in Cloud Engineering·PinnedAWS: Running Flink Application on Kinesis Data Analytics(KDA)- Part 1Update [05/12/2020]: Amazon Kinesis Data Analytics(KDA) now supports Apache Flink v1.11. So please skip the Flink 1.8.2 workaround mentioned in this article. If you are here, you might have already heard of apache-flink. It is primarily used for distributed stream processing/aggregation. …Flink5 min read
Published in Javarevisited·PinnedBuild a Tiny Compiler in JavaAre you googling the questions “How to create a compiler in Java?”, “Tiny compiler in Java?”, “AST to Java bytecode”. Then you are in the right place. The word Tiny is subjective. But yeah, the code is simple enough to understand the end-to-end flow of compiler development. Talk is cheap. Show me the code! There you go…Compilers4 min read
Published in Distributed Systems Engineering·4 days agoJRAFT RheaKV — Example (Part 4)This will be a short session detailing how the RheaKV server and client are used. We have seen how the RheaKV client is structured. We are digging into how the RheaKV server is working. So far, so good. …Rheakv4 min read
Published in Distributed Systems Engineering·5 days agoDistributed Computing from a different dimensionLinked List Musical chair Sleep Sort RAFT Partially Synchronous SystemCloud Computing1 min readDistributed Computing from a different dimensionLinked ListMusical chairSleep SortRAFTPartially Synchronous System----
Published in Distributed Systems Engineering·5 days agoJRAFT RheaKV — Server (Part 3)So far we have covered the client-side. Today we will be looking into the server classes. So, we were invoking functions in the RheaKVRpcService (more acting like Feign client in spring-boot). The server side starts from KVCommandProcessor. Project StructureRheakv5 min read
Published in Distributed Systems Engineering·May 14JRAFT RheaKV — Client (Part 2)Picking up from where we left. This session will cover the rest of the items in the workflow, namely PD, RegionRouteTable & failover. PlacementDriverClient.javaRheakv9 min read
Published in Distributed Systems Engineering·May 13JRAFT RheaKV — Client (Part 1)Hey all, today we will look into RheaKV a project using RAFT. Previously we looked into mini-redis a project using tokio-rs. Since the project is a bit bigger than mini-redis, we will try creating a simpler version of rheaKV called rheakv-lite, and migrate relevant things one by one, to make…Rheakv7 min read
Published in Go Rust·May 8Rust Day 18: Number of 1 BitToday, I moved out to a new place. The entire day was spent mostly shifting. Couldn’t focus much on the DB side of things. Anyways, completing the LC streak. This is a well-known problem called hamming distance. Problem: Number of 1 Bits - LeetCode Write a function that takes an unsigned integer and returns the number of '1' bits it has (also known as the Hamming…leetcode.com Logic: The above logic is used in the Fenwick tree.Rust2 min read
Published in Go Rust·May 6Rust Day 17: Min StackThis is an interesting problem. Here we will be implementing a MinStack struct. Taking a step back, I started this rust journey 17 days back. Back then, rust was a bit hard to learn, and understand. But then, day by day, I progressed to read rust code, solve LC using…Rust2 min read