If you are a manager, I am sure you would have had similar conversations either in your 1:1 or during the interview. If you are considering becoming one, likely, you have asked those questions to someone or yourself. I have been working in this company for X years. …

Jupiter notebook brings a number of benefits in data engineering development: An interactive dev environment Develop with mocked data, and translating code into package code or tests profile, tune and understand your spark job interactively 1. Pre-requisite Install jupyter Install load spark lib Add your virtual environment into your notebook 2. Start your Jupiter run: jupyter notebook First, we need to locate your pyspark path with findspark

Event sourcing is not a new concept. In the ear of the serverless architecture world, events trigger nearly everything in lambda. If you had worked with worker pattern or actor pattern, you would find lambda are just another actor/worker but acting at the service level. The Event Sources: You can find…

a success (fun) story of akka.remote and Kinesis Sample Code First: https://github.com/zdjohn/akka-remote-kcl-exapmle A very brief intro of Kinesis Kinesis Stream is used to collect and process large streams of data records in real time. Streaming Data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and…

When you using async code with akka.net, you will come across this blog post. (https://petabridge.com/blog/akkadotnet-async-actors-using-pipeto/) And recently i ran into a problem, later turned out to be a really interesting one. So here is this blog. A simple overview of the problem In sample code I have created: A caller actor, which make (fake) API calls. …

I am using elasticsearch for the past year in a number of projects. It got me thinking a lot of maintaining data. And, general ways I handling data now and past. In this post, I will talk about time-based index. …

In the past year, at my work, we have used hangfire in production for a number of projects, however only recently we updated a to 1.5.7. The 1.5.0 is a big release. I feel i am really late to the party, but (Oh, Boy!), q⁰^p, So many fruity features. …