@jeqo

Looking for alternative ways to reproduce time-based conditions in Kafka Streams applications —e.g. if you’re doing some sort of join or windowing based on time— I ended up creating a CLI tool to support a couple of features: Record events from topics, including their timestamps and gap Replay events, including waiting periods between them SQLite is used a storage for recorded events, so events can be generated, updated, tweaked using SQL.

Kafka Streams is in many ways governed by the concept of time. For instance, as soon as stateful operations are used, the event-time drives how events are grouped, joined, and emitted. Stream-time is the concept within Kafka Streams representing the largest timestamp seen by the the stream application (per-partition). In comparison with wall-clock time (i.e. system time) — at the execution of an application — stream-time is driven by the data seen by the application. This ensures that the results produced by a Kafka Streams application are reproducible. One nuance of stream-time is that it needs incoming events to “tick”. This could represent an issue for events that are sparse in time, and we expect results to be produced more often (e.g. windows to be closed and emit, punctiation to be calculated). This is a known issue, and there are some proposals to overcome it in certain parts of the framework, e.g. KIP-424. This post covers a proof-of-concept instrumenting producers to emit contol messages to advance stream time.

Kafka quotas have been around for a while since initial versions of the project — though not necessarily being enabled in most deployments that I have seen. This post shares some thoughts on how to start adopting quotas and gives some practical advice, and a bit of the history of quotas in the Kafka project.

Shipping CLI binaries with Java hasn’t been the most user-friendly experience. Java is required to be installed on the client-side, starting Java applications (e.g. executable JAR) tend to be slower than to binary applications. GraalVM, and specifically native-image tooling, is aimed to tackle most of these issues with Java by enable building native binaries from Java applications. Even though this has been supported for a while now, reflection and other practices require additional configurations that make this process either unsupported or very cumbersome to implement. With the arrival of new frameworks that target the benefits of GraalVM, like Micronaut and Quarkus, it started to be possible and simpler to implement applications that included Kafka clients, and package them as native binaries. This post is going to explore the steps to package vanilla Kafka client applications —i.e. no framework— as native binaries.

Kafka topic partitions are replicated across brokers. Data loss happens when the brokers where replicas are located are unavailable or have fully failed. The worst scenario — and where is no much to do — is when all the brokers fail; then no remediation is possible. Replication allows to increase redundancy so this scenarios is less likely to happen. The following scenarios show different trade-offs that could increase the risk of lossing data:

Originally posted in Confluent Blog Apache Kafka® based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. One result of this is that producers and consumers don’t know about each other, as there is no direct communication between them. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

for a more updated version, check https://jeqo.github.io/posts/2019-03-26-importance-of-distributed-tracing-for-apache-kafka-based-applications/ Tracing is one of the hardest time in integration or microservice development: knowing how a request impact your different components, and if your components have behave as expected. This could be fairly easy if we have monolith where we have one database and with some queries or checking one log file you can validate everything went well. Once you introduce distributed components and asynchronous communication this starts to get more complex and tedious.

One of the most important features from Apache Kafka is how it manages Multiple Consumers. Each consumer group has a current offset, that determine at what point in a topic this consumer group has consume messages. So, each consumer group can manage its offset independently, by partition. This offers the possibility to rollback in time and reprocess messages from the beginning of a topic and regenerate the current status of the system. But how to do it (programmatically)?

In this post I will show how to use Docker containers to create and scale a Kafka cluster, and also how to create, scale and move topics inside the cluster.

Warming up before AMIS 25th Conference event where I will be presenting with my friend and colleague Arturo Viveros (@gugalnikov) about Oracle SOA Suite provisioning, I want to share some practices that help us to provide Oracle Database instances between developers and improve our productivity.

Ansible is an automation tool that is recognized for be simple and powerful at the same time. From my experience, I can say this is mainly because of its scripting language: YAML, and its agentless architecture.

After find some limitations on building SOA Docker image using Dockerfile (as volume access, default size image) I researched on how to improve building process and I found Packer (from the same guy that creates Vagrant).

I decided to implement a naive integration between Java EE applications and RxJava/Kafka/Avro, to publish and subscribe to events. You can go directly to that code, or check my approach:

Yesterday, I released a new version (0.2.0) of my Oracle Fusion Middleware Cookbook hosted on Chef Supermarket

Cool news came from Oracle a couple of weeks ago: Oracle WebLogic Server is now supported on Docker!.

In this post, I will show how to create an AWS EC2 Instance with an Oracle BPM 12c Quickstart Domain created. And I will use previous post for related tasks.

Provisioning tools have change the way we create software environments: How much time we spend installing OS, databases, configuring platforms, applications? Now you can translate this steps into code, getting the software development benefits and challenges into infrastructure: versioning, reuse, continuous improvement.

I was creating AWS EC2 instances to install Oracle Fusion Middleware products, and I found an issue: How to download Oracle’s installers if I want to use installers on several instances? This could consume a lot of network bandwith and I want to make this process repeatable, so I don’t want to wait 1 hour each installation only downloading files.

I’ve had the opportunity to present a review of this book Applied SOA Patterns on the Oracle Platform.

I have some months working with Vagrant and I think it’s owesome! Integration with almost all kind of virtualization platforms: VirtalBox, VMware, Docker, AWS EC2, Hyper-V and so on. Also it’s able to use differente Provisioners: Chef, Puppet, bash, Docker, Ansible. I really like it.