Kafka Emulator CLI: Record and Reply Records considering time-distance

Looking for alternative ways to reproduce time-based conditions in Kafka Streams applications —e.g. if you’re doing some sort of join or windowing based on time— I ended up creating a CLI tool to support a couple of features: Record events from topics, including their timestamps and gap Replay events, including waiting periods between them SQLite is used a storage for recorded events, so events can be generated, updated, tweaked using SQL.
Read more
poc kafka cli tools

Kafka Streams: Tick stream-time with control messages

Kafka Streams is in many ways governed by the concept of time. For instance, as soon as stateful operations are used, the event-time drives how events are grouped, joined, and emitted. Stream-time is the concept within Kafka Streams representing the largest timestamp seen by the the stream application (per-partition). In comparison with wall-clock time (i.e. system time) — at the execution of an application — stream-time is driven by the data seen by the application. This ensures that the results produced by a Kafka Streams application are reproducible. One nuance of stream-time is that it needs incoming events to “tick”. This could represent an issue for events that are sparse in time, and we expect results to be produced more often (e.g. windows to be closed and emit, punctiation to be calculated). This is a known issue, and there are some proposals to overcome it in certain parts of the framework, e.g. KIP-424. This post covers a proof-of-concept instrumenting producers to emit contol messages to advance stream time.
Read more
poc kafka-streams dev

Getting started with Kafka quotas

Kafka quotas have been around for a while since initial versions of the project — though not necessarily being enabled in most deployments that I have seen. This post shares some thoughts on how to start adopting quotas and gives some practical advice, and a bit of the history of quotas in the Kafka project.
Read more
kafka ops

Kafka client applications with GraalVM

Shipping CLI binaries with Java hasn’t been the most user-friendly experience. Java is required to be installed on the client-side, starting Java applications (e.g. executable JAR) tend to be slower than to binary applications. GraalVM, and specifically native-image tooling, is aimed to tackle most of these issues with Java by enable building native binaries from Java applications. Even though this has been supported for a while now, reflection and other practices require additional configurations that make this process either unsupported or very cumbersome to implement. With the arrival of new frameworks that target the benefits of GraalVM, like Micronaut and Quarkus, it started to be possible and simpler to implement applications that included Kafka clients, and package them as native binaries. This post is going to explore the steps to package vanilla Kafka client applications —i.e. no framework— as native binaries.
Read more
kafka cli graalvm dev


Kafka data loss scenarios

Kafka topic partitions are replicated across brokers. Data loss happens when the brokers where replicas are located are unavailable or have fully failed. The worst scenario — and where is no much to do — is when all the brokers fail; then no remediation is possible. Replication allows to increase redundancy so this scenarios is less likely to happen. The following scenarios show different trade-offs that could increase the risk of lossing data:
Read more
kafka durability ops


The Importance of Distributed Tracing for Apache Kafka Based Applications

Originally posted in Confluent Blog Apache Kafka® based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. One result of this is that producers and consumers don’t know about each other, as there is no direct communication between them. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.
Read more
event-driven distributed-tracing observability kafka zipkin


Tracing Kafka applications

for a more updated version, check Tracing is one of the hardest time in integration or microservice development: knowing how a request impact your different components, and if your components have behave as expected. This could be fairly easy if we have monolith where we have one database and with some queries or checking one log file you can validate everything went well. Once you introduce distributed components and asynchronous communication this starts to get more complex and tedious.
Read more
integration observability kafka tracing

Rewind Kafka Consumer Offsets

One of the most important features from Apache Kafka is how it manages Multiple Consumers. Each consumer group has a current offset, that determine at what point in a topic this consumer group has consume messages. So, each consumer group can manage its offset independently, by partition. This offers the possibility to rollback in time and reprocess messages from the beginning of a topic and regenerate the current status of the system. But how to do it (programmatically)?
Read more
integration kafka

Scaling Kafka with Docker Containers

In this post I will show how to use Docker containers to create and scale a Kafka cluster, and also how to create, scale and move topics inside the cluster.
Read more
devops integration kafka docker


Speed up your Oracle Database provisioning with Docker and Ansible

Warming up before AMIS 25th Conference event where I will be presenting with my friend and colleague Arturo Viveros (@gugalnikov) about Oracle SOA Suite provisioning, I want to share some practices that help us to provide Oracle Database instances between developers and improve our productivity.
Read more
devops oracle database ansible docker

Ansible - an agentless provisioning

Ansible is an automation tool that is recognized for be simple and powerful at the same time. From my experience, I can say this is mainly because of its scripting language: YAML, and its agentless architecture.
Read more
devops ansible docker


Oracle SOA Suite 12c Docker Image

After find some limitations on building SOA Docker image using Dockerfile (as volume access, default size image) I researched on how to improve building process and I found Packer (from the same guy that creates Vagrant).
Read more
devops oracle soa docker packer

Integrate Java EE 7 and Kafka using Avro and RxJava

I decided to implement a naive integration between Java EE applications and RxJava/Kafka/Avro, to publish and subscribe to events. You can go directly to that code, or check my approach:
Read more
development integration back-end java ee kafka avro rx

Oracle Fusion Middleware Chef Cookbook - New Release!

Yesterday, I released a new version (0.2.0) of my Oracle Fusion Middleware Cookbook hosted on Chef Supermarket
Read more
devops oracle soa chef

Docker image for Oracle SOA Suite 12c

Cool news came from Oracle a couple of weeks ago: Oracle WebLogic Server is now supported on Docker!.
Read more
devops oracle soa docker


Running Oracle BPM 12c on AWS using Vagrant and Chef

In this post, I will show how to create an AWS EC2 Instance with an Oracle BPM 12c Quickstart Domain created. And I will use previous post for related tasks.
Read more
devops oracle fmw bpm chef vagrant

Chef Cookbook for Oracle Fusion Middleware 12c

Provisioning tools have change the way we create software environments: How much time we spend installing OS, databases, configuring platforms, applications? Now you can translate this steps into code, getting the software development benefits and challenges into infrastructure: versioning, reuse, continuous improvement.
Read more
devops oracle fmw bpm chef

Create a NFS instance on AWS using Vagrant and Chef

I was creating AWS EC2 instances to install Oracle Fusion Middleware products, and I found an issue: How to download Oracle’s installers if I want to use installers on several instances? This could consume a lot of network bandwith and I want to make this process repeatable, so I don’t want to wait 1 hour each installation only downloading files.
Read more
devops nfs aws vagrant chef

Book Review - Applied SOA Patterns on the Oracle Platform

I’ve had the opportunity to present a review of this book Applied SOA Patterns on the Oracle Platform.
Read more
integration oracle soa book review

Vagrant and Chef Quickstart

I have some months working with Vagrant and I think it’s owesome! Integration with almost all kind of virtualization platforms: VirtalBox, VMware, Docker, AWS EC2, Hyper-V and so on. Also it’s able to use differente Provisioners: Chef, Puppet, bash, Docker, Ansible. I really like it.
Read more
Devops vagrant virtualbox