The Aussie NiFi Ninja

A blog for cyber security data engineering aficionados

March 25, 2017 / by Andre F de Miranda

Compact Profiles — Can Apache MiNiFi get even leaner?

Apache MiNiFi is a subproject of Apache NiFi, targeted at serving as a complementary that takes NiFi unique data collection approach to the source of the data.

As such MiNiFi strives to achieve a small and lightweight footprint, while still offering NiFi’s data provenance and full chain of custody of information.

The project currently consists of two independent agents:

  • MiNiFi (aka MiNiFi Java) — A Java based agent that shares most of its processors with NiFi’s code base but requires a functional instance of the Java Runtime Environment, and;
  • MiNiFi-CPP — A C++ implementation of a NiFi compatible agent that trades the vast numbers of JAVA processors for the performance offered by C++ applications;

The existence of two MiNiFi implementations represents one of the core qualities of the NiFi project: Flexibility and user empowerment.

By offering two different agents, users may balance the advantages and disadvantages of each of the implementations until they find a solution that fits its needs.

The Man Who Stared at Processes

Due to a number of reasons, my current deployments tend to float around MiNiFi and while the application itself is quite lightweight, I frequently get myself looking at java memory utilization and asking myself, really?!

To be honest, it is not as if MiNiFi is a memory intensive process — it is not — the issue that I have with Java based application is that the JVM itself always seems to be as memory hog.

So I have been playing with the idea of using MiNiFi with minimalist Java deployments such as those offered by JAVA 8 Compact Profiles and what sort of difference those make when running MiNiFi.

It turns out that while Java Embedded and compact profiles do in fact exist, using them is a bit cumbersome, main issue being the general lack of free/opensource binary java distributions built around those technologies.

But fear not! While, the absence of those distributions may look like an impediment, thanks to the work done by Waldemar Kozaczuk, you can now build an openjdk based compact profile in just a few minutes!

The secret is to use openjdk8_compact_profiles_builder and a binary distribution like Azul’s Zulu.

Building the profile is as simple as running

sh build_compact_profile.sh http://cdn.azul.com/zulu/bin/zulu8.19.0.1-jdk8.0.112-linux_x64.tar.gz 3 add_java_beans

Does it work with MiNiFi?

Well… MiNiFi seems to start and continue running without crashes but I am not yet confident it effectively works…

Therefore I strongly recommend you test it extensively before considering using it (specially considering that given compact profiles are not part of the MiNiFi testing environment!).

How much memory is saved?

Now comes the sad side of the story. It is all very subjective and dependent on the number of processors and flow you have, so “why bother”?

Good question!

Maybe you shouldn’t, but it was still worth the challenge… And so that you don’t go away feeling click baited, here’s the output of a vanilla MiNiFi instance (i.e. non-configured) running the generally available OpenJDK

32054 user 20 0 4887100 207516 19924 S 0.3 3.4 0:04.77 java

And here’s the same output when ran with a custom compact3 + Java beans profile built with build_compact_profile.sh

24293 user 20 0 3284264 175144 18852 S 0.0 2.9 0:03.57 java

As you can see both VSZ RSS and %MEM values dropped but are the savings worth the hassle of having to create a custom Java package?

I will leave that to your judgment…