The Aussie NiFi Ninja

A blog for cyber security data engineering aficionados

February 24, 2018 / by

Hi there NiFi, this is Teams...

In the previous post of this series we saw how easily you can use Apache NiFi to easily link between legacy applications and Microsoft Teams.

In this new post of the series we will demonstrate how to utilise NiFi to respond to interactions originating from Microsoft Teams.

Prerequisites

To complete this task you will need to fulfil the following prerequisites:

  • A working deployment of Apache NiFi;
    • Preferably the deployment should be publicly reachable from the Internet, so make sure you configure it to be secure.
    • If an Internet reachable instance cannot be arranged, you can still utilise an API building service such as nGrok to act as an intermediary between Teams and your NiFi instance.
  • An Office 365 license with access to Microsoft Teams (e.g. E3 trial);

Getting started

In the previous post we mentioned that Teams has two different types of webhooks:

  • Outgoing webhooks
  • Incoming webhooks (aka Incoming Webhook Connector)

In this exercise we will be using Outgoing webhooks. Outgoing webhooks enable Teams to call a web endpoint every time a chat message is sent to the “bot”. Once the bot receives the message it may respond to it using one of the available payload options.

Some useful information first…

Before we proceed with the instructions it may be helpful to remind the readers about the core difference between ListenHttp (the processor we used for testing) and HandleHttpRequest (the processor we are about to use).

ListenHttp is essentially an ingestion processor, it allows NiFi to act as an HTTP server, generating flowfile for each of the responses and returning to the client a configurable status. If it fails (e.g. no disk space) it then returns an error message.

HandleHttpRequest and its twin processor, HandleHttpResponse are completely different beasts. When used in tandem, both processors allow a data flow designer to create web APIs using NiFi. Yes. That is what you just read:

Web APIs using NiFi.

As the names suggest, HandleHttpRequest manages the ingestion of requests, creating a “web session” within NiFi that is used to map a particular flowfile to a particular request.

But what happens to that request? Well… the sky is the limit. You can do whatever you want. And as usual, if NiFi cannot do it out of the box, you can always “cheat” and use ExecuteScript to call one of your favourite scripting languages.

In this demo we demonstrate a simple flow that allow NiFi to respond to three questions.

Setting up the Outgoing Webhook

Step 1. Using the Team interface, click the team you want the bot to respond to, click the “…“ and navigate to the Apps tab.

Step 2. On the lower right corner you will see “Create an outgoing webhook”.

Step 3. Populate the details required.

(Please note Teams requires a HTTPS endpoint so you will need to configure NiFi to handle TLS connections or to use a reverse proxy to do so. We leave this out of this post so we don’t get distracted)

Step 4. Copy the token.

By this stage your Teams integration should be basically complete. All you need now is to configure NiFi to handle the data.

Integrating NiFi

Step 5. Drag an HandleHttpRequest processor to the canvas and configure it to match your URL (or reverse proxy settings in case you used one).

Screenshot 5

Step 6. As part of the configuration of HandleHttpRequest you will also need to setup a HTTP Context Map controller (This controller is the “glue” between the Request and the Reponse).

Feel free to adopt the default values.

Screenshot 6

Step 7. Drag a RouteOnAttribute processor to filter requests so that only your webhook requests get diverted for processing.

In my case I use the domain as filter but you can use any attribute.

Step 8. Because Outgoing webhook payloads are just a JSON payload we use the incredible EvaluateJsonPath processor to map one particular part of the payload $.text into an attribute.

Screenshot 7 Screenshot 8

Step 9. We then use the extracted JSON element to route to each of our possible responses.

Screenshot 9

Note that although the matching in place is quite simplistic, nothing prevents you from using more complex strategies to handle the messages received.

Step 10. As with MSFT Teams incoming webhooks we discussed in our previous blog post, the responses are just basic JSON. We replace the content of the flowfile with whatever we want to write back to the originating user…

Screenshot 10

Step 11. We finally return the payload to the HandleHttpResponse processor. Completing a flow that looks like this:

Screenshot 11

Our bot is ready to rock!

Testing the integrations

Test the Outgoing Webhook by navigating back to Teams and sending a message to your bot using @metions or private messages. After a few seconds you should see a response being sent back to you by the bot you just created.

Screenshot 12

Closing remarks

The NiFi flow presented above is very simplistic (e.g. ideally you should be doing HMAC validation to authenticate the messages you receive).

Nonetheless it serves as a useful foundation to how to implement a Microsoft Teams bot in Apache NiFi and highlight how the platform can be useful to those teams seeking to streamline how they ingest data, but also, those looking to orchestrate data driven activities.

In fact, with a bit of work around NLP, the bot we demonstrated above could be extended to react to actual commands, like for example being able to respond to a question such as “@NiFi-a-bot. Is the domain ‘www.evil.com’ present in my last week’s proxy logs?”.

For those looking at implementing such concept we recommend they start by reading this great article put together by the legendary Timothy Spann.

If you are interested in knowing more about Apache NiFi, make sure you follow us on twitter.