Custom Kibana LDAP authentication with nginx

The default version of Kibana does not come with an LDAP integration, making it difficult to integrate authentication in many organizations that rely on a Windows environment. However, through the use of a reverse proxy (nginx) and some simple Python code, we can get started creating our custom authentication scheme.

This solution is naturally not going to be as extensive as the integrated solution that Elastic has already built. But if you are just looking to authenticate people and tie them to different roles based on AD groups, then this solution should suffice.

Approach

We're going to start by making a tiny web app that handles user authentication and connects to our LDAP server. Furthermore, the web app will also utilize the Elasticsearch rest API to create and update users.

However, we still need a way to authenticate the newly created users with Kibana. Authenticating with Kibana can be done by many means, however in this article we will be leveraging Kibana's header-based authentication which enables the user to sign in using the “basic” authentication scheme.

We then place Kibana behind a reverse proxy, (in this instance nginx), that is capable of validating whether users are signed in to our web application. We then attach specific headers that authenticate every request forwarded to Kibana. This solution is efficient as it enables every user to have a corresponding user account within Elasticsearch, and you get to control the logic for how roles are associated with LDAP.

The simplified diagram below describes the initial login flow:

Web application

We will begin by crafting the web application, as nginx will be leveraging its functionality. For the purpose of this article, the Python-based Flask micro web framework will be utilized as it is a user-friendly option which requires a significantly lesser overhead compared to other alternatives.

The following three endpoints are required:

1. /login page — Enables our users to sign in
2. /authed endpoint — Used by nginx to check if the user is authenticated
3. /logout page — Logout page that matches the logout URL for Kibana, such that Kibana correctly signs out users.

The application will also be tasked with communicating with an LDAP server to propagate the user's input and then updating users within Elasticsearch. For the LDAP functionality, python-ldap will be used.

Authenticating with LDAP

In an ordinary Python dictionary, we define the relationship between roles in Kibana. The dictionary's keys tell us which group a user belongs to in our LDAP directory and what roles they should have in Kibana. The dictionary also has a secondary purpose: it defines some authorization rules. Any users who don't have any of the memberships in LDAP will be rejected.

roles = { 
    "cn=Company.OrdinaryEmployees,dc=company,dc=com": "Standard",
    "cn=Company.ElevatedEmployees,dc=company,dc=com": "Elevated"
}

The login_through_ldap method serves as our way to check if the users can authenticate with the LDAP server — simply pass along the credentials. If anything goes wrong, the LDAP library will raise an exception, which we will handle later. We then create a new bind using another user (this step is entirely optional), to perform the group membership lookup. If everything goes well, we return the connection.

def login_throug_ldap(username: str, password: str):
    conn = ldap.initialize("ldap://ldap-server:389");
    conn.protocol_version = 3
    conn.bind_s(username, password)
    conn.simple_bind_s(os.getenv("LDAP_USER"), os.getenv("LDAP_PASS"))
    return conn 

The get_ldap_memberships method utilizes the previous method. We construct a root distinguished name (DN) DC=company,DC=com from which we will look up users that should be able to authenticate with the application.

Apart from passing a root DN, we also pass the following parameters to search_st:

1. filterstr which serves as the query when looking up users.
2. attrlist takes a list of attributes we would like to be returned.

Lastly, the returned memberships are filtered using the roles dictionary, such that it only includes the memberships we care about, before returning it.

def get_ldap_memberships(username, password) -> list:
    ldap = login_throug_ldap(username, password)
    results = conn.search_st("DC=company,DC=com", scope=2, attrlist=["memberOf"], filterstr=f"(sAMAccountName={username})")
    team_cns = [cn.decode("utf-8").lower() for cn in results[0][1]['memberOf']]
    return [role for id, role in roles.items() if id.lower() in team_cns]

Syncing roles

Being able to authenticate users against our LDAP server is just one piece of the puzzle, and we're still lacking a crucial one: Granting authenticated users access to Kibana.

Below, an authentication header is defined. The credentials consist of a username and an access token that is tied to the user, which is used as the password. This header is used for subsequent requsts when having to update users within Kibana, and as such, the elastic_user should have the appropriate rights to perform this action.

elastic_user = os.getenv("ELASTIC_USER")
elastic_pass = os.getenv("ELASIC_PASS")
elastic_url = os.getenv("ELASTIC_URL")
auth_value = f"{elastic_user}:{elastic_pass}"
auth_headers = {
    "Authorization": "Basic " + base64.b64encode(bytes(auth_value, "utf-8")).decode("utf-8")
}

Using the requests library, we define the flow for updating and creating users within Kibana. We check if the user is already created, if not, we create it.

def create_user(username, roles):
    user_resp = requests.get(f"{elastic_url}/_security/user/{username}", headers=auth_headers)
    if user_resp.status_code == 404:
        create_user(username, roles)

def create_user(username, roles):
    data = {
        "password": md5(bytes(password_salt + username, "utf-8")).hexdigest(),
        "roles": roles,
        "full_name": "John Doe",
        "email": f"{username}@company.com"
    }
    resp = requests.post(f"{elastic_url}/_security/user/{username}", json=data, headers=auth_headers)

You may notice that we need to set a password when creating a user. This step is paramount, as the password used here will also be used when constructing the authorization header, which authenticates the user's requests with Kibana.

Ultimately, this password cannot be random. One option could be to use the user's password when they log in to our web application. However, to make things easier, we will use a pattern that is made up of a predetermined salt and the user's username, and then hashed with md5. This pattern means that our login is predictable and the user's authorization header can be recreated whenever they successfully log in.

Login page

Now that we can authenticate users against the LDAP server and synchronize them with Kibana, we just need a frontend that can authenticate users.

We start by enabling the session within our flask application:

@app.before_request
def before_request():
    session.modified = True

We create three methods: A login method responsible for displaying the login page, a doLogin method that validates the entered credentials are created and a logout method that destroys the session.

The login method is the initial entry point for every user and responds to GET requests. In this instance, a flask template is returned that shows a login dialog to the visiting user (the HTML is omitted from this article, but it should be fairly simple to create). When the user attempts to sign in, we respond with the same URL, just using POST instead.

@app.route("/")
def login():
    return render_template("base.html", msg=msg, base_url=base_url)

The do_login method authenticates the user, using the get_ldap_memberships method. Should this method raise an exception, we assume this is due to the user being rejected and redirect them back to the login page.

Otherwise, a list of relevant memberships is returned. In case this list is empty, we redirect the user back with a message saying they don't have access to this application.

The session["username"] variables are assigned to the authenticated user, such that the user is stored for the duration of the session. The create_user method ensures the user has access to Kibana.

@app.route('/', methods=['POST'])
def do_login():
    username = request.form.get('username').strip()
    password = request.form.get('password')

    try: 
        memberships = get_ldap_memberships(username, password)
        if len(memberships) == 0:
            return redirect(base_url + "?msg=denied")
        session["username"] = username
        create_user(username, roles)
    except Exception as e:
        return redirect(base_url + "?msg=invalid")

Finally, a logout method is created that simply removes the username variable from the session and redirects the user back to the login page.

@app.route('/logout')
def logout():
    if "username" not in session:
        return redirect(base_url)
    session.pop("username")
    return redirect(base_url + "?msg=logout")

Authenticated

Nginx needs an endpoint it can query to validate if the user is signed in to our web application. The /authed endpoint satisfies this requirement.

We check if the username is signed in. If not, a 401 is returned, telling nginx that the users is indeed not signed in. Otherwise, we attach an authorization header using the pattern described in the syncing roles. Take note of the name used for the authentication header, as it will be consumed by nginx later on.

@app.route('/authed')
def is_auth():
    if "username" in session:
        resp = Response("Authenticated", 200)
        username = session['username']
        password = md5(bytes(password_salt + username, "utf-8")).hexdigest()
        auth = bytes(username + ":" + password, "utf-8")
        resp.headers.add("X-AUTH", base64.b64encode(auth).decode("utf-8"))
        return resp 
    return Response("Unauthorized", 401)

Configure the reverse proxy

With the web application up and running, only configuring nginx remain.

The use case diagram, below, describes the flow nginx will authenticate each user request and how it handles unauthenticated users.

The flow is relatively simple, and can easily be incorporated thanks to nginx's auth_request directive. Let's examine the configuration:

http {
    upstream web-application {
        server 127.0.0.1:5222; # change port to match your setup
    }

    upstream kibana {
        server 127.0.0.1:5601;
    }

    server {
        location / {
            auth_request /auth-proxy/authed;
            auth_request_set $auth_login $upstream_http_x_auth;
            error_page 401 = @login;

            proxy_set_header Authorization "Basic $auth_login";
            proxy_pass http://kibana/;
        }

        location /api/security/logout {
            return 302 /auth-proxy/logout;
        }

        location @login {
            return 302 /auth-proxy;
        }


        location /auth-proxy {
            proxy_pass http://web-application/;
        }

        location /auth-proxy/authed {
            internal;
            proxy_pass http://web-application/authed;
        }
    }
}

Within the server block, we define how nginx should react to a default request, being forwarded to Kibana.

location / {
    auth_request /auth-proxy/authed;
    auth_request_set $auth_login $upstream_http_x_auth;
    error_page 401 = @login;

    proxy_set_header Authorization "Basic $auth_login";
    proxy_pass http://kibana/;
}

The auth_request directive causes nginx to send an HTTP request to /auth-proxy/authed. If this request receives a 401 or 403 response, nginx redirects us to the error page @login.

If we receive a 200 response, we know the user is signed in, and the web application sends an X-AUTH authorization header with the user's login details.

We can carefully extract this header using the auth_request_set directive, which stores the value of the $upstream_http_x_auth into an internal $auth_login variable.

The proxy_set_header allows us to attach a new authorization header referencing the $auth_login variable before proxying the entire request to Kibana.

Forwarding logout

Since the web application is responsible for handling user sessions, we need to forward Kibana's logout request to it.

Fortunately, this is rather easy, as we can simply replace the /api/security/logout URL that Kibana uses. The URL is instead pointed to the sign-out URL within our web application, as such:

location /api/security/logout {
    return 302 /auth-proxy/logout;
}

Users will now be redirected when they press the sign-out button in Kibana 🥳.

Conclusion time

Congratulations 🎉! If you have configured everything correctly, this setup enables your users to authenticate through LDAP using a small web application. Additionally, it also synchronizes roles with your organization's LDAP memberships, offloading this responsibly to a centralized solution.

While this setup provides a somewhat clean way to authenticate users, there are also some considerations:

• Roles are always updated - Every sign in causes the user's roles to be refreshed; ideal if you only rely on AD groups, however manually attached roles within Kibana are automatically overwritten.
• Request overhead - For every request sent to Kibana, nginx will make a request to the /authed endpoint, to ensure the user is remains authenticated.

Conclusively, even though this setup effectively bridges Kibana with LDAP and streamlines user authentication, it's important to continuously evaluate and enhance the system. By addressing its limitations and expanding its capabilities, you can further improve security, efficiency, and user experience.