Last updated at Mon, 06 Nov 2017 21:03:30 GMT

This guest post originally appeared on IBM developerWorks.

Overview

The aim of the article is to cover technical areas on how to load balance Rsyslog TCP/IP traffic from Redhat Server to Logentries (a third party solutions for real time log management, monitoring and analytics) and configuration used for load balancing network traffic flow between the connected servers.

Majority of virtual machine used for testing were provisioned by Softlayer Cloud Service located in Dallas DC data centers.

Netscaler VPX Standard appliance was also provisioned in Softlayer with a bandwidth of 200Mbps.

Server Architecture

pic

Configure Rsyslog TCP/IP traffic connection in Redhat 6 Linux Server

In Red Hat Enterprise Linux 3/4/5, the default system log tool is Syslogd which is provided by package Sysklogd, but since Red Hat Enterprise Linux 6, the Rsyslogd became the default.

Rsyslog package is also provided since Red Hat Enterprise Linux 5.2. Red Hat Enterprise Linux 6 edition with default Rsyslog package were used for testing.

However users can upgrade Rsyslog to a newer/ higher version. Configuring Rsyslog to forward traffic TCP data to remote location LogEntries using ( Netscaler Load Balancer) requires configuring a file.

Step:

  1. Login into Redhat Linux Server as root
  2. Stop Rsyslog service >> service rsyslog stop (By default rsyslog package should be installed on Redhat 6 server)
  3. Locate Rsyslog configuration file and then run command to edit file >> vi /etc/rsyslog.conf
  4. Edit configuration file as the image shown below
  5. Close and save /etc/rsyslog.conf


Configure Netscaler Load Balancer Network IP’s

Netscaler routes data between its network via Virtual IP and subnet IP address.

There are 3 Network IP required for Netscaler – Netscaler IP (NIP), subnet IP (SNIP) and virtual IP (VIP).

a) Virtual IP (VIP) >> Front end IP address that receives TCP traffic from Redhat server (rsyslog)
b) Subnet IP (SNIP) >> Routes data to Datahub server from Netscaler and within Netscaler appliances
c) Netscaler IP (NIP) >> Administer Netscaler configuration

Performance test was conducted with TCP protocol as the framework for sending message between servers.

Basically we are sending traffic from Redhat 6 Linux machine using rsyslog as forwarding client to Netscaler which in turns send data to LogEntries UI via Datahub.

We utilize NetScaler virtual IP as the target location to forward rsyslog log data through TCP protocols on a specified port.

Netscaler route packet data between Datahub server and Redhat 6 server using subnet IP (SNIP) from Netscaler network’s IP.

For Netscaler Load balancing method we use least connection without any persistence in network traffic with two Datahub service running on TCP, port 10000 and bound to the Virtual IP.

Netscaler configuration mode include – fast ramp, edge configuration, layer 3 mode, use subnet, client side keep alive and path mtu discovery.

Netscaler network IP’s address is required for load balance TCP connection. Contact your network service provider/administrator to setup Netscaler appliances with the appropriate network IP’s.

When using firewall in Softlayer VLAN make sure to enable ICMP protocol and the port used by NetScaler in your firewall settings.

Step:

  • Login to Netscaler admin page as root (see image below for details)

  • Check that Netscaler network IP’s are enabled on the admin page UI >> click configuration tab — system — network — IP’s

  • Configure Netscaler settings by selecting modes to the following (fast ramp, edge configuration, layer 3 mode, use subnet, client side keep alive, direct route advertisement, static route advertisement & path MTU discovery >> click configuration tab — system — settings — configure modes

Netscaler3

Configure Netscaler to use Datahub Server

Setting up a connection bridge between Netscaler load balancer servers and Datahub servers, involves creating a virtual server on Netscaler appliances with a Virtual IP (VIP ) and port and then bind Datahub service to the virtual servers.

Datahub service creates a connection pipe to the Datahub servers on port 10000, and users can bind as many Datahub service to Netscaler virtual server.

In other words users can bind as many Datahub servers to Netscaler virtual server. Users must make sure that Netscaler network IP’s and virtual server are enabled on the Netscaler Administrator UI page.

For distribution of network traffic, least Connection will be the preferred method for load balancing.

Netscaler Datahub service binding

Step:

  • Add a new datahub server in Netscaler >> click configuration tab — traffic management — load balancing — servers

  • Connect Datahub servers to a Netscaler service >> click configuration tab — traffic management tabs — load balancing — service

  • Under the services tab, add Datahub server health monitoring

  • Under the services tab, confirm Datahub service is now running on Netscaler

  • Modify Datahub service to Netscaler virtual servers to use the below settings >> click configuration tab — traffic management tab — load balancing — virtual servers

Basic settings  >> VIP address, specific port 80, redirection mode IP based, TCP protocol

Virtual service binding settings >> bind 2 Datahub service to Virtual IP (VIP)

Profiles configuration settings >> TCP.LAN profiles on Virtual IP (VIP)

Traffic settings – Client time-out 9000 sec, layer 2 parameters OFF, ICMP virtual server response

Configure Redhat 6 Server as a log forwarding client (Rsyslog) to Netscaler

The following procedure will set up Redhat server as a log forwarding agent to Netscaler appliances in Softlayer cloud by using rsyslog deafult package application and a configuration file to distribute traffic to Logentries cloud solution over the internet.

Step:

  1. Login to Redhat 6 Linux Server as root
  2. Stop Rsyslog service >> service rsyslog stop
  3. Change directory to /etc/rsyslog.d
  4. Create a configuration files with .conf extension name in the path /etc/rsyslog.d/  >> e.g netscaler.conf
  5. Edit the netscaler.conf file as shown in the image below

netscaler.conf file

  1. Working directory used by rsyslog
  2. Hostname of local machine – Redhat Server
  3. Rsyslog definition to monitor a specific log
  4. Interval polling for logs been followed by rsyslog
  5. Hostname template used for multiple log forwarding to remote location
  6. Rsyslog queue setting
  7. Log tagging to uniquely identify logs been followed by rsyslog >> Need to configure tags in LE UI dashboard
  8. Netscaler Virtual IP (VIP) address and port 80 used for front end connection to Netscaler application deployed in the Softlayer Cloud Services

Test Network Connection ( Redhat Server >> Netscaler >> Datahub Server)

Testing the different connection between the servers can be done using a ping request to check if servers is reachable.

Ping Test

  • Ping Netscaler Virtual IP, Subnet IP and Host IP from Redhat Server
  • Ping Netscaler Virtual IP, Subnet IP and Host IP from Datahub Server
  • Ping Both Redhat Server IP and Datahub Server IP from Netscaler application

Send TCP message from Redhat Server to Datahub Server on port 10000 using telnet

Verify if Redhat Server is sending data to Netscaler with tcp dump, netcat & telnet

  • telnet IP_ADDRESS 10000
  • nc -Z IP_ADDRESS 10000
  • netsat -taupn | grep syslog
  • tcpdump -A dst IP_ADDRESS

Send TCP message from Redhat Server to Logentries Cloud UI via Netscaler

  • Start Rsyslog service >> service rsyslog start
  • Cat a sample file to another file been monitored by Rsyslog — cat sample.log >> /path/to file/location.log