Scalable Logging in Rails using Log4r and graylog

Rails default logger is quite simple.but it becomes quite difficult to debug in production as the app grows big. Imagine  you have 5 different app servers and you have to grep all the log files. Some simple shell scripting could ease the pain but still log file is quite messed up and it still wont help you to analyse  which url returned 500, time taken by the api etc. It would then be great to have all logs in a centralised location easing up the pain of developers. Yes ! There is a gem for that! 🙂

Log4r is a powerful and flexible ruby gem used for logging in ruby inspired by the popular java library log4j. It supports multiple output destinations per log level, custom log levels etc. Checkout the library .

You might also need to check graylog. Graylog is an open source log management solution that can be used to monitor your logs.  It is built on the top of java, mongodb, elasticsearch . To know more about it click here. Graylog also comes with a web interface. Check out the link to see to setup graylog web interface using nginx as a reverse proxy .

I used gem  ‘log4r-gelf‘ along with log4r to sent my logs to graylog. All log4r and  ‘log4r-gelf‘ in your gem file

gem 'log4r'
gem 'log4r-gelf'

and do

bundle install 

2. Now create a yml file log4r.yml.

log4r_config:
 loggers:
 - name: production
 level: INFO
 trace : false
 outputters:
 - production
 - gelf

 - name: development
 level: DEBUG
 trace: true
 outputters:
 - development

 outputters:
 - type: DateFileOutputter
 name: production
 filename: production.log
 dirname: "log"
 formatter:
 date_pattern: '%H:%M:%S'
 pattern: '%d %l: %m '
 type: PatternFormatter

 - type: GelfOutputter
 name: gelf
 gelf_server: "example.graylog.in"
 gelf_port: "12219"
 level: INFO

3. Next in the application.rb

require 'log4r'
require 'log4r/yamlconfigurator'
require 'log4r/outputter/datefileoutputter'
include Log4r

module ChillrApi

 log4r_config = YAML.load_file(File.join(Rails.root, 'config', 'log4r.yml'))
 YamlConfigurator.decode_yaml( log4r_config["log4r_config"] )
 config.logger = Log4r::Logger[Rails.env]

end 

Script to Monitor RabbitMQ Queue Messages

Below is a code I used to monitor rabbitmq queues. We were  using a microservices architecture. In our architecture each services are communicating using a rabbitmq broker. Services tend to publish message to broker and the broker  forwards these messages. One bottle neck is in this communication part.Due to some memory issues the consumers that consumes these messages got hang and the messages in a particular queue went too high .So I decided to write a script to sent an SMS and alert in slack channel whenever the message count becomes greater than the threshold.Note that I have used a gem slack notifier link: https://github.com/stevenosloan/slack-notifier. You could also just curl

 require 'net/http'
 require 'uri'
 require 'json'
 require 'slack-notifier'
 require 'sevak_publisher'

 CONFIG = YAML.load_file(File.join(__dir__, 'rabbitmq_config.yml'))
 def monitor_rabbitmq
   rabbitmqctl_url = CONFIG['rabbitmqctl']['url']
   rabbitmqctl_user = CONFIG['rabbitmqctl']['username']
   rabbitmqctl_password = CONFIG['rabbitmqctl']['password']
   uri = URI.parse("#{rabbitmqctl_url}/api/queues")
   request = Net::HTTP::Get.new(uri)
   request.basic_auth(rabbitmqctl_user, rabbitmqctl_password)
   req_options = { use_ssl: uri.scheme == 'https' }
   response = Net::HTTP.start(uri.hostname, uri.port, req_options)  do |http|
      http.request(request)
   end
   queue_details = JSON.parse(response.body)
   queue_details.each do |queue|
     output = { name: queue['name'],
                messages: {
                  total: queue['messages'],
                  ready: queue['messages_ready'],
                  Unacknowlged: queue['messages_unacknowledged']
                  },
                 node: queue['node'],
                 state: queue['state'],
                consumers: queue['consumers'] }
      if output[:messages][:ready] > 100
         sent_alert_slack("RabbitMQ QUEUE High! \n #{output[:messages][:ready]} :\n #{output}")
       end
    end
   end
   def sent_alert_slack(message)
      notifier = Slack::Notifier.new CONFIG['slack_settings']    ['notification_api'],
                             channel: '#rabbitmq-monitoring',
                             username: 'notifier'
      notifier.ping message
    end
   begin
    puts "\n", Time.now
    monitor_rabbitmq
  rescue => e
    puts "Error: #{e.message}"
end