Basics of Apache Logging: A Definitive Guide

The need for an easy-to-use platform to log application data is exploding. Everyone wants to capture data about their users and their products. Apache Logging lets you do that easily, reliably, and at scale. This post captures everything related to including configuration and Apache log examples. Read on.

What is Apache Logging?

Apache Logging is an open-source project created to allow users to examine their logs efficiently. Apache Logging allows users to extract data from their log files and store it in other formats like CSV or XML. It also provides functions for comparing two different versions of the same log file.

The Apache Logging project is a software library that implements a logging service. It provides developers with a way to create and control logs, enabling them to see what is happening with the code.

Apache Logging services provide access to logs in a way that is easy for humans to read. The logs are stored in log files, which consist of messages showing the date and time, the server’s hostname, and the IP address of any client who requests a document.

Pick any Apache logs example you will find that they are used for debugging, tracing, monitoring and diagnostics. The logs can be used for testing, performance tuning, system administration, and even security analysis.

The Apache Logging project is the official successor of the Jakarta Commons Logging project. This open-source project contains log4j, a widely used tool for logging application behavior. It also contains the SLF4J API, which provides a framework for other loggers and allows them to be plugged into applications that use various other libraries, such as Apache Commons Logging or Google’s Guava.

How do I Enable Apache Logging? 

Apache Logging is typically a mechanism in a web server that allows users to track server activity. It can be enabled on your web server. Once enabled, it will start to log all the HTTP requests made by your visitors. It is a very handy feature of the Apache Webserver. It logs all the requests and responses and provides the ability to analyze the traffic in real-time. This helps you troubleshoot issues in your system quickly.

To enable Apache Logging, you need to configure a few directives in the webserver configuration file.

To enable Apache HTTP access logs, first you need to open HTTP configuration of Apache at /Applications/MAMP/conf/apache/httpd.conf

Next, find the code

#CustomLog logs/access_log combined

Replace this code with

CustomLog /Applications/MAMP/logs/apache_access_log combined

This will make sure all your access logs are logged in your default log directory, with the “Log Format” named “Combined”. This log format has some standard conventions.

Once done, restart your Apache server with the MAMP widget. You can also restart using the command line

$ /Applications/MAMP/bin/apache2/bin/apachectl restart

It is important to remember that if you enable the log directory, all your data will be written to /Applications/MAMP/Library/logs/access_log. This is not desirable. It is always ideal to store the access logs to /Applications/MAMP/logs/. This is where you could find MySQL, Apache error log, and PHP logs.

Types of Apache Logs 

Apache log structure is very flexible and easy to manage. There are two types of logs: Access logs and Error logs. 

Access Log

This is where all the information about the requests coming to the webserver is noted. Apache access log response time is also measured. The information can be anything like pages visited by the audience, requests success rate, and time is taken for the server to respond to the requests. To manage request logging, you need to be familiar with three configuration directories: LogFormat, CustomLog, and TransferLog.

Various other directories were available, but as Apache kept upgrading, these directories were deprecated. CustomLog can now achieve all the functions that these directories do. Few deprecated directories are RefererLog, CookieLog, RefererIgnore, and AgentLog.

Error Log

The information on errors encountered by the server during processing is found in the error log. It contains information on events unrelated to request serving and includes the diagnostics information about the server. The error log also contains the information that the access log doesn’t. Here is some log information that the error log offers:

• Different informational messages
• Critical Events
• Errors that occurred during request servicing (status 400-503)
• Standard error output
• Startup and shutdown messages

The error log has a standard format. Every line contains three fields: time, error level, and messages. You can also get some raw data in the error logs during some rare instances. These logs are created by using the ErrorLog directory configuration. 

Log Locations

The storage location of the error log and access log files depends on your operating system. Both these files are stored as separate entities on the server. Let’s have a look at the default storage location for various operating systems. 

Note: To change the apache log directory, use the #grep command.

• Linux Mint / Debian / Ubuntu

For the unencrypted sites, the httpd log file location in Linux is /etc/apache2/sites-available/000-default.conf. Similarly for the encrypted sites with SSL/TLS protection, the httpd log file location in Linux is /etc/apache2/sites-available/default-ssl.conf.

Here are the default derivatives for Linux OS –

Setting	Config File	Value/Path
Access Log	/etc/apache2/sites-available/000-default.conf	CustomLog ${APACHE_LOG_DIR}/access.log combined
Log Level	/etc/apache2/apache2.conf	WarnLogFormat “%v:%p %h %l %u %t “%r” %>s %O “%{Referer}i” “%{User-Agent}i”” vhost_combinedLogFormat “%h %l %u %t “%r”
Error Log	/etc/apache2/apache2.confwarn	ErrorLog ${APACHE_LOG_DIR}/error.log
Custom Log	/etc/apache2/conf-available/other-vhosts-access-log.conf	CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
Log Format	/etc/apache2/apache2.conf	%>s %O “%{Referer}i” “%{User-Agent}i”” combinedLogFormat “%h %l %u %t “%r” %>s %O” commonLogFormat “%{Referer}i -> %U” refererLogFormat “%{User-agent}i” agent

• CentOS / RedHat / Fedora

The main configuration file for the Red Hat distribution is located at /etc/httpd/conf/httpd.conf. The additional Virtual host config files can be placed in the directory /etc/httpd/conf.d. This directory is automatically read at the start by the server. Here are the default directives.

Setting	Config File	Value/Path
Access Log	/etc/httpd/conf/httpd.conf	/var/log/httpd/access_log
Error Log	/etc/httpd/conf/httpd.conf	/var/log/httpd/error_log
Log Level	/etc/httpd/conf/httpd.conf	warn
Custom Log	/etc/httpd/conf/httpd.conf	CustomLog “logs/access_log” combined

• OpenSUSE

Similarly, for the OpenSUSE Operating system, the default configuration for encrypted sites can be found at /etc/apache2/default-vhost-ssl.conf. The default virtual host config for the unencrypted sites can be found at /etc/apache2/default-vhost.conf. Here are the default directives – 

Setting	Config File	Value/Path
Access Log	/etc/apache2/sysconfig.d/global.conf	/var/log/apache2/access_log
Custom Log	/etc/apache2/sysconfig.d/global.conf	CustomLog /var/log/apache2/access_log combined
Error Log	/etc/apache2/httpd.conf	/var/log/apache2/error_log
Log Level	/etc/apache2/sysconfig.d/global.conf	warn

Configuring Apache Error and Access Logs 

In the Apache framework, you have high flexibility to adjust the logging behaviour both globally or for each file of the vhost. There are various directives that you can use to change the Apache log directory behaviour. The most common are log level and log format directives.

Log Level directive

The main feature of the log level directive is to determine the minimum security level for the events that are logged to a specific destination. The importance level of an event can range from “Emerg” to “Trace8”. This typically represents the severity level. An event with the “Emerg” level might lead to instability, whereas the “Trace8” level provides trace level messages. Apache change log level can be altered according to your requirement.

Log Format

The layout and formatting of log events are controlled by the Log format directive. The default Apache log format is CLF (Common Log Format). But Apache gives you the flexibility to change fields included in each log by enabling you to specify your format string. These are a few default CLF:

Log Format “%h %l %u %t \”%r\” %>s %b”

The format string is represented in the first parameter. It indicates the information regarding the log file and the written format of the log file. To decipher the log format, refer to the documentation provided by Apache. These are a few standard logging strings formats – 

Format String	Description
%%	Percentage Sign
%…a	Remote IP Address
%…A	Local IP Address
%…B	Bytes size for response (Excluding HTTP)
%…b	Bytes size for response (Excluding HTTP), Instead of Zero a dash (-) is used
%…D	Time to serve (microseconds)
%…f	Name of the File
%…h	Remote host
%…H	Request protocol
%…P	Process ID

These five fields are recommended as they are crucial for troubleshooting issues and monitoring server health – 

1. %>s: Requests HTTP status code. The final request after internal redirection is shown.
2. %U: The requested URL path excluding the additional query string is displayed.
3. %a: This is used to identify the traffic from a particular source. It displays the IP of the client who is making the request.
4. %T: Time taken to process the request in seconds. It is useful to measure the speed of the site.
5. %{Name}e: It is also known as request ID. On every request, it will log a unique identifier. This is primarily useful for tracking requests from your Apache server to your web server.

TransferLog

It is the basic request logging directive. It creates an access log with filename

TransferLog /var/www/logs/access_log

The TransferLog directive uses CLF by default; it records every log request on single line information. If the LogFormat directive was previously used in a configuration file, the TransferLog directive would utilize that format instead of the default CLF format.

CustomLog

This is the most powerful directive. Most of the time, it can replace the TransferLog directive and can be used alternatively. The CustomLog filename looks similar to that of TransferLog – 

CustomLog /var/www/logs/access_log custom

By default, Apache uses CLF. However, it doesn’t record many request parameters. To overcome this problem, the developers should at least change the configuration to a combined format. This configuration includes the Referer and UserAgent fields.

Apache Error Log Location 

The Error Log configuration directive is located in the main folder of the server. It can be accessed from the following location – 

ErrorLog /var/www/logs/error_log

The LogLevel directive makes sure that more information than necessary is not stored in the log. Typically, error logs are defined by various levels; all the levels above the specific level will be written to the log. The default level is set to warn.

Level	Description
emerg	Emergencies
alert	To act immediately (Alerts)
crit	Critical Conditions
error	Message of Error
warn	Message of Warning
notice	Significant condition but Normal
info	Informational Message
debug	Debugging information

Apache error log is a wonderful guide that lets you know if something bad has happened. But it does not have enough information like host details and the location of the error. In short, it fails to describe the error information.

Log Related Modules

The Apache web server offers many modules that can help you change the way Apache works or let you extend the Apache capabilities. Let’s have a peek at a few modules that help you add or change the logging behaviour – 

Mod_log_debug

This is an experimental module that might not be available in your Apache distribution. This module provides additional features for logging the debug messages. 

Mod_log_forensic

It is used to enable logging before and after the processing of a request. The developer can easily trace the events between the forensic log and normal log as every entry is assigned with a unique ID. The downside being custom formats are not supported by forensic loggers.

You should use forensic keywords to specify the forensic log of specific log files. This should be done after enabling the module. To add forensic data to normal logs, the LogFormat string %{forensic-id} pattern can also be used.

The forensic logs usually start with either a + or – symbol. The entry log for a particular request is represented as +, and the following entries for the same request are indicated by -.

Mod_logio

To measure the number of bytes used to send and receive a request, this module can be utilized. It displays three values: bytes received, sent, and transferred (Addition of both send and received). Also, changes in the size of the requests due to SSL and TLS encryption can be accurately accounted for. This module is by default included in Apache.

Also, Mod_logio can track the Time to First Byte (TTFB). This is enabled by the LogIOTrackTTFB on|off directive.

Mod_filter

This module offers several filter providers that are context-sensitive to the output chain. It is not specifically used for logging; it is rather used for extracting specific requests based on the filters. Many Apache packages offer this module at default, but few modules may require enabling. 

FAQs

1. How to turn off verbose logging apache-spark 

You need to modify the spark logging config file

• Go to the Spark home folder.
• To access all configuration files, navigate to subfolder conf.
• Create a new log4j.properties file from the existing template file of log4j.properties.template.
• Edit the default logging to warn in log4j.properties.

2. How to tell where Apache is logging?

We have already discussed the default storing directory for all the OS. In case you have changed your directory, this is how you can find where the Apache is logging using the command grep

• #grep CustomLog /usr/local/etc/apache2/httpd.conf

3. How to turn on rewrite logging Apache?

To enable mod_rewrite in your Apache (XAMPP, WAMP) follow these steps

• Navigate to httpd.conf and open it in a text editor.
• Enable the mod_rewrite command by removing the ‘#’ placed before the command line.
• Restart your WAMP or XAMPP server

4. How To Add Apache Common Logging API Into Tomcat Server?

To configure an alternative mode of logging in to Tomcat Server, we need to replace the existing JULI implementation with the intended common logging mechanism. 

• Download Tomcat’s existing hard-coded JCL support mechanism from Apache Tomcat server. 
• To download, use command -f extras.xml to retrieve tomcat-juli.jar and tomcat-juli-adapters.jar. 
• Copy the downloaded files to “$CATALINA_HOME/bin/”.
• Once done, retrieve the project logger and insert log4jx.y.z.jar in “$CATALINA_HOME/lib”. 
• Next, create a log4j.properties in “$CATALINA_HOME/lib”

5. How To Add Apache Common Logging API Into Java Web?

The process to add Apache Common Logging API into Java Web is the same as above. Apache Common Logging allows extensibility with other environments with Log4j. You must configure the log4j by the same process as mentioned in the previous answer, but write the same in Java Web. 

6. How To Encrypt Apache Access_logs?

Archived Apache Access_logs can be encrypted with access_log.1.g.gpg. However, this may not work for active log files. To encrypt them, we need to use the same gpg function and modify it to make the file a part of the logrotate configuration.