Access and Error Logs

Access and Error Logs

Log Files

A web server log is a record of the events having occurred on your web server. Both Apache and NGINX store two kinds of logs:

Access Log

Contains information about requests coming into the web server. This information can include what pages people are viewing, the success status of requests, and how long the request took to respond. Both Apache and NGNIX access logs use the Combined Log Format, so their outputs are similar. An access log record looks something like this:

10.185.248.71 - - [09/Jan/2015:19:12:06 +0000] 808840 “GET /inventoryService/inventory/purchaseItem?userId=20253471&itemId=23434300 HTTP/1.1” 500 17 “-“ “Apache-HttpClient/4.2.6 (java 1.5)”

Error Log

Contains information about errors that the web server encountered when processing requests, such as when files are missing. An Apache error log record looks something like this:

[Thu Mar 13 19:04:13 2014] [error] [client 50.0.134.125] File does not exist: /var/www/favicon.ico

The NGINX error log file format is a bit different and looks like this:

2023/08/21 20:48:03 [emerg] 3518613#3518613: bind() to [::]:80 failed (98: Unknown error

Location

Access and error log files are stored on individual web servers. By default on most Linux systems, you can find Apache logs at /var/log/apache, /var/log/apache2, or /var/log/httpd. Similarly, NGINX logs are often located at /var/log/nginx by default.

Apache General Configuration

The following general default logging configuration directives are specified in the absence of a specific virtual host container configuration.

Log Level Directive

This specifies log message severity. The default is “warn.”

Table of Level Severities

Note regarding a particular level: Levels are listed in order of descending severity. When triggered, a configured log level will log all events of equal level or greater.

Log Format

Apache offers a ton of flexibility for what you can log. You can find a full description of the Apache log fields listed in the Apache log documentation. Some notable fields include:

• %a: Client IP address of the request
• %A: Local IP address
• %b or %B: Size of response in bytes, excluding HTTP headers
• %D: The time taken to serve the request, in microseconds
• %h: Remote hostname
• %k: Number of keepalive requests handled on this connection
• %l: Remote logname
• %r: First line of request
• %s: Status
• %t: Time the request was received
• %T: The time taken to serve the request, in seconds
• %u: Remote user if the request was authenticated
• %U: The URL path requested, not including any query string
• %v: The canonical ServerName of the server serving the request
• %X: Connection status when the response is completed

You can configure a custom pattern inside your Apache configuration file and then define where you want those logs to be written. Here is an example of one log format you can choose. You read more on the mod_log_config documentation.

LogFormat “%h %l %u %t “%r” %>s %b “%{Referer}”"“"%{User-agent}”" combined

CustomLog log/access_log combined

Top 5 Log Fields

Here are some of the most valuable log fields when monitoring server health or for troubleshooting issues. You should consider including each of these in your Apache log format.

1. HttpStatusCode: This will tell you the exact status of the response. You can read the full list of status codes, but here’s the short version:

• 2xx is successful
• 3xx is a redirection
• 4xx is a client error (a bad request coming from the client or a request that’s not authorized)
• 5xx is a server error (the request was valid, but the server had a problem fulfilling it due to a processing error or timeout of some kind)

1. UrlPath: Which page the user was trying to load.
2. RemoteIPOrHost: The IP address making the request. Useful to drill down on traffic from a particular source.
3. RequestTimeSeconds: How long it took for the request to be processed. This will give you a good idea of how fast or slow your site is. If your site is too slow, you’ll probably need to optimize your servers or your application.
4. Unique_ID: Commonly known as a request ID, this helps you search and trace a particular request through to your web application server.

Log Format Notes

These default directives can be thought of as a recipe of formatting assigned to a nickname and used with a CustomLog directive. The format string represents several specifiers preceded with a “%” and the specifier character.

A specifier represented as %{Referrer}i means a variable value of type “i,” which in this case means the “Referrer” request header content. The “i” specifies content from the request header.

The “vhost_combined” following the format string, indicated in the example below, is simply a name assigned to the format. Apache calls these “nicknames.” Their use with a CustomLog directive is like this: CustomLog <path/to/log> <nickname>. This will log the format represented by the nickname to the specified log location.

Example:

LogFormat "%v:%p %h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i" vhost_combined

Formatting as JSON

Most log management solutions can automatically parse each of the fields out of the Apache log line. If yours does not support this or doesn’t support your desired format, you can also log in JSON. This more flexible format allows you to define your own fields, and it’s also self-documenting because the key names describe the data. The main downside is your logs will consume more volume because you will be including key names with every message. Here is an example of one potential format. You’d insert this in your Apache configuration file as described in the section above.

LogFormat "{ "time":"%t", "remoteIP":"%a", "host":"%V", "request":"%U", "query":"%q", "method":"%m", "status":"%>s", "userAgent":"%{User-agent}i", "referer":"%{Referer}i" }"

Once you’ve made the changes to the web server configuration file, you’ll need to restart the Apache web server service.

Combined Access Log Config

Default vhost combined access log config allows for a combined access log for those vhosts without a specific location config. Change this config if a new location is desired.

Example:

CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

The “vhost_combined” mentioned above is a label or name for a specific format.

Default Virtual Host Overrides

Virtual host (vhost) override logging config is specified in the respective distribution tables below. When duplicating these default files for a specific vhost config, change the default log specification to vhost-specific logs if desired. Leaving the default specification will log all vhost access and error log entries to the Apache default.

All log directives specified as general Apache config are overridable in a specific vhost configuration file.

Example standard port 80 vhost config for a site example.com:

ServerName example.com

ServerAdmin webmaster@example.com

DocumentRoot /var/www/example.com

LogLevel info ssl:warn

ErrorLog /var/www/example.com/logs/error.log

CustomLog /var/www/example.com/logs/access.log

Debian/Ubuntu/Linux Mint Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

Table of Default Directives

* Conditioned environment variable. Provides support for multiple Apache server instances.

** Environment variable. Used to dynamically set the initial log path.

Default Virtual Host Overrides

Virtual host config (port 80):

/etc/apache2/sites-available/000-default.conf

Virtual host config (port 443):

/etc/apache2/sites-available/default-ssl.conf

Red Hat/Fedora/CentOS Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

Table of Default Directives

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

Virtual Host Config

Provide a <virtualhost></virtualhost> container within the main config /etc/httpd/conf/httpd.conf or a config file specified within the /etc/httpd/conf.d directory, such as vhost.conf, which is then read by default upon server start.

openSUSE Configuration

The following default configuration directives are specified in the absence of specific virtual host container configuration for the Linux distributions listed.

Table of Default Directives

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

*** Conditioned on loaded mod_ssl.

Default Virtual Host Overrides

Virtual host config (port 80):

/etc/apache2/default-vhost.conf

Virtual host config (port 443):

/etc/apache2/default-vhost-ssl.conf

Apache Applicable Modules

The Apache web server offers a number of modules that either change the way Apache works or extend its capability.

mod_log_config

This module is compiled in by default in the Apache base. Directives Include:

BufferedLogs: Buffers log entries in memory before writing to disk. This is an On|Off switch.

CustomLog: Sets the filename and log file format.

LogFormat: Specifies log file format.

TransferLog: Specifies the log file location.

mod_log_debug

Configurable debug logging. This module is experimental and may or may not be provided in a package distribution. If provided, it will likely require enabling. Directives include:

LogMessage: User-defined messages logged to an error log.

Example: Adding configurable debug messages to a location tag similar to below.

LogMessage "/path/to/specific/directory has been requested by" ${REMOTE_ADDR}

mod_log_forensic

Allows forensic request logging and analysis. Often included in package distributions but may require enabling. This module supercedes mod_unique_id for forensic purposes.

A “+” sign precedes the unique identifier for an initial request log entry.

A “-” sign indicates subsequent log entries for the same request.

A final log entry for a given request includes a “-” sign preceding the unique identifier. This indicates final forensic request log entry. Directives include:

ForensicLog: Used to log requests for forensic analysis. Each log entry is assigned a unique identifier.

Forensic Log Format: Specifies a unique forensic ID to the beginning of the log string. This is not a directive, but rather information related to the id itself and the significance of the initial id character, which indicates first and subsequent log writes.

ForensicLog Filename: Specifies a filename for the forensic log location relative to the server root. This directive can be specified in the main server config, or within a vhost config.

Example: Initial forensic log entry calling https://localhost using default log format:

+3264:55958cb4:0|GET / HTTP/1.1|Host:localhost|Connection:keep-alive| Cache-Control:max-age=0| Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8| User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36| Accept-Encoding:gzip, deflate, sdch| Accept-Language:en-US,en;q=0.8| If-None-Match:"2cf6-519e8edfca429-gzip"| If-Modified-Since:Thu, 02 Jul 2015 18%3a51%3a39 GMT

Example: Ending forensic log entry from the above request.

-3264:55958cb4:0

Example: Standard port 80 vhost config for a site including a custom forensic log:

ServerName example.com ServerAdmin webmaster@example.com DocumentRoot /var/www/example.com LogLevel info ssl:warn ErrorLog /var/www/example.com/logs/error.log CustomLog /var/www/example.com/logs/access.log example CustomLog /var/www/example.com/logs/forensic.log forensic

Usage notes:

If an ending log entry is not written, which means there is no -<unique id> entry, the request did not complete, indicating script failure.

Once module is enabled, adding a %{forensic-id} variable to a regular log format string will include a forensic identifier in other logs, such as in the following:

LogFormat "%{forensic-id} %h %l %u %t "%r" %>s %O" common

mod_logio

This module adds three new logging formats to the mod_log_config format specifications, including byte quantities received, sent, and transferred (a combination of received and sent quantities). This module is normally included in the base Apache compile.

LogIOTrackTTFB: Enables time tracking between the initial request read time and the moment the first byte response is sent.

mod_filter: Provides context-sensitive filters to the output chain by registering any number of filter providers. mod_filter is not specific to logging but allows for extracting specific requests based on the filter provider. Context containers include main apache config, vhost config, within directory tags, and .htaccess files.

Employing this module allows for filtering requests containing such things as certain injection criteria and which IP address it’s from.

This module is provided by default in many of the package distributions but may require enabling. For logging, the FilterTrace directive posts information to the error log. Directives include:

• AddOutputFilterByType: Assigns an output filter to a particular media type.
• FilterChain: Configures a filter chain.
• FilterDeclare: Declares a smart filter.
• FilterProtocol: Causes the mod_filter to handle response headers correctly.
• FilterProvider: Registers filter providers.
• FilterTrace: Allows for debugging/diagnostic information to an error log prior to provider processing.

Example: Filtering in a vhost context container conditionally on filter and include modules:

# Declare a resource type filter:

FilterDeclare xss

# Register a provider:

FilterProvider xss INCLUDES %{REQUEST_FILENAME}="(/[<>]+)$"

# FilterProvider

…

# Build the chain:

FilterChain xss

# Declare a custom log:

CustomLog /var/www/log/xss.log xss

# Format the log entry:

LogFormat "%h %u %t "%r" %>s "%{Referer}i" "%{User-Agent}i"

mod_unique_id:

This module constructs an environment variable and a unique identifier for each request. It is often included in package distributions but may require enabling. This unique identifier is written to the access log.

This module has been superseded by mod_log_forensic for forensic purposes but is still supported for others.

Unlike the forensic identifier, the unique identifier is passed to the application handler via the environment variable UNIQUE_ID. This allows application developers to trace a request through the web server to the application server. It can be useful for debugging a request.

Apache spins off child processes to handle requests, and a child instance processes several requests at a time. As a result, it is sometimes desirable to use a unique identifier to identify a single request across multiple server instances and child processes.

Once enabled, the module provides an identifier by default to the application handler.

Example:

UNIQUE_ID: Vaf3en8AAQEAAAtoQlAAAAAA

The identifier is constructed from a 32-bit IP address, 32-bit process ID, 32-bit timestamp coupled to a 16-bit counter for tighter resolution than a single-second, 32-bit thread index. The timestamp component is UTC to prevent issues with daylight saving time adjustments. The application handler should treat the identifier as an opaque token only and not be dissected into constituents.

NGINX General Configuration

Like Apache, NGINX allows administrators to modify many aspects of web server logging. One of the key differences with NGINX is that the error log format cannot be modified.

In the sections below, we’ll look at different ways you can customize NGINX access and error logs. Note that you must restart (for example: systemctl restart nginx) or reload (for example: nginx -s reload) for configuration changes to take effect.

Log Level Directive

You can modify the access_log and error_log settings in NGINX configuration files. To change the error log severity, append a severity to the end of an error_log entry in the http section of a configuration file.

The format for an entry is:

[log file] [location] (severity);

For example, this entry will change the error_log at /var/log/nginx/error.log to debug.

error_log /var/log/nginx/error.log debug;

The supported severity levels, from most to least severe, are:

• emerg - Events that make the system unstable or unusable
• alert - Events that require immediate action
• crit - Critical events that may impact system performance but are less severe than alert or emerg
• error - Error conditions
• warn - Warning conditions
• notice - A notable but low-severity event
• info - Informational events
• debug - Granular debug-level logging

Setting a lower severity level will include logs for all the higher severities. For example, warn includes error, crit, alert, and emerg.

Configure NGINX log file location

NGINX configuration files can also configure where access and error logs are stored. To set a log file location, change the [location] parameter in the configuration file. For example, to change our error log file location to /var/log/nginx/debug/debug.log, we can use this configuration file entry:

error_log /var/log/nginx/debug/debug.log debug;

Customizing NGINX access logs

While you cannot customize the NGINX error log format, you can use the log_format syntax in the http section of a configuration file to customize access log formatting.

The general process is as follows:

1. Create the log_format
2. Add the name of the log_format to your access_log entry

NGINX supports many variables you can use to customize logs.

Here’s an example of a custom log format named “custom” that includes a timestamp in local time, address of the server, address of the client, request method (HTTP verb), and requested page.

log_format custom escape=default '"$time_local " , $server_addr", "$remote_addr", "$request_method", "$request_uri"';

We also need to add that format to our access log entry, like this:

access_log /var/log/nginx/access.log custom;

An example access log record using this format is:

"11/Nov/2023:18:27:10 +0000 " , 127.0.0.1", "192.168.11.11", "GET", "/pepnegg.html"

Formatting as JSON

Like Apache log files, NGINX access_log files support formatting logs as JSON using the log_format syntax. For example, a simple configuration for log_format that includes a “webserver” hardcoded as “my server,”  ISO 8601 timestamp, remote address, and page requested is:

log_format json-logs escape=json '{"webserver": "my server", "time": "$time_iso8601", "remote": "$remote_addr", "page": "$request_uri"}';

We can then add that json-logs format to our access log with this configuration:

access_log /var/log/nginx/access.log json-logs;

Here’s what an example access log record using that format:

{"webserver": "my server", "time": "2023-11-11T17:51:03+00:00", "remote": "127.0.0.1", "page": "/"}

Last updated: December 2022