log_format in advanced tab not working

[Treppengelaender]

I want to add a Custom Nginx Configuration to my Proxy Host.
I've added the command
access_log /path/to/logfile.log proxy;
with success.
But now i want to change the output of the logfile.
I thought it is as easy as declare a new log_format into the Custom Configuration Window.
But when i try to add something like
log_format custom '$remote_addr - $remote_user';
the Status from this Proxy Hosts toggles from Online to Offline.

Is there a good solution to customize the log_format? Is it just a bug and it shoult be in general possible to edit the log_format in the Custom Nginx Configuration Window?

Best Regards

[Highview-Lucas]

the project sets it's own log file format for some reason.

It would be nice to have the option of selecting one of the standard (apache/nginx/etc) ones, or overriding it entirely.

[wasn-eu]

I go with that
To use the standard analyzer tools we need the standard log format

[tuxmeister]

I honestly can't find a good reason for this project to have a non-standard log format by default. I've been fighting with telegraf and grok patterns (new stuff to me) for the better part of the afternoon just to get some metrics into Influx... just... why? I can't even find the source place where this is defined to change it before the container is configured.

@jc21 would adding the custom output format in one of the "advanced config" file appenders work?

EDIT: just to clarify, I have not been successful with grok, even though I'm using a constructor to help me, I still find that for most of my proxy hosts, the cache status is a dash so I can't make one pattern to rule them all (within my ability and patience right now).

[Highview-Lucas]

the log_format definitions need to go into the base nginx.conf, the actual logging line can just go into the host definition. (so you can't just put it into the custom configuration inside the webUI, as you've discovered)

/etc/nginx/nginx.conf inside the docker container has log_format entries on lines 46/47

You can comment out these two lines and put in standard ("combined") log_format definitions - i don't have them handy, I'm at work, nginx docs should have the standard ones though!

make sure you name them correctly, the lines will need to start with "log_format proxy" and "log_format standard"

this does leave us an opening to define log_formats and then select which one we want per host inside the WebUI (and by "we" i mean someone else ;) )

[Treppengelaender]

Yes, editing /etc/nginx/nginx.conf works. It is also possible to add a line log_format customtemplate 'YOUROPTIONSHERE';
But wouldn't updating the container destroy changes made in /etc/nginx/nginx.conf?
Is there a way to change the log_format or set a custom log_format in the /data directory in the container?

[Highview-Lucas]

I think I ended up putting /etc/nginx into a bind-mounted directory - changes persist through recreations - but if there's a huge change, it's likely to bite me.

[d33pjs]

Upvote this feature. Did anyone got that working with advanced configurations?

[chaptergy]

You could open a PR with this change (removing the log_format from nginx.conf) and explaining your reasons. Removing the two lines is not much effort. The worst thing that could happen is that he rejects it and you have wasted 5 minutes of your time.

[PathToLife]

You can specify a custom log_format by creating a file in

/data/nginx/custom/http_top.conf

then restarting the server.

whatever is inside there is injected directly into http {} of nginx.conf https://github.com/jc21/nginx-proxy-manager/blob/master/docker/rootfs/etc/nginx/nginx.conf

example below i added real-ip [Real-ip $http_cf_connecting_ip]

log_format proxy2 '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Real-ip $http_cf_connecting_ip] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';

then added in the dashboard access_log /data/logs/proxy2_host-7.log proxy2;

note log_format standard log_format proxy are already defined nginx does not let you override them

[d33pjs]

You can specify a custom log_format by creating a file in

/data/nginx/custom/http_top.conf

then restarting the server.

whatever is inside there is injected directly into http {} of nginx.conf https://github.com/jc21/nginx-proxy-manager/blob/master/docker/rootfs/etc/nginx/nginx.conf

example below i added real-ip [Real-ip $http_cf_connecting_ip]

log_format proxy2 '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Real-ip $http_cf_connecting_ip] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';

then added in the dashboard access_log /data/logs/proxy2_host-7.log proxy2;

note log_format standard log_format proxy are already defined nginx does not let you override them

This worked perfectly. Thank you so much!

[AMWN]

I go with that To use the standard analyzer tools we need the standard log format

Made a regexp for the custom proxy log_format. It's not perfect, but works for me for scraping with Promtail.

\[(?P<dateandtime>\d{2}\/[A-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|\-)\d{4})\] (?P<upstream_cache_status>-) (?P<upstream_status>[^ ]*) (?P<status>[^ ]*) - (?P<request_method>[\w\.]+) (?P<scheme>[\w\.]+) (?P<host>[^"]*) "(?P<request_uri>[^"]*)" \[Client (?P<remote_addr>[\w\.]+)\] \[Length (?P<bytessent>\d+)\] \[Gzip (?P<gzip_ratio>[^ ])\] \[Sent-to (?P<Sentto>[^"]*)\] "(?P<http_user_agent>[^"]*)" "(?P<http_referer>[^"]*)"

[TehloWasTaken]

You can specify a custom log_format by creating a file in

/data/nginx/custom/http_top.conf

then restarting the server.

whatever is inside there is injected directly into http {} of nginx.conf https://github.com/jc21/nginx-proxy-manager/blob/master/docker/rootfs/etc/nginx/nginx.conf

example below i added real-ip [Real-ip $http_cf_connecting_ip]

log_format proxy2 '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Real-ip $http_cf_connecting_ip] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';

then added in the dashboard access_log /data/logs/proxy2_host-7.log proxy2;

note log_format standard log_format proxy are already defined nginx does not let you override them

Hey, thanks for posting this. I just tried doing this but it isn't working properly and it isn't reading my config file to apply the custom log format. Could you maybe help me out?

I am trying to use the log format from this link:
https://grafana.com/grafana/dashboards/12559

I applied it in the http_top.conf in the proper folder, but it just isn't reading it and still applying the standard format. Help?

[motionrus]

You can specify a custom log_format by creating a file in

/data/nginx/custom/http_top.conf

then restarting the server.

whatever is inside there is injected directly into http {} of nginx.conf https://github.com/jc21/nginx-proxy-manager/blob/master/docker/rootfs/etc/nginx/nginx.conf

example below i added real-ip [Real-ip $http_cf_connecting_ip]

log_format proxy2 '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Real-ip $http_cf_connecting_ip] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';

then added in the dashboard access_log /data/logs/proxy2_host-7.log proxy2;

note log_format standard log_format proxy are already defined nginx does not let you override them

My data/nginx/proxy_host/1.conf file now duplicates the logs

  access_log /data/logs/proxy-host-1_access.log proxy;
  error_log /data/logs/proxy-host-1_error.log warn;
  access_log /data/logs/proxy-host-1_access.log upstream_time ;

If I look at the proxy-host-1_access.log file, I see duplication

==> data/logs/proxy-host-1_access.log <==
[12/Jan/2023:05:24:15 +0000] - 304 304 - GET https motionrus.site "/" [Client 77.51.203.17] [Length 0] [Gzip -] [Sent-to 172.31.21.178] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
77.51.203.17 - - [12/Jan/2023:05:24:15 +0000] "GET / HTTP/2.0" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"rt=0.001 uct="0.000" uht="0.004" urt="0.004"

Here is my data/nginx/custom/http_top.conf

    log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent"'
                             'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

Does anyone know how to solve the problem with duplication?

[okworx]

Made a regexp for the custom proxy log_format. It's not perfect, but works for me for scraping with Promtail.

\[(?P<dateandtime>\d{2}\/[A-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2} (\+|\-)\d{4})\] (?P<upstream_cache_status>-) (?P<upstream_status>[^ ]*) (?P<status>[^ ]*) - (?P<request_method>[\w\.]+) (?P<scheme>[\w\.]+) (?P<host>[^"]*) "(?P<request_uri>[^"]*)" \[Client (?P<remote_addr>[\w\.]+)\] \[Length (?P<bytessent>\d+)\] \[Gzip (?P<gzip_ratio>[^ ])\] \[Sent-to (?P<Sentto>[^"]*)\] "(?P<http_user_agent>[^"]*)" "(?P<http_referer>[^"]*)"

Here's a regex that works for import into Matomo together with --log-date-format='%d/%b/%Y:%H:%M:%S %z':

'\[(?P<date>\d{2}\/[A-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2}\s+(\+|\-)\d{4})\]\s+(?P<upstream_cache_status>-)\s+(?P<upstream_status>[^ ]*)\s+(?P<status>[^ ]*)\s+-\s+(?P<request_method>[\w\.]+)\s+(?P<scheme>[\w\.]+)\s+(?P<host>[^"]*)\s+"(?P<path>[^"]*)"\s+\[Client (?P<ip>[\w\.]+)\]\s+\[Length (?P<bytessent>\d+)\]\s+\[Gzip\s+(?P<gzip_ratio>[\d\.\-]+)\]\s+\[Sent-to (?P<Sentto>[^"]*)\]\s+"(?P<user_agent>[^"]*)"\s+"(?P<referrer>[^"]*)"'