Error 403 with Nginx in front

Think I'm hitting the same issue with 1.41.3

Login page loads and if I enter valid credentials it appears to login but redirects to a blank page with the path /oauth2/authorize. At a glance, it appears that headers are not being passed correctly.

We are using a comparatively simple setup: Fusionauth + Nginx in a single docker image. Our nginx config:

server {
    listen 8443 ssl;
    listen [::]:8443 ssl;
    server_name _;

    ssl_certificate /etc/ssl/certs/nginx.crt;
    ssl_certificate_key /etc/ssl/private/nginx.key;

    access_log            /var/log/nginx/access.log;

    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Port 8443;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Server $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        Accept-Encoding "";

    location / {
        proxy_http_version 1.1;
        proxy_pass http://127.0.0.1:9011/;
    }
}

We are seeing a similar issue. The API appears to be working from our application perspective but we are seeing a similar 403 after attempting to log in to the admin console using the existing username and password. Our nginx configuration is similar:

server {
	listen 80;
	listen [::]:80;

	location / {
		rewrite ^ https://$host$request_uri? permanent;
	}
}

error_log /var/log/nginx/error.log info;

server {
    listen 8443 ssl;
    listen [::]:8443 ssl;
    server_name _;

    ssl_certificate /etc/ssl/certs/nginx.crt;
    ssl_certificate_key /etc/ssl/private/nginx.key;

    access_log            /var/log/nginx/access.log;

    proxy_set_header        X-Forwarded-Proto https;
    proxy_set_header        X-Forwarded-Port 8443;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Server $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        Accept-Encoding "";

    location / {
        proxy_http_version 1.1;
        proxy_pass http://127.0.0.1:9011/;
    }
}

@zradick Hmmm. In 1.41, the underlying web server changed, so I wonder if there's are changes to that contributed nginx config that are required.

I think it might be related to https://fusionauth.io/community/forum/topic/2254/having-an-issue-with-nginx-in-front-of-fusionauth

Does making the nginx change mentioned there solve the issue for you?

@dan Thanks, we are running the nginx tightly coupled with our fusion auth deploy (we modified the Docker file in order to start both together in a single container) so that we can deploy it to AWS Fargate. I'm not sure exactly why we were getting a 403 hitting the admin endpoint for auth through the LB, however we are closing off ingress to the public internet anyway and we are able to successfully auth when tunneled through our bastion host and the application API is working so we can at least upgrade to 1.41.3 for the security patch there.

@johnmiller Hiya,

Did you try the fix mentioned here: https://fusionauth.io/community/forum/topic/2254/having-an-issue-with-nginx-in-front-of-fusionauth

If not, what issue are you seeing in particular?

The Nginx 403 Forbidden error occurs when a client attempts to access a section of the webserver without adequate permissions. When Nginx accesses a directory, it attempts to generate an index of its files and present them to the browser or client. However, the default configuration disables directory indexing, leading to the display of the 403 forbidden error instead.

Incorrect Index File

The try_files tries the literal path you specify in relation to the defined root directive and sets the internal file pointer. If you have directory indexing off, and is having this problem, it's probably because the try_files you are using has a directory option:

location / {
  try_files $uri $uri/ /index.html index.php;
}

to

location / {
  try_files $uri /index.html index.php;
}

Incorrectly set permissions

This error can also result from files and directories having incorrectly set permissions. In order to resolve this , change the directories permission to 755 and the file permissions to 644 . Make sure that the user running the Nginx process owns the files. For example, set user to www-data:

sudo chown -R www-data:www-data *

Finally, set the directory and file permissions as:

sudo chmod 755 {dir}
sudo chmod 644 {files}

I'm having the same problem. Unable to solve it so far.
@johnmiller @maburns @zradick did you manage to get past this problem, and if so, how?
Thanks for any advice!

Here's some info about our setup.

• FA behind nginx reverse proxy, docker based. 3 environments (dev, staging, production). Has been working for many months without trouble.
• A few days ago, upgraded FA in the DEV env, from 1.28 to 1.47.1
• Since then, when trying to log into FA, the POST request to /oauth2/authorize is responded by a blank page (HTTP 403)
• Tried a lot of stuff, but to no avail so far. We're using the recommended nginx config (https://github.com/FusionAuth/fusionauth-contrib/blob/master/Reverse Proxy Configurations/nginx/nginx.ssl.conf)
• For debugging purposes, I've built up a 4th environment with the exact same setup (nginx reverse proxy and FA). The FA installation (1.47.1) was installed from scratch. The problem is not showing in that environment.
• This leads to the suspicion that the 403 could be related to existing data/config in the existing installation. But which? And how to access logs, when not being able to log into FA?
• The logs in stdout of the FA container are not anything related to the error.

Any ideas? How can I see logs of why FA decides for a 403 in /oauth2/authorize ?

@mark Are you able to access the system through the APIs? https://fusionauth.io/docs/v1/tech/apis/system#export-system-logs

@mark-robustelli Unfortunately, no. We're running FusionAuth in a Docker container. For that scenario, the doc says "When running FusionAuth on Docker or other container service where logs are written to stdout and not written to the file system, this API will return an empty archive.".

@mark I know this is a long shot, but can you access other pages after you attempt to log in as is occuring with this post https://fusionauth.io/community/forum/topic/2488/dashboard-homepage-timeout/3 ?

@mark-robustelli Unfortunately not. I've tried it several times, with different browsers (to make sure it's not a cache issue etc.). It seems that when logging in, the page https://<myurl>/oauth2/authorize (which results in a 403) is not really logging me in. If afterwards I go to https://<myurl>/admin/application or https://<myurl>/admin/tenant, I'm still redirected to the login form.

I really would like to see FusionAuth's application logs, I'm pretty sure I'd see something there why the 403 occurs. Is there really no way? I need to get this issue fixed.

You can set up a Docker log driver to direct log output somewhere else:

• https://fusionauth.io/docs/v1/tech/admin-guide/monitor#log-files
• https://fusionauth.io/community/forum/topic/1549/want-access-to-my-docker-logs

OK so I was finally able to track it down fix it. Some short notes below, maybe it helps anyone else running into the same problem.
Thanks to @mark-robustelli and @dan for their inputs.

1. I accessed FusionAuth directly on port 9011, completely bypassing the reverse proxy
2. In the CORS filter config, I activated "Debug enabled" which in the system logs showed me java.lang.IllegalArgumentException: Illegal character in authority at index 7 and java.lang.NullPointerException: Cannot read field "actionURI" because "actionInvocation" is null
3. Going from there, I completely disabled the CORS filter
4. This immediately got rid of the 403 and everything worked again, accessing FA using the reverse proxy
5. The FA UI told me that I'm missing two headers in my reverse proxy config: X-Forwarded-Proto and X-Forwarded-Port, which I then configured (https/443 in my case)
6. After that I enabled the CORS filter again (all default values) and since, everything is working correctly, like before

@mark Awesome!