@mark-robustelli I made some headway in finding out when the token is blank!

It seems that the first time that I make a sign in with Google, the token is returned every time.

Any subsequent calls to get identity provider links may or may not return the token. The only way to guarentee it comes back is to completely "forget" the oauth2 link in accounts.google.com, specifically here: https://myaccount.google.com/connections.

My solution is to save the token the first time it comes through in the database. There's one problem with this solution is that I don't have a refresh token for the user, so if this token expires I won't be able to refresh it, but I am only assuming that Fusion will refresh the token and return a new one for me. I have no idea if this is true or not.

Since I can't see the source code of FusionAuth and the docs aren't clear about what kind of token this is, I can't say if this is going to work for sure.

If you can give me more clarity into how this works

@mark-robustelli I see now from here: that it is the refresh token, I see now that I should store the refresh token the first time and initially refresh that token to get an access token. Thank you @dan!

IMHO this one tiny detail is not clear enough, even though it in there. For example, in the Link API it could mention again that the token is a refresh token. Instead it says this:

This is treated as an opaque token as the type varies by identity provider, this value may not be returned by all identity providers. When provided, this token is typically a long lived access or refresh token, but consult individual identity provider documentation for specifics.

That's not true though is it? It's a refresh token when one is available.

While the details may be in the docs, IMHO it isn't extremely clear because OAuth is confusing for people in the first place, and FusionAuth is asking us to perform some but not all parts of the OAuth flow. The answer is spread throughout the docs. In the end if we want to create a custom login page with Google page we need to first:

1. Create an OpenID for Identity Provider for Google.
2. Create the Sign in Google URL on our own to call the sign in page.
3. When the flow comes back to our page we need to exchange the code with the FusionAuth API.
4. When that returns to our redirected url, we need to get the link API to get the refresh token in the token.
5. Exchange that refresh token for an access token, and I guess refresh the token on our own?

However, in a flow without FusionAuth we'd get the access token and refresh token together, store both, and use the access token until it is expired, then use the refresh token.

While it is possible to figure this out, it's just not as clear as the rest of the docs. Everything up to this point was well written and clear. This was hard to figure out from my POV.

@mark-robustelli Thank you for the quick response. I appreciate you taking the time to consider the issue.

You raise a valid point about the possibility of pod issues. To clarify, here's additional context and observations:

Relevant Pods: The specific pods attempting to communicate with FusionAuth remain consistently up and healthy throughout the periods where FusionAuth becomes unreachable.

External Connectivity: Successful communication with external services like Google and Gravatar demonstrates that broader network connectivity from our pods is unaffected.

Dedicated Service: FusionAuth is a separate, dedicated service. The issue lies in our GKE cluster's ability to reach it sporadically.
Given this additional information, I'm now leaning towards these potential areas for investigation:

GKE Egress Rules: I have meticulously examined firewall rules and configurations within GKE that might selectively block traffic to FusionAuth. It's possible a misconfiguration could be causing this intermittent issue. However it seems unlikely as it is intermittent. It also doesn't happen after a deploy or anythign related to GKE availability.

FusionAuth Ingress Rules: I have double-checked the FusionAuth server's settings to ensure there aren't any firewall or IP-based restrictions accidentally preventing connections originating from our GKE cluster. There was one that i saw it went down and I added an allowed IP at the time. It didn't immediately solve it but it did resolve within the next few minutes.

Next Steps:

Would you have any additional guidance or specific areas I should focus on? Any insights on potential pitfalls in GKE's network setup or FusionAuth's configuration that might cause this behavior would be greatly appreciated.

The fact that it is intermittent is a problem that makes this difficult to solve.

Thanks again!

Hello FusionAuth Support and Community,

I'm facing a critical issue with a dedicated FusionAuth instance and would greatly appreciate your expertise. Here's the situation:

Problem Description

Sporadic Unavailability & Downtime: Our FusionAuth dedicated instance becomes randomly unreachable specifically from within our Google Kubernetes Engine (GKE) cluster. This causes our authenticated portion of the site to be unavailable. This happens every once in a while. It happened twice this week two days in a row, and happened once before about a month ago.

Accessibility Contrast: Intriguingly, the instance remains accessible from our personal computers during these unavailability periods.

Timeout from Pods: When attempting a curl request from a pod within the GKE cluster, we consistently get a "connect ETIMEDOUT" error for the FusionAuth instance's API endpoint.

Resolves Itself: The issue mysteriously resolves itself within approximately 30 minutes.

Server Logs

The following server logs accompany the timeout:

preplan-api-7465b86756-dwgnw ClientResponse { 
preplan-api-7465b86756-dwgnw exception: FetchError: request to https://[obfuscated-instance-url]/api/identity-provider/login failed, reason: connect ETIMEDOUT [obfuscated-ip]:443 
... [Stack Trace]
}

Troubleshooting Steps (So Far)

Verified Instance Status: The dedicated instance shows no signs of being down when accessed outside the GKE cluster.

General Connectivity: Our pods have regular internet connectivity otherwise (able to curl google.com).

Whitelisting: We have whitelisted our NGINX Load Balancer IP address from our fusionauth instance settings.

Environment Details

FusionAuth Version: 1.47.1

GKE Setup: gke running a network pool of 4 nodes with our API replicated 10 times. No other issues with our cluster and site is otherwise available.

Request for Guidance

I would sincerely appreciate the community's help in figuring out:

Potential Root Causes: What could explain this temporary, selective unavailability of FusionAuth only from within our GKE cluster?

Network Configuration Issues: Are there specific firewall rules, routing, or DNS settings within GKE to examine?

Troubleshooting Techniques: Any recommended strategies to further diagnose this connectivity problem?

Thank you in advance for your insights and assistance!

@jacob-0 Thank you all for your help with this issue! Thanks for the fast turnaround and the hard work here! Thanks again everyone!

@mark-robustelli I see now from here: that it is the refresh token, I see now that I should store the refresh token the first time and initially refresh that token to get an access token. Thank you @dan!

IMHO this one tiny detail is not clear enough, even though it in there. For example, in the Link API it could mention again that the token is a refresh token. Instead it says this:

This is treated as an opaque token as the type varies by identity provider, this value may not be returned by all identity providers. When provided, this token is typically a long lived access or refresh token, but consult individual identity provider documentation for specifics.

That's not true though is it? It's a refresh token when one is available.

While the details may be in the docs, IMHO it isn't extremely clear because OAuth is confusing for people in the first place, and FusionAuth is asking us to perform some but not all parts of the OAuth flow. The answer is spread throughout the docs. In the end if we want to create a custom login page with Google page we need to first:

1. Create an OpenID for Identity Provider for Google.
2. Create the Sign in Google URL on our own to call the sign in page.
3. When the flow comes back to our page we need to exchange the code with the FusionAuth API.
4. When that returns to our redirected url, we need to get the link API to get the refresh token in the token.
5. Exchange that refresh token for an access token, and I guess refresh the token on our own?

However, in a flow without FusionAuth we'd get the access token and refresh token together, store both, and use the access token until it is expired, then use the refresh token.

While it is possible to figure this out, it's just not as clear as the rest of the docs. Everything up to this point was well written and clear. This was hard to figure out from my POV.

@mark-robustelli Thanks for your response!
I think our paths are getting crossed and I am still confused.

Can you answer these questions:

1. token that comes back from the retrieveUserLinksByUserId in the TypeScript SDK or Retrieve A Link(s) in the Link API. Is it supposed to be the access token?
2. If that is the access token, how does the token get refreshed? Does that happen via FusionAuth?
3. I read the OpenID documentation from Google and their API returns access_token, refresh_token, and id_token keys in JSON, but nothing like token, so somewhere this is getting translated, but how does the id_token get translated or where is the functionality of the refresh token?

@mark-robustelli I made some headway in finding out when the token is blank!

It seems that the first time that I make a sign in with Google, the token is returned every time.

Any subsequent calls to get identity provider links may or may not return the token. The only way to guarentee it comes back is to completely "forget" the oauth2 link in accounts.google.com, specifically here: https://myaccount.google.com/connections.

My solution is to save the token the first time it comes through in the database. There's one problem with this solution is that I don't have a refresh token for the user, so if this token expires I won't be able to refresh it, but I am only assuming that Fusion will refresh the token and return a new one for me. I have no idea if this is true or not.

Since I can't see the source code of FusionAuth and the docs aren't clear about what kind of token this is, I can't say if this is going to work for sure.

If you can give me more clarity into how this works

@mark-robustelli Yes I have it set up. This is the reply I get from fusionauth

{
  "identityProviderLinks": [
    {
      "displayName": "jsch...@gmail.com",
      "identityProviderId": "c43c909e-ada2-4731-b840-9e30e928c2a2",
      "identityProviderName": "Google OpenID",
      "identityProviderType": "OpenIDConnect",
      "identityProviderUserId": "110...3166",
      "insertInstant": 1701322547795,
      "lastLoginInstant": 1701636612288,
      "tenantId": "17470e59-34e8-4833-9c80-080d3abe09b3",
      "token": "",
      "userId": "ea855097-9b97-48fc-99ee-b9975a7e7a8b"
    }
  ]
}

As you can see in this response from the server, the token is blank.

@mark-robustelli I do have it set up as an idp.
I am using openId because I could not get Google working from the API. I am not using the hosted login. I have built my own.
I use the identity provider API route to complete the sign in.
I just need the token. And I need to know that fusion is refreshing the tokens on my behalf.

@mark-robustelli Here's how I am doing it with the Sign in with Google flow:

Sign in with Google button on the home page has a generated url . That function looks like this for me:

function generateGoogleOAuth2LoginUrl() {
  const { google } = require("googleapis");
  const redirectUrl = `${process.env.FRONT_ROOT}/sign-up/login-redirect`;
  const oauth2Client = new google.auth.OAuth2(
    process.env.GOOGLE_OAUTH_CLIENT_ID,
    process.env.GOOGLE_OAUTH_SECRET,
    redirectUrl
  );
  let scopes = [
    "https://www.googleapis.com/auth/calendar",
    "https://www.googleapis.com/auth/userinfo.email",
    "https://www.googleapis.com/auth/userinfo.profile",
    "openid",
  ];

  return oauth2Client.generateAuthUrl({
    access_type: "offline",
    scope: scopes,
  });
}

From there when the user logs in we are placed at /sign-up/login-redirect

I use the typescript fusionauth client to log in with google

async function loginWithGoogle(code) {
  return client.identityProviderLogin({
    data: {
      code,
      redirect_uri: `${process.env.FRONT_ROOT}/sign-up/login-redirect`,
    },
    applicationId: process.env.FUSION_AUTH_APP_ID,
    identityProviderId: process.env.GOOGLE_IDENTITY_PROVIDER_ID,
  });
}

This works and from there I create my own internal user linked to FusionAuth

From there I call to get the token from the link api

async function getLinkedAccounts(userId) {
  const { response } = await client.retrieveUserLinksByUserId(
    process.env.GOOGLE_IDENTITY_PROVIDER_ID,
    userId
  );
  return response;
}

This works and gives me the user I am looking for however the token field is blank. Everything else is populated. I would call Google myself but only one of us can have the token.

One assumption I am making is that FusionAuth is handling the token refresh on my behalf.

The other way is with Username and Password authentication and I do all the Google calling myself, and I keep the credentials stored on my database.

@mark-robustelli BTW I did add the appropriate scopes there:

https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid