lichess.org
Donate

Lichess APi response code 404 after 3 minutes

#1
Rewrite on right Categorie.

Hi,
i download autmatically my game every 2 days (48h) from lichess api , for internal Stats.

Is possible that at 2024-03-02T00:24:39Z there were and anomaly response from lichess server , restart lichess server or other error for get 404 http_response code instead 200 with game ?

From log,
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 ==> Start download pgn game from lichess api
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 ==> End download pgn game from lichess api
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 ==> Detected 404 http_response code

## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - START - 2024-02-29
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - SEARCH_USER - {"documents":[{"_id":"Feroptimist","status":"active","insertDate":"2024-02-28T09:35:46.764Z","firstCachedDate":"2024-02-28T10:38:28.000Z","lastRefresh":"2024-02-28T10:38:28.000Z","lastDateTouch":"2024-02-28T10:38:28.000Z"}]}
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - USER - Feroptimist
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - LOCK_USER -
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - LOCKED_USER - Utente Lockato Per aggiornamento
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - UPDATE_USER - Pulizia Cache Plain attuale
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - UPDATE_USER - Avvio donwload incontri
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 # ||| # refresh - DELETE_NEW_USER - Utente inesistente su lichess settato per forget.
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 # ||| # refresh - END - Fine
#2
Hi!

Cool that you've made an application to download your games!

> Is possible that at 2024-03-02T00:24:39Z there were and anomaly response from lichess server , restart lichess server or other error for get 404 http_response code instead 200 with game ?

I don't know what happened, but I wouldn't rule out any of the things you've mentioned - many things can be involved when making HTTP requests :)

So if it has been working fine most of the time,
you could probably just chalk this up on bad luck...

...or if you would like to do some more "investigation",
maybe you could share some more information about how you are downloading the games?

Which Lichess API URL/s are you using to export the games?

By looking at the Lichess API documentation at lichess.org/api ,
I'm guessing that you might be using lichess.org/api#tag/Games/operation/apiGamesUser ?

Or do you maybe use a combination of multiple different endpoints?

One way to reproduce a 404 response code with the URL I guessed,
is to provide a username which doesn't exist in Lichess,
i.e "lichess.org/api/games/user/NONEXISTINGUSERNAME"

If you have hardcoded "Feroptimist" in your application,
then that scenario sounds unlikely though...

But maybe it isn't hardcoded,
and you take it from some variable which maybe has been initialized with an unexpected value...

So one thing you could do to gain more information in case the 404 shows up again in the future,
would be to add in your log what the request URL (and its parameters) that you finally send looked like,
i.e something like "lichess.org/api/games/user/Feroptimist"
(I see you have some json-data in your log now, but that is possibly the data you use to "build" the request URL - not the built request URL itself).

One thing that caught my attention in the topic title is "response code 404 after 3 minutes".
A response code is typically given from the server immediately, and then the data (games) is sent from the server.
Maybe it is possible to log more information about the response,
such as the source address of the responder (which typically would be Lichess, but maybe there is some HTTP Proxy involved in this unexpected scenario - and the request never reaches Lichess)

Hope your investigation is fruitful, should you pursue such a path!
#3
We call every 48h :
"lichess.org/api/games/user/{username}?evals=true&opening=true&accuracy=true";
All calls comes from www.feroxosplain.it

Unfortunately we don't have more verbose log and we cannot increase verbosity , but on coding, detect 404 response is on time , because if lichess reponse is 404 we delete all user cached information , the user don't exist more.

User Feroptimist exist, is my profile.

For user setting in code , we engaged mongo Support to know if there were some inconsistency error in database, but this not append, and variable value "Feroptimist" don't changed in document structure , we called right username.
Response http_code 404 in 3 minute, looks like , very high time response and some error during response and middle or end of comunication, never append before.

Thanks for support
#4
Wow, you have your own site - cool!

Hmm,
it is difficult to know what is happening without knowing the inner workings of the application...
... but one interpretation of those logs,
could be that it is maybe MongoDB update which causes the 404 response code,
and not Lichess?

> ## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 ==> Start download pgn game from lichess api
> ## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 ==> End download pgn game from lichess api
> ## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 ==> Detected 404 http_response code
...
> ## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - UPDATE_USER - Avvio donwload incontri
> ## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 # ||| # refresh - DELETE_NEW_USER - Utente inesistente su

Maybe that Start download and End download works perfectly fine,
but then when the MongoDB is to be updated - it is missing the user in the database?

Do you think that scenario could fit what is happening? (That would mean that MongoDB is accessed over HTTP ?)

Or are you fairly certain that the 404 reponse code is from Lichess?
#5
Yes, i'm sure 404 come from lichess,

Explain log , and thee is not async function in all operation, all operatons are sequentially.

## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - START - 2024-02-29 <== normal declare of start
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - SEARCH_USER - {"documents":[{"_id":"Feroptimist","status":"active","insertDate":"2024-02-28T09:35:46.764Z","firstCachedDate":"2024-02-28T10:38:28.000Z","lastRefresh":"2024-02-28T10:38:28.000Z","lastDateTouch":"2024-02-28T10:38:28.000Z"}]} <== Document in mongo response , we have a document with correct _id.
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - USER - Feroptimist <== Detected username
## B1AA782A84033337465CB079 - 2024-03-02T00:24:39 # ||| # refresh - LOCK_USER - <== Internal Information lock for deny double loading
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - LOCKED_USER - Utente Lockato Per aggiornamento <== Mongo Response OK , User Locked cannot execute double check
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - UPDATE_USER - Pulizia Cache Plain attuale <== Clea n cache OK
## B1AA782A84033337465CB079 - 2024-03-02T00:24:40 # ||| # refresh - UPDATE_USER - Avvio donwload incontri <== Start download
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 # ||| # refresh - DELETE_NEW_USER - Utente inesistente su lichess settato per forget. <== User not exists on lichess, because we get 404 http_code
## B1AA782A84033337465CB079 - 2024-03-02T00:27:17 # ||| # refresh - END - Fine <== End of execution

Method request is a simple curl:
Last example for reload user with OK status after delete:

## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - START -
## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - SEARCH_NEW_USER - {"document":{"_id":"Feroptimist","status":"waitForPrimaryCache","insertDate":"2024-03-02T07:49:44.927Z","firstCachedDate":null,"lastRefresh":null,"lastDateTouch":"2024-03-02T07:49:44.927Z"}}
## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - NEW_USER - Feroptimist
## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - LOCK_NEW_USER - {"matchedCount":1,"modifiedCount":1}
## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - LOCKED_NEW_USER - User Locked For Update
## A81F53FB69E486D278E26E77 - 2024-03-02T08:53:14 # ||| # add - BUILD_NEW_USER - Start donwload incontri
## A81F53FB69E486D278E26E77 - 2024-03-02T08:55:51 # ||| # add - BUILD_NEW_USER - Start build environment.
## A81F53FB69E486D278E26E77 - 2024-03-02T08:55:51 # ||| # add - BUILD_NEW_USER - Start zipFile
## A81F53FB69E486D278E26E77 - 2024-03-02T08:55:52 # ||| # add - BUILD_NEW_USER - Unlock user
## A81F53FB69E486D278E26E77 - 2024-03-02T08:55:52 # ||| # add - BUILD_NEW_USER - End
## A81F53FB69E486D278E26E77 - 2024-03-02T08:55:52 # ||| # add - END - Fine

Function to call lichess for add o update is the same.

I think there were something betweern my host and lichess in that moment.
#6
A question :
Are you using specific header to trcaking by a Hexadecimal Code transaction ??
I can put in my requests headers to view what append in your system log, view the Hexacode like a Transaction_ID for lichess in other case if append again.
#8
> Yes, i'm sure 404 come from lichess,

Ah, cool.

> Are you using specific header to trcaking by a Hexadecimal Code transaction ??

I don't know what the access logs of Lichess looks like.
Maybe you would need to contact Lichess support directly to inquire if such support is available.
But then, I'm not sure what you would ask about... It seems you don't have a problem at the moment?

It was a one time thing?
Maybe if you figure out how to reproduce the problem,
someone will be able to help you out.

> Unfortunately we don't have more verbose log and we cannot increase verbosity , but on coding, detect 404 response is on time , because if lichess reponse is 404 we delete all user cached information , the user don't exist more.

That you wouldn't be able to add logging seems somewhat troublesome,
for the quest to figure out how to reproduce the problem...

I'm hoping the 404 was a one time thing,
and that you'll only see response codes 200 Success in the future!
#10
We integrated little bit verbosity and append againt tonight 2024-03-05T00:28:36, lichess response is 404.

## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - SEARCH_USER - {"documents":[{"_id":"Feroptimist","status":"active","insertDate":"2024-03-02T07:49:44.927Z","firstCachedDate":"2024-03-02T08:55:52.000Z","lastRefresh":"2024-03-02T08:55:52.000Z","lastDateTouch":"2024-03-02T08:55:52.000Z"}]}
## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - USER - Feroptimist
## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - LOCK_USER -
## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - LOCKED_USER - Utente Lockato Per aggiornamento
## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - UPDATE_USER - Pulizia Cache Plain attuale
## 27E6BC75561C367144A245B3 - 2024-03-05T00:25:56 # ||| # refresh - UPDATE_USER - Avvio donwload incontri
## 27E6BC75561C367144A245B3 - 2024-03-05T00:28:36 # ||| # refresh - LICHESS_RESPONSE - 404
## 27E6BC75561C367144A245B3 - 2024-03-05T00:28:36 # ||| # refresh - DELETE_NEW_USER - Utente inesistente su lichess settato per forget.
## 27E6BC75561C367144A245B3 - 2024-03-05T00:28:36 # ||| # refresh - END - Fine

This topic has been archived and can no longer be replied to.