1timspalding
I'm going to start a system status thread, so members have a central place to find info on any general problems with the site. And a place to discuss them, if needed.
This isn't thread for reporting individual bugs. We understand that members (and even staff!) can't always distinguish between specific bugs and system problems, so please don't get mad at anyone for talking about a "mere" bug.
The top post (here) will always link to the current update:
CURRENT POST: https://www.librarything.com/topic/365080#8656883
This isn't thread for reporting individual bugs. We understand that members (and even staff!) can't always distinguish between specific bugs and system problems, so please don't get mad at anyone for talking about a "mere" bug.
The top post (here) will always link to the current update:
CURRENT POST: https://www.librarything.com/topic/365080#8656883
2timspalding
System Status: Saturday, October 19, 2024, 1:30pm Eastern
We have three issues going on now:
Explanation for problem 1-2:
Last week we had had problems with a server, which turned out to be an overheated processor—it probably overheated because it's now faulty. This server hosted a number of "virtual" servers, which handled various different services we need to run the site.
When the server went down, we lost a number of services outright. These Ganawa found new "homes" for quickly. So for example, our site search and library search were rehomed quickly. But we also lost "portions" of other services, including our non-logged-in traffic.
By "non-logged-in traffic" I mean that LibraryThing's main website is split into two sets of servers--one set for logged-in members and one set for logged-out members. We do this so that periodic "scraping" attacks from search engines or (increasingly) AI bots, don't affect the experience for logged-in members.
Thus, while we didn't lose our ability to serve traffic to non-logged-in members, we lost a lot of our "muscle," and have been struggling. We also lost some of our resources for image generation, and are struggling there too. (Struggling means, for example, that our non-logged-in webserver is running at 500% of normal load!)
Our sysadmin, Ganawa, is currently "finding homes" for all the remaining virtual servers we lost. That's the first priority. Some time next week we will also probably be going into the server farm to replace the chip. If that's successful, a lot of time will go into moving services around again.
If LibraryThing were much larger and richer, we might be able to handle problems with more grace. We are lucky to have a degree of redundancy--some instant, some requiring back-end work. But we aren't like Amazon where--apparently--they literally turn off entire data centers now and then to confirm that the other data centers can instantly step in for them!
We have three issues going on now:
- Slowness and Cloudflare error pages on non-logged-in LibraryThing. The site is slow when you are logged out, and pages may error out. So members reported Intermittent Login Errors. If this happens to you, just keep trying. Once you're in, things will be fast!
- Slow pics Some LibraryThing Images, such as profile images and covers, Are Slow or Fail To Load. The problem happens with new images or when an image needs to be made at a new size.
- Wiki saving. The Wiki has a problem with saving, which may be a "System"-level problem. See this bug.. We aren't sure what the problem is. The Wiki itself runs on MediaWiki/Wikipedia software, on servers separate from the rest of LibraryThing. (It's separated for security reasons.) We have only just started to look the issue, which might be caching, database problems or server problems. It's probably not related to 1-2.
Explanation for problem 1-2:
Last week we had had problems with a server, which turned out to be an overheated processor—it probably overheated because it's now faulty. This server hosted a number of "virtual" servers, which handled various different services we need to run the site.
When the server went down, we lost a number of services outright. These Ganawa found new "homes" for quickly. So for example, our site search and library search were rehomed quickly. But we also lost "portions" of other services, including our non-logged-in traffic.
By "non-logged-in traffic" I mean that LibraryThing's main website is split into two sets of servers--one set for logged-in members and one set for logged-out members. We do this so that periodic "scraping" attacks from search engines or (increasingly) AI bots, don't affect the experience for logged-in members.
Thus, while we didn't lose our ability to serve traffic to non-logged-in members, we lost a lot of our "muscle," and have been struggling. We also lost some of our resources for image generation, and are struggling there too. (Struggling means, for example, that our non-logged-in webserver is running at 500% of normal load!)
Our sysadmin, Ganawa, is currently "finding homes" for all the remaining virtual servers we lost. That's the first priority. Some time next week we will also probably be going into the server farm to replace the chip. If that's successful, a lot of time will go into moving services around again.
If LibraryThing were much larger and richer, we might be able to handle problems with more grace. We are lucky to have a degree of redundancy--some instant, some requiring back-end work. But we aren't like Amazon where--apparently--they literally turn off entire data centers now and then to confirm that the other data centers can instantly step in for them!
3Maddz
Can I make a suggestion? Put a perma-link to this thread on the home page, perhaps with a last updated date/time stamp. At least then there's a single place to check the system status.
4timspalding
>3 Maddz:
Thanks. I'm not sure we'll do exactly that, but I'd like to think of ways to make this a more general thing.
Thanks. I'm not sure we'll do exactly that, but I'd like to think of ways to make this a more general thing.
5norabelle414
Could you pin it to the top of the group, at least?
6ccatalfo
Just wanted to post a quick update: we are still working on all of this. We are _hoping_ to have a better "pics" setup running later today, as well as possibly for the non-logged-in site.
7conceptDawg
The pics system has been updated to use a whole new set of servers. So book covers and gallery images should be working at full-force again. Let us know if you see any issues.
This should alleviate point #2 in the original post.
This should alleviate point #2 in the original post.
8conceptDawg
We are hoping to roll out a new set of servers for our non-member requests in the morning. That should alleviate point #1 in the original post.
10MarthaJeanne
>9 timspalding: Thank you. That will be very useful.
11birder4106
Despite everything, this is good news.
Thank you for your efforts.
I was afraid that the delays were due to my PC, which is already a bit old.
Thank you for your efforts.
I was afraid that the delays were due to my PC, which is already a bit old.
12Avron
issue 3 may have been less of an issue than first thought. I have just gone to a page that appears to have saved when I attempted to do so at the weekend. It didn't display any changes then, does display them now.
13elenchus
>12 Avron:
Can confirm similar situation with me: my edit that didn't show over the weekend, now displays. So perhaps the edits were successful but somehow weren't displayed as such immediately.
Can confirm similar situation with me: my edit that didn't show over the weekend, now displays. So perhaps the edits were successful but somehow weren't displayed as such immediately.
14KeithChaffee
On the other hand, I just attempted to add something to the MysteryKit wiki, and it didn't save.
15conceptDawg
We are still working on the wiki (issue #3).
Issues #1 and #2 should be corrected and working normally (even better than before).
Issues #1 and #2 should be corrected and working normally (even better than before).
16Tess_W
probably the wrong place, probably a dumb q.......where are the instructions for adding a Wiki page to a current wiki?
17kristilabrie
>16 Tess_W: Is the wiki page you want to add to a current wiki in existence yet, or do you need to create a brand new wiki page to add to another?
If the page you want to link to already exists, you can just link to that page from the "landing" page you want to navigate from. If it doesn't exist yet, search for the nonexistent page by the title you want to use for it: you'll see a "Create the page "your page title here" on this wiki!" where you can click on the page title to create the new page.
The left-hand "Help" (under "Browse") on the Wiki (e.g. https://wiki.librarything.com/index.php/Main_Page) will give you various help articles on editing wikis in general, including how to start a wiki page, link to other wiki pages, etc.. https://www.mediawiki.org/wiki/Help:Contents
If the page you want to link to already exists, you can just link to that page from the "landing" page you want to navigate from. If it doesn't exist yet, search for the nonexistent page by the title you want to use for it: you'll see a "Create the page "your page title here" on this wiki!" where you can click on the page title to create the new page.
The left-hand "Help" (under "Browse") on the Wiki (e.g. https://wiki.librarything.com/index.php/Main_Page) will give you various help articles on editing wikis in general, including how to start a wiki page, link to other wiki pages, etc.. https://www.mediawiki.org/wiki/Help:Contents
18Tess_W
>17 kristilabrie: Thank you! I've been to all those pages, and there must be one little thing I'm missing!
To be explicit:
This page I created: https://wiki.librarything.com/index.php/Current_Quarterly_Read
need to go here:https://wiki.librarything.com/index.php/Reading_Through_Time_Quarterly_Theme_Reads
OR https://wiki.librarything.com/index.php/Reading_Through_Time_Quarterly_Theme_Rea... There is no edit button, so I could edit this to be the page I created.......
Any help is appreciated!
To be explicit:
This page I created: https://wiki.librarything.com/index.php/Current_Quarterly_Read
need to go here:https://wiki.librarything.com/index.php/Reading_Through_Time_Quarterly_Theme_Reads
OR https://wiki.librarything.com/index.php/Reading_Through_Time_Quarterly_Theme_Rea... There is no edit button, so I could edit this to be the page I created.......
Any help is appreciated!
19AnnieMod
>18 Tess_W: Look under the table of contents, all the way to the right. The edit for the almost whole page is there and opens: https://wiki.librarything.com/index.php?title=Reading_Through_Time_Quarterly_The...
PS: the link on the left menu opens https://wiki.librarything.com/index.php?title=Reading_Through_Time_Quarterly_The... which allows you to edit the whole page.
PS: the link on the left menu opens https://wiki.librarything.com/index.php?title=Reading_Through_Time_Quarterly_The... which allows you to edit the whole page.
20timspalding
System Status: Saturday, October 25, 2024, 6:00pm Eastern
There are no current issues.
1. The regular production servers are no longer having slowness problems.
2. The "pics" servers are in good shape.
3. The wiki-saving issue has been solved. It was an error in our Cloudlare caching logic. If you don't see a change, wait an hour or two. There's the potential for caching the change to caching.
Feel free to report issues, but remember that a bug is not a system issue. This is for system issues.
There are no current issues.
1. The regular production servers are no longer having slowness problems.
2. The "pics" servers are in good shape.
3. The wiki-saving issue has been solved. It was an error in our Cloudlare caching logic. If you don't see a change, wait an hour or two. There's the potential for caching the change to caching.
Feel free to report issues, but remember that a bug is not a system issue. This is for system issues.
22timspalding
Boo!
23bnielsen
>21 MrAndrew: and >22 timspalding: đź‘»
24timspalding
System Status: Saturday, October 29, 2024, 9:40pm Eastern
There are no current issues.
Last night we had problems with our library search systems—searching external libraries and Overcat. A load balancer and a search system died. It was fixed around 9am.
There are no current issues.
Last night we had problems with our library search systems—searching external libraries and Overcat. A load balancer and a search system died. It was fixed around 9am.
25MarthaJeanne
>24 timspalding: Your time is crazy mixed ip.
"Saturday, October 29, 2024, 9:40pm Eastern"
I think you mean Tuesday, October 29, 2024, 9:40am Eastern
"Saturday, October 29, 2024, 9:40pm Eastern"
I think you mean Tuesday, October 29, 2024, 9:40am Eastern
27Keeline
This may be a manifestation of #2 on the initial list. I am seeing some cases where cover images captured in the iOS app are not always being updated in the app and on the site. Usually I have just tried to capture the image again and it seems to work as I process upgrades in the collections.
I have one that is being a little stranger today ( https://www.librarything.com/work/27377612/edit/209068795 ) where the app shows the new cover with the dust jacket. The list view shows the cover. But the detail view ( linked above ) does not with shift-reloads (Chrome on Mac while logged in) or while in Incognito Mode.
So I am not sure if "There are no current issues" is 100% true.
It feels like a propagation issue but you know your systems better than I do as an outsider with extensive Linux sysadmin experience.
After enough strangeness in behavior with this one, I decided to look for any discussion of the issue and found this thread which seems to be connected and the sort of thing that could cause the issue.
ETA: In the time it took me to find the thread and write the reply, this particular example now loads the image. It still suggests a slower propagation of images between systems. But it does mean that if you look at the link for evidence of a problem, you may not find it.
ETA2: I will have to give up my day's project to upgrade my listings with new copies. Attempts to capture images are simply NOT working reliably. On this one https://www.librarything.com/work/15395575/book/274362012 the upload on the iOS app started and then timed out and defaulted to the generic book cover. The website does not show it either. I don't want to have to repeat my efforts for each cover I am attempting to update. Something is wrong in moving images from one system to another and I hope Tim or others can work on this.
James
I have one that is being a little stranger today ( https://www.librarything.com/work/27377612/edit/209068795 ) where the app shows the new cover with the dust jacket. The list view shows the cover. But the detail view ( linked above ) does not with shift-reloads (Chrome on Mac while logged in) or while in Incognito Mode.
So I am not sure if "There are no current issues" is 100% true.
It feels like a propagation issue but you know your systems better than I do as an outsider with extensive Linux sysadmin experience.
After enough strangeness in behavior with this one, I decided to look for any discussion of the issue and found this thread which seems to be connected and the sort of thing that could cause the issue.
ETA: In the time it took me to find the thread and write the reply, this particular example now loads the image. It still suggests a slower propagation of images between systems. But it does mean that if you look at the link for evidence of a problem, you may not find it.
ETA2: I will have to give up my day's project to upgrade my listings with new copies. Attempts to capture images are simply NOT working reliably. On this one https://www.librarything.com/work/15395575/book/274362012 the upload on the iOS app started and then timed out and defaulted to the generic book cover. The website does not show it either. I don't want to have to repeat my efforts for each cover I am attempting to update. Something is wrong in moving images from one system to another and I hope Tim or others can work on this.
James
