Ongoing Indexing Delays

Things are much better. See Message 29.

Over the weekend LibraryThing was up and down, and experienced other issues. We are now solidly up, but continue to have problems with system we use for indexing changes to members' catalogs.

As things stand, additions, deletions and edits work. You'll see the books in Add Books and your catalog. Features like clicking on tags and so forth work. But the indexing system for our catalog-search feature is delayed. We are reindexing members' catalogs regularly, but the reindexing is user-by-user, so expect delays.

Our plan is restart regular indexing Tuesday morning (September 10).

Thank you for your patience.

And thank you for keeping us informed!

Seconded! I appreciate the update.

Thanks for update. 😉

Thaks for the information

Yes, thanks for the update. It was a bit wonky at times, not the usual quick response so I knew something was up. Luckily I haven't hit any used book sales lately so have no piles of things to enter. Hm...that's a bummer in a way.

>1 timspalding:
Thank you for the update.

>1 timspalding: I had noticed many months ago that some of my French books were not indexed, and could not be retrieved through searching, and nobody at LT could explain why. I hope this re-indexing will fix things.

OMG. thank you so much - I was really overwrought...

I will take a nap. and try my serious load-ups tomorrow.

Pat M

Thanks for the update Tim, perhaps this is a good time for you to correct the number of reviews on my book? (It should be 5 reviews.)

Nelson

>1 timspalding: Gracias Por la informacion, y su magnifico servicio

>10 Garcia-Gonzalez: Do you mean this one with 8 reviews?

https://www.librarything.com/work/32274280

I'm so glad to hear this was a noted issue, I thought something was wrong with how I was typing/seeing lmao

Just sending some appreciation for the folks who make all this happen.

Thanks for the update. I thought I was doing something wrong.

Thanks so much for announcing what's happening -- and making sure that everyone sees it.

Thanks for keeping us informed. I guess this problem is also affecting the App as well - I've been unable to add books that way all day

>1 timspalding: thanks for all you do so well!

Thanks for the heads up!

I had some trouble accessing the site on Sunday, but assuming it was simply because I was in a place with poor 4G coverage (some other sites were also hard to reach, but LT seemed to be worst affected).

Noticed this as well. Thanks for your hard work to fix it!

Thanks for letting us know what's going on.

Whew! I thought it was me. Thanks for the update!

>1 timspalding: Thanks for the update. I was looking at the website and there's no apparent support contact or status updates.

Thank you!

It still won't let me sign in on the phone app. i never signed out on the PC so I don't if there is a problem there too.

>1 timspalding: Tim, thanks for the LT situation report. I usually start my day on the computer opening Library Thing (along with coffee). Today I started the day with the VA's "My Healthy Vet" -- NOT a user-friendly experience. Thank you for staying on top of LT.

>17 rmharris: The app still isn't working for me. Are you able to use it? If so, what sort of device and operating system do you use?

Okay, here's an update:

1. Site should be up and operating since the morning.
2. App should be too
3. New adds, edits and deletes have been going to the index since this morning.
4. Adds, edits and deletes made during the downtime are still processing. It's going member by member, so you either have the updates or you don't. It should finish in the next 12h. We'll post an update when it happens.

>28 jasbro:

Can you describe how it's not working?

>26 JohnWCuluris:

Still true?

>29 timspalding: Just out of interest, when you have time, can you tell us in general terms what caused the problem?

>30 ocrhdlg: Our queuing system (that normally handles changes to catalog data and the search indexes immediately) crashed and caused the queues to be corrupted. We had to rebuild the queues and then requeue the items that didn't get processed while the queues were down over the weekend.

>30 ocrhdlg: Yeah. It was a serious of interlocked problems. This is my summary. I'm not the sysadmin, so I may have gotten some details wrong.

1. We've had some problems with our queueing system (RabbitMQ). These servers went down over the weekend. They weren't perfect before that, but they were under repair and expected to last the weekend. We believe that bot traffic sent them over the edge (see #2). We fixed this by turning off index updates until developers could fix the system on Monday. The system was fixed yesterday, so new updates started going through. Unfortunately, because the queue was down, we had no record of all the edits we missed. So we've had to reindex everyone who did basically anything on the site during the downtime. That's many thousands of users, doing ALL their books. That's taking a while to eat down.

2. We've been having more and more problems with bad, bot traffic. We've always had a lot of bot traffic. Some was well-behaved (Google), some so-so (sometimes even large companies would ignore robots.txt, our robots directive, or scrape us too fast) and some was bad. But, apart from the DDoS attack we had a year ago, the bad traffic was mostly manageable.

Since LLMs took off, internet traffic has seen huge jumps, as everyone and their brother wants to scrape the whole web to build their AI system. (Some coverage here.) We've seen huge increases, both from established players and mysterious organizations. Even the good players are remarkably cavalier about obeying robots.txt. (There's a good news article about this). The bad ones are… bad.

Anyway, over the weekend we started to get tons of new traffic. Much was from Singapore IPs. (Singapore has been a common culprit for us. I think there are some big data centers there.) It caused us to go down a number of times. It was the worst kind of traffic--incessant and using many, many IPs to avoid being blocked! We solved the traffic problem with some new "rules" on Cloudflare and our servers.

3. Unfortunately, the rules we put in place were faulty--they always worked at first, and then went south when an error happened. So we've be up and happy, the developer or sysadmin would leave, and then the problem would pop up again. The mechanics of it looked like the problem was bad bot traffic, but actually it was the SOLUTION to the bad bot traffic. The problem was a timeout set backwards--a high number not a low. This morning ccatalfo figured out the setting was backwards, and reversed it. Everything started working again perfectly.

Anyway, that's what I got for you, as a non-sysadmin.

To give a little context of scale here: in the last 24 hours we've had over 10 million requests that have been managed by some of our bot rules. These aren't our total served, these are just the ones that we've managed by some rules that we have set up. Even more are getting through the cracks, some are being managed by CloudFlare before they get to our rules, and of course we also have legitimate traffic.

The numbers have been steadily increasing since the AI bots have come on the scene in the last year.

I really feel for all the hard work your team consistently puts in to manage this situation! Thanks for doing it.

>34 reconditereader: Yes, I agree. Many thanks to Tim and his colleagues for their trouble-shooting and these helpful explanations.

Thank you for explaining and for all your hard work. I had no idea what is going on behind the scenes.

That was in "interesting" tale of the "interstices of the interweb".

We take a lot for granted.

Meanwhile I have 3 books sitting in a queue waiting to be uploaded because I won't begin the REAL uploading until those three are in the list.

What is the status of the system? is it still indexing?

I made the error of starting this add collection process over the weekend and it was painful and of course crazy-making.

Thanks

In addition, the books that were added by hand (all 800) ... are not randomly appearing on the main screen but showing up alphabetically. As many of them are by the same author (hey this is a place with old people!) it does not reflect the wonderful variety we do have.

Is that is feature as of this weekend's event???

>29 timspalding: Can you describe how it's not working? No, not really at the moment. And having added 13 books to our library over the weekend (not trying the app), it may be a while before I have a quantity to try it again. (Hope not, but never know!) As I recall the last time I tried the app (still on a dinosaur iPhone 8 Plus, but at least it's iOS 16.7.10), it let me in, went all the way to the barcode scanner, "beeped" in my first item - and then it just froze. Going back a ways (maybe in August? July? before then?), earlier this week I found two twice-entered records from an earlier attempt with the app; as I recall, it was freezing up then too, and not realizing those works had actually made it to our library, I tried, tried again. If any other specifics would help, please let me know; thanks.

>That sure puts li'l ol' me with my obsessions & predilections in stark perspective!

>39 jasbro: The iPhone 8's freezing after the beep after a scan is kinda known issue - we had been stuck with it for years now. :)

The indexing is still going. Unfortunately a couple of the indexers died overnight and we had to restart them this morning. But progress continues. We will update as soon as it finishes but you might find your catalog done well before that. Searching for your most recently added book (or, more specifically, books added Sunday-Monday) and finding it in the results is a good indicator that the indexer has completed your catalog.

The iPhone 8 has a hardware issue (that ONLY presents on iPhone 8, no other variation of the iPhone). We've been trying to work around that issue with various permutations but haven't been able to solve it.

>42 conceptDawg: oh interesting. I have noticed an issue with iphone 8 and camera view for ISBN bar codes. It worked really well at first, now just drags (for some time)

>41 conceptDawg: Unfortunately a couple of the indexers died overnight ...

Since an "indexer" has in the past been a type of human worker in publishing, this phrase was highly alarming at first glance. I presume, though, that these were machines that needed to be "restarted."

>1 timspalding: i just joined and added one book to my library, “dead men do tell tales”. I want to read it but can’t find a button to do so. Help. Gary pederson

>45 garypederson: You cannot. LT is a cataloging site, not a library site. You cannot read books here.

LT contains links to places where you can search for the book and eventually find it (for free or more often to buy it) but the site itself does not contain any book. For these links, look at the Quick links menu on the work page.

>42 conceptDawg: Does that include the 8 Plus? (I’m guessing “yes,” but it seems worth asking.)

>44 paradoxosalpha: My thought exactly!

Thank you for all the work you do!

>41 conceptDawg: Could you say whether or not MARC imports are affected by this problem? I've been trying over the last few days to import several hundred works (mostly without ISBNs) but nothing has been processed AFAIK. Also, is there any way to view a list of background jobs and their status, so we could check?

I cannot upload only 4 books w universal import.

>49 GuildfordInstitute: We've had some recurring problems with our queuing system this week. I'll let "the other Chris" comment on this when he comes in on Monday. He's our resident import-queue guru—and also happens to be our MARC guru also, so bonus.

Queueing system might still be funky. I got about 100 of about 400 imported and then it stalled.

>49 GuildfordInstitute: the imports are really a separate system and so wouldn't have been affected by these queue issues. That said, the import system has its own set of ongoing "infelicities" (for example no way to see the status of current or past ones) which we need to address.

>52 ryasef: Sorry about this: yes, i can see on our side that only about a quarter have finished importing. File looks fine so need to figure out what's holding them up.

>53 ccatalfo: If it's helpful, it looks like all the Overcat ones worked but anything that wasn't found in Overcat (which was Option 1) is now stalled.

>53 ccatalfo: Can confirm the stall continues; I tried manually uploading 4 of them again and I think they are also stalled? For what it's worth, I did change my username between the big upload a day or two ago and now.

>55 ryasef: Thanks for the updates - we've restarted some indexing processes as of yesterday afternoon and that has kicked things into a higher, more normal, gear.

Could you try importing once again - it looks as if you aborted the last few (out of frustration, I bet)?

>57 pomotov:
By chance would that be December 10th, or has LT discovered the secret of time-travel and is sending an engineer to start the process early? Pre-emptive strike?

>58 pgmcc: It seemed like maybe they were summarizing the thread from earlier? If so, our reindexing being slow earlier this week is a separate issue from September's.

>59 kristilabrie:
The use of future tense in >57 pomotov: led me to believe September was a typo. As it happens September 10th 2024 was a Tuesday, as is December 10th 2024.

It would be a nice trick if LT had time-travelling software engineers to fix bugs that appear retrospectively. That would, of course, raise the question about fixing a bug in the past resulting in the present being different as the bug would not appear, so does the engineer return to the present find the world to be a totally different place? What happens if the engineer causes another bug in the past? Will that lead to total catastrophe? I am sure these issues would all be discussed in the user manual.

>60 pgmcc: The momentum of the time stream will ensure that there's a different but commensurate regression with every patch. The engineer returns to a system that is basically equally broken, but in a different place.

Of course, we can predict that the engineer spends the rest of their existence jumping back in time to fix a bug which causes a bug which they have to jump back in time to fix. This looks like a time loop but in fact it's just bad HR policy. (Maybe that only happens in Mirror Universe LT.)

It's a common spammer tactic to post a message which is an AI-generated summary of the thread, which makes the message seem relevant and therefore not suspicious, and then come back later and edit the message to include a spam link. Which is indeed what happened here.

>62 norabelle414: Great! Something new to look forward to. :)

>62 norabelle414:

Ah, the ongoing ensh*ttification of the 'net.