August 24, 2005
Outage Update: In order to speedily recover from a backlog of results to validate, we decided keep the public servers off-line throughout the evening (Pacific Time). We apologize for the inconvenience, but credit that has been pending for weeks is now being granted. More information in Technical News.
Technical News
August 24, 2005 - 22:00 UTC
Once the length of an outage goes beyond a certain point, the potential deluge of users hitting our servers upon return cannot possibly get any worse (because at some point everybody is waiting for work). That said, we decided to keep the scheduler off again for at least another night so we could battle the current problems some more.
To reiterate, the current problems are (a) a severe backlog of results to validate, and (b) the disks array holding the results/workunits is filling up. We let the "antique" deleter run all night last night - it has removed 70% of antique results so far (for more information about this see the previous posts below).
After our normal database backup, we then turned on all the backend processes (validation, assimilation, and regular file deletion) so that all those queues could drain. Actually, only the validator queue is the bottleneck, but as it drains the assimilator/file deleters have to work to keep up.
