From Counter-Vandalism Wiki
- 05:18 Krinkle: Moved CVNBot14 from email@example.com to cvn-app1.wmflabs
- 01:33 Krinkle: Moved CVN-ClerkBot to Labs
- 01:09 Krinkle: cvn-app1 somehow got disconnected (might be due to a netsplit). Manually killed processes, stillalive started them again.
- 05:21 Krinkle: Moved to labs: KrinkleBot2, KrinkleBot4, KrinkleBot5
- 05:13 Krinkle: Log files of processes running in our cluster at wmflabs: http://cvn.wmflabs.org/logs/
- 18:45 Krinkle: KrinkleBot7 has set foot on Labs. One small step for a bot, one giant leap for the Countervandalism Network!
- 09:59 Krinkle: KrinkleBot4 was restarted earlier today but got alt nick (KrinkleBot4_), not voiced and not working in #cvn-wp-es, fixed after reboot.
- 22:06 Krinkle: New bot running on ts-willow: KrinkleBot8 (#cvn-ja, ja.wiki scan channel)
- 20:12 Krinkle: Database maintenance (CVNBot14, KrinkleBot[2,4,5,7])
- 05:40 Seahorse: Yarrow does not have mono installed, restarting bots on Willow
- 05:39 Seahorse: Attempting to migrate SeahorseRC bots to yarrow
- 05:29 Seahorse: Toolserver ~cvn bots (And my bots) are performing poorly. They crash every 2 hours or so and often refuse to start from the web interface. Unable to access p_cvn.
- 03:33 Krinkle: Quited FurryRC4, was only flooding with database connection issues in cvn-broadcast
- 23:07 Krinkle: Toolserver issues cause ts-willow control panel to be broken. Affected bots: Servers#ts-willow. Bots currently running will continue to do so. But we can't stop/start any.
- 22:26 Krinkle: Toolserver had a short hickup earlier. KrinkleBot7 didn't recover. Restarted.
- 21:35 Krinkle: ts-willow issues have been resolved by Toolserver. Control panel back up. Starting KrinkleBot4.
- 15:47 Krinkle: Toolserver issues (affects bots on ts-willow) - bots currently running are okay, but control panel is unresponsive.
- 08:46 Krinkle: Quit SeahorseRC2 in #cvn-zh-scan, was double with FurryRC4
- 21:14 Krinkle: Fixed
CTFBot.init. Started Cluekie, restart works again now.
- 21:05 Krinkle: Cluekie has been down since a failed restart on March 24
- 22:41 Seahorse: SeahorseRC4 will be running as the primary bot for #cvn-simplewikis until further notice.
- 22:41 Firebolt: CVN-ClerkBot 1.2.2 deployed, cLogger retired
- 06:55 Krinkle: Quit cLogger and removed it from ts-willow/krinkle SGE crontab; now obsolete with CVN-ClerkBot 1.2.2
- 23:32 Krinkle: CVN Control Panel 0.8.3 deployed
- 23:01 Krinkle: ts-willow is back up
- 22:58 Krinkle: ts-willow is having issues. Bots are timing out one by one.
- 01:34 Krinkle: Deployed Console.msgs nl-rev19 (2012-03-14) to KrinkleBot7
- 03:07 Krinkle: CVN Control Panel 0.8.2 deployed
- 03:01 Krinkle: All swmtbots are back on 'mono' now
- 02:44 Krinkle: Testing CVNBot14 under 'mono' instead of 'mono.old'. Toolserver upgrade to new mono introduced a fatal bug for SWMTBot, previous version was made available under 'mono.old'. Bug is fixed?
- 02:40 Krinkle: r61 compiled successfully. Replaced _BotTemplate with a stock r61, and deployed to CVNBot14.
- 21:15 Krinkle: Taking CVNBot14 down for database maintenance and test run the new SWMTBot beta version on it (r61)
- 17:29 Krinkle: Compiled SWMTBot 1.20beta (r61) from source, uploading to ts-willow/cvn for testing later tonight
- 02:07 Seahorse: SeahorseRC9 to soon be moved to ts_cvn, renamed as a CVNBot.
- 23:16 Krinkle: IDEA:
- 10:23 Krinkle: SentryBot dropped out of
hyperspaceIRC. Still connected but not reporting anything. Restart fixed it.
- 06:41 Krinkle: cvn-srv2 dropped for a few minutes but is back up
- 21:44 Firebolt: Renamed CVNBot16 (for #cvn-newpages) to FurryRC3
- 02:59 Krinkle: CVN API stopped syncing last night after the swmtbot instances were moved to a new subdirectory. 5-minute crontab for CVN API sync has been fixed and is back up.
- 01:53 Seahorse: CVNBot16 migrated to ts ~cvn successfully. Also included on the control panel.
- 19:47 Seahorse: srv-2 went down for a very short period. The bots went down from a ping timeout (They weren't connected). I have restarted all of the bots successfully.
- 07:50 Seahorse: rcbot tests seem successful. No obvious lag from any other bots. I am quitting rcbot for tonight, and disabling it on the ts-cvn control panel.
- 00:55 Krinkle: Move of Cluekie complete. Stored in /h/cvn/bots/CTFBot. Moved start command to /h/cvn/backend/custom/cluekie-start.sh (modified to fit new paths)
- 21:29 Krinkle: Directory structure update complete (bots are now all in /h/cvn/bots and swmtbot instances in /h/cvn/bots/swmtbots). Re-started all ts-willow primary bots
- 20:53 Krinkle: Disabling all processes on ts-willow/cvn for slight directory reorganization
- 20:48 Krinkle: Moving Cluekie to ts-willow/cvn
- 20:46 Krinkle: Cluekie was also in ts-willow/krinkle long-run SGE queue, but kept disabled for now since its configuration need an update per cluenet-channel on the source irc server was renamed
- 20:45 Krinkle: cLogger was having issues due to the garbaged that ended up in the SGE qsub for ts-willow/krinkle. Queue cleared and minutely crontab submission re-enabled. Should be fine now.
- 18:53 Seahorse: Toolserver issues resolved. All bots started successfully.
- 03:50 Seahorse: CVNBot5 started for #cvn-meta
- 03:32 Seahorse: Wikimedia Toolserver CVN Bots semi-outage. Upon an attempt to restart them, an error is returned: + Abort (core dumped) nohup mono SWMTBot.exe (wd: ~/SeahorseRC2). Similar errors are now occurring even when using similar commands like "ls"
- 20:30 Krinkle: bug 34508 has been mostly fixed. Tracking progress here: http://etherpad.wikimedia.org/IRCBot-Messages
- 20:03 Seahorse: CVNBot14 behaving unusually. Lagging behind w/ edits and crashing upon certain commands such as getadmins.
- 06:07 Seahorse: Bugzilla ticket opened on behalf of CVN/others: https://bugzilla.wikimedia.org/show_bug.cgi?id=34508
- 20:07 Seahorse: Network restart. 14 bots were online. No lags before restarting. All 14 bots successfully restarted.
- 20:07 Seahorse: MediaWiki update has broken some functions of the bots. A patch will be released as soon as possible.
- 02:19 Seahorse: Cluekie not functioning because ClueBot-NG's ClueNet feed is not functioning. Asking ClueNet ops about it now.
- 02:00 Seahorse: Updated SentryBot's Admin & Bot lists via command. An update hadn't been performed for several weeks.
- 20:36 Krinkle: Taking down CVNBot14 for database clean up (removing old entries). Hoping to speed up the bot now that commons is having lots of activity and lots of gadget users querying the database as well
- 01:57 Krinkle: [ts-willow] Updated SWMTBot.ini of all bots to use 'mono.old' instead of 'mono' for restartarg
- 01:56 Krinkle: [ts-willow] Updated services.ini to use 'mono.old' instead of 'mono' for starting bots
- 22:50 Krinkle: Toolserver provided 'mono.old', ts-willow bots are back up
- 22:50 Krinkle: Toolserver updated mono and broke ts-willow bots. Currently running bots are OK. start/restart/quit is fatal.