Developer Log

March 29

• 01:33 Krinkle: Moved CVN-ClerkBot to Labs

March 14

• 01:09 Krinkle: cvn-app1 somehow got disconnected (might be due to a netsplit). Manually killed processes, stillalive started them again.

February 27

• 05:21 Krinkle: Moved to labs: KrinkleBot2, KrinkleBot4, KrinkleBot5
• 05:13 Krinkle: Log files of processes running in our cluster at wmflabs: http://cvn.wmflabs.org/logs/

February 26

• 18:45 Krinkle: KrinkleBot7 has set foot on Labs. One small step for a bot, one giant leap for the Countervandalism Network!

January 2

• 09:59 Krinkle: KrinkleBot4 was restarted earlier today but got alt nick (KrinkleBot4_), not voiced and not working in #cvn-wp-es, fixed after reboot.

November 26

• 22:06 Krinkle: New bot running on ts-willow: KrinkleBot8 (#cvn-ja, ja.wiki scan channel)

September 25

• 20:12 Krinkle: Database maintenance (CVNBot14, KrinkleBot[2,4,5,7])

June 11

• 05:40 Seahorse: Yarrow does not have mono installed, restarting bots on Willow
• 05:39 Seahorse: Attempting to migrate SeahorseRC bots to yarrow
• 05:29 Seahorse: Toolserver ~cvn bots (And my bots) are performing poorly. They crash every 2 hours or so and often refuse to start from the web interface. Unable to access p_cvn.

May 15

• 03:33 Krinkle: Quited FurryRC4, was only flooding with database connection issues in cvn-broadcast

May 14

• 23:07 Krinkle: Toolserver issues cause ts-willow control panel to be broken. Affected bots: Servers#ts-willow. Bots currently running will continue to do so. But we can't stop/start any.

May 7

• 22:26 Krinkle: Toolserver had a short hickup earlier. KrinkleBot7 didn't recover. Restarted.

April 30

• 21:35 Krinkle: ts-willow issues have been resolved by Toolserver. Control panel back up. Starting KrinkleBot4.
• 15:47 Krinkle: Toolserver issues (affects bots on ts-willow) - bots currently running are okay, but control panel is unresponsive.

April 14

• 08:46 Krinkle: Quit SeahorseRC2 in #cvn-zh-scan, was double with FurryRC4

April 3

• 21:14 Krinkle: Fixed restartarg in CTFBot.init. Started Cluekie, restart works again now.
• 21:05 Krinkle: Cluekie has been down since a failed restart on March 24

March 18

• 22:41 Seahorse: SeahorseRC4 will be running as the primary bot for #cvn-simplewikis until further notice.
• 22:41 Firebolt: CVN-ClerkBot 1.2.2 deployed, cLogger retired

March 16

• 06:55 Krinkle: Quit cLogger and removed it from ts-willow/krinkle SGE crontab; now obsolete with CVN-ClerkBot 1.2.2

March 15

• 23:32 Krinkle: CVN Control Panel 0.8.3 deployed
• 23:01 Krinkle: ts-willow is back up
• 22:58 Krinkle: ts-willow is having issues. Bots are timing out one by one.

March 14

• 01:34 Krinkle: Deployed Console.msgs nl-rev19 (2012-03-14) to KrinkleBot7

March 12

• 03:07 Krinkle: CVN Control Panel 0.8.2 deployed

March 11

• 03:01 Krinkle: All swmtbots are back on 'mono' now
• 02:44 Krinkle: Testing CVNBot14 under 'mono' instead of 'mono.old'. Toolserver upgrade to new mono introduced a fatal bug for SWMTBot, previous version was made available under 'mono.old'. Bug is fixed?
• 02:40 Krinkle: r61 compiled successfully. Replaced _BotTemplate with a stock r61, and deployed to CVNBot14.

March 10

• 21:15 Krinkle: Taking CVNBot14 down for database maintenance and test run the new SWMTBot beta version on it (r61)
• 17:29 Krinkle: Compiled SWMTBot 1.20beta (r61) from source, uploading to ts-willow/cvn for testing later tonight

March 9

• 02:07 Seahorse: SeahorseRC9 to soon be moved to ts_cvn, renamed as a CVNBot.

March 8

• 23:16 Krinkle: IDEA:
• 10:23 Krinkle: SentryBot dropped out of hyperspace IRC. Still connected but not reporting anything. Restart fixed it.
• 06:41 Krinkle: cvn-srv2 dropped for a few minutes but is back up

March 6

• 21:44 Firebolt: Renamed CVNBot16 (for #cvn-newpages) to FurryRC3

March 5

• 02:59 Krinkle: CVN API stopped syncing last night after the swmtbot instances were moved to a new subdirectory. 5-minute crontab for CVN API sync has been fixed and is back up.
• 01:53 Seahorse: CVNBot16 migrated to ts ~cvn successfully. Also included on the control panel.

March 4

• 19:47 Seahorse: srv-2 went down for a very short period. The bots went down from a ping timeout (They weren't connected). I have restarted all of the bots successfully.
• 07:50 Seahorse: rcbot tests seem successful. No obvious lag from any other bots. I am quitting rcbot for tonight, and disabling it on the ts-cvn control panel.
• 00:55 Krinkle: Move of Cluekie complete. Stored in /h/cvn/bots/CTFBot. Moved start command to /h/cvn/backend/custom/cluekie-start.sh (modified to fit new paths)

March 3

• 21:29 Krinkle: Directory structure update complete (bots are now all in /h/cvn/bots and swmtbot instances in /h/cvn/bots/swmtbots). Re-started all ts-willow primary bots
• 20:53 Krinkle: Disabling all processes on ts-willow/cvn for slight directory reorganization
• 20:48 Krinkle: Moving Cluekie to ts-willow/cvn
• 20:46 Krinkle: Cluekie was also in ts-willow/krinkle long-run SGE queue, but kept disabled for now since its configuration need an update per cluenet-channel on the source irc server was renamed
• 20:45 Krinkle: cLogger was having issues due to the garbaged that ended up in the SGE qsub for ts-willow/krinkle. Queue cleared and minutely crontab submission re-enabled. Should be fine now.
• 18:53 Seahorse: Toolserver issues resolved. All bots started successfully.
• 03:50 Seahorse: CVNBot5 started for #cvn-meta
• 03:32 Seahorse: Wikimedia Toolserver CVN Bots semi-outage. Upon an attempt to restart them, an error is returned: [1]+ Abort (core dumped) nohup mono SWMTBot.exe (wd: ~/SeahorseRC2). Similar errors are now occurring even when using similar commands like "ls"

February 26

• 20:30 Krinkle: bug 34508 has been mostly fixed. Tracking progress here: http://etherpad.wikimedia.org/IRCBot-Messages
• 20:03 Seahorse: CVNBot14 behaving unusually. Lagging behind w/ edits and crashing upon certain commands such as getadmins.

February 20

• 06:07 Seahorse: Bugzilla ticket opened on behalf of CVN/others: https://bugzilla.wikimedia.org/show_bug.cgi?id=34508

February 18

• 20:07 Seahorse: Network restart. 14 bots were online. No lags before restarting. All 14 bots successfully restarted.
• 20:07 Seahorse: MediaWiki update has broken some functions of the bots. A patch will be released as soon as possible.

February 5

• 02:19 Seahorse: Cluekie not functioning because ClueBot-NG's ClueNet feed is not functioning. Asking ClueNet ops about it now.
• 02:00 Seahorse: Updated SentryBot's Admin & Bot lists via command. An update hadn't been performed for several weeks.

January 29

• 20:36 Krinkle: Taking down CVNBot14 for database clean up (removing old entries). Hoping to speed up the bot now that commons is having lots of activity and lots of gadget users querying the database as well

January 16

• 01:57 Krinkle: [ts-willow] Updated SWMTBot.ini of all bots to use 'mono.old' instead of 'mono' for restartarg
• 01:56 Krinkle: [ts-willow] Updated services.ini to use 'mono.old' instead of 'mono' for starting bots

January 15

• 22:50 Krinkle: Toolserver provided 'mono.old', ts-willow bots are back up

January 11

• 22:50 Krinkle: Toolserver updated mono and broke ts-willow bots. Currently running bots are OK. start/restart/quit is fatal.

Archives

• ...
• Developer Log/Archive 4 (2010 Dec - 2011 Dec)
• Developer Log/Archive 3 (2010 July - 2010 Dec)
• Developer Log/Archive 2 (2008 Jan - 2009 Oct)
• Developer Log/Archive 1 (2006 Mar - 2007 Dec)