Developer Log

April 14

08:46 Krinkle: Quit SeahorseRC2 in #cvn-zh-scan, was double with FurryRC4

April 3

21:14 Krinkle: Fixed restartarg in CTFBot.init. Started Cluekie, restart works again now.
21:05 Krinkle: Cluekie has been down since a failed restart on March 24

March 18

22:41 Seahorse: SeahorseRC4 will be running as the primary bot for #cvn-simplewikis until further notice.
22:41 Firebolt: CVN-ClerkBot 1.2.2 deployed, cLogger retired

March 16

06:55 Krinkle: Quit cLogger and removed it from ts-willow/krinkle SGE crontab; now obsolete with CVN-ClerkBot 1.2.2

March 15

23:32 Krinkle: CVN Control Panel 0.8.3 deployed
23:01 Krinkle: ts-willow is back up
22:58 Krinkle: ts-willow is having issues. Bots are timing out one by one.

March 14

01:34 Krinkle: Deployed Console.msgs nl-rev19 (2012-03-14) to KrinkleBot7

March 12

03:07 Krinkle: CVN Control Panel 0.8.2 deployed

March 11

03:01 Krinkle: All swmtbots are back on 'mono' now
02:44 Krinkle: Testing CVNBot14 under 'mono' instead of 'mono.old'. Toolserver upgrade to new mono introduced a fatal bug for SWMTBot, previous version was made available under 'mono.old'. Bug is fixed?
02:40 Krinkle: r61 compiled successfully. Replaced _BotTemplate with a stock r61, and deployed to CVNBot14.

March 10

21:15 Krinkle: Taking CVNBot14 down for database maintenance and test run the new SWMTBot beta version on it (r61)
17:29 Krinkle: Compiled SWMTBot 1.20beta (r61) from source, uploading to ts-willow/cvn for testing later tonight

March 9

02:07 Seahorse: SeahorseRC9 to soon be moved to ts_cvn, renamed as a CVNBot.

March 8

23:16 Krinkle: IDEA:
10:23 Krinkle: SentryBot dropped out of ~~hyperspace~~ IRC. Still connected but not reporting anything. Restart fixed it.
06:41 Krinkle: cvn-srv2 dropped for a few minutes but is back up

March 6

21:44 Firebolt: Renamed CVNBot16 (for #cvn-newpages) to FurryRC3

March 5

02:59 Krinkle: CVN API stopped syncing last night after the swmtbot instances were moved to a new subdirectory. 5-minute crontab for CVN API sync has been fixed and is back up.
01:53 Seahorse: CVNBot16 migrated to ts ~cvn successfully. Also included on the control panel.

March 4

19:47 Seahorse: srv-2 went down for a very short period. The bots went down from a ping timeout (They weren't connected). I have restarted all of the bots successfully.
07:50 Seahorse: rcbot tests seem successful. No obvious lag from any other bots. I am quitting rcbot for tonight, and disabling it on the ts-cvn control panel.
00:55 Krinkle: Move of Cluekie complete. Stored in /h/cvn/bots/CTFBot. Moved start command to /h/cvn/backend/custom/cluekie-start.sh (modified to fit new paths)

March 3

21:29 Krinkle: Directory structure update complete (bots are now all in /h/cvn/bots and swmtbot instances in /h/cvn/bots/swmtbots). Re-started all ts-willow primary bots
20:53 Krinkle: Disabling all processes on ts-willow/cvn for slight directory reorganization
20:48 Krinkle: Moving Cluekie to ts-willow/cvn
20:46 Krinkle: Cluekie was also in ts-willow/krinkle long-run SGE queue, but kept disabled for now since its configuration need an update per cluenet-channel on the source irc server was renamed
20:45 Krinkle: cLogger was having issues due to the garbaged that ended up in the SGE qsub for ts-willow/krinkle. Queue cleared and minutely crontab submission re-enabled. Should be fine now.
18:53 Seahorse: Toolserver issues resolved. All bots started successfully.
03:50 Seahorse: CVNBot5 started for #cvn-meta
03:32 Seahorse: Wikimedia Toolserver CVN Bots semi-outage. Upon an attempt to restart them, an error is returned: [1]+ Abort (core dumped) nohup mono SWMTBot.exe (wd: ~/SeahorseRC2). Similar errors are now occurring even when using similar commands like "ls"

February 26

20:30 Krinkle: bug 34508 has been mostly fixed. Tracking progress here: http://etherpad.wikimedia.org/IRCBot-Messages
20:03 Seahorse: CVNBot14 behaving unusually. Lagging behind w/ edits and crashing upon certain commands such as getadmins.

February 20

06:07 Seahorse: Bugzilla ticket opened on behalf of CVN/others: https://bugzilla.wikimedia.org/show_bug.cgi?id=34508

February 18

20:07 Seahorse: Network restart. 14 bots were online. No lags before restarting. All 14 bots successfully restarted.
20:07 Seahorse: MediaWiki update has broken some functions of the bots. A patch will be released as soon as possible.

February 5

02:19 Seahorse: Cluekie not functioning because ClueBot-NG's ClueNet feed is not functioning. Asking ClueNet ops about it now.
02:00 Seahorse: Updated SentryBot's Admin & Bot lists via command. An update hadn't been performed for several weeks.

January 29

20:36 Krinkle: Taking down CVNBot14 for database clean up (removing old entries). Hoping to speed up the bot now that commons is having lots of activity and lots of gadget users querying the database as well

January 16

01:57 Krinkle: [ts-willow] Updated SWMTBot.ini of all bots to use 'mono.old' instead of 'mono' for restartarg
01:56 Krinkle: [ts-willow] Updated services.ini to use 'mono.old' instead of 'mono' for starting bots

January 15

22:50 Krinkle: Toolserver provided 'mono.old', ts-willow bots are back up

January 11

22:50 Krinkle: Toolserver updated mono and broke ts-willow bots. Currently running bots are OK. start/restart/quit is fatal.

December 28

20:07 Seahorse: Test

October 16

18:26 Krinkle: WikiaRC has been back up for a while now. Status OK
18:26 Krinkle: KrinkleBot4 restarted, seems OK
18:26 Krinkle: KrinkleBot4 crashed: mmap(...PROT_NONE...) failed \ Stacktrace: at (wrapper managed-to-native) string.InternalAllocateStr (int) <0x00003> at SWMTBot.SWMTUtils.replaceStrMax (string,char,char,int) <0x0008b>

October 3

20:27 Seahorse: All bots are properly functioning, except for WikiaRC which is missing and can not be started due to a login issue.

September 21

06:29 Krinkle: cvn-srv2 bots have been started. Should all be OK again now.

September 16

22:35 Krinkle: cvn-srv2 went down yesterday as a result of an internet connection drop at Alex's ISP. The server was also recently moved after which remote authentication was reset and has not been restored yet. Staff is locked out of the Bot Coordination panel. Awaiting server admin to reset authentication.

September 13

00:27 Krinkle: Cluekie has it's own SGE cronsub entry now for longrunning jobs at the Toolserver (every 5 minutes, like cLogger)

June 17

21:20 Seahorse: CVNBot8 was broadcasting a SQL logic error every time a list broadcast was made by another bot. The bot has been kicked from #cvn-broadcast and devoiced on #cvn-sw (MelosBot3 was voiced to replace it temporarily)
00:15 Krinkle: #cvn-broadcast went crazy, caused bots to run in a loop and lag. All bots were forced quit. ts-willow bots have been started, cvn-srv2 can't be started due to control panel issue.
00:15 Krinkle: Control panel on cvn-srv2 is inaccessible. Login rejects credentials.

June 16

20:13 Seahorse: MelosBots started, covering #cvn-sw and bRCBots 1 & 2 started covering #cvn-simplewikis and #cvn-meta
20:08 Seahorse: Login denied when attempting to login to srv2. Several bots missing or lagging severely probably due to a bad connection.

June 12

13:30 Krinkle: Domainname countervandalism.net unavailable. Cause unknown. Could be a DNS problem, or domain expired.

June 8

22:35 Krinkle: Cluekie started
21:54 Krinkle: irc.wikimedia.org is back up. Staying tuned for possible side-effects.
16:34 Krinkle: irc.wikimedia.org is down. CVNBots are okay. #wikimedia-tech is notified, but body systemadmin has responded yet.

June 6

11:33 Krinkle: Toolserver maintenance is over; ts-willow bots back up
10:12 Krinkle: Toolserver maintenance untill 12AM UTC. ts-willow bots may be unavailable during this time.

May 31

23:07 Krinkle: CVNBot7 wasn't running on cvn-srv2. Started now.

May 27

15:40 barras_: Restarted CVNbot2, CVNBot5, SWBot3, KrinkleBot4 and CVNBot14. All of them were online but did not report anything. Restarting fixed the issue.

May 24

19:55 Krinkle: cvn-srv2 is back up. Lauched 22 bots from Webmin.

May 23

18:57 Seahorse: srv2 bots have gone down, and the webpanel is not loading. It is assumed that the entire server is either shut off or simply not responding.

May 21

18:38 Krinkle: cvn-srv2 went down again. Back up now. 22 bots online.

May 20

00:07 Seahorse: srv2 seems to be functioning properly, all bots now online with the exception of WikiaRC

May 19

23:17 Krinkle: cvn-srv2 went down again. Looks like internet connection drop. ETA unknown

May 14

13:24 Krinkle: cvn-srv2 is back up. 22 bots restarted.
13:24 Krinkle: cvn-srv2 is back up. 22 bots restarted.
13:14 Krinkle: cvn-srv2 just went down. Looks like internet connection drop. ETA unknown

May 3

23:25 Krinkle: CVN server back up. Bots back up.
22:24 Krinkle: CVN server down. AlexZ / cbrown unreachable.

April 25

22:56 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

March 30

21:18 Krinkle: Toolserver was doing fine. One of the main Freenode servers failed, back up now. Toolserver hosted cvnbots are back up.
16:30 Krinkle: Toolserver is having network issues.
16:09 Krinkle: Cluekie is running again
12:36 barras_: Cluekie lagged and I quited it then.... now I see it is on none of our servers :s Krinkle help me ;)

March 27

00:32 Krinkle: Network restart. 22 bots were online. No lags before restarting. SWBot & SWBot3 didn't come back after restart, started manually. All 22 bots successfully restarted.

March 19

09:33 AlexZ_: CVN-wiki fixed and upgraded to 1.17alpha (r82950)

February 25

23:21 Seahorse: Issue resolved, and bots restarted.
23:15 Seahorse: CVN server unreachable, webmin panel not loading, and bots are down.

February 22

00:13 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

February 12

02:53 Seahorse: CVN Server's bots all have disconnected for an unknown reason, bots are currently being started again.

February 11

15:07 barras_: SeahorseRC2 quitted; currently running an old bot in #cvn-zh-scan from me. quiet it when the normal one comes back.
13:51 barras_: Restarted KrinkleBot7. Bot didn't show every event. Restart seems to have fixed it.

February 7

23:26 Krinkle: Random dropout/restart/disconnecting of bots on cvn-srv2. Processes in /home/cvn/bots/ terminated. Network restarted. 22 bots up now.

February 1

02:27 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 31

01:57 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 27

01:05 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 24

20:03 AlexZ_: .dyndns.org addressed has expired

January 23

11:20 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 22

18:03 Krinkle: WikiaRC was in #cvn-bots but not in #cvn-wikia. Complaints about relay bots to local channels of individual wikia wikis not working followed. A restart fixed it for now.
04:53 AlexZ_: rebooted RG after becoming unresponsive and reporting various DNS errors. "ERR P0000-00-00T00:01:09 sntpc: Failed to resolve ntpserver 'ntp.ucsd.edu': Host name lookup failure." Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 21

00:54 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 18

19:29 Krinkle: New channel: #cvn-wikia-zelda (Zeldapedia countervandalism <http://zelda.wikia.com/>). Local op(s): Furry

January 17

22:04 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.
01:47 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 16

22:54 AlexZ_: rebooted srv2 after 58 days
13:58 Krinkle: Network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 13

23:49 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.
02:23 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 12

01:52 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 11

05:44 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 10

02:57 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 9

00:50 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 8

03:23 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 7

02:16 Krinkle: Daily network restart. 23 bots were online. No lags before restarting. All 23 bots successfully restarted.

January 6

02:13 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 5

00:13 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 4

14:53 Krinkle: Added cLogger as longrun cronsub that auto-restarts in crontab @ krinkle @ toolserver
02:08 Krinkle: Daily network restart. 23 bots were online. No lags before restarting. All 23 bots successfully restarted.

January 3

04:56 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

January 2

03:30 Krinkle: Daily network restart. 21 bots were online. No lags before restarting. All 21 bots successfully restarted.

January 1

20:11 Krinkle: The new fa.wikinews is now monitored as well. Loaded in CVNBot15 (#cvn-wp-fa : Scan-channel for all Persian (fa) wikis)
18:09 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 31

04:22 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 30

04:06 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 29

21:36 Seahorse: CVNBot10 became extremely slow and laggy due to an unbotlisted bot doing extremely rapid constant page deletions on ksh.wikipedia. The reason it may have been unlisted is because it was on the whitelist which I had to remove from it. Problem solved and bot restarted.
03:53 AlexZ_: updating webgui URL to https://srv2.countervandalism.net:1568/

December 28

01:45 Krinkle: Daily network restart. 22 bots were online. No lags before restarting. All 22 bots successfully restarted.

December 27

17:09 Krinkle: Setting feedFilterEventDelete:2 back to feedFilterEventDelete:4 for CVNBot14 @ Toolserver (too many deletions flood the channel, expection of requester was lower)
15:55 Seahorse: CVNBot10 Started by Krinkle, was down for almost a day and failed to start first attempt.
04:03 Krinkle: Setting feedFilterEventDelete:4 to feedFilterEventDelete:2 for CVNBot14 @ Toolserver (showing delete actions, per IRC request)
03:48 Krinkle: Daily network restart. 20 bots were online. No lags before restarting. All 20 bots successfully restarted.

December 26

16:41 Krinkle: KrinkleBot7 apparantly missed the addition of a proxyblockbot to the botlist and added 12,000 rows. Rows deleted and database 'vacuum'ed. Bot back up.
16:25 Krinkle: KrinkleBot7 down for database optimalization. Keeping in the terminal for a short run to keep an eye on the debug log.
16:19 Seahorse: KrinkleBot7 lately has been showing 3+ min lag, restarted twice but the problem has not fixed. This may be SWMTBot's general lag-for-big-wikis problem.
10:16 AlexZ_: cvnwiki seems to be working again (check w/ webhost) & botserver needs eventual reboot to apply some updates.
10:12 AlexZ_: rebooted bots
10:06 AlexZ_: cvnwiki is dying 126: Incorrect key file for table '/ramdisk/mysql/tmp/#sql_2f54_0.MYI'; try to repair it (127.0.0.1)
10:04 AlexZ_: Network restored after submitting reboot request to RG via UI.
09:59 AlexZ_: 2010-12-26T01:39:58-08:00 lmd: ipnet0: UP on bridge1 with 75.37.205.255/22 GW:75.37.204.1
09:58 AlexZ_: 2010-12-26T01:39:16-08:00 vrsip: Broadband connection lost
09:22 AlexZ_: Running a broadband link test on RG, this may cause network issues.
04:40 Krinkle: Daily network restart. All bots were on. No lags. CVNBot6 didn't boot back up. WikiaRC did leave but not come back. Started both manually.
02:56 Krinkle: Issue with #cvn-unifications solved, SULWatcher's are back up. It was down since 01:50, 26 December 2010 according to logs.
02:52 Krinkle: #cvn-unifications has been empty for a few hours. Bots seem to be functioning properly. Investigating possible relation to WMF issues earlier today and/or the toolserver replication issues because of that.
00:18 Krinkle: Cluekie back up. Runs version 0.0.4-beta (latest trunk)

December 25

23:55 Krinkle: Cluekie will be down for maintaince and back shortly!
21:21 J-FelizNavidad: Restarted SULWatcher
18:38 Krinkle: Banned Jules from #countervandalism, no known operator, not approved in the network.
18:22 Krinkle: First publicly logged message
17:57 Krinkle: Testing Clogger - From toolserver
16:20 Krinkle: Testing with right log enabled
16:04 Krinkle: Alright, we're up !
00:00 Krinkle: Demo..