Commit Graph

90 Commits

Author SHA1 Message Date
Justin Clark-Casey (justincc)
686b22da6e On shutdown (job engine stop), don't allow the ObjectDisposedException on BlockingCollection.Take() to propogate if the running thread checked IsRunning before the stop thread set it and disposed of the canellation source.
Looks to address http://opensimulator.org/mantis/view.php?id=7453
2015-02-25 20:04:53 +00:00
Justin Clark-Casey (justincc)
6834393821 Fix bug in JobEngine where an attempt to restart after stop would trigger an exception because the cancellation source was not recreated. 2015-02-04 00:10:44 +00:00
Justin Clark-Casey (justincc)
0c31eb0a5d Add the missing deleted and re-added JobEngine class from the previous commit 8e1e8a0 2015-01-12 20:59:58 +00:00
Justin Clark-Casey (justincc)
8e1e8a0920 Make the performance controlling job processing threads introduced in conference code use a generic JobEngine class rather than 4 slightly different copy/pasted versions. 2015-01-12 20:56:37 +00:00
Robert Adams
7c2aeb9e8e Fix cut-and-paste error that made StatsManager web fetch queries
fail for container specification.
2015-01-01 09:39:07 -08:00
Justin Clark-Casey (justincc)
86367d7219 refactor: Move methods to start a monitored thread, start work in its own thread and run work in the jobengine from Watchdog to a WorkManager class.
This is to achieve a clean separation of concerns - the watchdog is an inappropriate place for work management.
Also adds a WorkManager.RunInThreadPool() class which feeds through to Util.FireAndForget.
Also switches around the name and obj arguments to the new RunInThread() and RunJob() methods so that the callback obj comes after the callback as seen in the SDK and elsewhere
2014-11-25 23:56:32 +00:00
Justin Clark-Casey (justincc)
4b45a1b6ad Change jobengine logging command to "debug jobengine log <int>" rather than loglevel, in common with similar commands. 2014-11-25 23:23:11 +00:00
Justin Clark-Casey (justincc)
28d1dbfee4 Move conditionals which control whether a task is placed in the JobEngine inside Watchdog.RunJob() (renamed from RunWhenPossible) and generalize them. 2014-11-25 23:23:11 +00:00
Justin Clark-Casey (justincc)
6c043dd6ff If an exception makes it to the top of a JobEngine request, catch and log instead of letting it terminate the simulator... 2014-11-25 23:21:38 +00:00
Justin Clark-Casey (justincc)
d3ed5de77d Add loglevel to jobengine that can be controlled via "debug jobengine loglevel <level>".
Defaults to 0
Level 1 currently does verbose logging about every queued and processed job.
2014-11-25 23:18:38 +00:00
Justin Clark-Casey (justincc)
9fb3065d33 Temporarily add root agent rez attachments work to job engine if it is running rather than as a fire and forget.
Experiment to see if serializing attachment rez and send initial data jobs improves other parts of sim performance.
2014-11-25 23:18:38 +00:00
Justin Clark-Casey (justincc)
859b1c5ba4 Add experimental job engine to see if queueing some existing async work during root agent entry to a region improves perf rather than always attempting to execute everything concurrently
Job engine is controlled via "debug jobengine start|stop|status".
Can only currently be enabled and disabled dynamically at runtime.
Currently only applies to code sending initial region data (objects, other avatar data) to a client.
2014-11-25 23:18:37 +00:00
Justin Clark-Casey (justincc)
23a0946e36 Fix long-lived thread name logging. 2014-09-26 01:10:49 +01:00
BlueWall
96a2449122 Undo "Revert "Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects.""
Fix for break in next commit

This reverts commit 376fab1402.
2014-09-17 20:03:54 -04:00
BlueWall
376fab1402 Revert "Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects."
This reverts commit af286d5fcb.
Issue with Jenkins builds
2014-09-17 18:11:41 -04:00
BlueWall
af286d5fcb Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects. 2014-09-17 17:56:10 -04:00
Justin Clark-Casey (justincc)
b4ce71df1e Make proper fix for last commit wrt Mantis 7317 by replacing disallowed c char and not literal 'c' 2014-09-09 18:57:02 +01:00
Justin Clark-Casey (justincc)
e271607bef For stat names containing periods, replace with '#' rather than throw exception
In relation to http://opensimulator.org/mantis/view.php?id=7317
2014-09-09 18:56:58 +01:00
Justin Clark-Casey (justincc)
41f2f3132b For monitoring purposes, start non-timeout tasks (which do not currently use a threadpool) via Watchdog.RunInThread() rather than Util.RunThreadNoTimeout()
The functionality is the same but this allow us to monitor such tasks via "show threads" and abort them for test purposes, etc.
Also extends thread names to provide more info (e.g. SendInitialDataToClient says what client the task is for).
2014-09-05 23:20:59 +01:00
Justin Clark-Casey (justincc)
f1f935ed95 Add 'server' stats information to pCampbot, as used elsewhere in OpenSimulator
This adds the "show stats", "stats record", etc. commands and information on available Threadpool threads, etc.
It also adds the Watchdog which logs warnings if time between executions is unexpectedly large.
2014-08-13 19:53:42 +01:00
Justin Clark-Casey (justincc)
cc61681484 Revert "Write UDP statistics to the log, not just the console (e.g., "show queues")"
Fixes http://opensimulator.org/mantis/view.php?id=7280
It can't be done this way because the stats data needs to show up on the console at all log levels, not just debug.
But this means setting it to log at fatal, which is not appropriate for this stuff in the log.
I understand the desire but this has to be done some other way, perhaps by (yet another) config parameter.
Also, this was already being done with the ClientStatsReport but that also should be done in another way, I think.

This reverts commit 5d53412766.
2014-07-25 01:56:41 +01:00
Justin Clark-Casey (justincc)
3355bedaeb minor: Limit processor related stats to 3 decimal places instead of all the places.
Easier to read and analyze, and probably still too much detail (1 dp would probably be fine)
2014-07-21 23:08:15 +01:00
Justin Clark-Casey (justincc)
200dcee1b7 Fix CPU processor use reporting on Mono.
Despite the comments in the code, it appears that the issue where the .NET performance counter was wrongly idle time time on Mono was fixed in 2009.
https://bugzilla.novell.com/show_bug.cgi?id=468625
Which means that the workaround is no longer necessary and produces bad results instead.
2014-07-21 22:55:38 +01:00
Oren Hurvitz
5d53412766 Write UDP statistics to the log, not just the console (e.g., "show queues") 2014-07-21 09:10:19 +01:00
Justin Clark-Casey (justincc)
8ae5ab809f Fix issue with running "stats record start|stop" console command
In commit e6080a38 (Wed Mar 19 00:29:36 2014) I renamed this from "debug stats record start|stop"
Unfortunately, I didn't do this fully so before this commit "stats record start|stop" will report a usage failure with the old debug text.
Unfortunately this is in the 0.8 release.  The workaround is to repeat the last command twice (e.g. "stats record start start")
2014-07-08 18:49:57 +01:00
Justin Clark-Casey (justincc)
5450b1b024 Change assembly versions to 0.8.1 2014-06-17 18:37:15 +01:00
Justin Clark-Casey (justincc)
47e7febebc Add monitored thread last update times as stats in "show stats all"
These have the format server.thread.<thread-name>
2014-03-21 02:06:10 +00:00
Justin Clark-Casey (justincc)
9858766516 minor: Make "stats show" an alias for "show stats" command. 2014-03-19 00:35:49 +00:00
Justin Clark-Casey (justincc)
e6080a38c5 Rename "debug stats record" command to "stats record" for consistency. Move from debug to general command category 2014-03-19 00:29:36 +00:00
Justin Clark-Casey (justincc)
1d0a9e521c Allow a snapshot of stats to be dumped to a file with a "stats save" command 2014-03-19 00:28:57 +00:00
Justin Clark-Casey (justincc)
8ce3fa646b Allow "show stats" console command to take a full stat name and display only that stat.
For example, scene.test.RootAgents will show only the RootAgents stat in the scene named "test"
2014-03-18 20:21:47 +00:00
Justin Clark-Casey (justincc)
7cab41f422 refactor: replace verbose checks with String.IsNullOrEmpty where applicable.
Thanks to Kira for this patch from http://opensimulator.org/mantis/view.php?id=6845
2013-11-15 21:45:08 +00:00
Justin Clark-Casey (justincc)
42bdf44658 Bump OPenSimulator version and assembly versions up to 0.8.0 Dev 2013-10-04 23:33:47 +01:00
Justin Clark-Casey (justincc)
a9f9b0da9d minor: Correct typo on "debug stats record start" message 2013-08-23 00:13:31 +01:00
Justin Clark-Casey (justincc)
217c8deae5 minor: remove mono compiler warning in StatsManager 2013-08-17 00:51:21 +01:00
Robert Adams
216f5afe54 Stats treaking. Update ToOSDMap for Stat and PercentageStat to return
all the various numbers that have been added to the console output.
Break out EventHistogram from CounterStat.
2013-08-10 09:09:52 -07:00
Robert Adams
d9bd6e6b5b Add parameter and explanation of ManagedStats return to OpenSimDefaults.ini.
Add 'callback' query parameter to managed stats return to return function
form of JSON data.
2013-08-08 09:45:30 -07:00
Robert Adams
50c163ae6c Add a JSON web fetch of the statististics managed by StatsManager.
Disabled by default. Enable by setting
[Startup]ManagedStatsRemoteFetchURI="Something"
and thereafter "http://ServerHTTPPort/Something/" will return all the managed
stats (equivilent to "show stats all" console command).
Accepts queries "cat=", "cont=" and "stat=" to specify statistic category,
container and statistic names. The special name "all" is the default and returns
all values in that group.
2013-08-08 09:45:01 -07:00
Justin Clark-Casey (justincc)
d6d5d4ebd0 Add file missing from last commit 4c2f6de 2013-08-06 18:32:16 +01:00
Justin Clark-Casey (justincc)
4c2f6de8e4 Add the experimental ability to dump stats (result of command "show stats all") to file OpenSimStats.log every 5 seconds.
This can currently only be activated with the console command "debug stats record start".
Off by default.
Records to file OpenSimStats.log for simulator and RobustStats.log for ROBUST
2013-08-06 18:29:33 +01:00
Justin Clark-Casey (justincc)
76bd3de2fd Add checks monitoring framework to provide alerts if certain conditions do not hold.
Not yet in use.
2013-08-05 19:22:47 +01:00
Justin Clark-Casey (justincc)
8efe4bfc2e Make "abnormal thread terminations" into "ClientLogoutsDueToNoReceives" and add this to the StatsManager
This reflects the actual use of this stat - it hasn't recorded general exceptions for some time.
Make the sim extra stats collector draw the data from the stats manager rather than maintaing this data itself.
2013-07-29 23:18:29 +01:00
Justin Clark-Casey (justincc)
9a4a513b5e Correct issue where the last instance of a sampled stat was shown 3x larger than it should have been (though internal use was correct) 2013-07-23 23:31:35 +01:00
Justin Clark-Casey (justincc)
76e46d0158 Improve spacing between data and units on console stats display 2013-07-23 17:23:16 +01:00
Justin Clark-Casey (justincc)
90528c23d9 For stats which can show average change over time, show the last sample as well as the average.
This is somewhat cryptic at the moment, need to improve documentation.
2013-07-23 01:13:13 +01:00
Robert Adams
fa02f28dbf Add ToOSDMap() overrides to the Stat subclass CounterStat.
Add a GetStatsAsOSDMap method to StatsManager which allows the filtered
fetching of stats for eventual returning over the internets.
2013-07-12 14:04:14 -07:00
Justin Clark-Casey (justincc)
085a87060a Change "ObjectMemory" stat to "HeapMemory" to align with other stat names. Also round this and ProcessMemory to three decimal places in common with other memory stats.
I believe leaving out such minor info makes stats easier to read
2013-06-20 00:52:39 +01:00
Justin Clark-Casey (justincc)
05790ba1cf Allow more than one stat category to be specified in "show stats"
e.g. "show stats httpserver.9000 server.network"
2013-06-20 00:45:56 +01:00
Justin Clark-Casey (justincc)
5b1a9f84fd minor: Change "memory churn" terminology in statistics to "heap allocation rate" since this is more generally meaningful 2013-06-20 00:32:12 +01:00
Justin Clark-Casey (justincc)
d97333255d Fix minor bug where the check whether to display SmartThreadPool stats was accidentally != null rather than == FireAndForgetMethod.SmartThreadPool
Due to another check this had no practical effect
2013-06-20 00:22:44 +01:00