Justin Clark-Casey (justincc)
686b22da6e
On shutdown (job engine stop), don't allow the ObjectDisposedException on BlockingCollection.Take() to propogate if the running thread checked IsRunning before the stop thread set it and disposed of the canellation source.
...
Looks to address http://opensimulator.org/mantis/view.php?id=7453
2015-02-25 20:04:53 +00:00
Justin Clark-Casey (justincc)
6834393821
Fix bug in JobEngine where an attempt to restart after stop would trigger an exception because the cancellation source was not recreated.
2015-02-04 00:10:44 +00:00
Justin Clark-Casey (justincc)
0c31eb0a5d
Add the missing deleted and re-added JobEngine class from the previous commit 8e1e8a0
2015-01-12 20:59:58 +00:00
Justin Clark-Casey (justincc)
8e1e8a0920
Make the performance controlling job processing threads introduced in conference code use a generic JobEngine class rather than 4 slightly different copy/pasted versions.
2015-01-12 20:56:37 +00:00
Robert Adams
7c2aeb9e8e
Fix cut-and-paste error that made StatsManager web fetch queries
...
fail for container specification.
2015-01-01 09:39:07 -08:00
Justin Clark-Casey (justincc)
86367d7219
refactor: Move methods to start a monitored thread, start work in its own thread and run work in the jobengine from Watchdog to a WorkManager class.
...
This is to achieve a clean separation of concerns - the watchdog is an inappropriate place for work management.
Also adds a WorkManager.RunInThreadPool() class which feeds through to Util.FireAndForget.
Also switches around the name and obj arguments to the new RunInThread() and RunJob() methods so that the callback obj comes after the callback as seen in the SDK and elsewhere
2014-11-25 23:56:32 +00:00
Justin Clark-Casey (justincc)
4b45a1b6ad
Change jobengine logging command to "debug jobengine log <int>" rather than loglevel, in common with similar commands.
2014-11-25 23:23:11 +00:00
Justin Clark-Casey (justincc)
28d1dbfee4
Move conditionals which control whether a task is placed in the JobEngine inside Watchdog.RunJob() (renamed from RunWhenPossible) and generalize them.
2014-11-25 23:23:11 +00:00
Justin Clark-Casey (justincc)
6c043dd6ff
If an exception makes it to the top of a JobEngine request, catch and log instead of letting it terminate the simulator...
2014-11-25 23:21:38 +00:00
Justin Clark-Casey (justincc)
d3ed5de77d
Add loglevel to jobengine that can be controlled via "debug jobengine loglevel <level>".
...
Defaults to 0
Level 1 currently does verbose logging about every queued and processed job.
2014-11-25 23:18:38 +00:00
Justin Clark-Casey (justincc)
9fb3065d33
Temporarily add root agent rez attachments work to job engine if it is running rather than as a fire and forget.
...
Experiment to see if serializing attachment rez and send initial data jobs improves other parts of sim performance.
2014-11-25 23:18:38 +00:00
Justin Clark-Casey (justincc)
859b1c5ba4
Add experimental job engine to see if queueing some existing async work during root agent entry to a region improves perf rather than always attempting to execute everything concurrently
...
Job engine is controlled via "debug jobengine start|stop|status".
Can only currently be enabled and disabled dynamically at runtime.
Currently only applies to code sending initial region data (objects, other avatar data) to a client.
2014-11-25 23:18:37 +00:00
Justin Clark-Casey (justincc)
23a0946e36
Fix long-lived thread name logging.
2014-09-26 01:10:49 +01:00
BlueWall
96a2449122
Undo "Revert "Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects.""
...
Fix for break in next commit
This reverts commit 376fab1402 .
2014-09-17 20:03:54 -04:00
BlueWall
376fab1402
Revert "Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects."
...
This reverts commit af286d5fcb .
Issue with Jenkins builds
2014-09-17 18:11:41 -04:00
BlueWall
af286d5fcb
Small changes to threading to send thread names to unmanaged threads. Needs Mono 3.6+ to see thread names in utilities like top -H . Some formatting of the thread name to fin in the 16 byte limit on Linux. Please test on Windows to see if the work has any adverse effects.
2014-09-17 17:56:10 -04:00
Justin Clark-Casey (justincc)
b4ce71df1e
Make proper fix for last commit wrt Mantis 7317 by replacing disallowed c char and not literal 'c'
2014-09-09 18:57:02 +01:00
Justin Clark-Casey (justincc)
e271607bef
For stat names containing periods, replace with '#' rather than throw exception
...
In relation to http://opensimulator.org/mantis/view.php?id=7317
2014-09-09 18:56:58 +01:00
Justin Clark-Casey (justincc)
41f2f3132b
For monitoring purposes, start non-timeout tasks (which do not currently use a threadpool) via Watchdog.RunInThread() rather than Util.RunThreadNoTimeout()
...
The functionality is the same but this allow us to monitor such tasks via "show threads" and abort them for test purposes, etc.
Also extends thread names to provide more info (e.g. SendInitialDataToClient says what client the task is for).
2014-09-05 23:20:59 +01:00
Justin Clark-Casey (justincc)
f1f935ed95
Add 'server' stats information to pCampbot, as used elsewhere in OpenSimulator
...
This adds the "show stats", "stats record", etc. commands and information on available Threadpool threads, etc.
It also adds the Watchdog which logs warnings if time between executions is unexpectedly large.
2014-08-13 19:53:42 +01:00
Justin Clark-Casey (justincc)
cc61681484
Revert "Write UDP statistics to the log, not just the console (e.g., "show queues")"
...
Fixes http://opensimulator.org/mantis/view.php?id=7280
It can't be done this way because the stats data needs to show up on the console at all log levels, not just debug.
But this means setting it to log at fatal, which is not appropriate for this stuff in the log.
I understand the desire but this has to be done some other way, perhaps by (yet another) config parameter.
Also, this was already being done with the ClientStatsReport but that also should be done in another way, I think.
This reverts commit 5d53412766 .
2014-07-25 01:56:41 +01:00
Justin Clark-Casey (justincc)
3355bedaeb
minor: Limit processor related stats to 3 decimal places instead of all the places.
...
Easier to read and analyze, and probably still too much detail (1 dp would probably be fine)
2014-07-21 23:08:15 +01:00
Justin Clark-Casey (justincc)
200dcee1b7
Fix CPU processor use reporting on Mono.
...
Despite the comments in the code, it appears that the issue where the .NET performance counter was wrongly idle time time on Mono was fixed in 2009.
https://bugzilla.novell.com/show_bug.cgi?id=468625
Which means that the workaround is no longer necessary and produces bad results instead.
2014-07-21 22:55:38 +01:00
Oren Hurvitz
5d53412766
Write UDP statistics to the log, not just the console (e.g., "show queues")
2014-07-21 09:10:19 +01:00
Justin Clark-Casey (justincc)
8ae5ab809f
Fix issue with running "stats record start|stop" console command
...
In commit e6080a38 (Wed Mar 19 00:29:36 2014) I renamed this from "debug stats record start|stop"
Unfortunately, I didn't do this fully so before this commit "stats record start|stop" will report a usage failure with the old debug text.
Unfortunately this is in the 0.8 release. The workaround is to repeat the last command twice (e.g. "stats record start start")
2014-07-08 18:49:57 +01:00
Justin Clark-Casey (justincc)
5450b1b024
Change assembly versions to 0.8.1
2014-06-17 18:37:15 +01:00
Justin Clark-Casey (justincc)
47e7febebc
Add monitored thread last update times as stats in "show stats all"
...
These have the format server.thread.<thread-name>
2014-03-21 02:06:10 +00:00
Justin Clark-Casey (justincc)
9858766516
minor: Make "stats show" an alias for "show stats" command.
2014-03-19 00:35:49 +00:00
Justin Clark-Casey (justincc)
e6080a38c5
Rename "debug stats record" command to "stats record" for consistency. Move from debug to general command category
2014-03-19 00:29:36 +00:00
Justin Clark-Casey (justincc)
1d0a9e521c
Allow a snapshot of stats to be dumped to a file with a "stats save" command
2014-03-19 00:28:57 +00:00
Justin Clark-Casey (justincc)
8ce3fa646b
Allow "show stats" console command to take a full stat name and display only that stat.
...
For example, scene.test.RootAgents will show only the RootAgents stat in the scene named "test"
2014-03-18 20:21:47 +00:00
Justin Clark-Casey (justincc)
7cab41f422
refactor: replace verbose checks with String.IsNullOrEmpty where applicable.
...
Thanks to Kira for this patch from http://opensimulator.org/mantis/view.php?id=6845
2013-11-15 21:45:08 +00:00
Justin Clark-Casey (justincc)
42bdf44658
Bump OPenSimulator version and assembly versions up to 0.8.0 Dev
2013-10-04 23:33:47 +01:00
Justin Clark-Casey (justincc)
a9f9b0da9d
minor: Correct typo on "debug stats record start" message
2013-08-23 00:13:31 +01:00
Justin Clark-Casey (justincc)
217c8deae5
minor: remove mono compiler warning in StatsManager
2013-08-17 00:51:21 +01:00
Robert Adams
216f5afe54
Stats treaking. Update ToOSDMap for Stat and PercentageStat to return
...
all the various numbers that have been added to the console output.
Break out EventHistogram from CounterStat.
2013-08-10 09:09:52 -07:00
Robert Adams
d9bd6e6b5b
Add parameter and explanation of ManagedStats return to OpenSimDefaults.ini.
...
Add 'callback' query parameter to managed stats return to return function
form of JSON data.
2013-08-08 09:45:30 -07:00
Robert Adams
50c163ae6c
Add a JSON web fetch of the statististics managed by StatsManager.
...
Disabled by default. Enable by setting
[Startup]ManagedStatsRemoteFetchURI="Something"
and thereafter "http://ServerHTTPPort/Something/ " will return all the managed
stats (equivilent to "show stats all" console command).
Accepts queries "cat=", "cont=" and "stat=" to specify statistic category,
container and statistic names. The special name "all" is the default and returns
all values in that group.
2013-08-08 09:45:01 -07:00
Justin Clark-Casey (justincc)
d6d5d4ebd0
Add file missing from last commit 4c2f6de
2013-08-06 18:32:16 +01:00
Justin Clark-Casey (justincc)
4c2f6de8e4
Add the experimental ability to dump stats (result of command "show stats all") to file OpenSimStats.log every 5 seconds.
...
This can currently only be activated with the console command "debug stats record start".
Off by default.
Records to file OpenSimStats.log for simulator and RobustStats.log for ROBUST
2013-08-06 18:29:33 +01:00
Justin Clark-Casey (justincc)
76bd3de2fd
Add checks monitoring framework to provide alerts if certain conditions do not hold.
...
Not yet in use.
2013-08-05 19:22:47 +01:00
Justin Clark-Casey (justincc)
8efe4bfc2e
Make "abnormal thread terminations" into "ClientLogoutsDueToNoReceives" and add this to the StatsManager
...
This reflects the actual use of this stat - it hasn't recorded general exceptions for some time.
Make the sim extra stats collector draw the data from the stats manager rather than maintaing this data itself.
2013-07-29 23:18:29 +01:00
Justin Clark-Casey (justincc)
9a4a513b5e
Correct issue where the last instance of a sampled stat was shown 3x larger than it should have been (though internal use was correct)
2013-07-23 23:31:35 +01:00
Justin Clark-Casey (justincc)
76e46d0158
Improve spacing between data and units on console stats display
2013-07-23 17:23:16 +01:00
Justin Clark-Casey (justincc)
90528c23d9
For stats which can show average change over time, show the last sample as well as the average.
...
This is somewhat cryptic at the moment, need to improve documentation.
2013-07-23 01:13:13 +01:00
Robert Adams
fa02f28dbf
Add ToOSDMap() overrides to the Stat subclass CounterStat.
...
Add a GetStatsAsOSDMap method to StatsManager which allows the filtered
fetching of stats for eventual returning over the internets.
2013-07-12 14:04:14 -07:00
Justin Clark-Casey (justincc)
085a87060a
Change "ObjectMemory" stat to "HeapMemory" to align with other stat names. Also round this and ProcessMemory to three decimal places in common with other memory stats.
...
I believe leaving out such minor info makes stats easier to read
2013-06-20 00:52:39 +01:00
Justin Clark-Casey (justincc)
05790ba1cf
Allow more than one stat category to be specified in "show stats"
...
e.g. "show stats httpserver.9000 server.network"
2013-06-20 00:45:56 +01:00
Justin Clark-Casey (justincc)
5b1a9f84fd
minor: Change "memory churn" terminology in statistics to "heap allocation rate" since this is more generally meaningful
2013-06-20 00:32:12 +01:00
Justin Clark-Casey (justincc)
d97333255d
Fix minor bug where the check whether to display SmartThreadPool stats was accidentally != null rather than == FireAndForgetMethod.SmartThreadPool
...
Due to another check this had no practical effect
2013-06-20 00:22:44 +01:00