Printable Version of Topic

Posted by: Imhotep Jan 11 09, 11:17

Greetings.

Starting at 07:00 EST (12:00 GMT) on Tuesday, 13th Jan I'll be doing some tweaking on the server configuration to optimize speeds and page loads. I don't anticipate going more than 2 hours on it. During that period you might notice short intermittant "page cannot be displayed" or "internal server error" pages on the forums. The www site should only be minimally affected. Outages should be no longer than a minute or so as I load the new configuration and test whether it works or not. If it does then I run the benchmarking program against it and if it doesn't make things faster than revert the change. During benchmarking you should not notice any anomolies.

I

Posted by: Imhotep Jan 11 09, 17:49

Hmm, OK, did some nondestructive tweaking and maybe I don't need to do any more.

Right after the switch:
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking forums.mosaicmusings.net (be patient).....done


Server Software: Apache/2.2.10
Server Hostname: forums.mosaicmusings.net
Server Port: 80

Document Path: /index.php
Document Length: 123620 bytes

Concurrency Level: 10
Time taken for tests: 2.252 seconds
Complete requests: 10
Failed requests: 0
Write errors: 0
Total transferred: 1240070 bytes
HTML transferred: 1236200 bytes
Requests per second: 4.44 [#/sec] (mean)
Time per request: 2251.860 [ms] (mean)
Time per request: 225.186 [ms] (mean, across all concurrent requests)
Transfer rate: 537.78 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 0.9 2 3
Processing: 2200 2225 23.2 2243 2249
Waiting: 202 223 20.9 240 246
Total: 2203 2226 23.8 2245 2252

Percentage of the requests served within a certain time (ms)
50% 2245
66% 2248
75% 2249
80% 2251
90% 2252
95% 2252
98% 2252
99% 2252
100% 2252 (longest request)
[pclark@c3po ~]$

And now after the latest tweaks:

[root@c3po ~]# ab -n 1000 -c 10 http://forums.mosaicmusings.net/index.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking forums.mosaicmusings.net (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software: Apache/2.2.10
Server Hostname: forums.mosaicmusings.net
Server Port: 80

Document Path: /index.php
Document Length: 124070 bytes

Concurrency Level: 10
Time taken for tests: 6.996 seconds
Complete requests: 1000
Failed requests: 13
(Connect: 0, Receive: 0, Length: 13, Exceptions: 0)
Write errors: 0
Total transferred: 124447029 bytes
HTML transferred: 124070029 bytes
Requests per second: 142.94 [#/sec] (mean)
Time per request: 69.958 [ms] (mean)
Time per request: 6.996 [ms] (mean, across all concurrent requests)
Transfer rate: 17371.94 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 3
Processing: 48 70 16.8 65 171
Waiting: 45 67 16.6 62 166
Total: 48 70 16.8 65 172

Percentage of the requests served within a certain time (ms)
50% 65
66% 73
75% 79
80% 83
90% 93
95% 102
98% 113
99% 125
100% 172 (longest request)
[root@c3po ~]#

7 seconds for 1000 requests. Was 2.2 seconds for 10. Presuming this benchmarking is accurate I think I've gotten it as good as it can get.

Anyone with any access, slow page issues, missing images, anything out of the ordinary, no matter how slight you might think it is please PM me with it so I can look into it. Nothing is a bother to us except for slow pages that we might not know about so don't worry about it.

Thanks.

Posted by: AMETHYST Jan 11 09, 17:50

Thanks for the heads up Pete... It must be quite exciting to handle things like that and actually know what you're talking about. Hugs, Liz

Posted by: Imhotep Jan 11 09, 17:53

It's an interesting mental exercise, I'll give it that much Always looking for better, faster ways to generate the forum pages. Keeps it interesting for a non-poetic techie like me :)

Posted by: AMETHYST Jan 11 09, 18:49

Well see that just goes to show there is room for everyone in poetry! LOL NAH, Nah ... LOL

Posted by: Imhotep Jan 12 09, 18:46

Perhaps. Perhaps not. One never knows.

I will be doing some tweaking tomorrow morning just to see if I can eek some more performance out of the system. I'd like to see it get close to the peformance I'm getting from the www.mosaicmusings.net page. Might be a little hard with the forum software but one can but try. I'll update this thread when I'm done playing around.

Posted by: Imhotep Jan 13 09, 08:09

OK, couple of more tweaks and looks like another slight improvement in the numbers:
[root@c3po ~]# ab -c 10 -n 1000 -k -q forums.mosaicmusings.net/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking forums.mosaicmusings.net (be patient).....done


Server Software: Apache/2.2.10
Server Hostname: forums.mosaicmusings.net
Server Port: 80

Document Path: /
Document Length: 124069 bytes

Concurrency Level: 10
Time taken for tests: 6.609 seconds
Complete requests: 1000
Failed requests: 404
(Connect: 0, Receive: 0, Length: 404, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 124422763 bytes
HTML transferred: 124063763 bytes
Requests per second: 151.31 [#/sec] (mean)
Time per request: 66.090 [ms] (mean)
Time per request: 6.609 [ms] (mean, across all concurrent requests)
Transfer rate: 18385.04 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 47 66 16.5 60 141
Waiting: 45 63 16.1 58 138
Total: 47 66 16.5 60 141

Percentage of the requests served within a certain time (ms)
50% 60
66% 68
75% 75
80% 79
90% 89
95% 99
98% 111
99% 122
100% 141 (longest request)
[root@c3po ~]#

(.3 seconds faster, 6.6 instead of 6.9 and the system handling 9 requests/second more than it did before).

I don't forsee any more 'internal server error' or 'page cannot be displayed' messages, but I'll be looking around to see if there's anything else to prod.

Enjoy.
I

Posted by: Cleo_Serapis Jan 13 09, 11:58

Cool stuff!

Posted by: Imhotep Jan 13 09, 15:25

OK, installed the 2nd hard drive and made them redundant. I'm done tweaking things for now. Course, when the new version of Fedora comes out in June I'll have to do it all over again. Oh well

Enjoy and PM me any problems.

Posted by: Psyche Jan 13 09, 19:58

Wow, Peter, you're a genius! Or do they call you people nerds....LOL....?

I'll tell you right away that I've just logged in a lot faster, big difference! I never got to downloading Firefox, which I should, but Old Father Time gets older & older, the young man said....(Lewis Carrolll....haha...).

What would we poets do without you?! There must be some poetical streak in you, it just comes out a bit techie. Perhaps we need Arnie to transform your thread into a techie poem. Arnieeeeee....?????

Thanks a lot, congrats,
Syl*** ....and a big hug to Liz, how're ya doin', Liz dear? It's great to have you back.

Posted by: ohsteve Jan 14 09, 06:42

Peter, great job. is the tweaking done thru hardware or soft ware? Hardware i can understand, software leaves me baffled most times. Oh and (jokingly) what is anomolies, are they molies that don't have any ID? undercover molies? when they turn thier backs on you, do you see molie asses? Sorry couldn't resist.
Steve

Posted by: Imhotep Jan 15 09, 12:05

Ya know, no idea... I think they come from the same place gruntled employees come from.... You can't find them when you're looking for them, they only come out when you're not lookin.

Posted by: Imhotep Jan 16 09, 13:53

Almost missed this part, sorry.

All the tweaking at this point is done through software. The underlying operating system is Linux (a Unix flavor). You can set up how many processes that are running are dedicated to serving web pages (more and things serve faster, less and it's slower, too many and you eat up too much RAM and it gets slower again), optimizing the database (SQL) for how big a RAM chunk it has to work with, how often it flushes changed data out to disk, etc etc etc (again with the too much/too few balance) that kind of thing. It's really just experimenting with each hardware configuration (number of processors, number of cores on those cpu's, how much RAM is in the system, how fast the hard drives are for handling the writes), getting Java accellerators configured and working, etc etc etc. There is a lot of black magic, guesses, and crossed fingers going on when actually doing the tweaks, but once in place you generally don't have to change them very often (if at all). Sometimes when a forum's database gets really large (1GB+) you need to up system RAM by 2-4x the size of the database so that the entire thing can run off memory rather than constant swapping out to disk (the busier the forum is the bigger a problem that becomes) it's not so much the size of the underlying databases, but the number of simultanious online users. More users online at any given time, more random queries to different parts of the databases happen which creates more swapping because the system needs to read something that it doesn't have in memory right now.. but oh, darn, I just wrote that piece to disk and need it so let's write this piece.. shucks, I wanted that... ... ... so you dump as much RAM into the system as you can so it's only ever writing changed data back to the hard drives and running as much as possible out of RAM. This server can handle up to 128GB of RAM on the motherboard, our current database is less than 300MB. I don't think we'll have to worry about this for a while - I think the bandwidth we have available will be a limiting factor long before server capacity or speed.

Posted by: ohsteve Jan 16 09, 16:22

Pete, Thanks, the scary part is I understood almost all of that, 128GB of RAM wow...I used to work on tropospheric scatter radio systems in the military that had a bandwidth of 8Ghz with FM modulation. I have been out for 16 years now, most of the stuff I worked with was made in the 1950's, so makes me wonder what they use now, probably mostly SatComm with at least 512Kb encryption. Not sure I want to know...lol.
Steve