| Red Squirrel - Jan-28-2005 server time |
| After almost a week, I have the server up and running again! Now I just need to fix things here and there but mostly everything is smooth, including email, which is the most complicated part. (email really takes out the simple out of SMTP, since it's far from simple) |
| Red Squirrel - Jan-25-2005 server time |
| For some reason it always shows like 16MB of ram remaining... but hopefully that's not right. Mem: 514000K av, 507612K used, 6388K free, 0K shrd, 11044K buff Swap: 1052248K av, 0K used, 1052248K free 442124K cached mind you, I'm restoring 1Gig worth of DBs now, but still, 6MB free? |
| Cold Drink - Jan-25-2005 server time | ||
More on how you use the system than specific applications, but yeah. Right now I'm using about as much resources as I ever do. I've got open a web browser and am compiling in the background, My memory usage is at 256MB RAM (all I have), and about 260MB swap, and 131MB in /tmp. For me, I could get by with /tmp mounted as tmpfs and still have half my swap (1G) left over This is with: - Apache 2 allways running - Another instance of Apache2 allways running for Subversion - MySQL - Postfix - CyrusIMAPd - Samba - OpenLDAP (although I never use it...) |
| Red Squirrel - Jan-25-2005 server time | ||
And we got success!
Yes, that's success, because it means php mysql and apache are working properly. I just need to restore all the databases from the backups. Wait a sec... what backups?!?! J/K |
| Red Squirrel - Jan-25-2005 server time |
| Oh I see so it's more of a thing that would matter for an online server. So in my case it's not that much of an issue. I guess another way on an online server would be to make a script that periodicly does a DU -hs in that folder and sends an email, so it could be easly monitored. |
| Cold Drink - Jan-25-2005 server time |
| My swap is only 1G |
| Red Squirrel - Jan-25-2005 server time |
| My swap is only 1 Gig, should probably be more... And for /tmp it's just on the same partition as everything else. Should it be on a seperate one? BTW I got apache and mysql installed, now on to php, and then, making them all work together. |
| Cold Drink - Jan-25-2005 server time |
| This is all such a tough subject Anyway, Im curious as to how much swap space you have, and do you have /tmp mounted at a partition or pointing to memory with tempfs or something. I was going to move my /tmp to tmpfs, but considering how much RAM that would wind up eating... |
| Red Squirrel - Jan-25-2005 server time |
| So I guess it has to do more with the distro then the file system then? Since I did hear Ext3 is just as reliable as ntfs, just have trouble believing after what happened. Just glad I did not loose data, but I did not go through every single file so I might have some corrupted data on my 100GB partition. I'll only know once apache/php/mysql is installed and that I run all my site, if there's parse errors I'll know stuff got corrupted. |
| wtd - Jan-25-2005 server time |
| I used Fedora Core 2 and had problems with unexpected shutdowns and data loss/corruption. I've never had the same problem with Ubuntu Linux, and I've given it a fair amount of stress. Both cases involved ext3. |
| Red Squirrel - Jan-25-2005 server time |
| No I don't think so anyway. I know httpd.conf pretty well, it's the apache installation string, php, mysql etc stuff that I can confused with but the actual config file for apache I know fairly well, and I had not changed it for a long time before it happened. Getting samba working right now, hopeing to challange apache/php/mysql either after or tomorow, then email. For email I'll attempt to use maildirs instead of mboxes, that should improve mail processing, since it might even be email that caused the server to crash, but I just happened to refresh a page so apache was the first program to crash from not enough memory. That's a possibility too. Especialy with all the spam rules I have, spamassassin takes a few minutes per email. |
| wldkos - Jan-25-2005 server time |
| well, when you make manual changes to apache, or any program for that matter, you really need to know what you need and do not need, because if the program is locking up, it might have been something in the httpd.conf file that you misunderstood. |
| Red Squirrel - Jan-25-2005 server time |
| Thanks but I already started reinstalling it last night. Got DNS working so far, need to do the rest, hoping to finish by the end of the week. |
| Cold Drink - Jan-25-2005 server time |
| http://www.tortuga.com.au/products/upsd/ You should be able to fine a proper RPM out there somewhere for RedHat. |
| Red Squirrel - Jan-25-2005 server time |
| Yeah too late now unless I raw read the drive. But apache was acting up for long I just never got around to troubleshooting it, never realized it was going to actually freeze. My next step is to figure a way to get my ups to shut down the machine properly, in case we ever get a power outage. Would not want this to happen again. |
| Cold Drink - Jan-25-2005 server time | ||||
Everything will be seperate. That's the beauty of chroot If you really want to, you can use loopback devices to mount files as the filesystem for another distro's install. That way, you have much, much lower chance of cross-contamination.
Well it depends. dmesg isn't there all the time, and isn't always that useful for this type of thing. My system has an "everything" log, but in this case I would look at/in /var/log/kern or .../kernel as well as .../daemon and .../messages. Of course, that wouldn't have helped it if happened to me Either way its too late now |
| wldkos - Jan-24-2005 server time |
| There is something that will list the last operations that your system was doing before it crashed (linux by default logs everything I appoligize for the lame help, but there are a few places that have a recording of the problem, and most likely the program that went all buggy should have a log of what happened too. |
| Red Squirrel - Jan-24-2005 server time |
| Yeah that's one way of doing it. What about installing stuff and general file management, would it be contain in it's seperate "space"? But I think to not complicate things I'll wait till I can afford a seperate box. I priced one for fun and for a low end one it would of costed me about 400 bucks to build which is pretty good. I should try to get a micro ATX case so I can just put it under my bed or something when not in use. Kind of funny since the athlon 64 chip was cheaper then the 32bit one of lower GHz. My supplier makes me wonder sometimes lol. |
| Cold Drink - Jan-24-2005 server time |
| Well, there are two things you can do. A lot of people use Virtual PC or VM Ware and run different Linux distros out of that, so you can try whatever you like. You *should* be able to do something similar without a comercial product. Depending on the distro's install procedure, you should be able to install to a directory on an existing partition. Then, while still in your "real" distro, you can just chroot in to the new one. Of course you can't overlap services or anything, but you can use XVnc with a login manager to create X in X sessions. Then you can play all you want while keeping your main distro up and stable |
| Red Squirrel - Jan-24-2005 server time |
| Actually I do allot of automated backups in bash which do me just fine, what's nice about cron is that it sends out an email so I have an address set for this. I got red hat back on the system, sure it's outdated, but think of all the people still using windows 95, that's much worse. I was thinking of using a different distro but rather not sacrifice the time to download. I think in the future I'll want to build a low end box, like a celeron or something, so I can test out new distros. I do need to be able to play more with linux without worrying about breaking stuff so a test machine would be great for that. |
| Cold Drink - Jan-24-2005 server time |
| Hmm using Red Hat? So I guess we found the problem I use Gentoo myself, but considering how long it takes to get it up and running, I wouldn't reccomend it if you need your system up, which it seems you do On a side note, I"m in the process of setting up automated backups to Subversion repositories, the whole lot of which will be compressed and dumped to a Zip 250. You might find it useful |
| Red Squirrel - Jan-24-2005 server time | ||
Yeah well right now I'm in the process of clean installing, since this crash totally fubared everything. All the data is suprisingly still there, but 3 hours of chaos really got me scared, and it's pretty bad for a simple power down at the wrong time. I don't think it was doing I/O at the time either, which is what I find odd. It was basically doing a BSOD but in linux, I went down and it was just a bunch of code and weird out of memory errors but it got jammed there so I hit the 3 key salute, did not do much, so hit reset. I'm downloading RH9 right now but it's only at 50% and I have another CD to go. So debating wether or not I should just put back 8.0. I want to get this done and over with asap since I need it for my business and other stuff. |
| Joe - Jan-24-2005 server time |
| I've had windows for about 7 months and i can count on my fingers the amount of times it crashed. |
| Cold Drink - Jan-24-2005 server time |
| I've had plenty of crashes and use EXT3 file systems without any data loss. I suppose it really depends on what you do. Everything should still be there, but there may be some lata loss in individual files. Unless you were performing IO on the files when the server went down, they should be untouched. I'd look a bit deeper into why the thing went down if I were you. That sounds too odd. |
| Red Squirrel - Jan-23-2005 server time |
| Nope, I thought the same about EXT3 until this happened... and it ain't the first time, though this time is very bad... Basically as soon as linux shuts down bad, you're screwed. At least in windows it will run scandisk and boot as if nothing happened, even if it was in the middle of something intense like a defrag or file transfer. Now I see why high end power backup systems are needed for linux servers. My 5 minute UPS just won't cut it during a power outage. Unless I'm there to shut down the server, which is usually the case. |
| Chris Vogel - Jan-23-2005 server time |
| That’s really bad! Were you not using ext3? I was under the impression that it was very reliable and could journalise like NTFS, but I’ve never tried it personally. |
| Red Squirrel - Jan-23-2005 server time |
| This is something I just learned the hard way. Sure linux in general is more reliable, it won't usually crash for no reason like windows does. But get this, my server spontaniously ran out of memory, so it just jammed up for no reason, then I went to reboot and it could not do the "scandisk" I had to run this command manually, and now I have tons of screw ups and it ain't done yet. Simply put, you can say goodbye to Zeroboard, and if something happends to Iceteks, youc an say good bye too, since the backups were ON that drive. Though I have database backups there, and elsewere, but the site itself is there. Now the perfect setup would be linux on NTFS. Linux file system is just too unreliable. At least you can shut off windows badly and not loose everything. Anyway, back to the server, I need to hold down a key so it keeps doing this thing, I might get lucky and not have lost zeroboard but things look really bad right now. I'll probably be switching to a windows based server if this keeps up. It's really too bad since despite being super complicated and user unfriendly, I gave linux a chance, but so many times it just threw it away. |