Hello,
Today i got a message that a linux server isn't booting at all.
After giving some instruction to the ISP Network Engineer they could fix the issue.
1. Type your root password
2. type command fsck
3. type y for yes.
After that the Linux kernel booted without any issue.
After that after trying to start the mysqld service, the linux server was not booting correctly.
I checked first the messages log by typing this command: ( last 100 lines of var log messages )
tail -n 100 /var/log/messages Apr 2 16:49:39 localhost kernel: ata1.00: configured for UDMA/33 Apr 2 16:49:39 localhost kernel: ata1: EH complete Apr 2 16:49:39 localhost kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 Apr 2 16:49:39 localhost kernel: ata1.00:
But i didn't notice anything wrong from this logs.
I continued to check the other log files like:
tail -n 100 /var/log/mysqld.log
[root@localhost ~]# tail -n 100 /var/log/mysqld.log InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html 180402 16:48:29 InnoDB: Assertion failure in thread 2129656688 in file fil/fil0fil.c line 4200 InnoDB: Failing assertion: ret InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:48:29 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8384512 read_buffer_size=131072 max_used_connections=0 max_threads=151 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337742 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/libexec/mysqld(my_print_stacktrace+0x2e) [0x84c712e] /usr/libexec/mysqld(handle_fatal_signal+0x484) [0x82ecba4] [0xbb9400] [0xbb9424] /lib/libc.so.6(gsignal+0x51) [0x235781] /lib/libc.so.6(abort+0x17a) [0x23705a] /usr/libexec/mysqld(fil_io+0x40b) [0x83c4a1b] /usr/libexec/mysqld() [0x83a7ff0] /usr/libexec/mysqld(buf_flush_batch+0x138) [0x83a87c8] /usr/libexec/mysqld(srv_master_thread+0xabf) [0x842accf] /lib/libpthread.so.0() [0xbddbc9] /lib/libc.so.6(clone+0x5e) [0x2ee04e] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 180402 16:48:29 mysqld_safe Number of processes running now: 0 180402 16:48:29 mysqld_safe mysqld restarted 180402 16:48:29 InnoDB: Initializing buffer pool, size = 768.0M 180402 16:48:29 InnoDB: Completed initialization of buffer pool InnoDB: Log scan progressed past the checkpoint lsn 8 2733752146 180402 16:48:29 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 8 2733790959 180402 16:48:30 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed 180402 16:48:30 InnoDB: Started; log sequence number 8 2733790959 180402 16:48:30 [Note] Event Scheduler: Loaded 0 events 180402 16:48:30 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution 180402 16:49:52 InnoDB: Error: Write to file ./ibdata1 failed at offset 0 1048576. InnoDB: 966656 bytes should have been written, only -1 were written. InnoDB: Operating system error number 5. InnoDB: Check that your OS and file system support files of this size. InnoDB: Check also that the disk is not full or a disk quota exceeded. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html 180402 16:49:52 InnoDB: Assertion failure in thread 2129656688 in file fil/fil0fil.c line 4200 InnoDB: Failing assertion: ret InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:49:52 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8384512 read_buffer_size=131072 max_used_connections=13 max_threads=151 thread_count=5 connection_count=5 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_siz[root@localhost ~]# tail -n 100 /var/log/mysqld.log InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html 180402 16:48:29 InnoDB: Assertion failure in thread 2129656688 in file fil/fil0fil.c line 4200 InnoDB: Failing assertion: ret InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:48:29 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.
I tried modifying the file
nano /etc/my.cnf to disable innodb and to try to reboot the mysql database but somehow i saw that i couldn't save the file at all. Immediately i saw that the issue is with the file system because it booted in only read-only file system.
tried to execute again
->fsck -n -f
-> fsck -n
-> fsck
[root@localhost ~]# fsck -f fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/mapper/VolGroup-lv_root: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/VolGroup-lv_root: ***** FILE SYSTEM WAS MODIFIED ***** /dev/mapper/VolGroup-lv_root: ***** REBOOT LINUX ***** /dev/mapper/VolGroup-lv_root: 73191/3276800 files (0.2% non-contiguous), 1070815/13107200 blocks
root@localhost ~]# fsck -n -f fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong (12036353, counted=12036385). Fix? no Free inodes count wrong (3203555, counted=3203609). Fix? no /dev/mapper/VolGroup-lv_root: ********** WARNING: Filesystem still has errors ********** /dev/mapper/VolGroup-lv_root: 73245/3276800 files (0.2% non-contiguous), 1070847/13107200 blocks
after just rebooting the linux i could start then mysqld and every service
[root@localhost ~]# reboot
root@localhost ~]# service mysqld restart Stopping mysqld: [ OK ] Starting mysqld: [ OK ] [root@localhost ~]# service radiusd restart Stopping RADIUS server: [FAILED] Starting RADIUS server: [ OK ]
Error creating directory: Read-only file system