Technical troubleshooting and more

While trying to start one of Vertica nodes you may face a data consistency problem. from vertica.log <PANIC> @v_vdb_node0005: VX001/2973: Data consistency problems found; startup aborted         HINT:  Check that all file systems are properly mounted.  Also, the --force option can be used to delete corrupted data and recover from the cluster         LOCATION:  mainEntryPoint, /scratch_a/release/svrtar5575/vbuild/vertica/Basics/vertica.cpp:1613 so.. don't PANIC :-) solution: restart the problematic node with force flag which will repair the corruptions from buddy nodes.  [dbadmin ]$ /opt/vertica/bin/admintools -t restart_node -d $db_name -s $host --force and the result: *** Restarting nodes for database vdb ***         restart host node0005 with catalog v_vdb_node0005_catalog         issuing multi-node restart         Starting nodes:                 v_vdb_node0005 (node0005)         Starting Vertica on all nodes. Please wait, databa

if you face this kind of error with your remote NFS: [root@Vertica000 ~]# mount /files/application/Rremote3 mount.nfs: backgrounding "10.0.0.2:/files/application/remoteFiles" mount.nfs: mount options: "bg,hard,nointr,rsize=65536,wsize=65536,tcp,actimeo=0,vers=3,timeo=600,addr=10.0.0.2" look for the problem in the log file: cat /var/log/messages | grep mount  mount to NFS server '10.0.0.2' failed: timed out, retrying Solution: In most of the cases, you have a problem with your iptables in the destination server login as root to dest server (10.0.0.2) in my case and type this command: iptables --flush  the go back to your origin server to try remount the problematic NFS file system of course this is in case nfs server was installed and functioning properly. Good luck.

The problem: [oracle@host1 ~]$ crontab -l Authentication token is no longer valid; new one required You (oracle) are not allowed to access to (crontab) because of pam configuration. Solution: when using PAM configuration and switching between users from root with su command (or using sudo from authenticated user using auth key), when the password of the user is expired or about to be expired you may face this problem. * switch to root * change the problematic account password (using passwd command) Check yourself: * su - [account] * crontab -l good luck :-)