Skip to main content

@v_vdb_node0005: VX001/2973: Data consistency problems found; startup aborted

While trying to start one of Vertica nodes you may face a data consistency problem.

from vertica.log

<PANIC> @v_vdb_node0005: VX001/2973: Data consistency problems found; startup aborted
        HINT:  Check that all file systems are properly mounted.  Also, the --force option can be used to delete corrupted data and recover from the cluster
        LOCATION:  mainEntryPoint, /scratch_a/release/svrtar5575/vbuild/vertica/Basics/vertica.cpp:1613


so.. don't PANIC :-)

solution:

restart the problematic node with force flag which will repair the corruptions from buddy nodes.

 [dbadmin ]$ /opt/vertica/bin/admintools -t restart_node -d $db_name -s $host --force

and the result:

*** Restarting nodes for database vdb ***
        restart host node0005 with catalog v_vdb_node0005_catalog
        issuing multi-node restart
        Starting nodes:
                v_vdb_node0005 (node0005)
        Starting Vertica on all nodes. Please wait, databases with large catalog may take a while to initialize.
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (DOWN)
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (DOWN)
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (DOWN)
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (DOWN)
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (RECOVERING)
        Node Status: v_vdb_node0001: (UP) v_vdb_node0005: (UP)


Comments

  1. ClaudiaJanuary 22, 2019 at 5:53 AM



    Hello,

    What if I encounter this situation on a single node?
    I tried to use this command :
    /opt/vertica/bin/admintools -t restart_node -d $db_name -s $host --force
    but since there is only a single node, restart doesn't work.

    Can you help me with that?

    Thanks a lot!

    ReplyDelete
  2. AceFebruary 21, 2019 at 12:18 AM

    try to restore using "rollback database to last good Epoch" - in the advanced menu

    ReplyDelete

Post a Comment

Popular posts from this blog

ORA-27104: system-defined limits for shared memory was misconfigured

I faced this error while trying to restore & recover of a PDB (pluggable database) part of the log file and the solution is described here below: log: initialization parameters used for automatic instance: db_name=CDB db_unique_name=gbux_pitr_PDB1_CDB compatible=12.1.0.2.0 db_block_size=8192 db_files=200 diagnostic_dest=/oracle/app/oracle _system_trig_enabled=FALSE sga_target=1888M processes=200 db_create_file_dest=/oracle/auxilary log_archive_dest_1='location=/oracle/auxilary' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 07/31/2016 16:22:20 RMAN-04014: startup failed: ORA-27104: system-defined limits for shared
6 comments
Read more

mount.nfs: backgrounding

if you face this kind of error with your remote NFS: [root@Vertica000 ~]# mount /files/application/Rremote3 mount.nfs: backgrounding "10.0.0.2:/files/application/remoteFiles" mount.nfs: mount options: "bg,hard,nointr,rsize=65536,wsize=65536,tcp,actimeo=0,vers=3,timeo=600,addr=10.0.0.2" look for the problem in the log file: cat /var/log/messages | grep mount  mount to NFS server '10.0.0.2' failed: timed out, retrying Solution: In most of the cases, you have a problem with your iptables in the destination server login as root to dest server (10.0.0.2) in my case and type this command: iptables --flush  the go back to your origin server to try remount the problematic NFS file system of course this is in case nfs server was installed and functioning properly. Good luck.
1 comment
Read more

ora-65035 while create pluggable database from another one

or : ORA-21000: error number argument to raise_application_error of -65035 is out of range Cause: this error caused because of an unrecoverable transaction in the source DB. Solution: 1. either stop and start your pluggable source DB 2. identify the open transaction and kill it if you can. use this query to do so: select * from v$transaction t,v$session  s where t.ses_addr = s.saddr;
Post a Comment
Read more