Technical troubleshooting and more

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is Please contact your system administrator. Add correct host key in /home/user/.ssh/known_hosts to get rid of this message. Offending RSA key in /home/user/.ssh/known_hosts:[] RSA host key for 192.168.0.10 has changed and you have requested strict checking. Host key verification failed. this can happen when you try to recreate the same machine again with the same IP but with different key, in this case you need to remove the record of the destination server from your local  known_hosts server and then you can re login.

After trying to run a failed upgrade to Vertica DB (network issues) we run into a situation that  /opt/vertica/config folder permissions turns to root:root we couldn't run even admintools to see what's going on then we realized it's a permission issues on the config folder in order to fix it - simply change this folder permissions chown -R dbadmin:verticadba /opt/vertica/config good luck!

in our case I was trying to add 5 nodes to Vertica 10 nodes cluster. while adding the new nodes we were facing some problems with the file system and we were forced to terminate the new machines. we fixed the VM image and recreated the nodes in order to scale out Vertica cluster again. during this process we were keep getting an errors message like: "node x already exists" while running the add nodes command (although we removed them from admintools.conf and from the cluster using: /opt/vertica/sbin/install_vertica --remove-hosts) finally, we found out a solution that was very helpful in our case and is not documented well in Vertica documentation, (we were not able to find it there) drop node command. DROP NODE <node_name>;   probably leftovers of the first scale out run were stayed in the database metadata itself, I guess that when we terminated the new nodes after the first try we didn't notice that one of the nodes were still part of the cluster

when running Vertica vnetperf utility you may face these errors below although you have no FW between servers and the communication is OK and no network errors are observed Vertica DB is up and running on these Vertica hosts as well [Connector Thread 172.16.8.11 ] Couldn't connect to 172.16.8.11 (family 2, attempt 0): Connection refused; errno=111 (Connection refused) [Connector Thread 172.16.8.10 ] Couldn't connect to 172.16.8.10 (family 2, attempt 0): Connection refused; errno=111 (Connection refused) [Connector Thread 172.16.8.12 ] Couldn't connect to 172.16.8.12 (family 2, attempt 0): Connection refused; errno=111 (Connection refused) [Connector Thread 172.16.8.11 ] Could not find anything to connect to for 172.16.8.11; errno=111 (Connection refused) [Connector Thread 172.16.8.10 ] Could not find anything to connect to for 172.16.8.10; errno=111 (Connection refused) [Connector Thread 172.16.8.12 ] Could not find anything to connect to for 172.16.8.12; err

let's say for example that you have 3 Vertica nodes and the inteconnect communication subnet was 10.0.0.X now for some reason you want to change the internal communication between the nodes to use another interface using this subnet:  192.168.1.X please follow these step by step guide in order to accomplish this target: ----------------------------------------------------------------- first change network from to use point-to-point mode 1. select set_control_mode('pt2pt'); select reload_spread(true); Check in the catalog directory if spread.conf got modified to broadcast mode. 2. Edit admintools.conf  controlmode controlmode = pt2pt 3. Use admintools to distribute database configuration and metadata to all cluster nodes 4. In order to make spread adjust new configuration you need to restart Vertica DB for each Vertica node run these steps to change IP used by interconnect 1. Stop Vertica on node 2.  Change hostname through vsql : alter node

After rebooting Vertica cluster I ran into this error in one of Vertica nodes startup.log: {   "node" : "v_vdb_node0003",   "stage" : "Database Halted",   "text" : "@v_vdb_node0003: VX001/3381: Failed to install new catalog\n\tLOCATION:  doInstallAndJoin, /scratch_a/release/svrtar3866/vbuild/vertica/Transaction/TransAPI.cpp:2886",   "timestamp" : "2018-01-17 08:32:40.880" } after wondering what can go wrong and trying to start the node in force mode without any success...  I saw this little message in Vertica.log file: 2018-01-17 08:30:23.210 Spread Client:0x9b39800 <ERROR> @v_vdb_node0003: {doInstallAndJoin} 42501/2812: Could not add location [/vertica_data2]: Permission denied then I realized that on this specific node we had another STORAGE LOCATION after remounting the /vertica_data2 file system and starting the node everything become normal :-) 

this is how it should look like: app.js index.js replicaSet.jade result: