Recently I have been playing with the Global Parallel File System, which is a clustered file system from IBM. When setting up a cluster you can configure GPFS to utilize SSH/SCP to send administrative tasks to the other nodes in the cluster.
The problem I ran into was that in my environment I do not run SSH over port 22 (for various reasons I wont get into). Needless to say once I configured SSH to listen on an alternate port GPFS stopped working.
After a bit of Googling I could not find anywhere in GPFS itself where you can specify the SSH port to connect to. All hope seemed lost, but then I remembered a little trick I learned long ago.
Change the ssh port
When GPFS is issuing an SSH command it is simply using the ssh client to connect. The ssh client, much like the sshd server has a configuration file. Within that configuration file you can specify what port to connect to by default. In order to force GPFS to connect to an SSH server via a non-standard port you can change the port in the SSH Client configuration file. After this change all ssh client connections will default to the specified port.
Below are instructions on how to change the ssh client & server ports to port 2022 and then test to ensure GPFS can communicate with all nodes.
The below steps will need to be executed on all hosts in the GPFS cluster.
Change the ssh server's default port
Edit the sshd_config file
Before changing the client configuration we need to tell the SSHD server to listen on an alternative port; we will be using port 2022 for our example.
[allhosts ~]# vi /etc/ssh/sshd_config
Before this change takes affect we will need to restart the SSHD service.
Before logging out open a new ssh connection to this server to validate SSHD is up and accepting connections. If it is not working, make sure you check your firewall rules.
[allhosts ~]# /etc/init.d/sshd restart
Change the ssh client's default port
At this point if you simply tried to type
ssh <server> the command would fail as the client is defaulting to port 22.
[host2 ~]# ssh host1 ssh: connect to host host1 port 22: Connection refused [host2 ~]# ssh host1 -p 2022 [email protected]'s password:
Edit the ssh_config file
In order to make the ssh client default to port 2022 we will need to edit the ssh_config file.
[allhosts ~]# vi /etc/ssh/ssh_config
# Port 22
Test ssh connectivity
Now all ssh connectivity on hosts where you changed ssh_config will default to port 2022.
[host1 ~]# ssh host2 [email protected]'s password: [host2 ~]# ssh host1 [email protected]'s password:
Test GPFS communication
Before SSH Change:
# mmgetstate -a Node number Node name GPFS state ------------------------------------------ 1 HOST1 unknown 3 HOST3 unknown 4 HOST2 unknown 5 HOST4 unknown 6 HOST5 active
After SSH Change:
# mmgetstate -a Node number Node name GPFS state ------------------------------------------ 1 HOST1 active 3 HOST3 active 4 HOST2 active 5 HOST4 active 6 HOST5 active
While the above examples are geared for GPFS this concept can be applied to any service that uses ssh to communicate that also uses the servers ssh client. This method would not apply to any applications that use their own ssh client implementation.
Recently Benjamin published his first book; Red Hat Enterprise Linux Troubleshooting Guide. In addition to writing, he has several Open Source projects focused on making Ops easier. These projects include Automatron, a project enabling auto-healing infrastructure for the masses.
Identify, capture and resolve common issues faced by Red Hat Enterprise Linux administrators using best practices and advanced troubleshooting techniques
What people are saying:
Excellent, excellent resource for practical guidance on how to troubleshoot a wide variety of problems on Red Hat Linux. I particularly enjoyed how the author made sure to provide solid background and practical examples. I have a lot of experience on Red Hat but still came away with some great practical tools to add to my toolkit. - Amazon Review