Segmentation fault w/ SSH

[Sheets]

I seem to be getting segmentation faults when I use ssh/scp. These may have started after the last software upgrade. I run OS X 10.4.8 on a Macbook Pro.

In anycase, when I ssh into a remote server I get this error in verbose mode:

OpenSSH_4.2p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data /etc/ssh_config
debug1: Connecting to [computer connecting to] port 22.
debug1: Connection established.
debug1: identity file /Users/[username]/.ssh/id_rsa type -1
debug1: identity file /Users/[username]/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.9p1
debug1: match: OpenSSH_3.9p1 pat OpenSSH_3.*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.2
Segmentation fault

Anyone else encountered this? I'm at a complete loss on what to do....

cheers.

[bankai]

It seems the step after this is when ssh checks the ~/.ssh/config file - do you have anything in this file ( on the remote machine or locally ) ?

[Sheets]

Well, the ~.ssh/config file doesn't seem to exist.

[ignorant question]should it? And what's its general form?[/ignorant question]

[Mailman42]

When you open a shell, and type -> ls -la what is the output?

I have the ~/.ssh directory.

If you do not, then (also in a shell) type -> mkdir .ssh <- this will create the directory for you.

[Sheets]

There is one file, "known_hosts".

[yellow]

Might I suggest increasing verbosity?

[scottj]

I'm getting the exact same error after applying the Feb 15th updates (Security, Java and DST). Unfortunately, I applied all three at once, so I can't say for sure which one caused it, but I would guess it was Security.

What's strange is that my SSH was working fine after the prior round of updates, which apparently caused the problem for Sheets. I use SSH all day, everyday, and it was definitely this last round that killed it.

My computer is a vanilla Macbook Core2Duo running 10.4.8 with a few common apps installed, and no OS modifications.

My output, with maximum verbosity:

OpenSSH_4.2p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to xxx.xxx.xxx.xxx port 22.
debug1: Connection established.
debug1: identity file /Users/scottj/.ssh/identity type -1
debug1: identity file /Users/scottj/.ssh/id_rsa type -1
debug1: identity file /Users/scottj/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.2
debug1: match: OpenSSH_4.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.2
debug2: fd 3 setting O_NONBLOCK
Segmentation fault

[hayne]

I don't know if you are prepared for this level of effort, but you might get more info about the reason for the crash by running 'ssh' under the debugger.
The debugger 'gdb' comes with the developer tools (Xcode) which is an optional install on the Tiger DVD.

To run 'ssh' under the debugger, you would run the command:

gdb /usr/bin/ssh

and then (after a short while) you will get to a gdb prompt. At that prompt you can type:
run
to start the 'ssh' program
or you can add the command-line parameters that you would normally give to 'ssh' after the 'run' command - for example:
run -l fred myserver.mydomain.com

[Sheets]

Well, using gdb did not give me much insight into the problem. The result was:

(gdb) run ssh username@address
`/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos' has changed; re-reading symbols.
`/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore' has changed; re-reading symbols.
`/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices' has changed; re-reading symbols.
`/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit' has changed; re-reading symbols.
`/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata' has changed; re-reading symbols.
`/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices' has changed; re-reading symbols.
`/usr/lib/libresolv.9.dylib' has changed; re-reading symbols.
Starting program: /usr/bin/ssh ssh username@address
Reading symbols for shared libraries + done
ssh: ssh: No address associated with nodename

Program exited with code 0377.

[hayne]

The message "No address associated with nodename" would seem to indicate a DNS problem - i.e. that there was a problem getting the IP address for the hostname you used.
Try using an explicit IP address in your SSH command to sidestep DNS issues.

[scottj]

Here's the output I got from gdb:

gdb) run xxx.xxx.xxx.xxx
Starting program: /usr/bin/ssh xxx.xxx.xxx.xxx
Reading symbols for shared libraries ................. done

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0xc30cd448
0x90b3fd2a in __cxa_get_globals ()

And for good measure, here's what the crashreporter log shows:

**********

Host Name: Boomer
Date/Time: 2007-02-16 11:12:37.228 -0800
OS Version: 10.4.8 (Build 8N1106)
Report Version: 4

Command: ssh
Path: /usr/bin/ssh
Parent: bash [248]

Version: ??? (???)

PID: 250
Thread: 0

Exception: EXC_BAD_ACCESS (0x0001)
Codes: KERN_INVALID_ADDRESS (0x0001) at 0xb1deab44

Thread 0 Crashed:
0 libstdc++.6.dylib 0x90b3fd2a __cxa_get_globals + 324
1 libstdc++.6.dylib 0x90b4042a __gxx_personality_v0 + 658
2 libgcc_s.1.dylib 0x90bd4acb _Unwind_RaiseException + 147
3 libstdc++.6.dylib 0x90b40747 __cxa_throw + 87
4 edu.mit.Kerberos 0x94b5b1f4 CCIContextDataMachIPCStub::OpenCCache(std::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&) + 314
5 edu.mit.Kerberos 0x94b5af9a CCEContext::OpenCCache(cc_context_d*, char const*, cc_ccache_d**) + 160
6 edu.mit.Kerberos 0x94b5ad1a cc_open + 64
7 edu.mit.Kerberos 0x94b5abb2 krb5_stdcc_resolve + 182
8 edu.mit.Kerberos 0x94b6015d __KLGetCCacheByName + 254
9 edu.mit.Kerberos 0x94b5fe46 __KLAcquireInitialTicketsForCache + 179
10 edu.mit.Kerberos 0x94b5fd3b krb5int_cc_default + 85
11 edu.mit.Kerberos 0x94b511cb krb5_gss_acquire_cred + 2395
12 edu.mit.Kerberos 0x94b5fccd kg_get_defcred + 73
13 edu.mit.Kerberos 0x94b5e9d0 krb5_gss_init_sec_context + 208
14 ssh 0x00027d42 0x1000 + 159042
15 ssh 0x00028283 0x1000 + 160387
16 ssh 0x00027675 0x1000 + 157301
17 ssh 0x000278d2 0x1000 + 157906
18 ssh 0x0000db53 0x1000 + 52051
19 ssh 0x0000b875 0x1000 + 43125
20 ssh 0x00003d60 0x1000 + 11616
21 ssh 0x00001a9e 0x1000 + 2718
22 ssh 0x000019c5 0x1000 + 2501

Thread 0 crashed with X86 Thread State (32-bit):
eax: 0x00000000 ebx: 0x90b406fd ecx: 0xbfffee3c edx: 0xa4b535a0
edi: 0xb1deab44 esi: 0xa4b535a0 ebp: 0xbfffeda8 esp: 0xbfffed70
ss: 0x0000001f efl: 0x00010246 eip: 0x90b3fd2a cs: 0x00000017
ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037

Binary Images Description:
0x1000 - 0x36fff ssh /usr/bin/ssh
0x8fe00000 - 0x8fe49fff dyld 46.9 /usr/lib/dyld
0x90000000 - 0x9016ffff libSystem.B.dylib /usr/lib/libSystem.B.dylib
0x901bf000 - 0x901c1fff libmathCommon.A.dylib /usr/lib/system/libmathCommon.A.dylib
0x90809000 - 0x908d1fff com.apple.CoreFoundation 6.4.6 (368.27) /System/Library/Frameworks/CoreFoundation.framework/Versi
ons/A/CoreFoundation
0x9090f000 - 0x9090ffff com.apple.CoreServices 10.4 (???) /System/Library/Frameworks/CoreServices.framework/Version
s/A/CoreServices
0x90911000 - 0x90a04fff libicucore.A.dylib /usr/lib/libicucore.A.dylib
0x90a54000 - 0x90ad3fff libobjc.A.dylib /usr/lib/libobjc.A.dylib
0x90afc000 - 0x90b60fff libstdc++.6.dylib /usr/lib/libstdc++.6.dylib
0x90bcf000 - 0x90bd6fff libgcc_s.1.dylib /usr/lib/libgcc_s.1.dylib
0x90bdb000 - 0x90c4efff com.apple.framework.IOKit 1.4.6 (???) /System/Library/Frameworks/IOKit.framework/Versions/A/IOK
it
0x90c63000 - 0x90c75fff libauto.dylib /usr/lib/libauto.dylib
0x90c7b000 - 0x90f21fff com.apple.CoreServices.CarbonCore 682.16 /System/Library/Frameworks/CoreServices.framework
/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
0x90f64000 - 0x90fccfff com.apple.CoreServices.OSServices 4.1 /System/Library/Frameworks/CoreServices.framework/Version
s/A/Frameworks/OSServices.framework/Versions/A/OSServices
0x91005000 - 0x91043fff com.apple.CFNetwork 129.19 /System/Library/Frameworks/CoreServices.framework/Versions/A/Fram
eworks/CFNetwork.framework/Versions/A/CFNetwork
0x91056000 - 0x91066fff com.apple.WebServices 1.1.3 (1.1.0) /System/Library/Frameworks/CoreServices.framework/Version
s/A/Frameworks/WebServicesCore.framework/Versions/A/WebServicesCore
0x91071000 - 0x910f0fff com.apple.SearchKit 1.0.5 /System/Library/Frameworks/CoreServices.framework/Versions/A/Fram
eworks/SearchKit.framework/Versions/A/SearchKit
0x9112a000 - 0x91148fff com.apple.Metadata 10.4.4 (121.36) /System/Library/Frameworks/CoreServices.framework/Version
s/A/Frameworks/Metadata.framework/Versions/A/Metadata
0x91154000 - 0x91162fff libz.1.dylib /usr/lib/libz.1.dylib
0x91165000 - 0x91304fff com.apple.security 4.5.2 (29774) /System/Library/Frameworks/Security.framework/Versions/A/
Security
0x91402000 - 0x9140afff com.apple.DiskArbitration 2.1.1 /System/Library/Frameworks/DiskArbitration.framework/Versions/A/D
iskArbitration
0x91411000 - 0x91437fff com.apple.SystemConfiguration 1.8.6 /System/Library/Frameworks/SystemConfiguration.framework/
Versions/A/SystemConfiguration
0x91449000 - 0x91450fff libbsm.dylib /usr/lib/libbsm.dylib
0x9193e000 - 0x919f0fff libcrypto.0.9.7.dylib /usr/lib/libcrypto.0.9.7.dylib
0x94981000 - 0x9499efff libresolv.9.dylib /usr/lib/libresolv.9.dylib
0x94b49000 - 0x94c21fff edu.mit.Kerberos 5.5.23 /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos

**********

-Scott

[yellow]

Before we go further, please log in as another user and verify that SSH is broken there as well.

I find it odd that the kerberos storage file should be referenced at all when using SSH.

0x94b49000 - 0x94c21fff edu.mit.Kerberos 5.5.23 /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos

You might want to try renaming that file and trying again. I'm not sure if a reboot is necessary betwixt renaming and trying to use SSH. I suspect not, but I'm not 100% sure. It can be found in /Library/Preferences/

[scottj]

Tried logging in as a different user, and I do still get the same segfault when I run ssh.

I'm not exactly clear which file you suggest I try to rename. I didn't find anything related to kerberos in either /Library/Preferences/ or ~/Library/Preferences/. Can you clarify?

[yellow]

/Library/Preferences/edu.mit.Kerberos

[scottj]

I don't have that file. Should I create it?

[yellow]

Nope. It's not needed by SSH.

I am perplexed as to why/how it was referenced in that crashlog.


Anyway, as far as I see it, you have a couple of options to choose from here.

1) I'd try doing some reinstallation of patches. Start with the 10.4.8 Combo Updater patch. See if that helps/has no effect. Assuming it doesn't, I'd look for the last patch (post 10.4.8 Combo) that referenced ssh and reinstall that.

2) Try using fink to download/compile a newer version of OpenSSH and see if that works for you.

3) Download the source for a newer version of OpenSSH and compile/install it on your own.

[podpearson]

Seems like a few people, including myself, are having this problem. Try googling ssh segmentation fault macosx.

I've contacted Apple about this at http://www.apple.com/macosx/feedback/ . I'd suggest other people with the same problem do likewise.

I'm also having a problem with sftp - are other people?

Regards
- Pod

[drtromo]

I've been having these kinds of problems (crashes on __cxa_get_globals related to Kerberos) for months now on my MacBook, although I haven't noticed ssh being affected. For me, it's usually things related to the KeyChain, which then takes out AFP, any secure mail, etc.

This is a completely phenomenological argument, but I started from the base 10.4 archive-n-install and worked my way through all of the updates, each time tar'ing up the following directories:

/System/Library/CoreServices/{Kerberos.app,KerberosAgent.app}
/System/Library/Frameworks/Kerberos.framework
/usr/lib/*sasl2*

I found a set that did work and now I just untar them over the "bad" copies. It does seem to restore most functionality. Unfortunately, anytime there's a update_prebinding event, it hoses those and I have to untar again.

I haven't tried to figure out why this works or what other side-effects it has, only that it restores enough functionality for me to get by without having to do an archive-n-install every week or two.

Caveat emptor...

[scottj]

As has been pointed out here and elsewhere, the problem I had was indeed due to a problem in the Kerberos framework introduced when I applied the latest security updates.

There's a post near the end of this thread (http://discussions.apple.com/thread.jspa?messageID=4096888#4096888) that details how to fix it using the 10.4.8. combo updater and a cool little utility called Pacifist.

-S