Login  
Search All Forums
Dart Home | PowerTCP SFTP & SSH for .NET | Custom Development Reply | PowerTCP SFTP & SSH for .NET Topics | Forums   
AuthorForum: PowerTCP SFTP & SSH for .NET
Topic: SFTP Thread Being Aborted on Large Directories
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 15, 2011 11:53 PM
Currently working with PowerSFTP and have been having no issues using it, but recently an issue has cropped on in a single instance.

I'm using the SFTP to create a .NET3.5 reporting tool that reads certain directories on a file share. While most of the reports run fine, one in particular is timing out regularly after only a minute or two.

The code:
Public Shared Sub ProcessDirectory(ByVal targetDirectory As String, _
                   ByVal oSFTP As Dart.Ssh.Sftp, _
                   ByVal blnFindFiles As Boolean, _
                   ByVal strFileName As String, _
                   ByRef blnFileFound As Boolean, _
                   ByVal blnLogFiles As Boolean, _
                   ByVal dteStartDate As Date, _
                   ByVal dteEndDate As Date, _
                   ByVal intFileSize As Integer, _
                   ByRef dtblAllFiles As DataTable)

  Dim lstSubFiles As List(Of Dart.Ssh.ListEntry)
  Dim leSubFiles As Dart.Ssh.ListEntry
  Dim dteCreationDate As Date

  Try
   lstSubFiles = oSFTP.List(targetDirectory)
  Catch ex As Exception
   Throw New Exception("An error has occured listing files for " & targetDirectory & " : " & ex.ToString)
  End Try

  For Each leSubFiles In lstSubFiles
   If leSubFiles.Attributes.IsDirectory Then
    If leSubFiles.FileName <> BACKUPDATA And leSubFiles.FileName <> CURRENT And leSubFiles.FileName <> PARENT Then
     ProcessDirectory(leSubFiles.Path, oSFTP, blnFindFiles, strFileName, blnFileFound, blnLogFiles, dteStartDate, dteEndDate, intFileSize, dtblAllFiles)
    Else
     'Ignore this directory
    End If

After a certain point the oSFTP.List command is failing with the error:
System.Threading.ThreadAbortException: Thread was being aborted.
  at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
  at 3.7.0(8 , Boolean , Byte[] , Int32 , Int32 )
  at 6.1.Read(Byte[] b, Int32 off, Int32 len)
  at 6.6.5(Byte[] , Int32 , Int32 )
  at 6.6.1(Int32 )
  at 6.6.0(Byte[] , String )
  at 6.6.0(String )
  at Dart.Ssh.Sftp.List(String directoryPath)

This app is running under IIS7 so I made some changes to the site's settings based on that, but it's having no effect. What's weird is that other reports call this ProcessDirectory command without failing, and this particular report has worked in the past, but now it is having a 100% failure rate.

Any advice?
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 16, 2011 12:58 AM
To makes things more confusing, if debug is set to true, this error does not occur.
Nick B (Admin)

From: Utica, NY USA
Posts: 619
Member Since: 05/25/10

Extra Support Options
Custom Application Development

 posted June 16, 2011 9:51 AM
Hello,

What version of the control are you using?

Can you provide us with a small, self-contained sample that reproduces your issue? (Seems like it would require access to your server) Please send it into support@dart.com, if possible.

Do your other reports call list on the same directory, or a different directory? You say this worked in the past; are you able to determine if any changes have been made to the directory, and if this issue is caused by those changes?

You say that this report times out after a minute or two; is this longer or shorter than expected?

You say that if debug is set to true, the error doesn't present. Perhaps it is overriding JIT Debugging? Have you tried enabling or disabling the JIT Debugger? http://msdn.microsoft.com/en-us/library/5hs4b7a6.aspx
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 17, 2011 12:04 AM
Hi Nick,
I'm running version 4.3 of the control.

I'll whip up a code sample and send it to you, but cut out any server access stuff. The main thing it does is read all files in a directory and its subdirectories and adds each file to a datatable if it matches a certain set of criteria.

The other reports are running fine when calling on the same directory - but they have a much smaller sample size. The one report that is failing has to iterate over ALL subdirectories and files in the given directory, while the others only read certain subdirectories. Speaking with our file server gurus, they haven't changed anything within the past month.

The actual report, when debugging is turned on and its working, takes 30 minutes+ due to the sheer volume of directories and files it has to cover. The report in question is one the staff who will eventually use it would run once every six months or so.

JIT was enabled, so I disabled it all and tried again, still getting the same issue.

Thanks!
Nick B (Admin)

From: Utica, NY USA
Posts: 619
Member Since: 05/25/10

Extra Support Options
Custom Application Development

 posted June 17, 2011 9:14 AM
I will wait for your sample, but I suspect the issue could lay just as much with your server, than with your application.

Are you able to determine if the list fails on a specific directory, or file?

Have any files or directories changed on your server between when it worked and when it stopped working?
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 28, 2011 3:22 AM
Hi Nick,
Thanks for your help on all this. Turns out the thread being aborted error was due to a time out issue, not the FTP at all.

However, I have come across a new issue. I'm using the SFTP tool to check for the presence of files on the server:

Public Shared Function FindFiles(ByVal strMcouId As String _
                 , ByVal oSFTP As Sftp _
                 , ByVal strFileName As String) As Boolean

  Dim dtblAllFiles As DataTable
  Dim lstFileList As List(Of Dart.Ssh.ListEntry)
  Dim leFile As Dart.Ssh.ListEntry
  Dim blnFileFound As Boolean = False
  Dim blnFindFiles As Boolean = True
  Dim blnLogFiles As Boolean = False
  Dim dteStartDate As Date = CDate("01/01/1000")
  Dim dteEndDate As Date
  Dim intFileSize As Integer = 0
  Dim strFilePath As String

  dtblAllFiles = New DataTable
  dtblAllFiles.Columns.Add("DIRECTORY")
  dtblAllFiles.Columns.Add("FILENAME")
  dtblAllFiles.Columns.Add("FILESIZE")
  dtblAllFiles.Columns.Add("CREATIONDATE")

  lstFileList = New List(Of Dart.Ssh.ListEntry)

  Try
   'We don't check for an empty string, as that is valid and would give us the root path
   lstFileList = oSFTP.List(ConfigurationManager.AppSettings("MoodleFilePath") & "/" & strMcouId)
  Catch ex As Exception
   Throw New Exception("An error has occured attempting to read files for Course " & strMcouId & " " & ex.ToString)
  End Try

  If lstFileList.Count > 0 Then
   For Each leFile In lstFileList
    If leFile.Attributes.IsDirectory Then
     'Course folders have numeric names, only want to check those
     If IsNumeric(leFile.FileName) Then
      blnFileFound = FindFileInDirectory(leFile.Path, oSFTP, blnFileFound, strFileName, dteStartDate, dteEndDate, intFileSize)
      If blnFileFound Then Return blnFileFound
     ElseIf leFile.FileName <> BACKUPDATA And leFile.FileName <> CURRENT And leFile.FileName <> PARENT _
       And leFile.FileName <> PRIVATEDIR And leFile.FileName <> TEMP Then
      'Do nothing, we don't want to process these directories
     Else
      'We have iterated over every course
      Return blnFileFound
     End If
    Else
     'Initialise the file path
     strFilePath = leFile.Path.Substring(0, leFile.Path.Length - leFile.FileName.Length)
     Try
      oSFTP.GetAttributes(strFilePath & strFileName, False)
      'If the above does not throw an exception, we've found the file
      blnFileFound = True
      Return blnFileFound
     Catch exFileNotFound As Dart.Ssh.SftpException
      blnFileFound = False
      Return blnFileFound
     Catch ex As Exception
      Throw New Exception("File not found " & ex.ToString & Environment.NewLine & "File: " & strFileName & _
                "Path: " & strFilePath)
     End Try

    End If
   Next
  Else
   'No files found at all
   blnFileFound = False
   Return blnFileFound
  End If

 End Function

However, it is failing to find files that are definitely there. One directory has only two files in it but they're both failing to be read:

http://i.imgur.com/TCdfK.png

The account SFTP is using is itumoodle and as you can see, it has read and write access.

In the code above, though, the lstFileList object only has two objects in it (. and ..)

Am I using the wrong commands or is there a deeper permissions issue here? Logging on directly to the server as itumoodle allows you to make directories, edit files, etc.

Finally, to make things more confusing, the SFTP tool CAN view this file :

http://i.imgur.com/LTolj.png

Any thoughts?
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 28, 2011 3:24 AM
As a test, I inserted the following code (note that itumoodle user account does have create directory access):

  Try
   oSFTP.CreateDirectory(ConfigurationManager.AppSettings("MoodleFilePath") & "/" & strMcouId, 101666)
  Catch ex As Exception
   Throw New Exception("Could not create a directory in " & ConfigurationManager.AppSettings("MoodleFilePath") & "/" & strMcouId & _
             Environment.NewLine & ex.ToString)
  End Try

However, this fails with the following exception:

System.Exception: Could not create a directory in /m0/moodledata/1374
Dart.Ssh.SftpException: Failure (SSH_FX_FAILURE: An error occurred, but no specific error code exists to describe the failure.)
 at 6.6.3(UInt32 )
 at 6.6.0(String , Int32 )
 at Dart.Ssh.Sftp.CreateDirectory(String path, Int32 posixPermissions)
Nick B (Admin)

From: Utica, NY USA
Posts: 619
Member Since: 05/25/10

Extra Support Options
Custom Application Development

 posted June 28, 2011 9:48 AM
Hello,

You seem to be doing some work with the results of the list(); if you use our included SFTP Client sample, do you see the expected files?

If you do not, if you use a third-party SFTP client, do you see the expected files?

Does the 1374 directory already exist in that location?

If you use our included SFTP Client sample, are you able to create that directory in the same location?

If you cannot, if you use a third-party SFTP client, can you create that directory in the same location?
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 28, 2011 8:49 PM
Using the VB3.5 version of the SFTP Client, I can view the files and create directories - using the exact same account and private key/public key pair, which is why I'm confused. I'm wondering if it may be due to not using the tool properly.

The 1374 directory already exists - I run a "directory exists" check before attempting to list it's contents.


Here's the code I use to authenticate and connect to the server in my application:
 Public Shared Function OpenConnection() As Sftp
  ' Create the connection to the remote server
  Dim ftpSecure As Sftp
  Dim loginData As SshLoginData
  Dim strPrivateKey As String = ""

  'Load the private key data
  Dim fsKeyFile As FileStream = New FileStream(System.Web.HttpContext.Current.Server.MapPath(My.Resources.Ssh_Private_Key), FileMode.Open, FileAccess.Read)
  Dim bByte() As Byte = New Byte(999) {}
  Dim intFileLength As Integer = fsKeyFile.Read(bByte, 0, bByte.Length)
  Do While intFileLength > 0
   strPrivateKey += Encoding.ASCII.GetString(bByte, 0, intFileLength)
   intFileLength = fsKeyFile.Read(bByte, 0, bByte.Length)
  Loop
  fsKeyFile.Close()

  'Set the login settings for SFTP
  loginData = New SshLoginData(ConfigurationManager.AppSettings("MoodleHost"), ConfigurationManager.AppSettings("PortNumber"), _
                 ConfigurationManager.AppSettings("MoodleUser"), "", strPrivateKey)

  ftpSecure = New Sftp

  With ftpSecure
   .Connection.Proxy = Nothing
   .Connection.Port = loginData.Port
  End With

  Try
   ftpSecure.Connect(loginData.HostNameOrAddress)
   ftpSecure.Authenticate(loginData.Username, loginData.Password, loginData.Pem)
  Catch ex As IOException
   ' Lost our connection. Let's reconnect.
   System.Web.HttpContext.Current.Trace.Warn("Connection", "Connection failed")
   OpenConnection()
  Catch ex As Exception
   Throw New Exception("An error has occurred logging into the Moodle file share : " & ex.ToString)
  End Try

  Return ftpSecure
 End Function

It connects on port 22. The connect function is called by creating a new instance of Dart.SFTP:

Dim oSFTP as new Sftp
oSFTP = SFTPHelper.OpenConnection()
Nick B (Admin)

From: Utica, NY USA
Posts: 619
Member Since: 05/25/10

Extra Support Options
Custom Application Development

 posted June 29, 2011 9:38 AM
Does the directory you're trying to create already exist?

Nothing looks wrong with the code you've provided there; I'd recommend examining our SFTP Client to see what the differences are, and try to isolate your issue.
ScottGage_UniSA

From: Mawson Lakes, Australia
Posts: 7
Member Since: 05/25/11		 posted June 29, 2011 9:42 PM
Hi Nick,
Thanks for your help - and yes, it does turn out that the issue is with my code! :)

The problem was this little if statement here:
If leSubFiles.FileName <> BACKUPDATA And leSubFiles.FileName <> CURRENT And leSubFiles.FileName <> PARENT Then

I didn't want to get stuck in a loop of going to the parent directory or anything, but of course since I used the AND method, this statement would never be true (the filename would have to be three things at once!) so I was bailing without reading any files.

I've changed the code around to ignore the directory if it's parent OR current OR backup, and things are working fine now.
Reply | PowerTCP SFTP & SSH for .NET Topics | Forums   
This site is powered by PowerTCP WebServer Tool PowerTCP WebServer for ActiveX