[navidemad]

Problem

rb_tinytds_connect calls dbopen(), and dbopen() does the login. During the login, the DBPROCESS has no userdata yet. When the login fails, FreeTDS calls tinytds_err_handler or tinytds_msg_handler. These handlers find no userdata, so they raise at once.

A raise from C is a longjmp: it leaves dbopen() before FreeTDS can clean up. Thus FreeTDS never frees the half-open connection:

• the DBPROCESS,
• the socket and the connection objects (tds_alloc_socket, tds_init_connection),
• the error strings (dbperror, tds_alloc_client_sqlstate).

Each failed attempt leaks 70 KB to 150 KB. A client that retries against a server that is not available loses this memory on each retry.

A related leak is in the same function. If the server negotiates a TDS version older than 7.3, rb_tinytds_connect raises after dbopen() returns a connection. At that point cwrap->closed is still 1, so the free function does not call dbclose(). The open connection leaks.

Fix

1. During dbopen(), the handlers keep the first error in a tinytds_login_error struct. The struct is on the stack of rb_tinytds_connect. The handlers return to FreeTDS and do not raise. FreeTDS then stops the login and frees its objects.
2. After dbopen() returns, rb_tinytds_connect raises the kept error. If dbopen() returned a connection, the function closes it first.
3. Before the "older than 7.3" error, rb_tinytds_connect closes the connection.

The raised exception does not change. It is the first error that the handlers receive, with the same message, source, severity, db_error_number and os_error_number. During the login, info messages stay ignored, as before. During dbopen(), the error handler returns INT_CANCEL, also for SYBETIME. FreeTDS answers INT_TIMEOUT with a cancel and waits again, so only INT_CANCEL stops the login. The login timeout test checks that the error comes before 10 seconds.

The pointer to the struct is a static variable. dbopen() runs with the GVL held, so only one login is in progress at a time. A handler reads the pointer only when its DBPROCESS has no userdata.

rb_tinytds_connect must build the exception without a DBPROCESS. Thus this PR moves the creation of the TinyTds::Error object from rb_tinytds_raise_error into a new rb_tinytds_new_error function. rb_tinytds_raise_error calls it and keeps its behavior.

Measurements

The tool is ruby_memcheck 3.0.1. It runs Ruby under Valgrind and removes the false positives that Ruby causes. A "definitely lost" block is a block with no pointer to it at exit.

Scenario	Attempts	master	This PR
Wrong username	5	443,394 bytes	32,879 bytes
Wrong username	20	1,674,939 bytes	32,879 bytes
Connection refused (port 1)	5	386,984 bytes	32,879 bytes
Connection refused (port 1)	20	1,449,299 bytes	32,879 bytes
tds_version: "7.1"	5	771,989 bytes	32,879 bytes
tds_version: "7.1"	20	2,989,319 bytes	32,879 bytes

On master, each attempt adds about 82 KB (wrong username), 71 KB (connection refused) or 148 KB (TDS 7.1). With this PR, the total stays the same for any number of attempts. The 32,879 bytes that remain come from the one-time dbinit allocations (the FreeTDS context).

Reproduction script

# MODE=login_error N=20 ruby_memcheck -Ilib leak_repro.rb
require "tiny_tds"

opts = { host: "mssql", username: "tinytds", password: "", database: "tinytdstest", login_timeout: 5 }
opts = case ENV.fetch("MODE")
  when "login_error" then opts.merge(username: "willnotwork")
  when "refused" then opts.merge(host: "127.0.0.1", port: 1)
  when "old_tds" then opts.merge(tds_version: "7.1")
  end

Integer(ENV.fetch("N", "10")).times do
  TinyTds::Client.new(**opts)
rescue TinyTds::Error
  nil
end

Validation

• bundle exec rake test: 124 tests, 538 assertions, 0 failures, 0 errors.
• The existing tests check the raised error for an unknown dataserver (error handler), a wrong username (message handler) and a login timeout (SYBETIME through toxiproxy).
• A new test checks the "older than 7.3" error. It covers the new dbclose() call, and it checks that a new connection works after the error.
• The full test suite under ruby_memcheck (binary_name: "tiny_tds") shows no invalid read or write, and no leak from dbopen(). The remaining 4,136-byte blocks come from nogvl_setup. Free the non-blocking error array before a raise #613 fixes them. With both PRs, only the one-time dbinit blocks remain.
• astyle --options=astyle.conf does not change the C files. standardrb reports no offense in test/client_test.rb.

Environment: Ruby 4.0.7 on Linux aarch64, FreeTDS 1.5.19 built from source, SQL Server 2022 CU27 (mcr.microsoft.com/mssql/server:2022-latest), Valgrind 3.24.0.

Note

The draft v4 branch (#603) keeps the same dbopen() call and the same handlers. That branch has the same leaks.

[andyundso]

I am wondering if you could simplify that to just tinytds_errordata. because if that is not not NULL when returning from dbopen, then an error occured, otherwise not.

[navidemad]

Done in ae50c59: the wrapper struct is gone, the slot is a plain tinytds_errordata.

A pointer alone cannot carry both facts, though. The handlers use the static pointer to know that a login is in progress, so it must be set before dbopen(), even when no error comes. A handler without userdata outside dbopen() (for example from dblogin()) must still raise at once. Thus the slot uses is_message = 1 as "empty": the handlers keep errors only, never info messages.

While testing this, I found a bug in my first commit. For SYBETIME, the handler returned INT_TIMEOUT. During the login, FreeTDS answers INT_TIMEOUT with a cancel and waits again (tdserror: returning TDS_INT_CONTINUE), so a 1 s login timeout took about 100 s. The handler now returns INT_CANCEL for every deferred login error, and the login timeout test checks the duration.