Home/ Questions/Q 8615073
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T05:16:59+00:00

PROBLEM Loading fixtures for large text fields crashes the fixture-processing process and often even

  • 0

PROBLEM

Loading fixtures for large text fields crashes the fixture-processing process and often even corrupts the fixture (.yml) file.

ENVIRONMENT

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin10.8.0]; Rails 3.2.6; OSX 10.6.8; WebBrick; and MySQL ― 8 Gb RAM

BACKGROUND

1:  Our project displays a multitude of articles which can regularly
exceed 5-10,000 words.

2:  Escaped content assigned to records (even of moderate volume)
crashes fixture processes.

3:  rake db:fixtures:load will crash on the subject .yml file with the
following error/warning: Could not determine content-length of
response body. Set content-length of the response or set
Response#chunked = true.

4:  Even when we break this material down into what are quite modest-sized text fields of 1,000 words or less, db:fixtures:load will still crash on the subject .yml file.

5:  Files are often corrupted by the crash. For example, I can rebuild the fixture from scratch with empty field assignments; I can add and process (db:fixtures:load) a single field at a time until reaching an apparent processing volume limit, and when this ultimately crashes the fixture-loading process, errant characters are sometimes reported in the initial few characters of the file; and even when I delete the last field assignment, that file will no longer run.

6:  Experimentation has established that there’s an approximately 50k limit on the yml that a fixture can assign to a single record. I have many yml files which successfully assign to multiple records, but none which successfully assign more than 50k *to a single record. This experience is uniform across dozens of tables.

6.1:  For example, a principal file that suggests such a threshold only creates a single record:

6.1.1:  If I assign empty field contents to all text fields, rake db:fixtures:load succeeds in creating a record void of its text fields.

6.1.2:  But if I increase the size (k) inserted into that single record, field by field, as soon as I cross the 50k threshold (approximately), the process crashes with the following error:

rake aborted! Mysql2::Error: Got error 139 from storage engine: INSERT
INTO [table_name]

6.1.3:  Yet increasing MySQL buffer sizes has not resolved the problem. I have the exact same limitation.

6.1.3.1:  Here are my MySQL buffer sizes in my.cnf:

[mysqld]
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M

6.1.4:  I also set chunked to true in all instances of webrick/httpresponse.rb as follows:

def initialize(config)
  @config = config
  @logger = config[:Logger]
  @header = Hash.new
  @status = HTTPStatus::RC_OK
  @reason_phrase = nil
  @http_version = HTTPVersion::convert(@config[:HTTPVersion])
  @body = ''
  @keep_alive = true
  @cookies = []
  @request_method = nil
  @request_uri = nil
  @request_http_version = @http_version  # temporary
  @chunked = true # @chunked = false
  @filename = nil
  @sent_size = 0
end

None of these attempts had any effect on the issue.

7:  Thus I deduce the threshold applies to the k assigned to a single record, because I have a number of yml files that exceed 1M.

RELATED ISSUES?

8:  I see other questions report this same “Set content-length of the response” warning/error. The instances I read however report the error occurring when viewing pages in WebBrick. This isn’t the case in our situation at all (although we are now using WebBrick). We had the same problem when we were using Mongrel; but I don’t see any evidence that our issue relates at all to the local Ruby web server.

9:  204_304_keep_alive.patch: I also see that people suffering this error occurrence arising from web server processes are fixing it by applying the 204_304_keep_alive.patch. However, particularly as we aren’t suffering the error condition in a webserver process, this patch may or may not be relevant to our issue.

10:  I’ve suffered the same issue on several OSX systems, and on earlier Ruby 1.8/Rails 2.

QUESTION

Others must be assigning 50k to a single record with fixtures.

Does anyone know straight-away what we have to configure or otherwise change in our Ruby/Rails/OSX/MySQL environment to overcome this very troubling encumbrance?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    MySQL IS LIMITED TO 10 TEXT FIELDS

    There was a curve to conquering this one:

    The logical course of my experiment to resolve our issue first split our one large text field into 14 smaller text fields.

    Thus increasing our MySQL buffer size would have fixed our fixture/insert problems, if I hadn’t split the original text field into 14 separate text fields BEFORE I increased our buffer sizes. I presumed the 14 modest-sized text fields would have no preclusive effect on tests with the buffer size increases, but this afternoon I discovered a MySQL bug report that indicated (for an implementation suffering similar problems) that MySQL only supports up to 10 text fields. It just happened that when I added our 11th of 14 text fields, that I also crossed the 50k threshold.

    All the more reason to write articulate error handlers… it happens also that exceeding the 10 text field limit *engenders the same error as running out of buffer space!

    Anyway, I hope this experience will steer others around both situations. I have a lot of time in fixing our configuration.

    In any case, the rules of thumb are:

    1. Abide by the 10 text field limit for MySQL.
    2. Deploy sufficient buffer sizes in your /etc/my.cnf.

    Having the spare RAM, I’m running “my-large.cnf” for our project’s needs.

    • 0