Question

How many requests per second from a client can Loggly handle? I am only able to get around 10–20 requests processed per second and I am wondering if this is normal.

Answer 1

I just ran a bunch of tests and found that it can’t really handle much via a tcp connection using syslog-ng.

Here are my test results for anyone wanting to try it.
I used balabit’s “loggen” program for this and sent 200 byte messages to the tcp port assigned to me by loggly.
Note that although the syslog RFC (3164 at least) states that a log message should not exceed 1024 bytes, I used 200 byte packets just to be fair and because many messages are that small.

Signed up for a free account.
Configured a TCP connection for testing.
Tried sending various amounts, results:

Test 1: FAIL

loggen -iS -r 6000  -s 200 -I 100 logs.loggly.com 16225
Send error Broken pipe, results may be skewed.      
average rate = 1392.13 msg/sec, count=18296, time=13.142, (average) msg size=200, bandwidth=271.74 kB/sec

Test 2: FAIL

loggen -iS -r 4000  -s 200 -I 100 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.       
average rate = 2767.16 msg/sec, count=121146, time=43.779, (average) msg size=200, bandwidth=540.15 kB/sec

Test 3: FAIL

loggen -iS -r 2500  -s 200 -I 100 logs.loggly.com 16225  
Send error Broken pipe, results may be skewed.      
average rate = 1931.27 msg/sec, count=85878, time=44.467, (average) msg size=200, bandwidth=376.98 kB/sec

Test 4: FAIL

loggen -iS -r 2000  -s 200 -I 100 logs.loggly.com 16225    
Send error Broken pipe, results may be skewed.      
average rate = 1617.72 msg/sec, count=83134, time=51.389, (average) msg size=200, bandwidth=315.78 kB/sec

Test 5: FAIL

loggen -iS -r 1000  -s 200 -I 100 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.      
average rate = 936.50 msg/sec, count=63331, time=67.624, (average) msg size=200, bandwidth=182.81 kB/sec

Test 6: PASS for duration configured, FAIL for > 100 seconds – SEE TEST 7

loggen -iS -r 500  -s 200 -I 100 logs.loggly.com 16225  
average rate = 325.00 msg/sec, count=32501, time=100.001, (average) msg size=200, bandwidth=63.44 kB/sec

Test 7: FAIL – Ran a new test @500 EPS for a longer period and the pipe broke after 255 seconds:

loggen -iS -r 500  -s 200 -I 10000 logs.loggly.com 16225
Send error Broken pipe, results may be skewed.     
average rate = 323.35 msg/sec, count=82642, time=255.577, (average) msg size=200, bandwidth=63.12 kB/sec

Test 8: FAIL (ran for longer @ 200 EPS, but still failed)

loggen -iS -r 200  -s 200 -I 10000 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.      
average rate = 163.53 msg/sec, count=234090, time=1431.470, (average) msg size=200, bandwidth=31.92 kB/sec

Test 9: FAIL (again, ran longer but still failed)

loggen -iS -r 50  -s 200 -I 10000 logs.loggly.com 16225    
Send error Broken pipe, results may be skewed.    
average rate = 47.36 msg/sec, count=89325, time=1886.014, (average) msg size=200, bandwidth=9.25 kB/sec

Test 10: FAIL? (same results, but lost the connection again. Hard to believe they can’t handle 10 eps?)

loggen -iS -r 10  -s 200 -I 10000 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.  
average rate = 9.94 msg/sec, count=1568, time=157.770, (average) msg size=200, bandwidth=1.94 kB/sec

Did some web searching to see what loggly can actually do, but there’s only marketing material that says it is scalable, not how scalable it is.
I did find this:
http://twitter.com/jordansissel/status/5948244626509824
Which is only 22 events per second…

Full Disclosure: I am the founder of LogZilla, so I was testing out the competition because we are launching a cloud-based syslog solution.
My tests show that our software is able to handle anywhere from 2,000 to 12,000 events per second depending on which servers we’re using in the cloud.