Question

Just wondering, is there any quick way to count all the NULL values (from all columns) in a MySQL table?

Thanks for any idea!

Answer 1

If you want this done exclusively by MYSQL and without enumerating all of the columns take a look at this solution.

In this method you don’t have to maintain the number of database columns by hard coding them. If your table schema will get modified this method will work, and won’t require code change.

SET @db = 'testing'; -- database
SET @tb = 'fuzzysearch'; -- table
SET @x = ''; -- will hold the column names with ASCII method applied to retrieve the number of the first char
SET @numcolumns = 0; -- will hold the number of columns in the table

-- figure out how many columns we have
SELECT count(*) into @numcolumns FROM information_schema.columns where table_name=@tb and table_schema=@db;

-- we have to prepare some query from all columns of the table
SELECT group_concat(CONCAT('ASCII(',column_name,')') SEPARATOR ",") into @x from information_schema.columns where table_name=@tb and table_schema=@db;
-- after this query we have a variable separated with comma like
-- ASCII(col1),ASCII(col2),ASCII(col3)

-- we now generate a query to concat the columns using comma as separator (null values are omitted from concat)
-- then figgure out how many times the comma is in that substring (this is done by using length(value)-length(replace(value,',',''))
-- the number returned is how many non null columns we have in that column
-- then we deduct the number from the known number of columns, calculated previously
-- the +1 is added because there is no comma for single value
SET @s = CONCAT('SELECT @numcolumns - (length(CONCAT_WS(\',\',', @x, '))-length(replace(CONCAT_WS(\',\',', @x, '),\',\',\'\')) + 1) FROM ',@db,'.',@tb,';');
PREPARE stmt FROM @s;
EXECUTE stmt;
-- after this execution we have returned for each row the number of null columns
-- I will leave to you to add a sum() group call if you want to find the null values for the whole table
DEALLOCATE PREPARE stmt;

The ASCII is used to avoid reading, concatenating very long columns for nothing, also ASCII makes us safe for values where the first char is a comma(,).

Since you are working with reports, you may find this helpful as this can be reused for each table if you put in a method.

I tried to let as many comments as possible.

Let’s split on pieces the above compact way (reverse way):

I wanted to end up having a query like this

SELECT totalcolumns - notnullcolumns from table; -- to return null columns for each row

While the first one is easy to calcule by running:

SELECT count(*) FROM information_schema.columns where table_name=@tb and table_schema=@db;

The second one the notnullcolumns is a bit of pain.
After a piece of examination of the functions available in MySQL, we detect that CONCAT_WS does not CONCAT null values

So running a query like this:

SELECT CONCAT_WS(",","First name",NULL,"Last Name");
returns: 'First name,Last Name'

This is good, we take rid of the null values from the enumeration.
But how do we get how many columns were actually concatenated?

Well that is tricky. We have to calculate the number of commas+1 to get the actually concatenated columns.

For this trick we used the following SQL notation

select length(value)-length(replace(value,',','')) +1 from table

Ok, so we have now the number of concatenated columns.

But the harder part is coming next.

We have to enumerate for CONCAT_WS() all values.
We need to have something like this:

SELECT CONCAT_WS(",",col1,col2,col3,col4,col5);

This is where we have to take use of the prepared statements, as we have to prepare an SQL query dynamically from yet unknown columns. We don’t know how many columns will be in our table.

So for this we use data from information_schema columns table. We need to pass the table name, but also the database name, as we might have the same table name in separate databases.

We need a query that returns col1,col2,col3,col4,col5 to us on the CONCAT_WS “string”

So for this we run a query

SELECT group_concat(column_name SEPARATOR ",") into @x from information_schema.columns where table_name=@tb and table_schema=@db;

One more thing to mention. When we used the length() and replace() method to find out how many columns were concatenated, we have to make sure we do not have commas among the values. But also take note that we can have really long values in our database cells. For both of this trick we use method ASCII(‘value’), which will return the ASCII char of the first char, which cannot be comma and will return null for null columns.

That being said we can compact all this in the above comprehensive solution.